variety-jones's blog

CF Stress: A highly customizable no-code tool to stress test CF problems
By variety-jones, 16 months ago, In English

cfstress.com

CF Stress is a highly customizable, no-code tool to stress test Codeforces problems.

You control all the parameters of the generator without writing a single line of code. Feeling lazy and want to pin down the error to an array of length 5? You can do that by adjusting the flags passed to the generator. Feeling adventurous and want to test your logic on test cases of length 1000? There's an option for that too. There's also a builtin diff checker for when you want to compare the output of large test cases. You can also seamlessly copy the input/output to your local machine.

Just plug in your submission ID and wait for the ticket to be processed.

It also has support for non-unique answers and interactive problems. The testcases are generated using testlib, the framework that's used to create the original test set.

Note: It only supports C++ and Python submissions for now. (For Java, I could not figure out how to make the name of the file the same as the class name).

A Sample Result

If this proves to be useful, I'll add more features, such as user accounts to view tickets in one place, emailing the user on successful completion of the ticket, better concurrency support to handle multiple tickets, revamped UI, etc.

If you have any feedback/suggestions, or if you'd like to collaborate, please let me know in the comments (or drop me an email).

  • Vote: I like it
  • +126
  • Vote: I do not like it

Comments Comments (58)
| Write comment?
»
16 months ago, # |
  Vote: I like it +8 Vote: I do not like it

Website is down rn.

→ Reply
  • »
    »
    16 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Fixed.

    → Reply
    • »
      »
      »
      16 months ago, # ^ |
        Vote: I like it +8 Vote: I do not like it

      Down again :D

      → Reply
      • »
        »
        »
        »
        16 months ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Fixed again.

        → Reply
        • »
          »
          »
          »
          »
          8 months ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          I have taken subscription 2 months before but now it is taking money from my account on its own for subscription ,plz fix this,i dont want subscription anymore.

          → Reply
          • »
            »
            »
            »
            »
            »
            8 months ago, # ^ |
              Vote: I like it 0 Vote: I do not like it

            I mean ...that's how subscriptions work.

            You could either cancel it from your end (Paypal/UPI/Razorpay), or we can cancel it from ours.

            We have cancelled your subscription.

            → Reply
            • »
              »
              »
              »
              »
              »
              »
              8 months ago, # ^ |
                Vote: I like it 0 Vote: I do not like it

              thanks a lott i was not aware of this,thx!!

              → Reply
»
16 months ago, # |
  Vote: I like it +11 Vote: I do not like it

It's not working for recent contests, like if I put 1631 E it does nothing

→ Reply
»
16 months ago, # |
  Vote: I like it +13 Vote: I do not like it

It seems like it would be super cool to have this in general, but the amount of work it would require to accurately create generators for all problems on CF is insane and not really feasible to expand in general without a colossal amount of work being put into it. From my testing, it only works on a handful of rounds. For instance: https://codeforces.com/contest/1627/problem/E doesn't work in the generator at all, and this was the first test I did.

→ Reply
  • »
    »
    16 months ago, # ^ |
      Vote: I like it +5 Vote: I do not like it

    Yeah, fair enough. Although I should've mentioned in the blog that I don't really intend to support older problems, I've already created CF Debug (currently offline) for that which tells you the smallest testcase from the existing test-set on which your submission fails. However, it's not effective for problems with multiple test cases, the recent norm on Codeforces.

    It only supports contests that happened after it was launched (i,e 22nd January, 2022 onwards). The plan was (and is) to update the generator repo a day or two after the contest ends. Considering the frequency of CF contests, I don't think it'd require more than 1 hour commitment per contest, as I already have templates prepared for array, trees, graphs, etc. Creating a good generator that covers corner cases is tricky, but creating a naive one that is correct should not take much time, I believe. It's a stress test after all.

    Depending upon the usefulness, I also plan to make at least the generator repo open-source. In that case, all we need is just 1 participant contributing a generator per problem, and the benefits can be reaped by many.

    → Reply
    • »
      »
      »
      11 months ago, # ^ |
        Vote: I like it -9 Vote: I do not like it

      Why it is asking for subscription plz give it free

      → Reply
»
16 months ago, # |
  Vote: I like it +4 Vote: I do not like it

Update: Added Contest 768 and made some minor improvements to the site:

  1. Querying for Java submissions should now correctly display an error, rather than displaying the last stored test case.

  2. The status page should now display the last ticket number.

  3. If a counter example could not be found, the result would be displayed instantly upon completion, (previously it was only displayed once a valid ticket was processed).

  4. Upon failures, help message to increase the constraints are displayed.

→ Reply
»
16 months ago, # |
  Vote: I like it +2 Vote: I do not like it

Update: Made some improvements to the site and added new features:

  1. The time limit for a single stress test has been reduced to 1 minute per submission (Earlier it was 3 minutes, which caused unnecessary delays if you accidentally enter an AC submission).
  2. A new feature of getting additional sample test cases for every problem has been added.
  3. The sample test case display page should now only display the Input and Expected Output (and not "Your output" and "Diff"). It also has new help messages for getting alternate counter examples.
→ Reply
»
16 months ago, # |
  Vote: I like it +1 Vote: I do not like it

Why website is down :(

→ Reply
  • »
    »
    16 months ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    Will need to investigate, but fixed for now.

    → Reply
    • »
      »
      »
      16 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      website down now.

      → Reply
      • »
        »
        »
        »
        16 months ago, # ^ |
          Vote: I like it +1 Vote: I do not like it

        Really? Seems to be working for me. (And the logs do not indicate any failure). Can you describe the exact issue that you faced?

        → Reply
»
14 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Thanks a lot for creating this awesome website .

→ Reply
»
13 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Does this not work without logging in? And how to reset password if you don't remember?

→ Reply
»
12 months ago, # |
  Vote: I like it 0 Vote: I do not like it

@variety-jones could you please add contest 839 to cf stress

→ Reply
  • »
    »
    12 months ago, # ^ |
      Vote: I like it +16 Vote: I do not like it

    Sorry, I currently do not add problems on demand. However, as a one time exception, I've added 839:C. You can find the counter example to your submission in Ticket 10633

    The tree has one path of length 1, and two paths of length 2. The expected path length would be

    $$$ \frac{1}{2} \times 1 + \frac{1}{4} \times 2 + \frac{1}{4} \times 2 = 1.5 $$$

    but your code prints $$$1.66667$$$

    → Reply
    • »
      »
      »
      12 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      could you please elaborate how did you get probability of first path as 1/2 and second path as 1/4 cause there total 3 paths and probability of first path is 1/3 and second is 2/3 expectation will be =(1/3)*1 + (2/3)*2 =5/3 =1.66667

      → Reply
      • »
        »
        »
        »
        12 months ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Having 3 paths does not imply that the all the 3 paths are equally likely.

        You start your journey at node $$$1$$$, which has edges to node $$$2$$$ and node $$$5$$$. With probability $$$\frac{1}{2}$$$, you'd end your journey at node $$$2$$$, and the remaining $$$\frac{1}{2}$$$ probability needs to be shared amongst all paths that start at node $$$5$$$. Since you make another choice at node $$$5$$$, nodes $$$3$$$ and $$$4$$$ will get an equal contribution of $$$\frac{1}{4}$$$ each. Hence, the result.

        → Reply
»
12 months ago, # |
  Vote: I like it 0 Vote: I do not like it

what will be the contest ID of edu round ??

→ Reply
»
11 months ago, # |
  Vote: I like it 0 Vote: I do not like it

please add generator for problem 1697 — C

→ Reply
»
11 months ago, # |
  Vote: I like it 0 Vote: I do not like it

I just tried this for the first time, however I see that Python support has been temporarily disabled. Any idea when it will be available again?

→ Reply
  • »
    »
    11 months ago, # ^ |
      Vote: I like it +6 Vote: I do not like it

    Hi, it might take some time for Python support to be added. If you need to test something urgently, you can DM me the details, and I'll test it out locally.

    → Reply
    • »
      »
      »
      11 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Thanks. I will try some local testing first, to see if I can work out the problem, but may get back to you.

      → Reply
»
11 months ago, # |
  Vote: I like it 0 Vote: I do not like it

can u please add today B..

→ Reply
»
11 months ago, # |
  Vote: I like it +4 Vote: I do not like it

Since when this tool became subscription based? When i tried to find stress cases for a problem its asking to purchase subscription. Looking at its popularity among codeforces' users, have you started making profit from it?

→ Reply
  • »
    »
    11 months ago, # ^ |
      Vote: I like it +12 Vote: I do not like it

    Maybe. Is there anything wrong with earning money for the efforts that you need to put in to maintain a product?

    And details about the upcoming subscription model has been present on the website for a long time now. I never promoted it as something that'd be forever free.

    → Reply
  • »
    »
    11 months ago, # ^ |
    ← Rev. 2   Vote: I like it 0 Vote: I do not like it

    It’s not like binarysearch.com I think. And I wouldn’t rule out binarysearch.com being a paid site in the future as well. It’s just challenging and a money/time sink to maintain a free website without it being open-sourced

    → Reply
»
11 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Have run into a little problem with the new subscription and login thing, respect the hustle btw, no disrespect. I was trying to use the service but must have made a typo when I made my password as there is no password confirmation. So I can't log into the account I just made, and there is no way for me to recover my typo'd password, so I can't use your service as of the moment unless I wanted to make a new codeforces account.

→ Reply
  • »
    »
    11 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Deleted your account. You should now be able to register again.

    For people running into this issue in future, DM me the email-id and CF handle that you used to register for the account. I might not respond, but your account should be reset within 24 hours, and you'll be able to register again.

    → Reply
    • »
      »
      »
      10 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      i have purchased the subscription and still it is showing the pricing..why??:(

      → Reply
»
10 months ago, # |
  Vote: I like it 0 Vote: I do not like it

There is no option for forgot password?

→ Reply
»
10 months ago, # |
  Vote: I like it 0 Vote: I do not like it

I searched 1702E which is after 22nd January but it is not showing anything :)

→ Reply
»
10 months ago, # |
  Vote: I like it 0 Vote: I do not like it

i have purchased the subscription and still it is showing the pricing..why??:(

→ Reply
  • »
    »
    10 months ago, # ^ |
      Vote: I like it +10 Vote: I do not like it

    Hi, it takes some time for the payments to be verified. You must have also gotten a similar message after completing the payment.

    Your subscription is activated now.

    Note for people stumbling on this thread: Your subscription should be activated within 12-24 hours after you make the payment.

    → Reply
»
10 months ago, # |
  Vote: I like it 0 Vote: I do not like it

There is no payment method for Turkey.

→ Reply
  • »
    »
    10 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    It's unfortunate that Paypal doesn't work in Turkey. I'm currently exploring alternate methods to accept payments from Turkey.

    Can you please DM me the preferred payment method that you use?

    → Reply
»
9 months ago, # |
  Vote: I like it 0 Vote: I do not like it

I just subscribed and i'm trying to submit 1679c with my submission 169759716 but it says error occurred while trying to download the submission , codeforces must be down, but cf is not down , i already subscribed :(

→ Reply
»
5 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Great tool! Could you please add 1472-f? Thanks so much!

→ Reply
  • »
    »
    5 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I can add it. But please note that it's not free, so you won't be able to use it without a subscription.

    → Reply