Not-Afraid's blog

By Not-Afraid, history, 18 months ago, In English

Hello Codeforces community.


Tl;dr: The script is specifically for Ubuntu and can be used to stress test `C++` solutions. It can be easily modified to work with python or java. To use it in windows you can download Ubuntu terminal which is officially available on Microsoft Store and then use it or you can modify the script to work in cmd by changing the .out files to .exe and a bit of syntax for cmd.

Note: If you don't want to read this lengthy blog, just go to this repo and follow the instructions of readme.

First let me tell you what stress testing is if you don't know, Stress testing is technique using which we can run our solution (which we think might fail/is failing) on a number of random test cases and match it's output with the output of a solution which is a brute force solution or accepted solution of someone else.
If you don't know why and when to use stress testing you can use this detailed article on Stress Testing by ADJA.

I want to share my script for stress testing your solution.

Requirements:
1) A solution which we want to test.
2) A brute force solution which gives correct solution.
3) A generator for generating test cases according to the problem.

About script:

code

Go through the code once, i have added comments and you will understand most of the part. To understand the script all we need is just basics of the language bash. So we have three .cpp files:


1) gen.cpp // our cpp solution for generating test cases 2) solution.cpp // our orignial solution 3) brute.cpp // our solution which uses brute force and guaranteed to give correct output

About test case generator:
For generating test cases you can either use "testlib.h" here (i personally don't use it because it takes 6-7 seconds on average to compile in my pc(idk there might be ways to reduce it) and also i need to remember the methods to generate things.) or write your own since most of the time you just need arrays, strings, trees or simple graphs or simple some integers.

I use the below code for generating test cases most of the times(file gen.cpp):

Code

You can use above code and modify it according to your needs. Just go through the code once and everything is self explanatory(i have added some comments too). Usage of above gen.cpp code:


1) to generate an array calling: gen_array(10, -5, 10); it will return an array(vector more specifically) with length 10 and elements in the range [-5, 10]. 2) to generate a tree calling: gen_tree(10): will return a tree with 10 nodes. 3) to generate a simple graph calling: gen_simple_graph(10, 12); will return a simple connected graph with 10 nodes and 12 edges. You can add things as you need them or use testlib which is the best if you know how to use it.

Usage:
Download this repository manually or by using git clone on terminal.

  • Copy your original solution which you expect might fail in the file solution.cpp.
  • Copy your brute force solution which is expected to give correct output in the file brute.cpp.
  • Change the gen.cpp file so as to generate test cases according to the question.

Now open your terminal in the directory where file s.sh resides and run command:
$ bash s.sh
or
$ ./s.sh
If it shows permission denied then give it execute permission using:
$ sudo chmod +x s.sh.

In file s.sh you can change the value of variable max_tests as you wish on how many random test you want to run the solution.

Verdict: the verdict of running file s.sh on every test case is either Accepted if your solution's output matches the brute solution output or Wrong Answer and will show the input on which the solution failed, the output of your solution and expected output according to the brute force solution on terminal and the script will be terminated.
If you wish to terminate the script at any moment you wish use the command ctrl + c in terminal.

quick demo: Here the output of first 3 test cases matched with the brite and as soon as it didn't, it printed the test case, output, and expected output to the console and terminated. Now you can debug your original solution and check where it goes wrong.
( )

You can watch this video by Errichto to understand more about stress testing.

Any suggestions or corrections are welcomed.

Apologies for my poor english.

 
 
 
 
  • Vote: I like it
  • +52
  • Vote: I do not like it

»
18 months ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

Suggestion — Maybe include the option to add a checker, because many problems have multiple possible solutions. Keep a default checker as a direct comparison of solutions, but allow the user to write his own.

  • »
    »
    17 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Thanks for the suggestion. I will try writing one and update it.

»
18 months ago, # |
  Vote: I like it 0 Vote: I do not like it

If you want to test cpp / c / java / python, you can checkout this

»
18 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Great Work!!

CF is unable to show the bash script properly. The git repo code is neat and clean.

»
14 months ago, # |
  Vote: I like it 0 Vote: I do not like it

How do I generate a binary string using the generator? please answer..

  • »
    »
    14 months ago, # ^ |
    Rev. 4   Vote: I like it +5 Vote: I do not like it

    I think this should do the job:

    Spoiler


    or maybe like this:

    Spoiler
    • »
      »
      »
      14 months ago, # ^ |
      Rev. 2   Vote: I like it 0 Vote: I do not like it

      Thank you very much for this and the script is working fine. Thanks again!

      Seriously You deserve 1000 upvotes.

»
14 months ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

sorry for a silly doubt but this wont work in cases when there are multiple possible solutions right

  • »
    »
    14 months ago, # ^ |
      Vote: I like it +5 Vote: I do not like it

    Yes, you are right. But for such case you can write custom checker to verify the solution instead of using diff.
    I never really needed it, so never wrote one else i would have definitely updated the script.

»
7 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Hey, I downloaded the ubuntu subsystem on Windows 10 and I tried to run this script. It gave an error like this -

Error

I use g++ 9.2.0 but still this error. Any way around this? Any help is appreciated!

  • »
    »
    7 months ago, # ^ |
      Vote: I like it +2 Vote: I do not like it

    Its probably saying that you dont have g++, try running g++ --version once there, it should show the version if its successful

    • »
      »
      »
      7 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      I didn't realize I had to install g++ in Ubuntu too. How stupid. Now it works. Thank You!

      • »
        »
        »
        »
        6 weeks ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        gibberish hey did it work after installing g++ like do i need to set path again i mean how to install g++ in ubuntu too ?? we already installed it right for normal ide i mean ...please tell me

        • »
          »
          »
          »
          »
          6 weeks ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          run these commands on a terminal:

          sudo apt-get update

          sudo apt-get upgrade

          sudo apt-get install g++

»
7 months ago, # |
Rev. 2   Vote: I like it +9 Vote: I do not like it

In case someone was looking for a stress testing script with a checker, I found this blog pretty useful.