MikeMirzayanov's blog

By MikeMirzayanov, 20 months ago, In English

Hello, Codeforces.

I have wanted for a long time and now, finally, I sat down and did it.

Many of you are familiar with my Testlib project. This is a library that helps you develop problems if you are using C++. In the case of problems for Codeforces rounds, its use is mandatory. It seems to me that it is almost a de facto standard for preparing materials for completely different olympiads and competitions.

The history of С++-Testlib is almost lost in centuries, I published the first version in 2005.

Nowadays, making edits to Testlib has become tricky. The code grown to thousands of lines, also it has too long a history. This increases the likelihood of accidentally changing some aspect of behavior, which will jeopardize the holding of a Codeforces round or another important competition. In short, making a bug in this code or changing behavior anywhere is highly discouraged.

So I recently took the plunge and wrote some tests for Testlib. Here's what tests can do now:

  • check that all cpp files in the repository are compiled (test test-000_compile-all-cpp)
  • check the plausibility of the behavior of a simple sval.cpp validator on a series of files (test-001_run-sval test)
  • check the plausibility of the behavior of the two checkers fcmp.cpp and wcmp.cpp on a series of files (test test-002_run-fcmp-wcmp)
  • check that the behavior of the random number generator has not changed (test test-003_run-rnd)
  • check the behavior of some functions and the behavior of InStream and opt in a series of unit tests (test test-004_use-test.h)

More good news. All this was automated through GitHub Actions. Tests are automatically run on any push or pull request. The launch is carried out on:

  • 6 different operating systems (ubuntu-18.04, ubuntu-22.04, macos-11, macos-12, windows-server-2019, windows-server-2022),
  • different versions of different C++ compilers (g ++, clang ++, msvc),
  • different C ++ standards
  • and both architectures (32 and 64 bits) are used.

In total tests run in 33 jobs! Look here https://github.com/MikeMirzayanov/testlib/actions/runs/2733832393

Of course, there are still few tests. Something else I will definitely add.

If you understand well how some functionality of Testlib works, then you can add a test for it. First of all, consider adding unit-tests to test-004_use-test.h (carefully study the design of this test).

Otherwise you might find it useful to use test-ref script in tests, which checks that a process has exited with the expected exit code, standard output, and standard error.

It is highly desirable to cover all new functionality with tests. If the functionality is such that it is not very difficult to write tests, then the presence of tests is required.

How to run tests locally?

You need to have installed bash and some tools. After just do cd tests && bash run.sh.

This will run all tests on all supported installed compilers on multiple C++ standards. Probably, just want to run tests on single g++ version using --std=c++11. You can use bash run.sh g++ v0 g++.

More examples:

  • bash run.sh — to run all tests on all compilers on multiple standards
  • bash run.sh g++ — to run all tests on g++ on multiple standards
  • bash run.sh g++ 11 v10 — to run all tests on g++-10 with --std=c++11
  • bash run.sh g++ 11 test-001_run-sval — to run test-001_run-sval on g++ with --std=c++11
  • bash run.sh test-001_run-sval — to run test-001_run-sval on all compilers on multiple standards
  • bash run.sh g++ 11 17 — to run all tests on g++ with --std=c++11 and with --std=c++17

Also you can use t.sh (or t.bat) to run shortcut of run.sh g++ 11 v0 $*. It is good idea to use it for local development to have quick response.

  • Vote: I like it
  • +711
  • Vote: I do not like it

| Write comment?
»
20 months ago, # |
  Vote: I like it -25 Vote: I do not like it

is it just me or does the typo look very concerning to yall also

»
20 months ago, # |
  Vote: I like it +35 Vote: I do not like it

Could you pleeease respond to this https://codeforces.com/blog/entry/105212

»
20 months ago, # |
  Vote: I like it 0 Vote: I do not like it

It's useful

»
20 months ago, # |
  Vote: I like it 0 Vote: I do not like it

check that the behavior of the random number generator has not changed

Looks like my issue will never be fixed because of this. So be aware on one int64 number generators!

  • »
    »
    20 months ago, # ^ |
      Vote: I like it +32 Vote: I do not like it

    This should be fixed with new version in registerGen, not just fixed and old tests changed.

»
20 months ago, # |
Rev. 2   Vote: I like it +5 Vote: I do not like it

I still have a question to ask on your test source code. Why are hand-written Bash files chosen instead of a build system? Using a build system let you define the dependency clearer, so no source need to be compiled if not changed. Re-running all tests on local took very long time :(. I mean, just plain Makefile is enough if minimalism aspect need to be considered, and it is still powerful.

  • »
    »
    20 months ago, # ^ |
      Vote: I like it +2 Vote: I do not like it

    In this case, the building is a small part of testing process. Either I don't know make very well, or replacing all the bash code with a Makefile would be very difficult. Even if it does, I'm sure the support will get worse since make is a much less common use tool than bash.

    Also I'm not sure it is good idea to migrate build process on make. Still, here we are not building a typical C++-project, but building in testing purpose. I guess there are differences. In addition, adding another dependency to the project and complicating its further support is also a dubious idea.

    Another point: specifically in the case of testlib in 95% of scenarios, the developer changes testlib.h and no incremental build will help, you have to rebuild nearly-everything from scratch.

    P.S. A couple of hours ago I was working on opts. I have run tests multiple times via t.bat test-004_use-test.h. It took about 10 seconds. I think we can live with it.

»
20 months ago, # |
Rev. 2   Vote: I like it +45 Vote: I do not like it

Tests for testlib?