[testlib] Checkers

Revision en4, by Zlobober, 2015-06-09 17:34:25


Checker is a program that should be written when task allows more than one correct solution. Although it seems that it is very easy to write checker, there are lots of important technical details that are easy to forget if you don't use a special library like testlib.h.

A common convention is that a checker should be a program taking three command-line arguments: the testdata filename, the participant output filename and the jury answer filename. Checker should read the contents of the input, output and answer, decide whether participant answer is correct (and optimal being compared to the jury's answer if there can be unoptimal answer in this task) and return one of several pre-defined verdicts:

Verdict testlib macro meaning
Ok _ok The output is correct, follows the output format and represents an optimal answer (if applicable in this problem).
Wrong Answer _wa The output is wrong or incorrect.
Presentation Error _pe The output doesn't follow output format specification of the task. Although, on Codeforces this verdict is being replaced by Wrong Answer since it's usually hard to distinguish Presentation Error and Wrong Answer
Partially Correct _pc(score) If there is a partial scoring in the task, this verdict should be used in situation when solution is partially correct or to give solution some number of points. Here score should be an integer between 0 and 200 where 0 represents the lowest mark (no points) and 200 represents the highest mark (maximum possible number of points)
Fail _fail This verdict means that checker has encountered a critical internal error or that the jury's answer is incorrect or that contestant found the more optimal answer than jury. This verdict means that something wrong has happened and it requires a special investigation from jury.

Usually the verdict returned by checker is indicated by the return code of its executable, but it may possibly be transfered to the testing system by many other ways: by creating a special xml-file with checker outcome, by writing to stdout or somehow else. When using testlib.h for writing a checker all those system-dependent ways are combined in a single expression quitf(VERDICT, "comment", ...).

Simplest checker example

Problem statement: You are given two integers a and b ( - 1000 ≤ a, b ≤ 1000). Find their sum and output it.

Let's write a checker for this problem. It will be very simple:

#include "testlib.h"

int main(int argc, char* argv[]) {
    // This command initializes checker environment.
    registerTestlibCmd(argc, argv);
    // Now there are three global variables specifying testlib streams:
    // inf - stream with the testdata.
    // ouf - stream with the contestant output.
    // ans - stream with the jury answer.
    // All those streams provide the similar interface for reading data.

    // This function reads a single integer from the participant output that 
    // should be between -2000 and 2000. If it doesn't belong to the specified
    // range, checker finishes with verdict _pe and comment saying that [sum of numbers]
    // is outside of the specified range.
    int pans = ouf.readInt(-2000, 2000, "sum of numbers");
    // This function reads a single integer from the jury output. Here we suppose
    // that jury's answer is correct and we do not need to additionally verify it.
    int jans = ans.readInt(); // We suppose that jury's answer is correct
    if (pans == jans)
        quitf(_ok, "The sum is correct."); // This finishes checker with verdit OK.
        // quitf handles a comment like printf, i. e. you may use specifiers like
        // %d, %s etc in the comment.
        quitf(_wa, "The sum is wrong: expected = %d, found = %d", jans, pans);

Available methods

There are lots of methods useful for writing checkers.

| stream.readXXX | All methods of form readXXX (like readInt, readLong, readDouble, readToken etc) are common for all testlib uses: checkers, validators and interactors. TODO: put all such methods on the separate page. | | void quit(TResult verdict, string message);
void quit(TResult verdict, const char* message);
void quitf(TResult verdict, const char* message, ...); | Finishes the checker with a given verdict and comment. | | void quitif(bool condition, TResult verdict, const char* message, ...); | if condition is true then performs quitf(verdict, message, ...) | | void ensuref(bool condition, const char* message, ...); | An equivalent of assert. Checks that condition is true, otherwise finishes with _fail verdict. Useful for debugging checkers. |

TODO: finish this list.

readAns paradigm

Suppose you have a task that asks contestant to find a complex composite answer that is not just a single number. Example:

Problem statement You are given an undirected weighted graph and two vertices s and t. Find a simple path between s and t of maximum weight and output it.


  Rev. Lang. By When Δ Comment
en10 English Zlobober 2015-06-09 20:25:45 25
en9 English Zlobober 2015-06-09 20:24:31 0 (published)
en8 English Zlobober 2015-06-09 20:22:02 644
en7 English Zlobober 2015-06-09 20:03:43 1134
en6 English Zlobober 2015-06-09 19:54:32 72
en5 English Zlobober 2015-06-09 19:51:19 7104
en4 English Zlobober 2015-06-09 17:34:25 4655 Tiny change: 'blem). |\n| Wrong ' -
en3 English Zlobober 2015-06-09 16:37:33 198
en2 English Zlobober 2015-06-09 16:36:43 120
en1 English Zlobober 2015-06-09 16:36:29 730 Initial revision (saved to drafts)