accidentallygivenfuck's blog

By accidentallygivenfuck, history, 9 years ago, In English

Hello Codeforces,

I want to share with you a simple tool that I've been using for a while now that I named "Ineffable".

What is ineffable?

Simple command-line grader for local grading of solutions for problems of competitive programming contests. It is written in Python and it uses timeout Perl library to limit resources (time and memory limits) of program being tested.

It works on Ubuntu, and I guess it should work on any Linux/Unix system as long as Python and Perl are installed.

Screenshots


(Original quote is "Success consists of going from failure to failure without loss of enthusiasm")

Installation

Run the following commands on your terminal:

$ mkdir ~/.ineffable
$ cd ~/.ineffable
$ wget https://bitbucket.org/silap/ineffable/downloads/ineffable-1.0.zip
$ unzip master.zip
$ rm master.zip
$ mv silap-ineffable-*/* .
$ rm -r silap-ineffable-*/
$ sudo ln -s ~/.ineffable/ineffable /usr/local/bin/ineffable

Or alternatively you may run this command if you have curl installed:

$ bash < <(curl -sL https://bitbucket.org/silap/ineffable/raw/f208b207160eacd8ba4c30c078d1255e309c6b40/install.sh)

Configuration

Whenever you run ineffable it looks for ineff.json file. You can also tell ineffable which configuration file to use using this syntax: inefabble another.json.

Configuring ineffable is easy and quite intuitive. After you install ineffable you can view the example configuration file that is located at ~/.ineffable/ineff.json.example.

Available configuration options are as follows:

key example value Description
pkdesc "Balkan OI 2014 Day 2 Problem Ephesus" Package description. (Optional)
pbcode "steeple" Problem code. Ineffable searches for string {PROBLEM} in keys warmup, execute, inp, out and tstdir, and replaces them with pbcode value. (Optional)
tl 1000 Time limit (in milliseconds). (Required)
ml 65536 Memory limit (in kilobytes). (Required)
warmup "g++ -O2 -o solution solution.cpp" Command to be executed before the grading starts. For example, it can be used to compile C++ code. Grading stops if the command exits with non-zero code (such as, when compilation fails). (Optional)
execute "./solution" or "python solution.py" Command that runs the solution. Solution needs to read from stdin and write to stdout. (Required)
tstdir "steeple_tests" Directory where the test files are located. If ommited, it is assumed that tests are located in the current directory. (Optional)

There are 2 different ways to show ineffable which files are input files and which files are output files:

  1. Smart stars:

    "inp": "input*.txt",
    "out": "output*.txt"
  1. Listing files.

    # Syntax 1
    
    "iopairs": {
      "input1.txt": "output1.txt",
      "input2.txt": "output2.txt",
      "input3.txt": "output3.txt"
    }
    
    # Syntax 2
    
    "iopairs": {
      "inp": {"input1.txt", "input2.txt", "input3.txt"},
      "out": {"output1.txt", "output2.txt", "output3.txt"}
    }

See it at work (sample session)

Let's solve problem Cow Steeplechase of USACO 2011 November Contest.

$ # Create folder `steeple` in your desktop and another folder `steeple_tests` inside it.
$ mkdir Desktop/steeple
$ cd Desktop/steeple
$ mkdir steeple_tests
$ cd steeple_tests
$
$ # Download and extract test data into `steeple/steeple_tests/`.
$ wget -q http://usaco.org/current/data/steeple.zip
$ unzip -q steeple.zip
$ rm steeple.zip
$ cd ..
$
$ # Create file `ineff.json` and configure.
$ touch ineff.json
$ echo '{
  "pkdesc": "Cow Steeplechase (USACO Gold November 2011 Contest)",
  "pbcode": "steeple",
  "tl": 3000,
  "ml": 65536,
  "warmup": "g++ -std=c++11 -O2 -o {PROBLEM} {PROBLEM}.cpp",
  "execute": "./{PROBLEM}",
  "tstdir": "{PROBLEM}_tests/",
  "inp": "I.*",
  "out": "O.*"
}' >ineff.json
$
$ # Create `steeple.cpp` and open it with your favorite editor to code your solution.
$ touch steeple.cpp
$ vim steeple.cpp
$ # ...
$
$ # Test your solution
$ ineffable

As I haven't spent much effort in making ineffable, there may be some bugs. I won't be able to fix the bugs at least till the IOI 2015 ends.

Ineffable is a simple project and anyone who knows some Python can alter it for their own purpose or even add some features. Project is available on Bitbucket (sorry no Github, it is blocked in my country :/).

EDIT: "Installation" section updated.
EDIT 2: Download links updated (to be able to keep number of downloads :P).

| Write comment?
»
9 years ago, # |
  Vote: I like it +5 Vote: I do not like it

This looks amazing!

»
9 years ago, # |
  Vote: I like it 0 Vote: I do not like it

Alternative installation method was corrected. Sorry for any inconvenience. :p

»
9 years ago, # |
  Vote: I like it +6 Vote: I do not like it

GOOD JOB.

»
9 years ago, # |
Rev. 2   Vote: I like it +5 Vote: I do not like it

Very handsome tool! I will write you about bugs and features!

P.S. Nice quotes for AC :)

»
8 years ago, # |
  Vote: I like it +4 Vote: I do not like it

How to check problems with many possible answers?

»
8 years ago, # |
Rev. 4   Vote: I like it +1 Vote: I do not like it

Tested on Mac OS X (10.11) and found out that timeout script is not compatible with it! also I think you made a mistake in your installation scripts! the zip file isn't named master.zip, but it is named ineffable-1.0.zip! please correct it :D

PS: I read that source code, and everything seemed amazing, but one thing! In my opinion, instead of using diff, it's better to provide an option like checker in Codeforces which checks the output! This option can be useful for problems which have many answers for some tests and it's allowed to output any of them.

Good Luck :D

  • »
    »
    8 years ago, # ^ |
      Vote: I like it +1 Vote: I do not like it

    Thank you.

    After my last IOI, I haven't used it myself. zemen proposed some improvements, but unfortunately I (unwillingly) ignored them. Primary reasons were that I was not very familiar with git at that time and couldn't find time to learn more about git. Now that I have a bit more free time, I think I can get my hands on that and add the checker feature. Thanks for the suggestion.

»
8 years ago, # |
  Vote: I like it -18 Vote: I do not like it

Amazing thanks. Just wondering if it is available for android?

  • »
    »
    8 years ago, # ^ |
      Vote: I like it +11 Vote: I do not like it

    Sorry. Don't know how I missed your comment before.

    I don't know a way in which android version would be helpful at all. In what specific case would you prefer to use android version over command line version?

»
8 years ago, # |
  Vote: I like it +5 Vote: I do not like it

The download zip seems to be unavailable

»
7 years ago, # |
Rev. 2   Vote: I like it 0 Vote: I do not like it

Download link is broken!

EDIT: I found it. Followed the instructions changing
wget https://bitbucket.org/silap/ineffable/downloads/ineffable-1.0.zip
to
wget https://bitbucket.org/silap/ineffable/get/b21131af3eef.zip and master.zip to b21131af3eef.zip.
Great tool by the way!