kznil96's blog

Pretests passed but failed system test (with 0 hacks throughout contest)
By kznil96, history, 4 years ago, In English

My submission #84516509 for question B of Codeforces Round 651 (Div 2) passed the pretests but failed the system test.

From what I understood, this usually means that successful hacks were presented during the hacking phase/ competition itself and my solution fails on the hack cases.

However, from what I can see, there are 0 successful hacking attempts for this problem throughout the course of the competition.

Can anyone shed light on what happened (solution passes pretests but fail system tests with 0 successful hacking attempts) ? It is quite discouraging (the bug was minor which resulted in runtime error, and I promptly spotted and fixed it as soon as system testing is over) that I was initially expecting a delta positive from this contest the moment it is over, but judging from my ranking after the system test, it is likely to be a delta negative instead.

(P/S: Needless to say, I did not use any randomised algorithm/ functions which would result in a different output. All output were exactly reproducible and is dependent on the input only.)

  • Vote: I like it
  • -38
  • Vote: I do not like it

»
4 years ago, # |
  Vote: I like it 0 Vote: I do not like it

You have got a runtime error on test 7, klick the link in the left down corner to see it.

I assume it is because 

    elif len(EVEN) > 0:
        EVEN.remove(EVEN[0])
        EVEN.remove(EVEN[1])

The intension is to remove first two elements, but the second remove removes the third. If that exists.

→ Reply
  • »
    »
    4 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    Yes, I understood where the mistake lies.

    I just do not understand how could my code has passed the pretests in the first place.

    Since there were 0 hacking attempts, this would mean that my code passed test 7 when I first submitted it, then failed on system test.

    I could not have found any other explanation otherwise.

    → Reply
    • »
      »
      »
      4 years ago, # ^ |
        Vote: I like it +3 Vote: I do not like it

      Your attempt has passed the pretests, it is true, but you should consider that there were only 6 pretests, and therefore 7th test is considered to be system test.

      → Reply
»
4 years ago, # |
  Vote: I like it +5 Vote: I do not like it

The pretests only contain a subset of all the tests. You can think of the pretests as some preliminary test. System test is a more thorough test containing a lot more tests than the pretests.

→ Reply
  • »
    »
    4 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I see, that explains a lot!

    Thanks!

    → Reply
    • »
      »
      »
      4 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      The main reason for this is to reduce the load on the server during the contest (and thus avoid queueforces). This is also the reason multi-test cases exist.

      The authors try to make pretests as strong as possible, but there is always a small possibility that your code passes the pretests but fails the system tests. (With multi-test cases you can fit even more test cases while still reducing the load)

      → Reply
»
4 years ago, # |
  Vote: I like it 0 Vote: I do not like it

In regular rounds system tests are completely separate from hacks. They're made by the problemsetters and they're run after the round ended. If you're hacked in contest you'll be notified and the verdict on your submission will be "hacked". Only in educational-style rounds are successful hacks added to the system tests for all submissions.

→ Reply