### MikeMirzayanov's blog

By MikeMirzayanov, 3 months ago, translation, ,

Hi Codeforces!

Meet a small innovation on Codeforces — difficulties of problems (and at the same time a new widget filtering problems in the archive). For all the problems of the archive, I’ve calculated the difficulties in the scale of the rating of participants. Approximately this means that if the rating of the problem is equal to yours, then on a typical round you would solve the problem with a probability of 0.5. And, in general, if your rating is ri, and the problem rating is rj, then the problem during the round can be solved approximately with probability:

For example, if the rating of a problem is less than yours by 200, then the expected probability of solving the problem is 0.75. With a difference of 400 rating points, the probability increases to 0.9.

For convenient search of problems in the archive, you can now use a special widget:

With it, you can find not only problems that have all the chosen tags, but also which have at least one tag from the list.

A difficulty of a problem is also displayed when choosing problems in a mashup.

I hope that now you will be able to do more effective practice, and the process of making new mashups for trainings will become easier.

•
• +3188
•

 » 3 months ago, # |   +191 The new pop-ups about judgment verdicts of submissions are definitely helpful! During contests, I don't have to stare at the submissions to see whether I passed the pretests anymore!
•  » » 3 months ago, # ^ |   +175 Thanks. Popups go from the internals of the judging server directly to browsers bypassing the database. So no reason to press F5 (but you can if you like it).
•  » » » 3 months ago, # ^ |   +96 Codeforces is already a more competitive programming site than topcoder
•  » » » 2 months ago, # ^ |   +5 Hey, could you add a feature to show only the unsolved problems in the problem set?? That would be great.
 » 3 months ago, # |   +6 Does the difficulty consider the ratings of participants who solve a problem after contest, or only during contest?
•  » » 3 months ago, # ^ |   +104 During contests only.
•  » » » 3 months ago, # ^ |   +27 Does it take into account unofficial participation during contest?
•  » » » 3 months ago, # ^ | ← Rev. 2 →   -34 Virtual contests should be taken into account too? Because the difficulty level can't just be decided by the limited amount of applicants in a contest. That's a small survey group to send out to everyone remaining.
•  » » » » 3 months ago, # ^ |   +61
 » 3 months ago, # |   +49 Any plans on using this to have a dynamic score on the contest?
 » 3 months ago, # |   +86 Very cool!Would it be possible to add an option to view problem rating within the statement of the problem? Right now I can only see it in "problemset" tab. This would be helpful.
 » 3 months ago, # |   +34 Btw,I don't like to filter problems by their tags cos it gives a great hint and we think accordingly,eg to identify a problem of dp is its main trick ...
•  » » 3 months ago, # ^ |   +48 If your goal is to get better at solving problems generally then it's the right mindset. Though if you have just learned a new technique or want to practice a given topic, then tags can be useful.
•  » » » 3 months ago, # ^ |   +1 true,generally i focus more on problem solving more and more of various types,so i try to solve it in my way first ....and learn new algo whenever a problem on it is encountered ..... and try to relate problems ....this strategy is not new...dreamoon blog
 » 3 months ago, # | ← Rev. 2 →   +24 Is the difficulty rating of a problem accessible via Codeforces API? Great feature by the way, now I can tell that the problems I solve are too easy ;-;
 » 3 months ago, # |   +3 It works great, thanks MikeMirzayanov
 » 3 months ago, # |   +58 codeforces is always innovative.proud to be a part of such a great polygon.Thanks MikeMirzayanov.
 » 3 months ago, # |   +27 great feature! I really want this type of thing.could you add this problem rating info to the codeforces API? I use codeforces API for automatically select random problem that nobody in my problem solving group solved. I used 'solved count' for judging problem difficulty. but solved count can't represent accurate difficulty. so I want to use this problem rating instead of solved count in codeforces API!
 » 3 months ago, # | ← Rev. 2 →   +10 I want to hide the problem tags and filter problems with difficulty. But it doesn't show the unsolved problems. Is it intended or a bug? upd: fixed.
•  » » 3 months ago, # ^ |   -8 Dude, if you know the tag whats the point of hiding it!
 » 3 months ago, # |   0 Currently, if I'm logged in and enter a difficulty in the "Filter Problems" widget, it only lists problems that I've already solved. This doesn't happen if I'm logged out or don't enter a difficulty, in which case it also lists unsolved problems.
•  » » 3 months ago, # ^ | ← Rev. 2 →   0 under the "filter problems" box there is a "settings" box. Check the box "Show tags for unsolved problems". **Sorry my mistake. I didn't understand it on the first place.
 » 3 months ago, # |   0 Really nice. Is it possible to share on how did you arrive at the formula ?
•  » » 3 months ago, # ^ |   +2 It's the typical ELO formula.https://en.m.wikipedia.org/wiki/Elo_rating_system#Theory
 » 3 months ago, # | ← Rev. 2 →   0 It would be nice if it will filter according to user rating by default.
 » 3 months ago, # |   +37 Best thing about Codeforces . Innovations always going on.
 » 3 months ago, # |   0 In order to list unsolved problems when filtering by difficulty, I need to check the option "Show tags for unsolved problems". It would be great if there was an option to filter by difficulty without having to show tags for unsolved problems, as these can give hints which spoil the problem.
 » 3 months ago, # | ← Rev. 2 →   0 In ten years we'll look back and ask, "Where were you when Codeforces ASCENDED?"And I'll say, I was in the jacuzzi with a hot girl by my side. Just kidding, I don't even have a jacuzzi.
 » 3 months ago, # | ← Rev. 2 →   +40 Awkward... https://codeforces.com/blog/entry/46304
•  » » 3 months ago, # ^ |   +8 codeforces should hire you!
 » 3 months ago, # |   +36 Really nice feature, I love it a lot. However, the ratings are in intervals of 100. Why is that? Can we have them in intervals of 10, at least?
 » 3 months ago, # |   +3 Thank you ! Codeforces is my love.
 » 3 months ago, # |   +35 Felt like I was reading a problem statement
 » 3 months ago, # | ← Rev. 2 →   +101 Hello, thank you very much for the cool addition! But, you can see the rating of the problem only if you select it from the problemset... Would it be possible to see at when you are direct on the problem / task page? ( Because you may have selected the task from a contest page, not from the problemset page )
 » 3 months ago, # |   +3 This is a great feauture! Good job, MikeMirzayanov! I have a small suggestion. It would be great if these difficulty ratings are also shown on the dashboard of previous contests. This would help while upsolving a particular contest as opposed to isolated problems
 » 3 months ago, # |   0 great feature !! i hope in the future add categories by (A,B,C)
 » 3 months ago, # |   0 Some problems have no rating, what about them?
 » 3 months ago, # |   -14 How will you calculate problem rating before contest ?Also, in problemset, how it has been calculated ?
 » 3 months ago, # |   0 Nice feature, but... it probably looks mainly on number of users who solved a problem. That's not always accurate on ICPC contests. "Monsters and Potions" has rating 2500 [don't read if you are planning to solve Southern Subregional!], just because everyone was too lazy to read a long statement during the contest. That problem is rather 1500. Isn't it better to disable this feature for ICPC contests? Leave it only for problems from rounds?
•  » » 3 months ago, # ^ |   0 It's for practiceAlso, i think it's some kind of weighted average from the rating of all the people that solved the problem on-contest, there is no direct correlation between score and number of solutions if you scroll through a bunch of examples on the problemset
•  » » 3 months ago, # ^ |   0 Number of people who solved it during the contest or in general? If it is in general then it should be changed to only official solutions.
•  » » » 3 months ago, # ^ |   0 Imagine a contest with 13 problems: 5 are very easy, 2 are very hard, and the remaining 6 have the same mid-level difficulty.What happens on the contest. Firstly top teams solve 4 of 5 very easy problems. Weaker teams follow them. Then one of the top teams finds one of mid-level problems and solves it, because it is easy for a top team. Another top team finds another mid-level problem and solves it, because it is easy for this team too. Third top team finds another mid-level problem and solves it.What we have now? Four very easy problems are opened, and three mid-level problems are opened. Noobs follow the monitor and go for these three mid-level problems. The last, 5th very easy problems remains unopened until, say, the end of the 2nd hour. Top teams discover it, say "LOL why nobody solved it" and solve it. Noobs don't have time for it, because they are busy with mid-level problems — they have more AC counts! In the end, the 5th very easy problem becomes like 8th or 9th solvable problem.This is what happened on that Southern Subregional. Surely, it's wrong to estimate difficulty by only AC counts. Something like 10% percentile of rating can say more in this case — if a noob team solves it, everybody can too, they just didn't try it on the contest.
•  » » » » 3 months ago, # ^ |   0 Wait, sorry, i was talking about regular contests. I should have written it in the main blog not under your comment, sorry again.
•  » » » » 2 months ago, # ^ |   0 Reminds me of CERC 2014 where problem L was solved by 3 teams. I was the first person to solve it there in-contest and I was thinking "wtf why did nobody solve this yet". It wasn't super easy, but something like the first or second mid-level problem and the organisers said they were disappointed so few teams solved it.
•  » » 3 months ago, # ^ |   +8 Arguably, parsing the statement of the problem and understanding it is part of solving the problem. Or rather I should say realizing that a problem is easy is part of the difficulty of the problem.
 » 3 months ago, # |   +12 It is a great innovation for me! Now I can try to solve problems whose difficulties is around my current ratings so as to improve myself more efficiently. Thanks Codeforces!Besides, I suggests that difficulty ratings are also shown on the dashboard of previous contests. It will be more convenient for me to solve problems.
 » 3 months ago, # |   0 It is just lovely ....
 » 3 months ago, # |   0 It's really useful change, as always. I think, that training will be even more effective. Thanks, MikeMirzayanov!
 » 3 months ago, # |   0 Thank you very much @MikeMirzayanov .
 » 3 months ago, # | ← Rev. 2 →   0 Epic! i would suggest to show difficulties in problems statement beneath the problem tags , Thank you MikeMirzayanov
 » 3 months ago, # | ← Rev. 2 →   +8 Thanks for the awesome feature.I suggest problem difficulty should be displayed on contest page such as this after contest.
 » 3 months ago, # |   0 A little addon Can you guys also add Hide Solved problems in all searches? MikeMirzayanov
 » 3 months ago, # | ← Rev. 4 →   0 Great feature ! i hope its help me to find a suit problem for me :)Can I suggest something ...I suggest that this platform support dark theme mode , coz sometime i code at night , just my thought :|
•  » » 3 months ago, # ^ |   0 I basically wrote my own chrome ext. to add some color to cf. Much faster than waiting around for a feature.
 » 3 months ago, # |   +1 Really helpful , thanks
 » 3 months ago, # |   +4 This is just so cool! Do expected problem difficulty also consider no. of submissions, and how old the problem is? I've noticed very old div1 C-E are basically same hardness as modern day div2C-E. Also, older problems will have more submissions simply because they're older.
 » 3 months ago, # |   0 Feature request : Virtual contest picker for multiple users, such that all of them haven't solved any problem in the contest. Thanks.
•  » » 2 months ago, # ^ |   +2 There is already a site for that made by Laggy.
•  » » » 2 months ago, # ^ |   +8 and BasselBakr
 » 3 months ago, # | ← Rev. 2 →   +40 Very cool!One question: consider a problem from a 5h ACM contest; if I have a rating 2500 and the problem has 2500, does it mean that I (my team?) would have had a probability 0.5 to solve it during a "typical" 2h individual CF contest, or during a "typical" 5h ACM team contest?
 » 3 months ago, # | ← Rev. 7 →   0 Thank you for the great feature, MikeMirzayanov!If I may suggest, it might be a good idea to add a parameter for rating deviation, as in the Glicko rating system. This would provide credibility to the rating of the problem, and distinguish ratings of new problems with few solvers from ratings of older ones.
 » 3 months ago, # |   0 The difficulty should be visible also on the problem page like the tags
 » 3 months ago, # |   +1 Great news! Small question: is there any bias in difficulty for D and E problems just because they're 4th and 5th among all the contest problems? For instance, problem D may have 2500 instead of real 1800 just because problem C from its contest is a long implementation problem and participants had not enough time after it to solve D.
•  » » 3 months ago, # ^ | ← Rev. 3 →   0 Yeah, the difficulty estimation isn't trivial. Besides, there's some categories of problems than are more frequent and popular than others. Consider DP,graphs,greedy,constructive algorithms as opposed to geometry and number theory. I don't meant that they are less interesting, but they have fewer submissions because their skip rate is high. At first I thought that the solution is to segregate the tags into very popular, moderate and unpopular categories and then using different parameters in each category to estimate the problem rating. But some groups of problems like the implementation problems have mixed reactions from contestants because even if they're easier than similar rates problem in a popular category, people might skip them therefore boosting the rating, or they're hard and interesting and have solves similar to popular category, but they're still treated with other implementation problems, thereby making it hard to judge whether their rating will be higher or lower than the true difficulty.
 » 3 months ago, # |   +6 Thank you for this system, but, unfortunately, sometimes rating of tasks very strange and has big error
 » 3 months ago, # |   +8 Congratulations to Codeforces team for this excellent job. I'm curious, did you based the work on this, also amazing, blog post? If so, what modifications did you apply? How do you handle things like div2E problems that no one solved during the competition?
 » 3 months ago, # |   0 I cannot understand this new system someone please tell easily please
 » 3 months ago, # |   +40 MikeMirzayanov I always wanted to say this, but could not think of where to say this. So, this maybe the right place. Codeforces has a hell lot of amazing tutorials. I have been benefitted a lot by reading them. But the problem is they are buried somewhere in the thousands of blog posts already on codeforces. There are some tutorials on topics that I don't even know exist. I don't know that codeforces has tutorial blogs on those topics. I usually come across them when I see them see in the recent actions section when someone comments on those blog posts. As I am not on a great level in terms of my ratings, there are many topics that I don't know of like segment tree beats, Berlekamp-Massey and so on. Also, there are a lot of tutorials on common topics but they way they have been explained in codeforces blog make them a lot more practical and helpful than other resources or tutorials on the internet like I would never have known that ternary search can be coded in a way similar to the binary search as explained here and also the blogs on maths topics like the mobius function and blog posts on geometry and other stuff. But again, the problem is I don't know many of them exist but when I find them, they are a treat to read. I believe that there are many other people who have been and will be benefitted by the tutorials available on codeforces. There have been some blog posts that have tried to collect the available tutorials but they include only a handful of these. So, it would be amazing if a separate section can be created on top for the tutorials that are available on codeforces in which one can see the topics and the tutorial blog posts on them that are available on codeforces.I know it's a lot of work but maybe the community can help somehow by proposing the available tutorial blog posts to be added to the section.
•  » » 3 months ago, # ^ |   0 "Favorite" anything you think is possibly useful. Google search works too. Just don't search using cf search.
 » 3 months ago, # | ← Rev. 2 →   0 that's great it's very helpful
 » 3 months ago, # |   +2 Realy good development.We can learn about the difficult and we don't need to search question anymore.
 » 3 months ago, # | ← Rev. 2 →   0 deleted
 » 3 months ago, # |   +2 There is every reason to say codeforces is the best, I just love it. Thanks MikeMirzayanov
 » 3 months ago, # |   -40 There is a issue, what if in a contest somebody accepts a problem with their second account, that way it wouldn't be accurate
•  » » 3 months ago, # ^ |   -14 No one has second account.(in ideal) :))
 » 3 months ago, # | ← Rev. 3 →   0 With it, you can find not only problems that have all the chosen tags, but also which have at least one tag from the list.So if I search for problem on binary-search and geometry, the result should contain all the problems which have at least one of these tags, right? I tried with a bunch of tags to list all related problems but the result had no problems.Or am I missing something?
 » 3 months ago, # |   +54 Thanks for a new feature — looks interesting!Is there any reason why all the numbers are rounded to 100? Is there anything bad in scale with smaller jumps (maybe even unit jumps, like we have for rating itself).Could you please share code used to calculate this rating? At least its algorithmic part. I can think of more than one model which can be used to get such approximation, so I'm curious what exactly stays behind it.Also, are there any pitfalls/issues that we are already aware of? Some things that I am already suspicious about are how various kinds of contests are handled (i.e. individual/team events, longer rounds etc. — same non-trivial problem will often be much more solvable in 3h round compared to 2h round) and how rating inflation is handled (I guess for contests from 2010-2012 the score that you get from those ratings will be somewhat misleading because back then even 2200+ was a big deal — but I didn't go through old problems and their ratings thoughtfully to see if there is anything suspicious).
 » 3 months ago, # |   -7 why watermelon problem has 1200 difficulty. I think this problem have a high difficulty.(It is easy problem i think)
•  » » 3 months ago, # ^ |   0 I don't know why but I have already seen some interesting difficulties on some easy problems too. The accuracy of those ratings might be improved by taking the AC ratios into consideration. Also, since it had been a long time, it is possible to see some awkward difficulties on old problems, because the ratings and some ideas have been changed.
•  » » » 2 months ago, # ^ |   -8 Yes. You are right.
•  » » » » 2 months ago, # ^ |   -8 I think the levels of problems are unfair.
 » 3 months ago, # |   +22 Is there any way we can see the problem ratings from the problem or contest page itself? Right now, it seems to only be the problemset page, which makes it hard to find for the rating of a particular problem (I guess one workaround right now is try to make a mashup).
 » 3 months ago, # |   +33 1A(Theatre Square)'s difficulty is 1300? I think it could be lower.
 » 3 months ago, # |   -26 OK..But it will not take any effect on me.......LOL
 » 3 months ago, # | ← Rev. 2 →   0 I'm really wondering about the formula . After playing around with it a bit, I've found that the probabilities it gives seem more extreme (closer to 0 or 1; farther from 0.5) than I would believe once you consider that Codeforces contests only have partial feedback (like pretest and systest phase). While I do understand where the formula came from, I feel that the definition of it being the probability that one would solve it in contest isn't really precise; maybe it would be something more like the probability that one would solve it in a contest with full feedback?
•  » » 2 months ago, # ^ |   -8 I think that you can test this formula in several contests.
 » 3 months ago, # |   +25 Is there any way to hide/turn off the scores displayed? Like tags, it gives the extra information that can make you think a certain way/alter the problem solving process in a way that could be undesirable.
 » 2 months ago, # |   +6 Is it also possible to show the rating of the problem in the problem page? I think it would be great! Thanks :)
 » 2 months ago, # |   +1 How about adding other tags?
 » 2 months ago, # |   0 Thank i need it.
 » 2 months ago, # |   +14 Why not add the difficulty in the tag? It seems that the difficulty can only be seen at Problemset page.
 » 2 months ago, # |   0 Yeah.
 » 2 months ago, # |   +11 Please, add also difficulty of the problem to the problem page.Because now we can see difficulties on the Problemset page, but cannot see at the single problem page. And it's required to visit problemset page, try to find this task to see rating
 » 2 months ago, # |   0 Good job !!Thanks ^_^
 » 2 months ago, # |   -11 codeforces is the best website for coding.
 » 2 months ago, # |   0 Very nice new stuff comming to CF, I stopped practicing 2 years ago after my ACMICPC run came to an end, however now I can practice and learn new stuff for future contests on CF, and coaching new teams, feels good man!!!!
 » 2 months ago, # |   +4 Thanks for the awesome feature.. I have one suggestion or request.. Please give an option to just see problem rating in tags instead of showing both rating and topics in tags. I want to see rating but not topics..
•  » » 2 months ago, # ^ |   0 Also it would be nice if difficulty can be viewed in contest's problems page to see the difficulty of all contest problems without having to check each problem separately.
 » 2 months ago, # |   +3 When will problem difficulty score be added to codeforces API?
 » 2 months ago, # | ← Rev. 2 →   +25 MikeMirzayanov Its possible separate tags with difficulty? I want to see the difficulty of problems, but don't see tags.
•  » » 2 months ago, # ^ |   0 Go to your profile Select Settings Scroll Down and in between multiple options, you will find Show tags for unsolved problems. uncheck it and no tags will be shown.
•  » » » 2 months ago, # ^ |   +8 Yeah, I know, but I want to see the rating of problems.
•  » » » » 2 months ago, # ^ |   +1 I am actually able to see the ratings of problems even after disabling it. Can someone tell me how to insert an image in the comment ?
•  » » » » » 2 months ago, # ^ | ← Rev. 2 →   +5 ImageI want to see the rating in problem, not in problemset.
 » 2 months ago, # |   +15 Thats the best thing Codeforces has ever done, now there is no need of SPOJ to practice specific types of problem.Thanks, team Codeforces.
 » 2 months ago, # |   0 The feature of difficulty level will help for all and will help us to raise level of practice we do normally..Thanks a lot for all the features
 » 2 months ago, # |   0 The pop-up feature of judgement verdicts is not working in my browser. What can be the reason and how does it look like? I still have to press F5 continuously.
 » 2 months ago, # |   0 Will there be such things for sgu problems too?Or maybe tags at least??
 » 2 months ago, # | ← Rev. 2 →   +11 How long after contests should we expect to wait for difficulties to come out? I'm asking because the ones for the two previous contests (here and here) have not been updated. UPD: Thanks, they have been updated now.
 » 2 months ago, # |   0 Hi MikeMirzayanov thanks for all the hard work you are putting into codeforces... Thanks to your hard work, Now we can filter problems according to their points and inclusively e.g (1800-2300) with tag.. Is it possible to add a feature to filter problems by the number of user who solved that problem inclusively for example (1200-2200 here i meant that the problem is solved by >= 1200 and <= 2200 users)... We can sort ascending or descendingly by user count... But it might be quite helpful if this feature of filtering problem is added... Just a personal point of view..
 » 2 months ago, # |   0 Can somebody explain why 1071C - Triple Flips has 2900 rating and 1071D - Familiar Operations has 2700, despite the fact that the first was solved by 40 persons, while the second by 20 during the contest?
 » 2 months ago, # |   0 ahmed_aly any consideration to add this to the CF ladders on A2OJ? The current difficulty measure scale of Div2C — 4, Div2D — 5, Div2E — 6, isn't always that accurate. For example, I'm currently doing the Div2E ladder, and there was a problem labeled level 6 difficulty that was pretty easy (had 1800 CF rating) and a problem labeled level 4 difficulty that wasn't super easy (had 2100 CF rating).If you don't think it's worth doing, that's fine. I love A2OJ a lot right now, so it's just a suggestion.
 » 2 months ago, # |   0 pop-ups about judgment verdicts of our submissions have stopped working (since the DDOS attack I guess). Is it just me or others are facing the same issue? Please fix it. It was a cool update. Thanks