I am currently struggling with understanding TopCoder rating system. It's pretty cool they published the exact formulas, but I really lack the background for it, the description. I'll try to gather here this information, which is missing there. It's the beginning, but I hope with time it will get more complete. Comments and improvement hints welcome!
TopCoder rating system is an example of Bayesian rating system. In general this probably means that potential of each coder is decribed by 2 values: the mean and the standard deviation (SD). When comparing 2 coders, not only their rating is considered, but also the volatility. Some Bayesian rating systems show rankings based not only by the mean value, but compute it as a function rank(mean, SD). TopCoder uses simiplified approach and ranks coders using only the first variable — mean value.
- performedAs — the hypothetical value of rating received as a result of a tournament. Unfortunately not presently in stats. But from the formulas it turns out to be easy to calculate: it's oldRating + 6 * ratingGain, assuming weight equal to 0.2. The exact weight is very easy to calculate, so one may compute their individual value. Full formula for performed as is
oldRating + ratingGain * ((1 / weight) + 1)
- rating — mean of performedAs values.
- volatility — standard deviation of performedAs values. Do not mistake it with standard deviation of rating values. The greater the volatility, the less predictable coder. Less predictable coder will be considered closer to the average rating among all coders participating in a contest. Coder with low volatility will be expected to present the level exactly denoted by his rating. Don't think that big volatility makes rating changes faster. It just means that a coder's rating varied a lot during last several matches.
- weight (w) — after 20 competitions its value is equal to 0.25. We may assume it a constant value. Then w / (w + 1) is 0.2 which is reciprocal to 5. This reciprocal is an essential value as it denotes the number of tournaments contained in the history. If 1 / w is 5, that means that rating system adds the most recent tournament to previous mean values as if it contained 5 entries. In other words, all previous tournaments are compressed to pretend there were only 5 tournaments.
- ratingChange — after performedAs was calculated it's being added to the mean value of performedAs. If you try to convert the formula for the mean value after adding i + 1 element to a serie, which has a known mean value, you would receive exactly the same formula as this one. That means the current performedAs value is added to the mean, as 6th element to hypothetical 5 elements existing before.
- volatilityChange — similarly as rating change, this is computed as if 6th element was added to previous serie of 5 elements.
Explanatory forum threads and articles
- Understanding your algorithm rating — and why it changes — a must read Featured Article. No idea why these articles are so well hidden in TopCoder contents.
- A Harsh Rating System? (page 4 linked) — a very long (19 pages) thread discussing why one may want to drop his rating to get rating grow faster after next several rounds.
- Ratings question — Why did I drop? Because high volatility plus low rating pushed your expected peformance high.