Following this thread I have suggested an improvement to a rating system to fix an "issue" with different people using different criteria during rating which results in some people strongly leaning towards giving many 5-starts left and right.

The proposed solution to this subjectivity of ratings was to add an extra step after the ratings are given, but before the scores for the games are calculated, to adjust all participants ratings so that they average at 3, and them use those adjusted scores at game level to compute average scores.

This way if a person was handing out 5-stars left and right those high ratings will actually not be that more valuable than 3-s. Similarly if a grumpy Grinch was 1-starring everything for no reason, those ratings would also not be harming the big picture.

As I mentioned, I suggested this "solution" mostly as a joke in answer to a person complaining about those "generous" raters, but it looks like the other participants of the thread (all 3 of them) actually liked the idea, so I am bringing it here for a broader discussion.

One thing that was pointed out by @huvaakoodia is that the system should not be set up as a black box. Meaning people should be fully aware of the existence of the system, and how it would affect the ratings they have given. And I fully agree, because there are drawbacks to this system.

The obvious drawback of this system is that is some "happy" rater 5-starred a game that actually deserved it - this well-earned 5 star rating will also be degraded down to about 3.

Second drawback is - more calculations required to put this system into place. Though it is actually no that many calculations. I didn't describe the math in the previous thread, but here I will. Feel free to ignore this bit if you aren't interested.

So, first of all, just to get this out of the way, what I am about to write is not the only way to "fix" the ratings to force them to average to 3. Actually there is an infinite number of ways to modify the ratings. For example we can just treat every rating as 3 and this will do the trick. A bunch of 3s will always average to 3. This is not a reasonable way to do the trick, but it is a way. :) But let's not use it.

Instead we take the exitsing ratings and compute their average. Let's call this existing average as X. Now we can multiply each value by 3 / X, and this will actually do the trick. But it might give us ratings that are out of bounds. For example if a rater was generous and the average is more than 3, then the rare 1 she might have given will turn into something less than 1. And if a game accumulates enough of such degraded 1s it might drag it's final score below 1 which will be... really discouraging to it's creator - scoring less than minimal score...

So how do we avoid this? Easy. Before we compute the average of existing ratings we substract 1 from each score (treat 1 as 0, 2 as 1, and so on up to treating 5 as 4). Then we use the step like previous - multiplying each of those ratings by 2/X and ((Notice that since we subtracted 1 from everywhere we want to force the average to be 2 rather than 3) and then we add 1 back to each value. This way we get come values for scores that are all no smaller than 1, and all average to 3! Yay! No?

Actually no. We are not yet done. Notice how we only checked the situation how a 1-star ratings from a generous rater were degraded below 1 and fixed it? But there is a possibility of an opposite scenario, when a rater was grumpy, gave a lot of bad marks, and so the correction is supposed to increase it's values - including pushing occasional 5-stars beyond 5. And once again - this introduces a possibility of a game scoring more than maximal rating, which, while not discouraging to the creator might have a bad effect on everyone else. So we don't want this to happen as well.

So how do we fix it? Actually the same way we fixed the problem with 1s! We just modify the rating before the multiplication turning 5 into 0 so it can't be modified by multiplying. Previously we subtracted 1 from each rating to do the trick. And now we need to subtract each value from 5.

But that leaves us with two possible modifications. Which one do we use? This question is pretty simple to answer if we know the average of original scores. This X we defined originally. If X is greater than 3 than we need to push the ratings down, so we are worried about preserving 1s and use the first modification. If X is less than 3 then we want to push scores up and are worried about preserving 5s.

So to sum it up, here is a step-by-step algorithm: 1. Compute X as average of existing scores given by 1 person. 2. if X > 3 then Vm = 1 + ( (V - 1) * 2 / ( X - 1) ) 3. if X < 3 then Vm = 5 - ( (5 - V) * 2 / ( 5 - X) )

Here Vm is a modified value of rating and V is corresponding original value of 1 to 5.

Additionally to make the system even less intrusive we might set up a margin within which we just let the rating be. For example we might be fine if the original ratings average is between 2,5 and 3,5 and we don't want to modify such ratings. And furthermore - for people whose averages are outside this margin we might want to push their average only to the edge of this acceptable range. To do this let's define this margin as m (for the 2,5 to 3,5 being acceptable m is 0.5). This will modify the algorithm a bit:

1. Compute X as average.
2. if X > 3 + m then Vm = 1 + ( (V - 1) * ( 2 - m ) / (X - 1) )
3. if X < 3 - m then Vm = 5 - ( (5 - V) * ( 2 -- m ) / (5 - X) )
4. Othervise Vm = V

Please not that in the second we still subtract m from 2. This is because we are inverting the scores by subtracting them from 5...

Phew...

The math class is now over.

So what do you think about this system? Do we need it? Will it do more harm than good? Is it too complex to implement?

If you ask me, I think the best way to test this system would be to run it along some existing data and see what effect it has on the final scores. And I can even volunteer to write a set of DB queries computing those modified averages if I get access to a table with ratings given. I don't even need to have the people/game names for it (I would require some unique IDs to replace them though - to be able to tell which ratings were given by the same person... but I guess there are such id already in place...)