*Spin* magazine moves from a five star system to ten

The old rating system granted up to five stars but now the maximum number of stars is ten.  This signals that they wish to start exaggerating the quality of the product.  When there are only five stars you know that they are laying their reputation on the line when they grant five stars to a new CD.  (Michelin of course won't give a restaurant more than three stars.  They don't calculate out to the fourth decimal place along a scale of one thousand.)  If the music isn't good you can decide to stop trusting them.  But say they give a new release eight, nine, or who knows maybe eight and a half stars?  What exactly are they trying to say?  Yes they are putting their reputation on the line when they give ten stars, but this will happen so infrequently that it will be harder to judge their overall trustworthiness.

Evaluation systems with fewer and grosser distinctions are often more credible because they are easier to monitor.

Comments

could it be they're trying to catch up with pitchfork? how do you read their rating system?

Comments for this post are closed

Interesting. How many different ratings to the credit agencies give out?

According to Wikipedia, Moody's divides securities into: Aaa, Aa1, Aa2, Aa3 A1, A2, A3, Baa1, Baa2, Baa3, Ba1, Ba2, Ba3, B1, B2, B3, Caa1, Caa2, Caa3, Ca, C...

Comments for this post are closed

Spin... forgot about that one

Comments for this post are closed

If you find it easier to understand what they're saying when it's expressed in five stars rather than ten, divide by two.

Comments for this post are closed

The truly brave reviewer uses only one non-divisible star. I challenge you, Tyler, to start doing restaurant/book reviews this way.

Comments for this post are closed

What this analysis neglects is that the "reputation-staking" force of a high rating is as much a factor of its percentile as its absolute rating.

So if Pitchfork gives out 4 9+'s a year, I will pay close attention unless/until I stop trusting them.

More precisely, the "degree of reputation-staking" of a high rating depends on the percentage of albums that rate "significantly" lower. "Significantly" meaning that e.g. 8.7 and 8.5 are considered as having the same rating.

Now, if readers don't in fact interpret rating systems in a percentile way, then maybe Tyler is right that a finer-granularity rating system is an indication that the publication is trying to make a smaller splash with its ratings.

Comments for this post are closed

Interesting post.

★★★★★★★★½

Comments for this post are closed

This kind of thing all started when they made that amplifier that went to 11. Never should have done that.

Comments for this post are closed

In BKS Iyengar's Light on Yoga he goes up to sixty stars and feels the need to include half stars.

Comments for this post are closed

Don't we all just want someone to say "you will like this."

Maybe that's why we put up with friends.

Comments for this post are closed

It does appear to be a nod to Pitchfork. Pitchfork's problem, though, is that they almost never award newly released records a 10/10. The last was in 2005, for Scott Pollard's "Relaxation of the Asshole." Before that it was 2002, for "Yankee Hotel Foxtrot" by Wilco.

Comments for this post are closed

I'd also like to hear about the extension of this argument to grading systems. Do plus/minus systems add information about student performance, or subtract it?

Comments for this post are closed

I almost exclusively read film reviews through the filter of Metacritic, which converts all the written reviews to a 100 point scale. I find the small scales typical of newspapers uninformative.

I wrote more here.

Comments for this post are closed

A ten-star system is silly.

A distinction between 8 and 9 on something as subjective as an opinion about a movie or restaurant is meaningless to me. Who buys this CD instead of that one because of that difference, or worse, the difference between an 88 and an 85?

A coarser scale conveys useful information as to what broad category something fits into - it's a stinker, etc., without presuming to measure quality with unattainable precision.

Comments for this post are closed

I find the small scales typical of newspapers uninformative.

A finer grained system is only relevant if the rating criteria support that level of differentiation. In all probability it doesn't. I expect that if a movie critic had his memory wiped and re-reviewed a movie, the rating could easily fluctuate up or down a star in a 5 star system depending on the phase of the moon.

For a physics problem, you don't say the answer is 5.346 +/- 2

The drive to measure things to 3 digits of accuracy is simply a delusion for those who desperately want there to be accurate metrics for things as subjective as movies, books, or wine.

However, given there's a pretty big audience of people who want to believe it, why should a magazine disabuse them and lose sales?

Comments for this post are closed

"This all started when they made that amplifier that went to 11. Never should have done that."

Alright, but most blokes are playing at 10, and if they need it a bit louder, where can they go? Nowhere.

Comments for this post are closed

Comments for this post are closed