*Spin* magazine moves from a five star system to ten

by on February 20, 2009 at 6:14 am in Music | Permalink

The old rating system granted up to five stars but now the maximum number of stars is ten.  This signals that they wish to start exaggerating the quality of the product.  When there are only five stars you know that they are laying their reputation on the line when they grant five stars to a new CD.  (Michelin of course won't give a restaurant more than three stars.  They don't calculate out to the fourth decimal place along a scale of one thousand.)  If the music isn't good you can decide to stop trusting them.  But say they give a new release eight, nine, or who knows maybe eight and a half stars?  What exactly are they trying to say?  Yes they are putting their reputation on the line when they give ten stars, but this will happen so infrequently that it will be harder to judge their overall trustworthiness.

Evaluation systems with fewer and grosser distinctions are often more credible because they are easier to monitor.

1 Anonymous February 20, 2009 at 6:34 am

could it be they’re trying to catch up with pitchfork? how do you read their rating system?

2 Ben Maritz February 20, 2009 at 7:03 am

Interesting. How many different ratings to the credit agencies give out?

According to Wikipedia, Moody’s divides securities into: Aaa, Aa1, Aa2, Aa3 A1, A2, A3, Baa1, Baa2, Baa3, Ba1, Ba2, Ba3, B1, B2, B3, Caa1, Caa2, Caa3, Ca, C…

3 Steve February 20, 2009 at 7:08 am

Spin… forgot about that one

4 beamish February 20, 2009 at 8:32 am

If you find it easier to understand what they’re saying when it’s expressed in five stars rather than ten, divide by two.

5 David Jinkins February 20, 2009 at 8:54 am

The truly brave reviewer uses only one non-divisible star. I challenge you, Tyler, to start doing restaurant/book reviews this way.

6 mk February 20, 2009 at 9:24 am

What this analysis neglects is that the “reputation-staking” force of a high rating is as much a factor of its percentile as its absolute rating.

So if Pitchfork gives out 4 9+’s a year, I will pay close attention unless/until I stop trusting them.

More precisely, the “degree of reputation-staking” of a high rating depends on the percentage of albums that rate “significantly” lower. “Significantly” meaning that e.g. 8.7 and 8.5 are considered as having the same rating.

Now, if readers don’t in fact interpret rating systems in a percentile way, then maybe Tyler is right that a finer-granularity rating system is an indication that the publication is trying to make a smaller splash with its ratings.

7 Anonymous February 20, 2009 at 10:02 am

Interesting post.

★★★★★★★★½

8 john February 20, 2009 at 10:13 am

This kind of thing all started when they made that amplifier that went to 11. Never should have done that.

9 A. Spirts February 20, 2009 at 10:20 am

In BKS Iyengar’s Light on Yoga he goes up to sixty stars and feels the need to include half stars.

10 Andrew February 20, 2009 at 10:28 am

Don’t we all just want someone to say “you will like this.”

Maybe that’s why we put up with friends.

11 John Voorheis February 20, 2009 at 10:44 am

It does appear to be a nod to Pitchfork. Pitchfork’s problem, though, is that they almost never award newly released records a 10/10. The last was in 2005, for Scott Pollard’s “Relaxation of the Asshole.” Before that it was 2002, for “Yankee Hotel Foxtrot” by Wilco.

12 Norman Maynard February 20, 2009 at 1:00 pm

I’d also like to hear about the extension of this argument to grading systems. Do plus/minus systems add information about student performance, or subtract it?

13 Thomas B. February 20, 2009 at 4:59 pm

I almost exclusively read film reviews through the filter of Metacritic, which converts all the written reviews to a 100 point scale. I find the small scales typical of newspapers uninformative.

I wrote more here.

14 Bernard Yomtov February 20, 2009 at 7:01 pm

A ten-star system is silly.

A distinction between 8 and 9 on something as subjective as an opinion about a movie or restaurant is meaningless to me. Who buys this CD instead of that one because of that difference, or worse, the difference between an 88 and an 85?

A coarser scale conveys useful information as to what broad category something fits into – it’s a stinker, etc., without presuming to measure quality with unattainable precision.

15 Tom West February 20, 2009 at 10:10 pm

I find the small scales typical of newspapers uninformative.

A finer grained system is only relevant if the rating criteria support that level of differentiation. In all probability it doesn’t. I expect that if a movie critic had his memory wiped and re-reviewed a movie, the rating could easily fluctuate up or down a star in a 5 star system depending on the phase of the moon.

For a physics problem, you don’t say the answer is 5.346 +/- 2

The drive to measure things to 3 digits of accuracy is simply a delusion for those who desperately want there to be accurate metrics for things as subjective as movies, books, or wine.

However, given there’s a pretty big audience of people who want to believe it, why should a magazine disabuse them and lose sales?

16 Nigel Tufnel February 21, 2009 at 5:39 pm

“This all started when they made that amplifier that went to 11. Never should have done that.”

Alright, but most blokes are playing at 10, and if they need it a bit louder, where can they go? Nowhere.

Comments on this entry are closed.

Previous post:

Next post: