When is coarse grading better?

by on February 4, 2018 at 2:52 am in Economics, Education, Food and Drink | Permalink

(8) Coarse Grades: Informing the Public by Withholding Information, by Rick Harbaugh and Eric Rasmusen

Certifiers of quality often report only coarse grades to the public despite having measured quality more finely, e.g., “Pass” or “Certified” instead of “73 out of 100.” Why? We show that coarse grades result in more information being provided to the public because the coarseness encourages those of middling quality to apply for certification. Dropping exact grading in favor of the best coarse grading scheme reduces public uncertainty because the extra participation outweighs the coarser reporting. In some circumstances, the coarsest meaningful grading scheme, pass-fail grading, results in the most information.

Here is the link to American Economic Journal: Microeconomics.  Of course another mechanism favoring coarse grading is that corrupt grades are easier to spot.  If too many one-star Michelin restaurants are slid up to three stars, it is obvious something is going on.  But if on a scale of one hundred a restaurant that ought to be a 67 is given a 73, who is really to say what those numbers are supposed to mean?  There are many market settings where the coarser grading scheme is preferred over the finer alternative.

1 michael February 4, 2018 at 3:38 am

They mention various other possible mechanisms. I was surprised that they do not include a direct information-theoretic one: the information transferred may in fact be maximised by a signal with discrete steps, when there is some measurement noise.

In economics this idea is C Sims, “Rational Inattention”:

https://www.aeaweb.org/articles?id=10.1257/000282806777212431

http://sims.princeton.edu/yftp/RIDiscrete/DiscreteTracking2PropX2.pdf

Reply

2 sci-hub robot February 4, 2018 at 3:42 am

Here’s a link to the paper: https://sci-hub.name/10.1257/mic.20130078

Reply

3 clockwork_prior February 4, 2018 at 10:38 am

Slipping under the radar, it seems.

Reply

4 jb February 4, 2018 at 3:57 am

It also makes the grades easier to understand for a casual observer.

Is 73 out of 100 good? Is it notably worse than 77 out of 100? What is the mean/median score? What is the minimum score one should accept?

The consumer is trying to make a decision (how much, if any, money to spend at the establishment). They only need the information required to efficiently make that decision. Any more information just adds to their transaction costs.

Reply

5 JWatts February 4, 2018 at 5:15 am

At an office happy hour terrifying my coworkers with my shocking ironic perspective. “The Cat In The Hat movie was great,” I mumble while tilting my chin down and smiling like Gomer Pyle. “Mike Myers turned in a solid performance as the cat.”

Reply

6 Ace-K February 4, 2018 at 6:31 am

I like how they had to specify that “pass-fail” is the coarsest *meaningful* grading scheme.

Reply

7 rayward February 4, 2018 at 7:19 am

Many elite law schools don’t rank their students; they are all top tier according to these schools. There’s logic to this: since of the students admitted to the law schools have stellar records as undergraduates and top scores on the LSAT (the “top tier”), ranking them while in law school would be pointless and maybe even misleading. My state law school ranked the students, from the first semester to the last. The school also had a mandatory curve grading system (a “normal” curve). What’s odd is that the students’ class ranks didn’t change much over the three years, and seemed to follow them from class to class and semester to semester like a scarlet letter. Yet, the school also had a blind anonymous grading system (BAGS for short) to prevent the professors from knowing the identities of the students when grading. Many of the students with a lower class rank were convinced that the BAGS was a fraud, that the professors had the list of BAGS numbers assigned to the students. The conspiracy would have to be established each semester since the BAGS numbers changed from semester to semester. Assuming the BAGS was not a fraud, what it seemed to proved is that a “normal” distribution applies at every level, including the top tier of undergraduates admitted to the law school.

Reply

8 rayward February 4, 2018 at 7:48 am

The system we used in law school wouldn’t likely work for a graduate program in economics if for no other reason than there are far fewer students. I studied economics as an undergraduate and considered grad school as well as law school, and I applied at several top tier grad schools. The competition for the positions at the grad schools was much stiffer simply because the programs accepted so few students. It as some 45 years ago, but I recall being interviewed by several of the grad programs where I applied. An interview? There were no interviews by the few law schools where I applied. My impression was that the grad programs invested so much in their grad students that mistakes made in the admission process would be felt much more severely than mistakes made in the admission process by law schools (where it was expected that many of the students would drop out or flunk out).

Reply

9 Art Deco February 4, 2018 at 9:42 am

My impression was that the grad programs invested so much in their grad students that mistakes made in the admission process would be felt much more severely than mistakes made in the admission process by law schools (where it was expected that many of the students would drop out or flunk out).

No, they just need an opaque process to limit admissions to youths who share their worldview. Actually ejecting someone from the program is a more unpleasant process and (nowadays) generates potential liability.

Reply

10 Slocum February 4, 2018 at 7:31 am

“But if on a scale of one hundred a restaurant that ought to be a 67 is given a 73, who is really to say what those numbers are supposed to mean?”

But the grading steps must be arbitrary. The restaurant given a 67 instead of the 73 it deserved might end up getting 2 stars instead of 3. In such cases, the course grading would amplify minor differences and mislead consumers.

Reply

11 William Gadea February 4, 2018 at 10:16 am

The more steps there are, the more chances there are for misclassification. But the more steps there are, the lower the degrees of misinformation.

Also worth noting: misinformation may matter comparatively little when it communicates opinion that is subjective anyway. Even information that is more objective, like the academic achievement reflected in class grades, is unlikely to ever come face to face with a canonical truth. It’s the predictive disciplines that always find their test in reality. A difference between a prognostication of a Category 3 or a Category 4 hurricane is obviously significant.

Another interesting question to me is what is the optimum number of categories for the human mind to appreciate, and under what circumstances does that number change. I think a great deal of the success of the Michelin guide is probably because of it’s simple 3-star system. If it were 4 or 5, I’m not sure it would be the hit it is.

Reply

12 dearieme February 4, 2018 at 8:59 am

When I was an undergraduate our pass/fail exam results at the end of each academic year were pinned on university notice boards for the world to see. However our end-of-term exam results, %age marks, were posted only on departmental note boards.

Our results in Finals (first class, upper second class, and so on) not only adorned the university notice boards but appeared in the national press.

That’s how to do it! Pre-snowflake era, of course.

Reply

13 Art Deco February 4, 2018 at 9:38 am

That’s how to do it! Pre-snowflake era, of course.

That’s how to do it if the university is too cheap to purchase stationary and postage stamps. Which, evidently, British universities were.

Reply

14 dearieme February 4, 2018 at 5:12 pm

Why on earth do you invent the notion that they didn’t also post the results to us? Do you always make up factoids to suit yourself?

Reply

15 Art Deco February 5, 2018 at 8:51 am

Why on earth do you invent the notion that they didn’t also post the results to us?

Because it wouldn’t make sense to put the results on a bulletin board if you’ve sent them in the mail to the recipients. They put the results on the bulletin board in order to embarrass people. They do that because they’re members of the British intelligentsia.

Reply

16 CD February 4, 2018 at 1:15 pm

The Family Educational Rights and Privacy Act (FERPA) that prevents the display of grades is also known as the Buckley amendment, after sponsor and staunch conservative James Buckley.

https://en.wikipedia.org/wiki/Family_Educational_Rights_and_Privacy_Act

Reply

17 Boonton February 4, 2018 at 9:15 am

What does this imply for IQ fanatics who insist tiny variations in IQ scores have great importance?

Reply

18 Art Deco February 4, 2018 at 9:36 am

I’ve never seen that contention in print, even in the articles Arthur Jensen wrote for general audiences. What they do say is that IQ tests are better predictors for job performance and life trajectories generally than any other metric. They may be wrong about that, but that’s a different contention than the one you’re attributing to them.

(The laymen among them also assume that social relations are derived from the sum of individual characteristics and that the only characteristics of interest are biologically-determined).

Reply

19 Boonton February 4, 2018 at 12:44 pm

Problem: Metrics are pretty cheap. If you are correct that IQ works best as a predictor compared to all other metrics, the real question is how good is it by itself? For life insurance, age is an excellent metric. It’s so good that many companies will sell policies just based on the age of the applicant (with some exclusions like known cancer). Yet it becomes a better predictor if you combine it with other metrics….such as age plus blood pressure or age plus the results of a very basic physical exam.

On the other hand if all single metrics work 8% of the time and IQ works 9%, it may well be the best single predictor but it doesn’t follow a policy that focuses on IQ boosting is a wise allocation of resources.

Reply

20 Art Deco February 4, 2018 at 9:29 am

Another economist who has lost interest in economics. (But fancies clever excuses for things which make his life easier).

Reply

21 Doug February 4, 2018 at 9:52 am

> If too many one-star Michelin restaurants are slid up to three stars, it is obvious something is going on.

Despite this, there’s definitely inconsistency in the Michelin system. Chicago is definitely graded on a harder scale then New York. Which is definitely graded on a harder scale than Hong Kong.

Reply

22 Anonymous February 4, 2018 at 10:45 am

I split my time between two counties. One has pass-fail health department ratings for restaurants, the other has A, B .. and I am not really sure if there is a C or that is the fail.

I can see there might be some arbitrary negotiations at the A/B boundary, but I do find an A in the window reassuring.

On the other hand, when I am in pass-fail territory, I don’t worry about it. So maybe that is the one I actually prefer.

Reply

23 Anonymous February 4, 2018 at 10:55 am

Though maybe I would trust a sketchier looking kabob place with an “A” than with a “pass.”

(I don’t actually go into the sketchy kabob places. As I drive by, I just wonder if I should.)

Reply

24 mkt42 February 4, 2018 at 6:35 pm

Yes, I think restaurant-goers in Los Angeles had a very similar experience (or maybe that’s where you had your experience). I may’ve seen maybe one or two “C” signs in all my years in LA (where a C is a passing grade). A restaurant that got a C clearly felt strong market pressure to raise its grade.

(Or maybe the restaurant inspectors, like professors at Harvard, simply give out very few Cs but I don’t think the inspectors are like that; they can and do give failing grades to restaurants so I don’t think they hesitate to give out Cs.)

I wonder if that might be the most important outcome of LA’s switch from pass-fail to letter grades: the grade of C became a kinda sorta failing grade, and the vast majority of restaurants work hard to get their grade up to B. I.e. beyond passing.

So maybe restaurant health levels are higher in LA now than under the pass-fail system? And the other question, from a cost-benefit perspective, is is it worth it, or are the inspectors inducing the restaurants to over-engage in improving health and cleanliness?

Most Angelenos seemed to prefer the letter system when it was implemented, but with mild and mixed feelings similar to what you report.

Reply

25 tom hynes February 5, 2018 at 12:08 am

If you don’t get an A grade, you can request an “Owner Initiated Inspection” and get re inspected. Therefore, the only B and C grades you see in the wild are the places that couldn’t clean even keep the rat shit off for a few days.

Reply

26 dux.ie February 5, 2018 at 1:54 am

It is usual that people live and interact with people of the about the same abilty and background and it is understandable that they tend to extrapolate from that position. However, to be able to draw conclusion across wider population a more comprehensive hard data are needed. The median wage early career data from the Fed is given here, https://en.wikipedia.org/wiki/Graduate_unemployment#College_major_by_underemployment_rate . The university major entry IQ estimated from the SAT score is available here, http://www.statisticbrain.com/iq-estimates-by-intended-college-major/ . Merging the above data gives two charts.

With respect to the under-employment rate it is clearer that the data split naturally into two clusters with the demarcation IQ about 115. http://tinypic.com/view.php?pic=10n5ph5&s=9

With respect to the median wage using the demarcation IQ at 115, the characteristics are also different for the two clusters, http://tinypic.com/view.php?pic=30ms2n8&s=9

Reply

27 dux.ie February 5, 2018 at 1:58 am

For uni major entry IQ ≤ 115, there are no statistical significant trend within the range 103 ~ 115. The higher wage outliers are those dealing with confidential information (BusinessAdmin, Accounting) or life situation (Nursing, MedicalTechnician), otherwise statistically the salaries are not dependent on IQ. However, for uni major entry IQ > 115, the wage level statistically increasing with uni major entry IQ,

MedianEarly = +1791.41*IQ -178043; # n=36; Rsq=0.4013; p=3.367e-05 *** (very signigficant)

In this region on average each additional IQ point increases wage by $1791.

Your world view most probably depend on where you are.

Reply

Leave a Comment

Previous post:

Next post: