What makes most restaurant reviews worthless

Not just the general reason why they are bad, but rather a very specific reason.  Caitlin Dewey reports about:

…a new paper appropriately titled “Demographics, Weather and Online Reviews.” The study analyzed 1.1 million online reviews of 840,000 restaurants, looking for exogenous — or external — factors in the data. In other words, they wanted to figure out what makes us like or dislike a restaurant, beside the restaurant itself.

The results can be surprising. The diners’ education levels? No effect on actual ratings. Population of the area? Again, not so much.

But reviewers consistently gave worse ratings when it was raining or snowing outside than when it was clear. And reviewers usually liked restaurants better on warm and cool days, rather than very hot or very cold ones.

In researcher Saeideh Bakhshi’s words: “The best reviews are written on sunny days between 70 and 100 degrees … a nice day can lead to a nice review. A rainy day can mean a miserable one.”

Not surprisingly, restaurants in California and Hawaii are popular.


The specific reason seems to reinforce the more general reason.

Restaurant reviews are hardly worthless and it is the height of arrogance and condescension to suggest that they are.

Seriously can't tell if this comment is genuine, or it's sarcastically mocking the self-righteousness of the average yelp reviewer.

Well, it is up there in the heights of arrogance and condescension to suggest that saying reviews are worthless is the height of arrogance and condescension, but reviews aren't worthless for anyone capable of exercising a modicum of judgement,

I live in New York. The restaurants here generally all share the same weather, so I'm not sure this finding should factor in at all when I'm comparing a few options to pick a place for dinner. Maybe this explains why Yelp, for example is "useless" in comparing restaurants in Manhattan and LA, but how often is it really used for that?

Restaurants are reviewed on different days.

Most established restaurants in NYC have hundreds if not thousands of reviews. Unless some of them have unique policies that skew their reviews towards better or worse weather, I would expect this effect to be minimal.

The point is more that it makes it difficult to compare reviews in your area across temporal gaps.

So you're trying to decide whether to go to restaurant A -- either vs. restaurant B or against a generalized field of "I know generally how good a restaurant has to be for me to be interested." There is a review of restaurant A, dated in the last three months. The review is a rave review. Does that reflect the quality of the restaurant or the quality of the day? Likely, you have no idea how nice that day was. If you were comparing to another review on the same day of restaurant B, perhaps it would be a fair comparison, but it is very unlikely to be the case that there is another review the same day.

Yes, the distinction between individual reviews and average scores is helpful. I generally wouldn't recommend basing a dining decision on the brief reaction of an internet commenter you know nothing about. But Yelp's aggregate ratings remain quite useful.

Funny I have the opposite sense. I don't go by the numbers, just read some comments (sometimes reading other comments by the same commentors to get a feel for their reviewing style). Some people put in zero stars because delivery took too long, if I'm eating in I don't care.

I probably overweight good writing though.

That's exactly what came to my mind.

I tend to generally agree with the average Yelp ratings for my area. Assuming the place has been open for a while and the number of reviews isn't too few it is quite useful.

Eat frozen pizza at home when it's raining.

I'm going to have to give the "frozen pizza" a 1 Pepperoni out of 5.

It's better if you warm it up.

I found reviewers to have lower baseline "good" ratings in New York than they did in Boston.

And I find that reviewers have a lower baseline "Excellent" rating in Boston than in New York!

It's possible, in fact likely, that both of these comments are correct. "Good" in NYC is not very good, but "excellent" is very good indeed, both relative to Boston. I.e., the top-end in New York goes higher, but the average is over-priced junk. It's just like housing.

If bad weather puts restaurant reviewers in a bad mood, it may do the same to restaurant staff. Either way the important lesson is not to eat out when the weather is ugly.

So you will enjoy eating out more when the weather is nice. However, what's important is how much more you will enjoy eating out to eating in depending on the weather. So eating out when the weather is bad may bring only 5 hedons, while it will bring 7 hedons when the weather is good, but staying in may bring only 1 hedon when the weather is bad, and 5 while the weather is good. Thus eating out gains me more when the weather is bad.

The type of person probably plays a part. I've written maybe five reviews of anything in my life. I'm free with recommendations to friends and acquaintances, but I'm not interested in giving Yelp or Amazon free labor. There's nothing in it for me. That leads me to think it is a certain type of person who writes the bulk of on-line reviews, a person who gets satisfaction from the act.

Then there is the probability that some number are written by robots. I've seen a lot of Amazon reviews that scan like they were written by Eurasians who spam my site with offers for cheap sneakers.

In the past month you have written reviews of Whole Foods, Home Depot and you local light bulb store. You review everything.

Ok, I've written three reviews, not five.

Some of you people are such sticklers.

That's nothing new and not unique to restaurants. It has been known since at least when I was an undergrad in the '90s that the #1 thing that will impact a high school student's opinion of a school from a campus visit is the weather that day. Opinions of campuses can flip if they visit a second time under opposite conditions. This was in Boston, which is a pretty good case study because of the close proximity of many large campuses that are very diverse in character that many prospective students visit in back-to-back days twice (once before applying, once after being accepted). Harvard, BC and Tufts are very "campus-y". BU and Northeastern are very urban in character. MIT has a foot in both camps.

So what you're saying it, we need seasonally-adjusted Yelp! ratings.

Shouldn't be too hard for them to manage internally. 'Course, you'd have to use the day the review was posted rather than the day they actually went to the restaurant (which isn't necessarily known).

There may also be anchoring/expectation effects. Suppose the first N reviews for restaurant X just happen to cluster around rainy days. Based on past less-than-stellar reviews, reviewer N+1 adjusts expectations downward. This serves as an anchoring point, causing review N+1 to be lower than it otherwise would.

So be sure to open your new restaurant in the summer. Also, don't call it "Rainy Day Café."

Lower to middle rated reviews are typically the most honest due to the prevalence of bogus inflated reviews by shills. Amazon.com has had that problem with its e books where fake reviews are common.

What a very clever study! Great catch.

I doubt weather is the only explanatory variable. Could restaurants be better in California / Hawaii? There will clearly be different people reviewing restaurants in those places. In Hawaii, a higher percentage of people will be on vacation reviewing restaurants as well which should have a big impact on the ratings.

It rains almost every day in Hilo, HI, on the the west side of the Big Island and the food's pretty good.

I refuse to accept this.
I live in Denver.

840k restaurants and 1.1m reviews? Seems the reviews/restaurant are quite low.

This is stupid reasoning from a stupid article. Now let me go get my down jacket, I'm freezing!

I think Tyrone slipped out and accidentally wrote the first sentence of this post. He deserves at least a co-author credit.

How did they know what the weather was when the reviewers made their review? Surely they don't all start with "So I was in Chattanooga on a sunny Tuesday morning..."

The only way this would bias the aggregated restaurant scores is if the weather is somehow correlated to the choice of restaurant itself. This may actually be true in a few specific cases (for instance, restaurants with outdoor seating probably get more patrons and therefore more reviews on sunny days), but for most restaurants weather is an exogenous variable which introduces noise but does not introduce bias the overall aggregated score.

One important thing that the study highlights is that the weather does in fact affect your enjoyment of a restaurant. If we assume you can't predict the weather when you are making your restaurant reservations, then the aggregated yelp score actually gives a very accurate view of your expected enjoyment, because it already factors in the impact of historical weather patterns in the area.

Okay, as a shameless plug for a now defunct broker of weather risk, the clear answer is that restaurants need weather contingent pricing. I could see the reviews of the future:

"On a beautiful, sunny day in May, I had an excellent meal, albeit more expensive than I expected" followed by "On a rainy Sunday evening, I had a nice dinner, service was meh, but what a bargain"!

The perfect example of bad science covered by a really bad reporter. There's a reason you're supposed to have more than one source.

Well, what else is correlated with weather, or more specifically, the number of sunny days.

Ever live in NY. Ever hear New Yorkers whine about food. This is an example of where a bad weather climate tracks bad reviews.

And, it could also be that better weather places -- places which attract tourists -- also attract better restaurants, or only the best survive.

It seems to me that it would be interesting to see this same study conducted with professional food critics. That is, do food critics also fall prey to giving poorer ratings when the weather is "lousy?"

Since online reviews can be quite self-reinforcing (subsequent reviewers less likely to be anchored by already posted reviews), maybe a lesson here for when you want to open your restaurant - do it during good weather so the "seed" reviews that will serve as anchor will be more positive!

And/or make an extra effort for your customers on bad weather days to compensate - "here's a free coffee, I know it's cold out there today."

So ignore all restaurant reviews that don't specify the weather.

Given the above point that most comparisons are within-geography, only places with multiple microclimates (cold in SF, warm in Oakland) need adjustment.

I thought restaurant reviews should be based on experience with the restaurant, the cuisine, the internal ambiance etc.. If the weather influences reviews, is this an example of how people can be irrational, violating the rationality assumption of neoclassical economists?

Comments for this post are closed