The notion of a “skill anomalous position”

Here is from MIT Technology Review, surveying research on chess blunders and cognition by Ashton Anderson at Microsoft Research in New York City, Jon Kleinberg of Cornell, and Sendhil Mullainathan:

…Anderson and co have found evidence of an entirely counterintuitive phenomenon in which skill levels play the opposite role, so that skillful players are more likely to make an error than their lower-ranked counterparts. The team call these “skill anomalous positions.”

That’s an extraordinary discovery which will need some teasing apart in future work. “The existence of skill-anomalous positions is surprising, since there is a no a priori reason to believe that chess as a domain should contain common situations in which stronger players make more errors than weaker players,” say Anderson and co. Just why this happens isn’t clear.

I don’t, by the way, find the concept of skill anomalous positions to be so surprising.  Better chess players have more “chunking” and more intuitions.  Usually that knowledge adds value, but in a variety of counterintuitive positions it can lead players down the wrong paths.  For instance a beginner probably does not know that on average a Queen and Knight working together are more effective than a Queen and Bishop, yet this is not always true and the less tutored intuition will sometimes prove correct.  Similarly, the better player may think that an endgame of Bishops of opposite color is more likely to be drawn, and often that is true.  Yet in other situations those ill-matched Bishops can yield an attacking advantage to the player with the better command of space, and so on.

I believe there are analogous concepts for economics and also philosophy, probably for other disciplines too.  For instance in economics I wonder if a person with less knowledge of open economy macroeconomics might sometimes end up making better forecasts.  Many anti-elitist theories of politics imply these phenomena can be true in a broad range of situations, Brexit for instance according to some.

Comments

"Ignorance is Strength"

But joe daddy is okay with it.

i imagine film historians would make lousy executives, or at least there is no evidence to suggest they'd make good ones.

This may also be an example of a well studied phenomenon in the psychology of learning, often referred to as "u-shaped learning", where a person who is in the process of mastering a domain is misapplying a rule or heuristic they have learned but not yet mastered. It's called u-shaped because when you plot performance (x-axis time, y-axis performance) novices and experts outperform those at intermediate levels. For example, 1st year med students will outperform 4th year students on average most diagnosis tests, but graduates outperform both. Or how children will start of correctly inflecting irregular verbs (eg "broke"), then will go through a phase of over-applying the general past-tense rule (and thus say "breaked"), before eventually returning to correct form.

This occurs in almost over domain where you look for it. Learning is about discovering regularities. Expertise is about NOT over-applying them.

Interesting. I say to the guys I'm training 'when you hear hooves, think horses not zebras'. If we have at our disposal lots of patterns we sometimes try to make things fit the pattern as opposed to finding the best match.

Most are simple and straightforward, so start there. If you find it doesn't fit, start looking for the more arcane or rare patterns.

Your source for the claim about med students? I'm calling BS on that one.

There is much work on this, but you can get a good start by checking out various papers, books, and chapters by Patel and Groen. Here are two examples, google scholar their names and you will get many more:

Patel, V. L., Groen, G. J., & Scott, H. M. (1988). Biomedical knowledge in explanations of clinical problems by medical students. Medical education, 22(5), 398-406.
Patel, V. L., & Groen, G. J. (1991). The general and specific nature of medical expertise: A critical look. Toward a general theory of expertise: Prospects and limits, 93-125.

Most medical school curriculums don't teach much about disease the first year; it's all normal physiology and anatomy. How could they make a diagnosis without knowledge of disease?

Most medical school curriculums in the United States don't teach much about disease the first year; it's all normal physiology and anatomy. How could they make a diagnosis without knowledge of disease?

Perhaps some knowledge of diseases leads to misdiagnosing which can have much worse results than no diagnosis. Or simply acknowledging, "if it smells like a pig, its a pig" for a simple flu etc. rather then what ever exotic diagnosis 4th year med student is being taught.

The work wasnt evaluating standard testing done on the instructional curriculum, but instead experimental, giving them novel diagnosis tests that they may or may not have encountered. Check out the links I posted in the response above and you can get more information on what they looked at if you are interested.

Yes but if you are inexperienced enough to not recognize those standard positional advantages, will you have the skill to cash your advantage in case of an anomaly?

Probably the silliest mistake people with skill and experience is putting something in the wrong slot. We are pattern matching machines, and our brains go for matches, and the more skill or expertise the more slots we have to put things into. And the more confident we become at quickly deciding what slot of belongs in.

Anyone in a troubleshooting endeavor knows how wrong, very wrong we can be, and the good ones develop systems of thought that constantly check. And it usually means checking the mundane details, starting from the very basics of the endeavor.

Chess is safe, it you lose so what. But lots of things people do need to be right, and these basic checklist for checking arrangements can catch lots of mistakes.

I am reminded of Sochi game 6 where Carlsen in a superior position blundered ( 26.Kd2?), Vishy too focused on trying to get counterplay with a4-a3 blundered in turn 26..a4 missing the opportunity to completely turn the game around with 26... Nxe5! Carlsen went on to win the game and said afterwards he had been very lucky. A lesser player than Anand would likely have played the correct move in this position.

As far as forecasting there are some spectacular failures by experts: a well known one is : "I think there is a world market for maybe five computers." Thomas Watson, president of IBM in 1943

or more recently. " Mission accomplished" Bush on Iraq (5/1/2003)

May be Bush did accomplish his mission ; a tale of a father, a son and a ghouly host.

"Although Watson is well known for his alleged 1943 statement, 'I think there is a world market for maybe five computers,' there is scant evidence he said it."

https://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_attribution

I've heard Bush called lots of things; expert isn't one of them.

Here is a tangentially related phenomenon (documented by computer scientist RJ Lipton) in which "the proportion of large errors by human players in positions where computers judge them to be a tiny fraction of a pawn ahead is under half the rate in positions where the player is judged ever so slightly behind": https://rjlipton.wordpress.com/2016/01/21/a-chess-firewall-at-zero/.

Maybe it is just data mining. If you have a big enough data set and enough "forking paths", you will always find some fascinating counter-intuitive and completely spurious correlation with a low p-value.

Situations where more skillful players make more errors.
Situations where men make more/less errors than women.
Situations where gap between actual vs. predicted error rate is higher for skillful women than for lower-ranked men.

Beginner's Luck had to come from somewhere.

Was the Brexit vote a skill anomalous position? :)

IIRC, from "Zen Mind, Beginner's Mind", in the beginner's mind there are many choices, in the expert's mind, few. And if this has to have relevance to the UK vote, most of the experts, including TC in his recent long post, fail to mount any defense of the EU. The whole thing is explained by the stupidity, xenophobia and nostalgia of the majority. Forget any new terms; this is simply myopia. In ruder terms, it's having your head up your ass.

This reminds me of the scene in The Princess Bride where the villain Vizzini (played by Wallace Shawn) engages in mental gymnastics in his attempt to pick the goblet without poison. Those scholars at MIT may think they are smart but their "discovery" was made into a hilarious movie scene 30 years ago. Maybe they missed the movie. Recommended.

+1 for Princess Bride reference. However, Vizzini was clearly not the more skilled player.

Vizzini was clearly just trying to cheat. He swapped the goblets, then waited to make sure the man in black drank first.

Vizzini's error was putting down his dagger in the first place, allowing the more skilled combatant to approach without danger to his hostage.

After that, while the poisoned chalice riddle did have a solution that wouldn't end with a poisoned Vizzini, the man in black could just have easily killed Vizzini by the sword at that point.

Would he have killed an unarmed Vizzini? I don't think so.

I suppose sometimes an aeronautical engineer forgets things fall, something a novice would never do.

There is a whole genre of bridge stories usually called either "little old lady" stories, or after the bridge write Victor Mollo's anecdotes, Rueful Rabbit stories. In these hands, experts are fooled by plays and make demonstrably inferior decisions which they would never make with less expertise and thus lose hands to far less skilled players. The best of these stories even have the expert account for his skill difference and still get fooled.

"There is a literature on everything"

Similar dynamic in poker. Guy runs a perfect bluff; forgets that his opponent is too green/stubborn/drunk to fold in any case.

"Experts" sometimes over-generalize the value of their heuristics. Move along nothing to see here.

"I don’t, by the way, find the concept of skill anomalous positions to be so surprising."

Yeah, how is this different from the concept of a trap? The whole point of a trap in a game is to set up something that looks tempting, but eventually leads to a worse position. To make one, you have to make it look like a good outcome to an opponent of the level of skill you're facing (i.e., looks good at the level of the game tree they are able to search, say 10), but have the bad outcome not visible at that level of skill (i.e., far enough down the game tree such that the lower skilled player can't see it, say 13). Is the idea that the score needs to change sign twice? I.e., negative 0-7, positive 8-12, negative 13 on? So mid-skilled players fall for the trap?

I didn't read the article. Maybe this is all explained in there.

This idea of "skill-anomalous positions" could also be used to analyze ideology in roll-call voting: https://pileusblog.wordpress.com/2014/12/05/this-is-what-a-legislature-with-a-bunch-of-libertarians-looks-like/

The phrase "Innovation comes from the periphery" comes to mind. As does Orwell's "There are some ideas so wrong that only a very intelligent person could believe in them." Or consider investing, where quite often the more you know the worse you do (e.g., high IQ people who become day traders during booms compared to simple Index investors over the long haul).

Probably other explanations as well. Testing requires differently skilled players dealing with the same position. A testing bias could result from the novice merely trying to find a good move, where the expert assumes he needs to find something counterintuitive or tricky & discounts simple sound moves while overlooking tactical replies to his more complicated visions.

The Championship game is nothing like that; title matches are often tainted with blunders due to the pressure. Fischer's piece sac (which could have drawn with proper play to follow but was still an awful oversight), Tal in his rematch with Botvinnik - and if you gave Karpov a single move back in each of his five matches with Kasparov, there would have been a different situation & he might even have won them all.

But the mutual blunder is common even at the grandmaster level. One case from the '60s saw 2 GMs both overlook a mate in one for two turns. Players sometimes 'trust' the others skill to their detriment.

The paper is here: http://arxiv.org/pdf/1606.04956v1.pdf

They omitted a key character in the FEN strings (compact representation of a chess position) in Figure 6 (showing the skill anomalous positions) -- the character indicating whether it is white or black to move! All three positions are basic king and pawn endgames, but without the missing character I can't tell whether it is white failing to win an obviously won position (if white to move) or black failing to draw an obviously drawn position (if black to move). All three positions relate to the positional idea of "opposition", and should be trivial to solve for even a very low-rated player if they simply calculate out the next 2-3 moves. Higher rated players must be choosing not to do so, instead relying (like Tyler said) on intuitions about opposition that are not quite right in positions where the pawn is in front of the king as opposed to behind it.

Any other chess players care to comment?

Love to but I'm making a sandwich.

I couldn't understand the graph in the linked article.

I find myself wondering if "skill anomalous positions" exhibit such anomalies all the way to the top of the skill range. I could very well imagine mediocre players having a glaring weaknesses, while Magnus Carlsen might make fewer mistakes in all positions.

Comments for this post are closed