The evolution of successful prediction

Therefore, on balance, our results suggest that Danto was substantively correct. As the number of events being evaluated grows, successful predictions will be increasingly outnumbered by events that seem insignificant at the time, but which come to be viewed as important by future historians in part because of events that have not yet taken place. More generally, our results provide further evidence for the observation that the combination of nonlinearity, stochasticity and competition for scarce attention that is inherent to human systems poses serious difficulties for ex ante predictions—a pattern that has previously been noted in outcomes such as political events, success in cultural markets, the scientific impact of publications and the diffusion of information in social networks. Given that historical significance is typically evaluated on longer time scales than these other examples, it is especially vulnerable to unintended consequences, sensitivity to small fluctuations and reinterpretation of previous information in light of new discoveries or societal concerns. A further complication is that historical significance, even when it can be meaningfully assigned, is specific to observers whose evaluation may depend on their own idiosyncratic interests and priorities. Although we speak of history as a single entity, in reality there may be many histories, within each of which the same set of events may be recalled and evaluated differently.

That is from Joseph Risi, Amit Sharma, Rohan Shah, Matthew Connelly, and Duncan J. Watts in Nature, in their new piece “Predicting History.

Via William A. Benzon.


The number events occurring on the number line grows faster than predictions the can be unique. Looks like another prime number bounds problem. Unique predictions organize, or reorder the past uniquely, the ordering cannot be subdivided a prime numbers. But the look back must be finite, we cannot look as far as zero, and mostly a lot shorter. Most of the predictions will come from getting a 'complete sequence', looking back that has a unique and minimum generator of the typical events over the period. Historians make digits and fame with 'era's' or 'epoch', and it is mostly finding that complete sequence looking back.

'but which come to be viewed as important by future historians in part because of events that have not yet taken place'

And then, they may come to be viewed as unimportant by later future historians in part because of events that have not yet taken place when the first group of future historians made their judgments.

But then, would a rising rate of measles infection be considered a prediction when it occurs after a declining rate of measles vaccination, or is cause and effect something that is not included when talking about predicting things?

The Battle of Waterloo was very quickly recognized as historic.

Decisive battles are often in a different class of evident historic imporant than elections because the losers really, truly lose: e.g., the Napoleonic Wars finally ended because Napoleon was captured after losing at Waterloo and imprisoned on St. Helena.

What about the Battle of New Orleans, which made the young Andrew Jackson's reputation? (a proto-Donald Trump). It was also seen as historic, for Jackson, albeit a completely unnecessary battle if they had better communications.

Retrospectively, the Waterloo battle did not change the course of history and its importance is very exaggerated. In contrast, battles during the Hundred Years war changed the course of history of the world for ever despite being unnoticed. At the beginning of this war there were just a bunch of knights fighting with armors and swords and 100 years later European armies mastered enough military techniques to be able to conquer the world.

A battle that was recognized as historic at the time but has since been nearly forgotten was Michael de Ruyter's June 1667 raid on the English fleet at Medway, certainly one of the worst English defeats ever and the impetus for the Treaty of Breda, the result of which ironically allowed the continued English colonization of Mid-Atlantic North America despite the Dutch victory. Had the Dutch chosen to retain their North American colonies rather than those in the East Indies, it's likely that the US would be a Dutch-speaking country.

Yes, and the Dutch traded Manhattan for a small, 7 km long, uninhabited spice island. At the time, an economic case could probably have been made for this, but strategically it was foolish and short sighted.

Yet Napoleon's defeat at Moscow wasn't, even though it led to the original Napoleonic War finally ending because Napoleon was deposed and exiled to Elba.

Waterloo was Napoleon's encore, and when the curtain fell, everyone knew the show was at an end.

Particularly considering how the armies that beat him were poised to capture Paris less than a week after Waterloo,.

To an extent, decisive battles are just about PR. Which was more decisive, Stalingrad or Kursk? And how many people aware of Stalingrad are also aware of one of the largest armored battles in human history, the one that broke Nazi Germany?

The same is true of Midway - it was only in retrospect that it became clear that Midway was basically the equivalent of Kursk for the Japanese Empire.

Anybody who studies WWII knows about Kursk. What's interesting is that the Germans were lulled into thinking the Russians were stupid, based on their conduct earlier in WWII, and fell into a massive tank trap at Kursk in 1943. The Russians were just 'acting dumb', which the Germans did not adjust for.

'Anybody who studies WWII knows about Kursk.'

Of course, and anybody who has listened to the 1812 Overture is likely aware of the Battle of Moscow.

How many Americans born in the last 50 years have studied WWII in any sense that is not thoroughly centered on American battles? Which is completely fine - each country learns about its own history regarding the wars it has fought, and Americans are not too likely to know about how the British Empire fought a land war against the Japanese Empire (outside of Japan, of course).

According to A. Bullock (Hitler and Stalin), it was specifically Russia’s poor performance in the 1939-1940 Winter War in Finland that led Hitler to low-rate them. But this was a case of the fox and the hound--gaining Finland was not as motivating as not being exterminated by Germany.

That poor performance also led to Stalin deciding he needed to upgrade the Red Army. Of course, in 1939-1940, Hitler and Stalin were busy splitting Poland between them - and Stalin was not even convinced in 1941 that the Germans were going to invade - until they did. There is no fox and hound story to be made out of the Winter War and Barbarossa - the Nazis had always planned to subjugate the vast lands and the population of Untermenschen to the East.

Further, the initial successes of Barbarossa were truly stunning in the history of warfare, with entire armies of millions of men and their entire array of their equipment being wiped out. The slightly harder thing to know is why Hitler consistently ignored the planning of his military and acted earlier than was considered wise by those with more experience in warfare than Hitler.

@C_P - I'm not aware that Hitler's generals were cautioning aggression earlier that Hitler ignored, just that both Hitler and Stalin were fixed on Stalingrad and the Germans stopped prematurely in taking Moscow in 1941 due to (it is said) the weather. However, if Hitler indeed was cautious rather than bold (and I've heard claims he was bold when his generals were cautious, which initially worked out well for Hitler), it explains why the Germans fought so poorly after 1943. Speaking of what-might-have-been, the Nazi patriot (is there such a thing? I think so) Rommel would have, had he killed Hitler, negotiated a end to WWII and thus have been able to keep East Germany from becoming Red.

Bonus trivia: a large part of the Wehrmacht was, in WWII, drawn by horse. That sounds primitive but horses are sometimes the best form of transportation when roads are not Macadamized (never mind paved with asphalt or concrete), as they were not in the muddy, roadless plains of Russia. Horses are used by US Special Forces in Afghanistan today.

The fox and hound story (attributed to Aesop) refers to the relative motivational efficacy of desire for gain versus desire to not be killed.

Right, so Waterloo is the template for the historically decisive event that was almost instantly recognized as a hinge of history, which it indeed was.

But the vast fame of Waterloo also shows how hard it is to recognize historic events at the time.

As you point out, lots of other battles didn't seem as decisive at the time as they are now considered to be. E.g., Midway in 1942 was obviously a huge victory, but two months later the Japanese sank 3 US and 1 Australian heavy cruisers at Savo Island.

In comparing the two naval battles, it wasn't as obvious in 1942 as it is now that:

A. Aircraft carriers were vastly more valuable than gunships.
B. By losing all their carriers at Midway, the Japanese lost all their planes, and it turned out that Japan was hard-pressed to replace the ships, planes, pilots, and mechanics lost at Midway, whereas US industry and manpower made up for losses like Savo Island much more easily.

'Right, so Waterloo is the template for the historically decisive event'

Well, there was this battle at a place called Marathon - some what say that was the template, but then, we are talking ancient history.

'But the vast fame of Waterloo also shows how hard it is to recognize historic events at the time. '

Odd how Marathon has stood the test of thousands of years, while Waterloo is just as likely to suffer the fate of other decisive battles such as Lepanto or Salamis, though Napoleon does seem to have reached a certain mythic status that may continue for centuries more.

'By losing all their carriers at Midway'

The Japanese lost all of the carriers and the aircraft deployed for the battle - they did not lose all of their carriers or aircraft (six carriers and their planes were used to attack Pearl Harbor, for example). Further, Yamato considered Midway to be a decisive battle before it was fought, and it turns out he was right.

Marathon sent the Persians packing for a decade, but then they came back in 480, only to be beaten at Salamis. And even that wasn't the end, as there were two more battles the next year before the Persians went home.

Waterloo stands out as decisive in part because the personality of Napoleon looms so large, especially after his incredible escape from Elba and resumption of the rule of France.

This post by TC rings very true. For example, I've heard that Gregor Johann Mendel (Czech: Řehoř Jan Mendel; 20 July 1822 – 6 January 1884) was believed by some historians to actually not really be aware of genetics 'factors' but was more or less doing a sort of accounting; he didn't really appreciate what he was writing down. Only later did Morgan et al and that Dutch plagarist geneticist assume Mendel was aware of genes. Not sure how true that is, without reading the primary sources, but you see parallels with early inventions and simultaneous invention ( - how much was this ahead of its time and how much was imputed by later people? See the entry for the "Oruktor Amphibolos"

Bonus trivia: the Antikythera mechanism was the world's first computer! It even had carry and a form of memory! Charles Babbage predated by 2000 years! No patents back then that's why there's no steampunk revolution.

Human behavior can be manipulated. Why else would firms spend billions to manipulate behavior. The surveillance economy should greatly improve firms' ability to manipulate consumer behavior, and the surveillance state should greatly improve governments' ability to manipulate social and political behavior. The ability to manipulate human behavior is the ability to predict the future. What is the flaw in
my prediction? Human behavior is subject to self-deception. What? How many times have economists predicted that one more tax cut for the wealthy will produce a flood of investment in productive capital, great advancements in technology and human capital, and unparalleled increases in economic growth, only to be wrong. Economists are no less likely than others to predict what they want to believe, it's just that their wrongness is more visible. [An aside, not all economists are prisoners of their beliefs. Indeed, there's a cadre of economists who believe that investor and consumer behavior can be manipulated, not with conventional monetary and fiscal tools, but by manipulating behavior by creating expectations of future events. In other words, if investors and consumers can be manipulated into believing a future of economic well-being, investors and consumers will make it self-fulfilling by their actions today. The field of behavioral economics is growing, with adherents across the ideological spectrum. Relying on data that reflects the past to predict future events will soon be viewed as preposterous as the crystal ball.]

Apparently, History is more fun than economics.

Anyway, behavioral market participants in 2019 love bonds with real yields of 0.65% (30-year UST 2.65%, inflation 2%); but 1981 market participants hated bonds with real yields of 9% (30-year UST 14%, inflation 5%). In Europe they love bonds with negative yields. Go figure.

Reliance on economic data is preposterous. Likely, that's why the Fed and economics professionals are persistently blindsided by events.

Compare Tyler's quote with a passage from T. S. Eliot, "Tradition and the Individual Talent", 1920:

No poet, no artist of any art, has his complete meaning alone. His significance, his appreciation is the appreciation of his relation to the dead poets and artists. You cannot value him alone; you must set him, for contrast and comparison, among the dead. I mean this as a principle of æsthetic, not merely historical, criticism. The necessity that he shall conform, that he shall cohere, is not one-sided; what happens when a new work of art is created is something that happens simultaneously to all the works of art which preceded it. The existing monuments form an ideal order among themselves, which is modified by the introduction of the new (the really new) work of art among them. The existing order is complete before the new work arrives; for order to persist after the supervention of novelty, the whole existing order must be, if ever so slightly, altered; and so the relations, proportions, values of each work of art toward the whole are readjusted; and this is conformity between the old and the new. Whoever has approved this idea of order, of the form of European, of English literature, will not find it preposterous that the past should be altered by the present as much as the present is directed by the past. And the poet who is aware of this will be aware of great difficulties and responsibilities.

This observer predicts that various aspects of the "Patriot Act" and its ancillary legalities will produce a caste system in the US. People that fail background checks and are rejected for air travel will quickly realize that once a background check for employment, housing, travel, etc. has a negative return it will always be so. While being unemployable and unhousable, these people will still need food, clothing and a place to sleep. Ergo, they will join the underground economy, working for cash. Businesses will be formed to use their labor and provide them with necessities. This inevitability is being generally ignored but shall come to pass none the less. As time passes, the number of these people can only increase, since once on a list, one is never expunged. They are also likely to at some point be represented and led by a particularly charismatic individual. That's when the fun will begin.

Astute observation. Related:
"The equilibrium state in an anarchy is a subsistence-level free-for-all."
Squatters and Squatter's Rights--

Sounds like a critique of classical economics and shout out for network economics. See, Oxford Handbook of the Economics of Networks, and Stanford economist Matthew Jackson’s new book on network economics.

Or more broadly, complexity economics.

It's been clear for a long time that economies are complex systems, but economists to date have heavily resisted the implications of that - that the economy is not predictable, that the future is dominated by unknown unknowns and feedback loops, that the adaptive nature of the system makes the use of historical responses to shocks in modelling problematic, etc.

I liked the abstract. It makes sense to me. It syncs with my priors on chaos and path dependency (for the wont of a nail, a kingdom was lost).

I guess Tyler's title is ironic? It isn't the predictions that evolve. It's that observations are resampled.

When we permit ourselves to resample, that is.

See also the respect cycles of presidents Carter, Bush 43, and now Trump. Heroes as long as public perception is needed, chumps as soon as it's not. (Sometimes redeemed again by post presidential service.)

Comments for this post are closed