Algorithm Aversion

People don’t like deferring to what I earlier called an opaque intelligence. In a paper titled Algorithm Aversion the authors write:

Research shows that evidence-based algorithms more accurately predict the future than do human forecasters. Yet, when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. This phenomenon, which we call algorithm aversion, is costly, and it is important to understand its causes. We show that people are especially averse to algorithmic forecasters after seeing them perform, even when they see them outperform a human forecaster. This is because people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake. In five studies, participants either saw an algorithm make forecasts, a human make forecasts, both, or neither. They then decided whether to tie their incentives to the future predictions of the algorithm or the human. Participants who saw the algorithm perform were less confident in it, and less likely to choose it over an inferior human forecaster. This was true even among those who saw the algorithm outperform the human.

People who defer to the algorithm will outperform those who don’t, at least in the short run. In the long run, however, will reason atrophy when we defer, just as our map-reading skills have atrophied with GPS? Or will more of our limited resource of reason come to be better allocated according to comparative advantage?

Comments

Algorithmophobia. Another plank for the Democratic Party.

Evidence for this?

Compare the stream of Republican politicians who regard evolution as too controversial to touch.

vs the Anti vax, anti GMO, A(larmist)GW, food pyramid Democrats.
Republican ( and Democrat) evolution deniers are a fringe. The only person I know who does not believe in it is a Democrat.

It is the Democrat's anti-Science views that effect everyone more, given their default position of imposing their views onto all of society.

Evolution denial is the standard Republican view, but thanks for informing us that you only know 1 evolution denier, a Democrat. According to a Pew poll 48% of Republicans agree that “humans and other living things have existed in their present form since the beginning of time,” Only 43% of Republicans accept evolution.

http://www.pewforum.org/2013/12/30/publics-views-on-human-evolution/

On the other hand, 100% of Democrats believe that evolution magically stopped the instant humans left Africa.

That poll seems odd. Why would 54% of Republicans choose evolution in 2009 and only 43% in 2013.

What happened in 4 years to change beliefs so?

Nothing really, so the poll has something wrong with it, most likely.

Republicans may deny evolution, but it doesn't impact anything. Whether we evolved or not, we are here now, and I have no reason not to lie if it would help signal belonging in a group which furthers economic or political goals.

Being anti-vaccination is more complicated. Vaccines have value, but the current manufacture and distribution system is deeply flawed, and an anti-vax stance helps further the complete deregulation of medicine necessary to achieve competitive pricing and cost-effective over-the-counter distribution of all medical products.

So in order to attack the claim, you really need to understand the underlying objectives.

There's nothing inherently anti-science about being a leftist, just as there's nothing inherently anti-science about being a rightist.

Anti-science is orthogonal to the Left/Right axis. It's about time the commentariat here stopped making this confabulation without reproach.

There's nothing inherently stupid about being a leftist, just as there's nothing inherently stupid about being a rightist.

Stupidity is orthogonal to the Left/Right axis. It's about time the commentariat here stopped making this confabulation without reproach.

The human can learn. The algorithm might, but who can say for sure?

I can say. I create machine LEARNING algorithms for a living.

I like derek's name for this. But there is no need to get political - this research I believe to be important and I welcome studies like this. But it occurs to me that it has a major issue - it assumes that the experimental subjects have no prior experience with algorithmic predictions. In fact, we humans have lots of experience, and because of that, we quickly throw away models that perform badly. To use an example mentioned in the paper - traffic recommendations. Growing up in the NYC area, I quickly learned that when the electronic sign approaching the George Washington Bridge says "fastest route that way" you should go the other way. We have too many experiences with bad algorithms - from poor Netflix recommendations to bad customer support systems.

This does not invalidate the other reasons for aversion to algorithms and models. But it suggests that many of the studies that purportedly show the superiority of algorithms over humans may suffer from "survivor bias." Perhaps humans aren't quite as stupid as people (other humans!) think. We have had too many experiences with bad models so we quickly throw them away when they show errors.

We are not quite as quick to throw away humans when they show errors, because there is (still) the human element that makes us pause before ignoring a human being. When that trace of human-ness finally recedes, then we can all expect to live in more perfect times.

It is far simpler than that. If your exposure to algorithms is your iPad or commercially available wide distribution software you would get the impression that things work. Yes, survivorship bias.

I'm dealing with a wonderful algorithm in one of my customer's asset management software that has increased my invoices by about 35% by automatically generating the numbers. I said it won't last, and indeed word went out to managers about the very high maintenance costs. That algorithm will end up costing the company tens of millions of dollars. A very expensive nation wide rollout promising savings and data, lots of data.

In almost every computerized gizmo that I deal with the challenge is finding the bugs and working around them with no documentation. So any time I run into something like this I check the results, if it seems close and there is a time saving, OK, but never trust.

The economist really is a rare and marvellous creature.

While the natural scientists chip away at the hackneyed old problems-–– how are stars born? how does the eye see or the brain think? how to live on mars or power cities with sunlight? –– under the pretence of illuminating deep mysteries of the universe, our economist busies himself with solving problems we didn't even know we had: Why do lovers give their sweethearts flowers? Why don't intellectuals want to take orders from computers, or craftsmen from robots? Why do male and female humans brood so incessantly on the sunk costs of their past sweat and blood and faded dreams in planning for the future?

For which efforts he is much mocked and little adored. Yet he perseveres, in the pious hope that, at last seeing the problems for what they are, we can begin to put a stop to this irrationality, and take small steps toward a much, much better world.

So do economists generally follow the algorithmic or human-forecasting process for their professional economics analysis/decisions/choice?

How do they decide what to study and publish? How accurate are their economic analyses and forecasts overall (70, 80, 90, 99% ??) Are there any fundamental disagreements among professional economists and what process does this profession use to rectify any core disagreement and establish factual economic truths?

Economics use both, in my experience. The Lucas critique, that is, the idea that historical relationships change if fundamental factors (policies, technology, preferences) change, make economists more wary of algorithms than say machine-learning types. I consider the forecasting that I do to be a blend of algorithms and judgment and I would not want to give up either.

'Yet, when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster.'

As course, some people think that forecasting tends to have a deeper flaw than whether it is done by a human or algorithm. Here is one way to phrase that problem - 'past performance does not necessarily predict future results.' Or one slightly more relevant to a web site that also makes economic predictions, seeing as how the profession predicted recession 54 out of 49 times - http://www.ft.com/cms/s/2/14e323ee-e602-11e3-aeef-00144feabdc0.html ('Now Loungani, with a colleague, Hites Ahir, has returned to the topic in the wake of the economic crisis. The record of failure remains impressive. There were 77 countries under consideration, and 49 of them were in recession in 2009. Economists – as reflected in the averages published in a report called Consensus Forecasts – had not called a single one of these recessions by April 2008.')

But in a world where a forecaster's connections are more important than a forecaster's results, we tend to be presented many forecasts concerning Daesh (barbarians deserve perjorative names) and the need to react against it militarily from the same people who made forecasts concerning the desirability and/or necessity of invading Irag.

Almost as if forecasting is used to justify a pre-conceived policy framework.

No where is algorithm aversion stronger than in the medical profession, where judgment is deemed superior; it's deemed superior by patients even more than by physicians, who have an economic incentive to profess the superiority of judgment. What's the explanation for patients? Because human beings like to believe they are unique, and whatever ails them is unique to them and whatever treatment is appropriate can only be determined by the patient's own physician. Critics of Obamacare howl that it will take away the physician's judgment and force physicians to apply algorithms for diagnosis and treatment! Unfortunately for patients, the physician's judgment is more often harmful than helpful.

Would you care to offer relevance for this claim or is it a belief based on your unique experience?

there's this: http://fivethirtyeight.com/features/your-brain-is-primed-to-reach-false-conclusions/

I've seen reference to many studies of algorithmic superiority in medical situations - often showing that adding human decision-making to the algorithm degrades performance. Here's a link to one article on the subject:

https://gigaom.com/2013/02/11/researchers-say-ai-prescribes-better-treatment-than-doctors/

This is not accurate, rather we want to apply our own algorithms and don't want to subsidize/be in the same risk pool/be taxed by people/organizations that rely on judgment. With less government involvement we have choices, (although some people will choose poorly), and the ACA incentivizes flawed systems.

Nothing in the ACA requires use of an algorithm. Most of the dislike of the ACA involves Title I, which doesn't have much to do with what physicians do at all.

The algorithms are created by humans and just like any other human creation are hardly perfect.

As a former Network Engineer who had to write routing tables for RIP networks, I can tell you this is oh too true. One error, one faulty route path value, and the whole thing went to sh!t in a millisecond. While current technologies are slightly more robust, it doesn't take much to break them into a million pieces. Moreover, the dependency on antiquated and dangerously insecure other protocols (e.g., DNS) should tell you any algorithm can be gamed or tricked into producing a wrong or maliciously desired result.

DNS is a standard, not an algorithm. The standard does what it says it does, if you want to use a more secure option those have been around for a long time. The problem is the humans who control the internetworks won't choose to deploy them.

My algorithm is better

Than

Your algorithm.

You just can't see why

And,

I won't give you the code.

Take it on faith.

Amen brother.

Republicans need to stop standing in the way of Progress and start taking Science on Faith!

You can hunt truffles with a dog, or trust your own nose. Or you can use your comparative advantage as an economist to jargonize the situation.

What kind of dilettante hunts truffles with a dog? Pigs are where it's at.

Different algorithm.

I read somewhere that human brain volume has decreased since civilization, in a similar fashion to domesticated animals. We are the internet's genitals.

For a moment, I thought you said 'gentiles'...

Can't tell if deep or bullshit.

It's a Mcluhanism.

"In the long run, however, will reason atrophy when we defer, just as our map-reading skills have atrophied with GPS? Or will more of our limited resource of reason come to be better allocated according to comparative advantage?"

I think the latter. New tools have always separated those who rely on them from those who understand their limitations. This question goes back to Phaedrus. Learning to differentiate when to trust the algorithm from when it has insufficient or incorrect evidence will remain more art than science for some time. This allows many roles for humans beyond simply feeding the machine information: fine tuning the algorithms' forecasting, applying it to specific domains and (less successfully) interdisciplinary cases, and explaining the results to those who don't speak the technical language will all be future jobs.

This will be hard to adapt to for a generation, maybe less.

As a practitioner of what is called "data science" (ugh) I can tell you that often a model can have superior aggregate performance and yet make mistakes that appear "dumb" (obvious to the human because they have access to more information) or insane (wild mistakes on nonsensical, rare, or constructed inputs). So when a computer recognizes an image of a chair it uses pixel statistics, whereas people can also ask themselves "can I sit on that?" which lets them handle novel weird chairs better. However the universe of chairs is not changing that quickly and a human asked to label thousands of images a day starts to make some errors, hence we have super-human performance on image object recognition tasks, etc.

Thus algorithm aversion need not be lumped in with climate change denial and antivaccinism, because it makes a bit of sense :) I think its a combination of humans worrying about things that are not likely to happen in practice (i.e., they think they are in an adversarial or nonstationary environment but the statistics are actually fairly stationary), humans equating their peak rested fed and alert performance with their actual average performance under repetition, and the failure cases of humans and models being disjoint.

Good point. And, you can look at it a different way as well: The computer is trained to recognize a chair based on the pattern of four legged and three legged chairs that it is introduced to (all prior chairs created by man), but is unable to recognizes as a chair a one legged chair whose one end of the cloth seat is bolted to the wall to create its support.

If you rely on priors in your algorithm, you might limit or fail to recognize creativity.

I guess it comes down to using the right tool for the right job.

And yet climate change was sold by deferring to algorithms. The models say this or that. Then we find the models woefully incomplete, or even in some cases misleading. Those who publicize the flaws are called deniers.

I have heard too many lies from environmentalists over the decades to not assume they are lying as a starting point. Even when I agree with their position.

I guess one needs to assume good intent on the algorithm's programmers, obviously lacking in climate science.

Funny. I've heard too many lies from anti-environment lobbyists over the years to not assume they are lying as a starting point. Even when I agree with their position.

As a result, my default position is to trust in those smarter than I. In the case of climate change, it turns out those are those are climate scientists (who use algorithms) over lobbyists (who use bribery).

Lying is not the opposite of lying. Just because anti-environment lobbyists are full of shit does not mean that environment lobbyists arent also full of shit.

When you say map reading has atrophied you mean at the individual or population level?

I can still read a map directly from time to time. The issue is with increasing numbers of people who have never used a map...

Laura,

Men cannot read maps, but women can. Which is why Tyler is making the claim that map reading for MEN has atrophied. Which is also why it is hard for you to believe his comment, since you are a woman.

If you are a man, when you are in a low state of map reading and have difficulty in asking for directions, there is a phase shift in map reading where you begin to atrophy with the introduction of google maps.

i'd love to see the data from google map, if it existed. I'd bet 90% of men print the map, and 90% of women print the text instructions,.

I don't think this was Laura's point, but I agree with yours. I suspect, however, your numbers are too high even given the selection bias.

I peint both. If enough printed both your above statement could be true and there be no gender difference.

Has anyone calculated the economic benefits of not getting lost thanks to Internet mapping and gps?

Taxi drivers in Singapore rave about the app GrabTaxi, which allows them to bid for calls, with the closest taxi winning.

To answer whether algorithmic-deference will lead either to the atrophy of reason, or to its more efficient allocation, we would first have to decide the following question:

What CANNOT be performed, or at least modeled, by algorithms?

And if there is something, what do we call it?

Kurt Gödel called it "intuition" (in his later philosophy, as described in the books by Wang). So the distinction we might use is "algorithmic" vs. "intuitive".

Since humans have some algorithms inside of them (or so we say), it is probably not accurate to use the distinction, "algorithmic" vs. "human" forecasting.

Gödel saw intuition as the source of his questions as to whether an algorithmic machine can be complete and consistent. (Caution: it is not necessarily the source of the incompleteness or the inconsistency, if or when it exists -- he left those open -- but the source of the questions about them.)

Well, is there anything else that cannot come from algorithms? (I.e. is there anything material, as opposed to emotions or the qualia of consciousness?)

Two possibilities are: new innovations and emergent social structures. Some of these seem to be beyond real prediction, although afterwards, their emergences can be imperfectly retrodicted by algorithms.

It seems to us as if, in these instances, some sort of spark happens, then we employ reason -- afterwards -- to effectuate the new change.

If this is true, then reason will not atrophy.

P.S. I think "algorithmic aversion" may be due to one or more of several different things:
1. the feeling that novel emergence is indeed intuitive, and that it always trumps algorithm (which is clearly false);
2. other cognitive biases, in addition to #1, all of which you can learn your way outside of;
3. limited personal attention-budget, which you can improve but not eliminate;
4. rational inability or nescience, such as whatever it is that continues to generate paradox;
5. formal Gödelian incompleteness or inconsistency, which may be the cause of some of #3.

I would say my map reading skills have improved because of GPS and map apps. Do you rememberhow directions were once given? " Make a left at the gas station, a right at the school, if you see the Seven eleven you've gone too far..." Allowing computers to do what they're best at frees humans to do what they're best at.

I was puzzled by this claim as well. When using a GPS, you are constantly referring to a map. There is just someone giving directions, too.

Agreed. GPS allows me to look at a map and drive at the same time. Emphasis on look; no annoying voices for me, just an above-behind-isometric view of a charted path through a map.

Not to mention the use of advanced geographical information systems. People can conduct searches with maps updated using real-time Bayesian analysis, plot deliveries using historical and live traffic information by time of day, overlay political, economic, and demographic data, zoom to 1ft resolution via sat, and so on. We are probably better at reading maps, and we have unquestionably better maps.

It is not map reading per se which atrophies. Reliance on GPS tends to generate a deficit in environmental knowledge and situational awareness. The gas station, school, and seven eleven are likely to be there next month, but the battery in the GPS might be dead in a few hours.

Situational awareness is the big problem with GPS. Before we had nav systems, getting someplace by map required you to actually study the map, look at alternate routes, remember the names of intersections, etc. With GPS, you just follow the 'turn now' or 'take your second right' instructions, without having to engage your brain. If the GPS fails, you might not have a clue where you are other than vague geographic area.

This is a problem in General Aviation. Before GPS, non-instrument pilots navigated using 'pilotage'. This involves reading a map, looking for features on the map that you can see out your window, then orienting yourself in space that way ("Let's see, there's a river below me, and there's a sharp bend north on my left. Looking at the river on the map, I can see the bend, so I know exactly where I am. I should fly towards that mountain peak in the distance to keep me on track."

Pilots also had to learn how to estimate the wind and correct for it, check their fuel status against their flight plan prediction, and all the rest.

Now, a lot of light plane pilots just sit back and let the GPS tell them where they are. But if it fails... It can be hard to find where you are on a map, assuming you're still carrying one.

I had that happen to me once when I first started flying a plane with a GPS - it failed on me just as I was nearing Denver where there is a lot of controlled airspace, and when the display went out I realized I had no idea where the base of the controlled airspace was relative to my aircraft. I had to call flight service and request vectors to the airport through the controlled airspace. I had my map, but I was worried that I would crash through controlled airspace before I could reacquire my position. If I had been out in the boonies, that wouldn't have been an option.

I learned my lesson, and always followed along on a map after that.

People who don't use forecasts but instead optimize over the whole range of possible futures will outperform people who rely on human or algorithmic forecasters.

People know from personal experience that computers are unreliable and deal poorly with unexpected inputs. Even I as a programmer don't trust programs I've written much farther than I can throw my computer. Indeed, software developers spend most of their time working the kinks out of programs they write ("debugging"), and many kinks often remain after software is shipped (bug tracking systems are a multimillion dollar industry). When necessary, software developers can write highly reliable software, but it takes longer and the incentives don't exist for most applications.

This is not the relevant question. That humans don't like following opaque algorithms is a given. The relevant question is what is likely to happen? What is likely to happen, based on the way things have been going is that technological sophistication will proceed apace, more complicated and opaque algorithms will be adopted in more areas of our lives, anxiety and frustration will increase, people will feel even more alienated and less in control of their immediate environment, and economists will cheer it on because it will be materially more efficient.

Could it be that a faulty algorithm BECOMES true because people believe it is true,

And,

Through herd behavior and common belief of humans

Make the algorithm come true.

As an example: Think of "racial science" of the 1890"s to the 1930's as an algorithm, and what it led to in the form of the Nazi mass movement.

Algorithms are written by humans. Trusting in an algorithm is trusting in a human.

The only difference is that algorithms won't make math errors.

Reason for going with human forecaster rather than machine might be so that it is obvious who to blame if things go bad.

Basically, everyone, including experts, will have and trust their own algorithms.

The experts generally know the limits of the algorithm. So they know when to ignore the results and when to pay attention to them. The non-expert is left with trust.

People learn, algorithms may or may not (a point made above, but worth reiterating.

People tend to couch their predictions in terms other people will follow. Not just a result, but enough of a 'why' to allow others to start nodding their heads.

People understand when predictions are counterintuitive. That means they will double check 'weird' predictions for accuracy and provide appropriate extra evidence. The extra evidence helps, as does the perception that the person making it understands that what they are saying is counterintuitive and has done their homework. Machines tend to just spit out the answer, without justification, or acknowledgement that there might have been a mistake, so they checked it three times, and yes, this really weird thing is true.

People understand the CONSEQUENCES of their predictions, and will inherently hedge against disastrous outcomes. I computer will gleefully predict a 90% chance of something great, while casually omitting the 10% chance of horrible death. Following a machine may be better on the average, but there's a huge risk that the errors that machines make will be more disastrous than the errors people make.

People can empathize with the types of mistakes other people make. Sure, I could have forgotten that trivial detail that actually turns out to change things. Machine mistakes, however, tend to be glaringly obvious to people. If the machine could make such a stupid mistake, the rest of it must be worthless too. People are bad at comparing different TYPES of intelligences (I believe this point has been made here before, no?)

Paul Meehl wrote prolifically about this issue as far back as the 1950s, looking at psychologists aversion to statistical or actuarial prediction. Unfortunately, very little has changed since then. Here's a nice summary article: https://www.psych.umn.edu/faculty/grove/112clinicalversusstatisticalprediction.pdf

Trust the algorithm, don't trust the algorithm? I just had an idea for neat app!

I'm not sure I always want to trust an algorithm either, and I'm a computer engineer.

The problem with algorithms is that they are no better than the assumptions cooked into them and the quality of the process that created them. That means they're probably very good when following the normal case or when their inputs remain in the region modeled. The problem comes when they have bugs, or have inputs that take them out of the region they were designed for. Then they can give spectacular errors that can wipe out all the good they've done.

Would you trust a trading algorithm that was a couple of percentage points better than a human when stocks are trading normally, but which might make a spectacular error during a crash that would wipe you out?

I accept that navigation software probably does a better job of defining efficient routes than a human would. But on the other hand, I've had a bug in my navigation database direct me to drive into an empty field. I've had the 'shortest route' include a very hilly gravel road covered in snow because it was very slightly shorter than the nice four-lane highway. No human would suggest that, and the reason I can use my nav software is because I'm in the loop and can reject obviously bad or dangerous choices.

But if you were blind and in a self-driving car, what would you rather have? An algorithm that is a 10% more efficient driver most of the time, but once in a while will do something stupid or dangerous? Or a human who will be less efficient but won't drive you off a cliff?

Lest you think these are silly concerns, Consider the Ariane Rocket's guidance system. The algorithms there were very efficient, but nobody tested a specific case of the actual motion the rocket went through, and in one code path, when the rocket went through vertical the trig functions attempted to do a divide by 0 and the whole thing crashed and the rocket was a total loss.

Then there is the GIGO problem - another space mission was lost because one team was entering data into the navigation software in metric units while another used imperial. In my industry we go to great effort to try to notify the user of potential data entry errors with sanity checks, but you can't catch them all.

There are other examples of algorithms that worked well for a time and then failed spectacularly. The same is true of medical diagnoses algorithms - they work well when the illness matches the common path of symptoms, but overuse can cause a doctor to become disengaged and fall back on 'cut and paste' medicine, following the algorithm's diagnosis without engaging mentally with the patient's actual overall condition. So more rare illnesses or illnesses with confounding symptoms may get missed. Good in the general case, fatally bad with the outliers.

And the problem for the general public is they simply have no ability to determine whether an algorithm is perfectly safe, or only 'perfectly safe' until it's not. It's hard to blame them for being leery.

Comments for this post are closed