The Importance of Selection Effects

by on September 13, 2011 at 11:01 am in Data Source, Economics, Science | Permalink

I love this example of the importance of selection effects:

During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage!

The RAF was initially confused. Can you explain?

You can find the answer in the extension or at the link.

Wald had data only on the planes that returned to Britain so the bullet holes that Wald saw were all in places where a plane could be hit and still survive. The planes that were shot down were probably hit in different places than those that returned so Wald recommended adding armor to the places where the surviving planes were lucky enough not to have been hit.

Right Wing-nut September 13, 2011 at 11:04 am

That should be an interview question!

Brian Moore September 13, 2011 at 11:20 am

One of the commenters at the link makes the good point that this is dependent on the assumption of randomized aiming; which, based on the low level of air combat technology at the time, was a pretty good assumption. :)

techreseller September 13, 2011 at 11:30 am

Very cool. I am going to try this one on my kids.

Andrew' September 13, 2011 at 4:51 pm

What? Shooting at them with autocannons?

gruff September 13, 2011 at 7:08 pm

No, armouring where they don’t get hit, obviously.

Rahul September 14, 2011 at 2:45 am

It only works if some of his kids never return home…….

James Clary September 14, 2011 at 9:45 am

This thread is full of win, on a related note, my wife is completely against my natural selection method of raising children

John Thacker September 13, 2011 at 11:38 am

I’ve heard this story many times before, but I don’t think he actually did it for the British, but for the US. Reprint of one of his papers on the subject.

tkehler September 13, 2011 at 11:39 am

Hmmm. Can “aiming” be “randomized”? I know what you mean but technically shouldn’t it be randomized firing/shooting? Aiming seems to imply something different.

Brian Moore September 13, 2011 at 4:22 pm

bad word choice on my part :)

Andrew' September 13, 2011 at 4:49 pm

To paraphrase Herm Edwards: You aim to hit the plane! Hullo…

Pub Editor September 13, 2011 at 5:00 pm

Sometimes, Germans aiming for the stars end up hitting London…

John Thacker September 13, 2011 at 11:41 am

Here’s another source for the story, article from 1984 in the Journal of the ASA. I still don’t think that this was for the RAF, but for the US Navy.

Alex Tabarrok September 13, 2011 at 11:47 am

Thanks to John Thacker for the links!

Chris Frashure September 13, 2011 at 11:42 am

I got the right answer as soon as I read it. Should I allow myself to feel good about that?

MD September 13, 2011 at 12:18 pm

Just don’t brag about it. Somebody might shoot you down.

Yog Sotthoth September 13, 2011 at 11:50 am

His assumption that the bullet holes are uniformly distributed across parts of the plane in the full uncensored sample effectively gave him the equivalent of the uncensored dataset. With this assumption, he then has an estimate of the full distribution of damage outcomes for each bullet location. You still have to assume something about adding armor to these locations will improve the outcomes. It wouldn’t make sense to add the armor to a part of the plan that for some reason can’t be fortified for some physical reason.

Silas Barta September 14, 2011 at 11:02 pm

Right, and similarly, his advice would tell you to put all the armor on the top of the bomber where no one’s going to hit it anyway! (At least not the anti aircraft artillery…)

celestus September 13, 2011 at 12:12 pm

Heard it already, but worth hearing again.

Jim Glass September 13, 2011 at 6:16 pm

The US Navy notably did the same thing when upgrading its aircraft fighting in the Pacific.

Gastarbeiter September 13, 2011 at 6:24 pm

survivorship bias

john malpas September 13, 2011 at 7:49 pm

you would think that the armour shoud protect the driver

Stephen September 13, 2011 at 8:36 pm

Keeping the plane from hitting the ground really hard goes a long way to protecting the driver.

Also bomber crew are cheaper to replace than bombers!

Rahul September 14, 2011 at 2:46 am

If you lose the crew you pretty surely lose the bomber.

Stephen September 14, 2011 at 10:49 pm

The great majority of the crew aren’t driving the plane. Of those who are driving, 50% could be critically wounded or dead and the plane would still arrive back home.

Jason Leonforte September 14, 2011 at 12:26 pm

Taking out a wing or an engine is probably a lot easier then trying to take out the cockpit.

Hasdrubal September 14, 2011 at 1:24 pm

Meh, driers are a redundant system.

Anon September 13, 2011 at 8:09 pm

Accuracy was poor, true, but angle of approach might not be random. For example, fighters might prefer an approach that made them difficult to see.

MattieF September 13, 2011 at 8:30 pm

When one of these stories goes around, I like to follow links back to the original source, and see where someone turned “boring fact A” into “wild speculation Z”. I didn’t read the Wald report yet, but I’m already amazed that it seems to actually exist.

Mike September 13, 2011 at 11:57 pm

You dont need to destroy a plane. You just need to damage it enough and gravity will destroy it.

There’s a similar axiom for submarines, but unlike airplanes the subs’s natural enemy surrounds it completely, not just on one side.

If you want a thrill, go to YouTube and watch the video of the Israeli F-15 pilot who landed his plane after it lost one of its wings in a collision.

Noumenon September 14, 2011 at 1:17 am

The video is here but I didn’t find it too interesting with the sound off.

Steve September 14, 2011 at 1:48 am

A young Freeman Dyson did statistical analysis for the RAF in WWII. His book “Disturbing the Universe” has some good stories along these lines.

Filip September 14, 2011 at 2:46 am

There’s a similar story here about soldiers’ helmets.

NAME REDACTED September 14, 2011 at 6:38 am

Assuming that damage was relatively evenly distributed, the damaged places on aircraft that survived was survivable. Likely, the ones who didn’t survive, sustained damage in those areas where the ones that survived didn’t.

NAME REDACTED September 14, 2011 at 6:39 am

Woot! was right!

Ryan King September 14, 2011 at 9:19 am

Isn’t this actually an ascertainment bias rather than a selection bias? There was nothing endogenous to the bombers. Did the lingo change at some point?

[ this is jerry ] September 14, 2011 at 10:24 am

“Two thousand black box parameters from planes that do not crash can be harvested via computer analysis to further increase airline safety.”
http://www.scientificamerican.com/podcast/episode.cfm?id=safe-air-landings-have-useful-black-11-09-14irline safety. “

[ this is jerry ] September 14, 2011 at 10:25 am
Urso September 14, 2011 at 11:35 am

I’ve also heard that the statisticians insisted that bombers should not have any defensive weaponry. B 17s and 24s were brimming with machine guns to defend against German fighters. But those machine guns meant additional drag, weight, and crew members. The wonks figured that if the guns were removed, the bombers could fly higher and faster, thus being more survivable. Moreover, they could carry a heavier bomb load, meaning fewer bombers would be needed on any mission. And the loss of any given bomber would result in fewer deaths, because the crew was smaller. Although the individual bombers would lose some defensive capacity, in the long run they’d suffer fewer casualties as a group.

It never happened because the crews hated the idea of flying combat missions in a plane that was incapable of shooting back at the guys who were shooting at you.

Tangurena September 14, 2011 at 3:29 pm

The only 2 WW2 bombers that I’m aware of that flew too fast and/or too high for interception were the Mosquito and B-29.

German interceptors flew more than 100mph faster than B-17 or B-24, so there was no way they could outrun fighter interceptors. Being large aircraft to begin with, losing the machine guns and gun crew wouldn’t get them to the speed needed to make them unnecessary.

Urso September 14, 2011 at 4:59 pm

*shrug* I’m not the one who wrote the memo

http://www.jcs-group.com/military/war1941aaf1/bomber7.html

Andy September 14, 2011 at 1:18 pm

Regarding anti-aircraft fire… some of you are entitrely on the wrong track. My late father was one of the aircrew who flew over Germany and I remember his descriptions of flak. There were TWO types of bomber kill. One was from German fighters. In pre-radar days they would come up behind the bombers looking for the little red glow from the exhaust stubs. They would then attack the bomber from close range astern. The chances were that the shells would take out the rear gunner and travel trhe length of the aircraft. This was pretty much unsurvivable by the rear gunner and they had a huge casualty rate. The other attack method was to lopp up and under the aircraft and rake the belly before dropping away in a stall.

Anti-aircraft flak was a different matter. The German technique was to light an area of the sky using searchlights. This formed a cone of light. The anti aircraft artillery was then trained to fill this box with shrapnel. AA shells are designed to explode at a set height and shower a wide area with shrapnel. This creates a “box” of flying splinters through which the bombers had to fly.
Dad said the AA shells looked very pretty coming up, then they would either fly by with a sinister whizzing noise or else explode with a shattering bang sending bits of metal through the fuselage. They regularly came back full of holes but still flying. One of Dad’s log book entried “Coned by sixty peeper(searchlights), shaky do!!” That was the time they brought the plane back full of holes and injured crew. I hope this gives some idea of the nature of the weaponry they were up against.
Also more sinister was the direct hit. These were rare but on a raid you would see sudden blinding white flares. These were called “scarecrow shells” and were said to be fired by the Germans to blind the pilots. In fact they were direct hits by shells and the aluminium airframe and everything in it would be consumed in a blinding flash. There were no scarecrow shells, just exploding aircraft. Hope this gives an idea of the nature of what they were up against. I think that ANY armour would be better than no armour at all against this lot.

Dave Tufte September 14, 2011 at 2:23 pm

There is a hand-drawn diagram of this argument posted at voluntaryXchange.

I do not know who did the diagram, or how old it is.

gamesliga October 3, 2011 at 4:07 pm

gamesliga bahis sitesi.
gamesliga yüksek oranlı iddaa oyna.
gamesliga gamesliga referans kodu servisi.
gamesliga gamesligaya üyelik üye olma
gamesliga referansı bahis sitesi.
gamesliga referans yüksek oranlı iddaa oyna.
gamesliga üyelik gamesliga referans kodu servisi.
gamesliga referans gamesligaya üyelik üye olma

Comments on this entry are closed.

Previous post:

Next post: