The Importance of Selection Effects

I love this example of the importance of selection effects:

During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage!

The RAF was initially confused. Can you explain?

You can find the answer in the extension or at the link.

Wald had data only on the planes that returned to Britain so the bullet holes that Wald saw were all in places where a plane could be hit and still survive. The planes that were shot down were probably hit in different places than those that returned so Wald recommended adding armor to the places where the surviving planes were lucky enough not to have been hit.


That should be an interview question!

One of the commenters at the link makes the good point that this is dependent on the assumption of randomized aiming; which, based on the low level of air combat technology at the time, was a pretty good assumption. :)

Very cool. I am going to try this one on my kids.

What? Shooting at them with autocannons?

No, armouring where they don't get hit, obviously.

It only works if some of his kids never return home.......

This thread is full of win, on a related note, my wife is completely against my natural selection method of raising children

Hmmm. Can "aiming" be "randomized"? I know what you mean but technically shouldn't it be randomized firing/shooting? Aiming seems to imply something different.

bad word choice on my part :)

To paraphrase Herm Edwards: You aim to hit the plane! Hullo...

Sometimes, Germans aiming for the stars end up hitting London...

I got the right answer as soon as I read it. Should I allow myself to feel good about that?

Just don't brag about it. Somebody might shoot you down.

His assumption that the bullet holes are uniformly distributed across parts of the plane in the full uncensored sample effectively gave him the equivalent of the uncensored dataset. With this assumption, he then has an estimate of the full distribution of damage outcomes for each bullet location. You still have to assume something about adding armor to these locations will improve the outcomes. It wouldn't make sense to add the armor to a part of the plan that for some reason can't be fortified for some physical reason.

Right, and similarly, his advice would tell you to put all the armor on the top of the bomber where no one's going to hit it anyway! (At least not the anti aircraft artillery...)

Heard it already, but worth hearing again.

The US Navy notably did the same thing when upgrading its aircraft fighting in the Pacific.

survivorship bias

you would think that the armour shoud protect the driver

Keeping the plane from hitting the ground really hard goes a long way to protecting the driver.

Also bomber crew are cheaper to replace than bombers!

If you lose the crew you pretty surely lose the bomber.

The great majority of the crew aren't driving the plane. Of those who are driving, 50% could be critically wounded or dead and the plane would still arrive back home.

Taking out a wing or an engine is probably a lot easier then trying to take out the cockpit.

Meh, driers are a redundant system.

Accuracy was poor, true, but angle of approach might not be random. For example, fighters might prefer an approach that made them difficult to see.

When one of these stories goes around, I like to follow links back to the original source, and see where someone turned "boring fact A" into "wild speculation Z". I didn't read the Wald report yet, but I'm already amazed that it seems to actually exist.

You dont need to destroy a plane. You just need to damage it enough and gravity will destroy it.

There's a similar axiom for submarines, but unlike airplanes the subs's natural enemy surrounds it completely, not just on one side.

If you want a thrill, go to YouTube and watch the video of the Israeli F-15 pilot who landed his plane after it lost one of its wings in a collision.

The video is here but I didn't find it too interesting with the sound off.

A young Freeman Dyson did statistical analysis for the RAF in WWII. His book "Disturbing the Universe" has some good stories along these lines.

There's a similar story here about soldiers' helmets.

Assuming that damage was relatively evenly distributed, the damaged places on aircraft that survived was survivable. Likely, the ones who didn't survive, sustained damage in those areas where the ones that survived didn't.

Woot! was right!

Isn't this actually an ascertainment bias rather than a selection bias? There was nothing endogenous to the bombers. Did the lingo change at some point?

"Two thousand black box parameters from planes that do not crash can be harvested via computer analysis to further increase airline safety." safety. "

Hmm. My link got eaten. Trying again:

I've also heard that the statisticians insisted that bombers should not have any defensive weaponry. B 17s and 24s were brimming with machine guns to defend against German fighters. But those machine guns meant additional drag, weight, and crew members. The wonks figured that if the guns were removed, the bombers could fly higher and faster, thus being more survivable. Moreover, they could carry a heavier bomb load, meaning fewer bombers would be needed on any mission. And the loss of any given bomber would result in fewer deaths, because the crew was smaller. Although the individual bombers would lose some defensive capacity, in the long run they'd suffer fewer casualties as a group.

It never happened because the crews hated the idea of flying combat missions in a plane that was incapable of shooting back at the guys who were shooting at you.

The only 2 WW2 bombers that I'm aware of that flew too fast and/or too high for interception were the Mosquito and B-29.

German interceptors flew more than 100mph faster than B-17 or B-24, so there was no way they could outrun fighter interceptors. Being large aircraft to begin with, losing the machine guns and gun crew wouldn't get them to the speed needed to make them unnecessary.

*shrug* I'm not the one who wrote the memo

Regarding anti-aircraft fire... some of you are entitrely on the wrong track. My late father was one of the aircrew who flew over Germany and I remember his descriptions of flak. There were TWO types of bomber kill. One was from German fighters. In pre-radar days they would come up behind the bombers looking for the little red glow from the exhaust stubs. They would then attack the bomber from close range astern. The chances were that the shells would take out the rear gunner and travel trhe length of the aircraft. This was pretty much unsurvivable by the rear gunner and they had a huge casualty rate. The other attack method was to lopp up and under the aircraft and rake the belly before dropping away in a stall.

Anti-aircraft flak was a different matter. The German technique was to light an area of the sky using searchlights. This formed a cone of light. The anti aircraft artillery was then trained to fill this box with shrapnel. AA shells are designed to explode at a set height and shower a wide area with shrapnel. This creates a "box" of flying splinters through which the bombers had to fly.
Dad said the AA shells looked very pretty coming up, then they would either fly by with a sinister whizzing noise or else explode with a shattering bang sending bits of metal through the fuselage. They regularly came back full of holes but still flying. One of Dad's log book entried "Coned by sixty peeper(searchlights), shaky do!!" That was the time they brought the plane back full of holes and injured crew. I hope this gives some idea of the nature of the weaponry they were up against.
Also more sinister was the direct hit. These were rare but on a raid you would see sudden blinding white flares. These were called "scarecrow shells" and were said to be fired by the Germans to blind the pilots. In fact they were direct hits by shells and the aluminium airframe and everything in it would be consumed in a blinding flash. There were no scarecrow shells, just exploding aircraft. Hope this gives an idea of the nature of what they were up against. I think that ANY armour would be better than no armour at all against this lot.

There is a hand-drawn diagram of this argument posted at voluntaryXchange.

I do not know who did the diagram, or how old it is.

Comments for this post are closed