## 20 September 2011

### The importance of "selection bias" in statistics

During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage!

This seems backward at first, but Wald realized his data came from bombers that survived. That is, the British were only able to analyze the bombers that returned to England; those that were shot down over enemy territory were not part of their sample. These bombers’ wounds showed where they could afford to be hit. Said another way, the undamaged areas on the survivors showed where the lost planes must have been hit because the planes hit in those areas did not return from their missions.

Wald assumed that the bullets were fired randomly, that no one could accurately aim for a particular part of the bomber. Instead they aimed in the general direction of the plane and sometimes got lucky. So, for example, if Wald saw that more bombers in his sample had bullet holes in the middle of the wings, he did not conclude that Nazis liked to aim for the middle of wings. He assumed that there must have been about as many bombers with bullet holes in every other part of the plane but that those with holes elsewhere were not part of his sample because they had been shot down
I suppose one could argue about whether the bullets were fired randomly, but the analytic principle involved is still an important one for students to learn.

Text credit John D. Cook at The Endeavor, via Neatorama; image via Mother Jones.

1. It really is a splendid bit of thinking, isn't it?

At first sight, it might appear that you could come to this conclusion just by doing the thought experiment in your head - and that may have been what he initially did. However, if you read (or in my case attempt to read) the paper in which he presented the idea, there's quite a considerable amount of maths gone into it.
Like many very clever ideas, it just seems blindindly obvious once you've had it pointed out.

Rather like evolution.

2. I might have to read his paper. I have to question if he considered that many parts are redundant out of necessity of the design and that the survivors largely lucked out that fewer than all of the redundancies were destroyed. The wing damage is a good example of this. The planes that could be surveyed might have been missing as much as half a wing on one side but that does not mean the plane would fly with two half wings.

3. Without a full load of fuel and bombs and ammunition, it's quite possible that it would fly with just the inboard wing panels.
Not well, of course, and the pilot would have to use rudder/elevator on account of having no ailerons, and you'd have to trust to luck on the flaps...

4. strange to take a picture of a Douglas DC-3/C-47 if talking about bombers but hey.....who aim I ?

5. I could have told you that hitting the cockpit or the wing fuel tanks would be a problem w/o any analysis.