Maths in a minute: Selection (and survivorship) bias
Selection bias happens when looking at a restricted data set leads you to wrong conclusions. An example is doing a voluntary survey to see if people like filling out surveys. You're only likely to get answers from people who do like filling out surveys, so the result tells you very little.
Other examples are using a basketball team to estimate the average height of the population, or care home residents to estimate the average age of the population. Both will give you skewed results.
These examples are quite obvious, but here's a more subtle one. In World War II the Statistical Research Group at Columbia University in the US was tasked with seeing how aircraft casualties could be reduced. The group looked at bomber planes that had returned from missions in Europe to see where they had been hit by bullets. They found that most of the damage was to the wings and the tail.
The conclusion seemed clear: reinforce the wings and the tail of planes since they are often hit.
One member of the team, Abraham Wald, then spotted a fatal error in this reasoning. The planes that were examined by the group were the ones that had survived their missions. But perhaps they survived because they were only hit in the wings and tail and nowhere else. What should be reinforced then, are the regions of the plane where the surviving bombers had not been hit.
The mistaken conclusion (to reinforce the wings and tail) is an example of survivorship bias: a situation where looking only at data that have survived a certain process (only planes that have returned) leads to an erroneous result. Survivorship bias is a particular type of selection bias.
The moral of the story is this: when analysing statistical data, make sure the data set you are looking at is as representative of the general population you want to study as it can possibly be. A gold standard in this context are randomised controlled trials — find out more in this Maths in a minute article.
About this article
This content was produced as part of our collaboration with the Isaac Newton Institute for Mathematical Sciences (INI) and the Newton Gateway to Mathematics.The INI is an international research centre and our neighbour here on the University of Cambridge's maths campus. The Newton Gateway is the impact initiative of the INI, which engages with users of mathematics. You can find all the content from the collaboration here.
