This is the first instalment of David Spiegelhalter's regular column called Understanding uncertainty.
League tables are controversial and for good reason. Few things are simple enough to be measured by a single outcome like, for example, the number of exam passes or successful heart operations. But even if we do accept a single yardstick, we haven't yet reckoned with chance, which by itself can produce apparent patterns to delight any tabloid editor.
Take the National Lottery as an example. As we'll see in a moment, a league table based on the whole history of the lottery appears to show that some numbers are preferentially drawn. Other numbers don't come up for ages. Clear evidence for cheating, or even the hand of god? Not necessarily; in fact the league table, taken at face-value, is completely spurious. What is at work here is simply chance. Let's have a look at the details.
Lucky numbers and unfortunate gaps
The UK National Lottery began on the 19th of November 1994 and there had been 1240 draws up to the 20th of October 2007. Six balls are randomly drawn from a set of forty-nine numbered balls. If you correctly predict the six numbers, you get a share of the jackpot. The animation below shows how often each of the 49 numbers came up in the first 1240 draws.
Starting from 1994, note how the leading number — the one that has been drawn most often — changes, until one number seems to gain a substantial lead. Click on Sort to rank the numbers, and Fast Forward to speed it all up. When the animation has finished, what you see is the league table ranking numbers by the frequency with which they were drawn. Thirty-eight is the clear winner.
By clicking on Histogram you can create a histogram which for each value n shows how many numbers have come up exactly n times. Start Dropping shows how that histogram is created (this is not available in the fast-forward mode). The distribution seems quite spread out, with some numbers appearing much more often than others.
We can take another angle on this by looking at the gap between each time a number comes up. The animation below keeps track of each time a specific number is not drawn.
This text is usually replaced by the Flash movie.
If you look carefully from the start of 2000, you'll see that the longest gap observed is 72, for number 17, which appeared in draw 435 on the 23rd of February 2000, but did not appear again until draw 508 on the 4th of November 2000. A back-of-the-envelope calculation shows that such a long gap is rare indeed. The chance of a particular number occurring in a single draw is 6/49, which means that the probability of a number not being picked is 1-6/49=43/49. The chance of this happening 72 times in a row is thus (43/49)72, which is roughly 0.000082, or 1 in 12,500!
What our league tables have shown us so far — a lucky number 38 and a gap of improbable size — doesn't seem to sit well with the assumption that the lottery is truly random. This is the sticky point at which you might be tempted to suspect the lottery of being rigged, or evoke cosmic interference. But with a little mathematical stamina it's possible to find out whether this is still within the realms of pure chance. So take a deep breath and sharpen your pencil.
The distribution of the number of appearances of each number
First you need to define a theoretical probability distribution based on the assumption that the lottery is random. Let stand for the number of balls chosen at each draw and for the number of balls in the bag, so in our case and . If the numbers are chosen at random, then for each the chance of being drawn is , so here . The chance of a number not being chosen at a particular draw is .
To work out the probability that in draws a particular number is drawn exactly times, you have to consider three factors. The first is that the number was picked times. The chance of this is . The second is that it was not picked times. The chance of this is . So is the chance of a particular sequence of picks and non-picks being observed. The third thing to notice is that overall there are
The animation below compares the actual distribution of numbers being drawn with the theoretical normal distribution. The theoretical distribution is shown in white with the actual histogram superimposed in grey. Of course the actual distribution is more jagged, but the theoretical distribution allows us to see whether the leading number is in fact surprisingly far in front.
This text is usually replaced by the Flash movie.
Below we see the final observed distribution with an approximate theoretical distribution superimposed. The fit looks good, suggesting that there is no real preference for particular numbers. The lucky number 38 is no evidence of bias.
The observed and the theoretical distribution.
The distribution of gaps between a number's appearances
You can work out the theoretical distribution of gaps between numbers in a similar way. Consider a particular number . Label a draw a success if number is drawn and a failure otherwise. As before, the chance of a success is . Let stand for the number of failures before the first success, in other words the gap before is drawn again. The chance of taking on any particular value is the same as the chance of observing a series of failures followed by a single success, so that
The observed and the theoretical distribution of gaps.
The maximum gap in the whole lottery history
But what about that maximum gap of 72, which we worked out to be extremely unlikely? It turns out that we asked slightly the wrong question, namely: "The number 17 has just been drawn, what is the chance that it will not be drawn within the next 72 draws?" In reality, though, we are looking at the results of 1240 draws, rather than just 72, and we are not interested specifically in the number 17. What we should have asked is: "After 1240 draws, what is the chance that any of the gaps between two draws of the same number is greater than 72?".
To work this out, we first need to know how many gaps there are in total. There have been 1240 draws, each featuring 6 numbers, so altogether 1240 × 6 = 7440 numbers have been drawn. Each time a number appears, there is a corresponding gap since it last appeared (this includes the initial gap until each number is drawn for the first time). Our question now becomes: "What is the chance that the longest of these 7440 gaps is at least 72?".
We can estimate this chance using our distribution:
This result suggests that a maximum gap of length 72 is not in the least surprising. In fact, there nearly is a fifty-fifty chance of a gap of at least that length occurring in 1240 lottery draws.
To check the accuracy of this result we can simulate "fictional" lottery histories on a computer, by picking 6 different numbers at random from 1 to 49, and then repeating this process 1240 times. The software we use contains random number generators that should ensure that each number really does have an equal chance of being chosen. We simulated 1000 full lottery histories and found the longest gap in each history which had the distribution shown below: 470 (47%) out of 1000 were 72 or more, which is similar to our approximate theoretical probability of 0.46. Good!
Testing for bias in the lottery
The good fit between the theoretical and observed distributions suggests that what we've observed in the 1240 draws is in keeping with the assumption that the lottery is random. Yet, the distributions do differ. How large can we allow this difference to become before we start to worry? In other words, how confident can we be of our conclusion that the lottery is random? Probability theory has an answer to this question too. In fact, there are many test statistics designed to identify different ways in which the lottery draws may not be entirely random. Let's consider one of the simplest.
The binomial distribution above says that after draws we expect any particular number to have occurred times. In the UK lottery this corresponds to , approximately . So, for example, after 1240 draws we would expect each number to have been chosen around times. Now add up the total number of times each number has occurred after draws and label these totals , , etc, up to .
For each number the expression is a measure of the discrepancy between what's observed and what's expected. If we add the for the 49 values of , we get an overall measure of discrepancy - it's called the chi-squared statistic and usually denoted by . To match our set-up we need to multiply this standard chi-squared statistic by the small amount . This is because, as already mentioned above, the totals are not quite independent. It's rather a tricky bit of statistical theory and we won't go into it here.
Just as it was possible to work out a theoretical distribution of the occurrences of each number based on the assumption that the lottery is random, it is possible to work out a theoretical distribution of . It is called a chi-squared distribution with, in this case, degrees of freedom. It tells us the chance of observing our particular value for assuming that the lottery is indeed random (and assuming that all the balls being drawn are statistically independent). If this chance is low, say less than 0.05, or 5%, then there's reason to worry.
We won't say much about the chi-squared distribution itself here. All we'll say is that in our particular case it tells us that our observed value of should be somewhere in the region of 48. It also tells us that the chance of our observed value being greater than 69 or less than 38.9 is only 0.05, or 5%, so we should watch out for observed values in those regions. The histogram below shows the observed values of the statistic where the draws have been combined in groups of fifty, so in each case . As it turns out, all observed values lie between the dotted lines at 60 and 38.9 — we can be reasonably confident that the lottery is indeed random.
Thus, the patterns we spotted in the lottery league table were red herrings. But what about those league tables that do not arise from pure chance? Can we separate chance from meaningful information in these cases? This is what we'll explore in the next issue of Plus. Our example will be the Premier League!
This article is based on an idea by Norman Fenton on considering the lottery results as a league table. John Haigh has also written about probability and the National Lottery, see Plus articles The UK National Lottery - a guide for beginners and Running a lottery - for beginners.
If you want to try your own analyses, a spreadsheet with the full lottery history can be downloaded from the main UK National lottery site.
About the authors
David Spiegelhalter is Winton Professor of the Public Understanding of Risk at the University of Cambridge.
Mike Pearson created the animations for this article.
Mike, David and the rest of their team are currently setting up a new website called Understanding uncertainty which will inform the public about all issues involving risk and uncertainty. The website will be launched in spring 2008.