When physicists at CERN announce their results, there is always a lot of talk about sigma levels. But what exactly are they?

Let's take the Higgs boson as an example. When it was discovered back in 2012, physicists didn't actually catch sight of the particle itself. They inferred it was there from an irregularity in the data they collected from the Large Hadron Collider (see here to find out more). But of course, such an irregularity could be down to mere chance: it could occur even if there wasn't any Higgs boson at all. Sigma levels reflect people's confidence that the result is not just down to chance. The higher the sigma level, the greater that confidence. Particle physicists require a sigma level of at least five to announce a new discovery, such as the Higgs boson.

"Five sigmas means that there is only a tiny chance of getting the result they did by just a fluke — that the Higgs boson wasn't there, and that because of a bit of bad luck or whatever, they got this blip in the [data]," says David Spiegelhalter, Winton Professor for the Public Understanding of Risk at the University of Cambridge. "[Five sigmas] means that there's a less than 1 in 3 million chance that they would have observed this result just as a matter of luck."

David Spiegelhalter

Physicists aren't the only scientists who quantify the confidence in their results in this way. People who test medical drugs in clinical trials, for example, also need to be confident that any effect they observe wasn't just a fluke and is really down to the drug's efficacy. But interestingly, medical scientists require a lower sigma level than particle physicists do. "To get a new drug passed and approved by regulatory authorities, you need to do two studies, but they only have to be two sigma discoveries," says Spiegelhalter. "People generally in science would consider three sigmas, that's a 1 in 1000 chance of getting the result they did just by chance, as being very strong evidence indeed."

Why the difference? Partly because people don't want to make fools of themselves. "People don't want to claim things and then later discover that they are wrong. It's happened before in particle physics and it's happened in genetics endless times — it's what they call false discoveries. Nobody likes to make false discoveries and then have to retract them. This is embarrassing. So you can say that the number of sigmas is how much you are prepared to try to avoid embarrassment. But it means you have to do a huge amount of more work to find something of five sigmas." Generally, it's more embarrassing to make a false claim about the fundamental nature of physical reality than, say, to be slightly wrong about the amount by which a drug reduces blood pressure, so the extra work particle physicists need to put in is worth it.

But there is a practical reason too. "If drugs required five sigmas, there would never be any drugs on the market. People would require huge numbers of patients to prove the drug is effective," says Spiegelhalter. "Whereas as it stands they can get by with a few hundreds or thousands of patients."

A third reason for the difference in sigma levels is a little more subtle. It relates to the fact that, the more you look for a sign of something, the more likely you are to find something that looks like the sign, but isn't. "In CERN they searched for the Higgs boson over an entire range of frequencies and they found one little blip," says Spiegelhalter. "The more places you look, the more your sigma level gets reduced, and therefore you have to go for a really high number to be really confident you haven't made a false discovery." This is called the look elsewhere effect. It plays an important role in genetics, where scientists look for signals in a huge number of different places. "This is why geneticists also demand very high sigma levels."

If you know a little bit about statistics, then all of this may sound familiar. In other areas of science people often use p-values to quantify their confidence in a result. So what's the relationship between p-values and sigma levels? "The sigma level is only a short hand for the p-value, which is the probability of observing something as extreme as you did if the thing that you are trying to find isn't actually there, or if the drug isn't effective, or the gene doesn't have a relationship with what you are interested in, and so on," explains Spiegelhalter.

To be confident of your result, the p-value therefore has to be very small, and becomes awkward to express: 1 in 1000, 1 in 3 million etc. "It's much easier to use the short-hand of 1, 2, 3, 4, or 5 sigma," says Spiegelhalter. "At CERN they actually calculate the p-value from a very complex statistical procedure and then translate it back to an effective number of sigmas."

So, next time you hear an announcement from CERN, remember: five sigmas are needed to claim a discovery. Signs for the famous Higgs boson existed at lower sigma levels long before 5 sigmas were reached. It would have been embarrassing, however, to claim a discovery earlier, only to find that the blip in the data was a fluke and the particle not there.