What happens when we're not looking? What happens when we do look? These two questions are the heart of the mystery of quantum mechanics.

By looking we really mean observing: for example, observing an experiment involving the smallest building blocks of matter, such as electrons. Quantum mechanics tells us that while we are not looking, such particles behave very differently from when we are looking. Before an observation is made, the particles do not occupy specific places in space and they do not travel at well-defined speeds. It's only when we make an observation, that particles assume specific locations and speeds (see here for more details). So quantum mechanics tells us that something special happens when an observation is made. But it doesn't tell us why or how that happens. It doesn't tell us what counts as an observation or an observer.

Quantum mechanics doesn't give us the whole story of what happens in between observations. It does tell us what happens to the mathematical object describing a system, the wave function, during those times. But what about the electrons and photons, and all the other stuff that makes up the world? What exactly is that stuff doing when we are not observing it?

Although quantum mechanics doesn't give us the answers to these questions, we do know something about the answers, if they exist. We know that they must be different from the answers people had before quantum mechanics, in the days of Newton and classical physics. The biggest difference may be something called contextuality — a part of the complicated relationship between observers and observations. It may be the most quantum thing about quantum mechanics.

The meaning of contextuality

"Context" means all the details that surround an event. Context gives us more information about what happened and how it happened: what else happened at the same time, or just before, or just after.

Not what you'd expect to find in your garage.

I asked physicist Ian Durham of St. Anselm College, in New Hampshire USA, to summarise it in a sentence. He said, "When you go to your garage, you get a car; you don't get a llama." Is a llama in your parking space unusual? Most likely very unusual if you live in a city like London. But maybe not if you live on a ranch in the Andes Mountains. The context — where you live — adds meaning to the basic event — a llama in your garage.

Contextuality in physics describes how or whether the details of an observation affect what is observed. We mean that the results of measurements can depend on how we made the measurement, or what combination of measurements we chose to do.

In physics this applies to measurements of quantities such an electron's spin: the answer you get for a particular direction of spin may vary depending on what you choose to measure alongside it. To see what this would mean if it applied to things we are familiar with, imagine measuring a person's height. If contextuality applied in this context, you might get a different value for height if you measured the person's weight along side it, than you would get if you measured the person's shoe size alongside it.

Is this weird? Maybe. It's weird if you think an electron's spin (or a person's height) had a preset value before you measured it, and you think the whole point of a measurement is to give you a true report of that value. You would expect to get that one answer when you make the measurement.

Definite theories

Durham suggested another way to picture contextuality. Imagine you're putting together a jigsaw puzzle. Somehow the box got lost, so you don't know what the picture is supposed to be. You put the puzzle together starting from the outer edges, and you find a picture of a cat. Then you break the puzzle up and put it together again, now starting from the inner pieces. It's the same pieces to the same puzzle, but this time you find the picture is — a llama.

How is reality pieced together?

This is weird if you think the story of the world should have a prefixed image, like a normal jigsaw puzzle. But does it? Or is the story of the world made up as it goes along, and changed by our choices?

Quantum mechanics doesn't tell us what electrons are doing when we are not observing those values. The electrons and other particles live secret, unknowable lives, as far as we can predict. A theory that tells us more might give us a complete picture of what electrons are doing at all times. It would also tell us the values of things we can measure like momentum or spin, even when we are not trying to measure them.

Classical Newtonian mechanics is a theory that has these features. Classical particles are like rocks. They have concrete positions and speeds. They have a real story about what they are doing when we don't look at them. Experiments show us the true values of those things.

Philosophers of science call this feature value definiteness, because the measurable things have definite values. So what about quantum mechanics? Could there be a "definite" theory underneath quantum mechanics? A theory that would give us a real story about the electrons, and that delivers the same predictions as quantum mechanics for experiments?

The Kochen-Specker theorem and contextuality

An argument called the Kochen–Specker theorem tells us what we can and can't have. The theorem is named for mathematicians Simon Kochen and Ernst Specker, who published their work in 1967. The physicist John Bell found a similar result around the same time and various other people have come up with simpler versions of the proof since then. Here's what it tells us.

Imagine you have invented a new theory to explain what quantum particles are really doing, even when we are not watching them. Your theory delivers the same predications as quantum mechanics for observations. But your theory has an added feature — it gives us a concrete story about the world. Electrons have definite speeds and spins and so on. Their speeds and spins have definite values that exist even when we are not trying to detect them. When we do try to detect them, your theory predicts that tests will show us those values that the speed and spin had just before we did the test. In other words, you theory has value definiteness. (An example of such a theory is pilot wave theory, you can find out more here.)

The Kochen-Specker theorem can be proved by a relatively simple argument involving colouring arrows within a sphere. See here to find out more.

The Kochen–Specker theorem tells us that your theory must be contextual. The values that your electron has for its speed and spin, those values must somehow or sometimes depend on how you try to measure them.

It may be tricky to give a clear explanation of how contextual effects could actually happen without upsetting those preset, definite values, or without changing what sort of things can have preset values. The preset values are not just "the spin of an electron". They must be things like "the spin as measured in this particular detector, while these other particular things are also measured," and so on. We seem to lose the point that we are trying to save – that there is something real in the world that exists outside of our particular observations.

There is an alternative – you can try to change your theory to avoid contextuality. The Kochen–Specker theorem says you must then give up value definiteness. The way you choose to measure things won't effect the values you get in the experiments, but those values didn't exist before you did the measurements! You can't say the electron had that speed and that spin until after you ran it into the detector and measured those values. Before the detector, we cannot say what is happening in the world.

This non-contextual option takes us back to the starting point — quantum mechanics as a mysterious theory that tells us nothing about what happens when we do not observe the world. Some are bothered by this option, but for some people, this option is not a problem. This is just quantum mechanics' message.

(Incidentally, the Kochen-Specker theorem has a surprisingly straightforward proof, involving colouring a sphere with two different colours — you can find out more here.)

Beyond quantum mechanics

Despite the various research efforts, the role of the observer in physics is still being debated.

Researchers continue to think about contextuality and to argue about whether we should want to avoid it. They have adapted the Kochen–Specker theorem to clarify the meaning of contextuality and to come up with different ways to talk about it.

An influential result came in 2004, when physicist Rob Spekkens from Canada's Perimeter Institute found a way to think about contextuality in theories besides quantum mechanics. His work makes it possible to ask if we can use a different theory in order to avoid contextuality, and what other weirdness we have to accept to avoid it.

Some people are looking for creative ways around the Kochen–Specker theorem. Matt Leifer of Chapman University in California told me he hopes we can avoid contextuality through retrocausality — that is by allowing future events to have an effect on the past. But even allowing that, he is not sure that we can get away from all appearances of contextuality. It may be a very deep part of the real world. He describes it as a sort of "unifying principle" for many other quantum things, like quantum tunnelling. If you want to avoid contextuality, you will still have to explain these things.

So is contextuality weird? Is a contextual world a problem? The answer for now is maybe – it all depends on how you look at it.

About this article

Brendan Foster is a writer, musician, and sometime physicist in Baltimore, Maryland, USA. He is also science programs coordinator for the Foundational Questions Institute, and cohost of the FQXi podcast, with Zeeya Merali and (sometimes) Puffy the Cat.

This article is part of our Who's watching? The physics of observers project, run in collaboration with FQXi. Click here to see more articles and videos about contextuality.