Is the Moon still there when you're not looking at it? You can't know because you'd have to look at it to check, but you probably think that it is. Why should the existence of something as neutral as the Moon depend on where we are looking?

The idea that a particle does not have a well-defined position in space but can be in several places at once can be translated into Schrödinger's famous thought experiment involving a cat that is simultaneously dead and alive.

Yet, the question, which was posed for illustrative purposes by Albert Einstein, highlights one of the major problems with quantum mechanics. While the theory doesn't apply to large things such as the Moon, it seems to suggest that reality, at the smallest scales, is fuzzy. According to the standard interpretation of quantum mechanics, particles, such as electrons or protons, don't have precise locations in space, they don't travel at well-defined speeds, and they don't have precise values for other physical quantities either. Only when we look — when we make a measurement — does a particle "assume" a location or speed through some mysterious mechanisms nobody understands. (See here to find out more about this problem of measurement.)

What's also strange is that nature appears fundamentally random. The location or speed a particle assumes when we measure it appears as if picked out of a hat. To use Einstein's famous phrase, it seems that God plays dice with nature.

Physicists believe that nature behaves in this strange way because of a mathematical object called the wave function. Just like Newton's laws of motion describe the behaviour of macroscopic objects, so the wave function is believed to describe the behaviour of tiny particles. You can use the wave function to calculate the probabilities of possible outcomes of an experiment, such as the particle appearing at point A or point B. Its predictions have been tested to a higher degree of accuracy than any other theory in physics and appear to be correct. But if you take the wave function as gospel, then you have to accept that, in general, particles really don't have well-defined values for properties like position or speed. The wave function simply doesn't speak that kind of language.

But why should we take the wave function as gospel? It does make correct predictions, so it's clearly onto something, but perhaps it's not all there is. Could we perhaps add something to quantum mechanics, to create a new theory which isn't random and gives us more of a clue as to what particles are doing when we are not looking?

Ride the wave

The answer is yes. An example is pilot wave theory, first proposed, and then abandoned, by one of the originators of quantum mechanics, Louis de Broglie, and later developed further by David Bohm. De Broglie insisted that particles as we normally think of them do exist. They travel through space along well-defined paths and at every moment in time have a precise location, speed and so on. Those paths, however, are not what we'd expect them to be from ordinary physics. The particles are guided, not by Newton's laws of motion, but by the wave function from quantum mechanics. "[The wave function] has another role, apart from the practical role of spitting out probabilities of outcomes of measurements," explains Jeremy Butterfield, a philosopher of physics at the University of Cambridge. "It also tells the individual particle where to go. It guides it. It's like a pilot of a big ocean liner." Hence the name "pilot wave theory".

A surfer's path is guided by the wave he or she is riding. According to pilot wave theory, a particle's path is also guided by a wave.

In contrast to ordinary quantum mechanics, pilot wave theory doesn't involve any fundamental randomness at all: particles don't just turn up in unpredictable places when we make a measurement. However, calculating their exact paths using pilot wave theory is a bit like predicting the outcome of a horse race. There are so many factors involved — the form of each horse, the quality of the turf, the jockeys' mood — you really have no chance in hell of getting it right. People resort to probabilities instead and pilot wave theory allows us to do the same. We might not be able to predict for certain where a particle will be when we measure it, but at least we can work out what chance it's got of being found at a particular spot. In both cases, the uncertainty involved is not down to nature being fundamentally random, but to our ignorance of the many factors involved.

This kind of use of probabilities to deal with our ignorance wasn't a new thing in physics when de Broglie first developed his theory. When people first studied the behaviour of gases in the 19th century, they soon realised they had no chance of tracking the motion of each individual molecule, and resorted to statistics instead, ending up with a theory now known as statistical mechanics.

Luckily (or rather, by de Broglie's ingenious design) the probabilities given by pilot wave theory match those given by quantum mechanics — so since quantum mechanics agrees with experiments, pilot wave theory does too. The crucial difference is that in ordinary quantum mechanics, the uncertainty is due to nature being fundamentally random, while in pilot wave theory it's due to our ignorance of all the details involved, just like in statistical mechanics.

Who's looking?

All this sounds remarkably sane, but it wouldn't be quantum physics if all the weirdness had gone away. And it hasn't. The theory says that any act of measurement interferes with the system you're measuring so much, it changes it drastically. "When you trundle on some massive [measuring] apparatus, which is billions of times bigger than your tiny [particle], you radically disturb it," explains Butterfield. The outcome of a measurement depends sensitively on the details of your measuring apparatus, and in particular, it depends on which quantities you decide to measure at the same time. And because the disturbance is so radical, that outcome is liable to tell you very little about what the system was actually doing before you looked. "In this view measurement is distorting, it's disturbing, it's not truth-telling — it's fibbing to you," says Butterfield. Even though the theory admits a definite reality that exists when we are not looking, you'll never catch that reality unawares. Physicists call this phenomenon quantum contextuality (you can find out more about contextuality here).

Crucially, this also means that certain properties can't be simultaneously measured to any desired degree of accuracy. The position and the momentum of a particle are an example: measuring the position disturbs the particle in a way that makes it hard to measure momentum and vice versa. This awkward relationship between position and momentum is summed up in Heisenberg's famous uncertainty principle from ordinary quantum mechanics. However, in ordinary quantum mechanics the principle is regarded as a fundamental uncertainty of nature, while in pilot wave theory it's down to in principle limitations of measurement.

Contextuality is also to blame for the most shocking aspect of pilot wave theory: "spooky action at a distance" to quote Einstein again. Several particles may be guided by the same wave function, and there are no geographical bounds to how far these particles can move apart. But because the guiding wave is the same, a measurement that affects a particle on Earth can also instantaneous affect another particle several hundred light years away. This weird effect sat uncomfortably with many of the originators of quantum mechanics, including Einstein, but since then it has been proved that quantum physics simply can't get away from it. And the phenomenon has also been verified in experiments: it's what physicists call quantum entanglement.

Pilot wave theory isn't the mainstream version of quantum mechanics. That's partly because it's hard to reconcile with Einstein's special relativity. It's only in the last twenty to thirty years that it has found slightly wider acceptance among physicists. A special boost came from a discovery involving droplets of silicone oil: when bouncing on a puddle of oil, these little droplets produce their own pilot waves and replicate the behaviour produced by pilot wave theory. They are guided by their pilot waves and, even though they themselves aren't quantum objects, replicate quantum phenomena. For example, they can create the same wave-like interference patterns that electrons produce in the famous double slit experiment. The video above explains how.

So what's right, pilot wave theory, orthodox quantum mechanics, or something else? Nobody knows and it doesn't seem likely we'll find out any time soon. To quote Nobel laureate Steven Weinberg, writing in the New York Review of Books, "Regarding [...] the future of quantum mechanics, I have to echo Viola in Twelfth Night: 'O time, thou must untangle this, not I.'"

About this article

Marianne Freiberger is Editor of Plus. She interviewed Jeremy Butterfield in Cambridge in February 2017.

This article is part of our Who's watching? The physics of observers project, run in collaboration with FQXi. Click here to see more articles and videos about contextuality.