This article was part of a project we ran to celebrate the International Year of Astronomy 2009. The project asked you to nominate the questions about the Universe you'd most like to have answered, and this is one of them. We out it to John D. Barrow, renowned cosmologist and Professor of Mathematical Sciences at the University of Cambridge. Here is his answer. The first part of the question has been answered in Plus by Martin Rees.

What is dark energy?

Dark energy is a term that appeared a few years ago, to distinguish it from something called dark matter. We've known for a long time that there's a lot more material in the Universe than we can see shining in the dark. We know this because we can see that stars in galaxies are moving around more quickly than they would if the only source of gravity pulling them was ordinary matter. Typically, there needs to be around ten times more material in a galaxy than we can see on our photographic plates — the extra material has been called dark matter. Identifying what dark matter is actually made of is an interesting problem, but it's not unexpected that the Universe should contain dark material [see Plus article What is dark matter? to find out more].

Exploding stars reveal dark energy

This image shows the light curve and spectrum of a supernova that was recently observed in the Centaurus A galaxy (marked by the red arrow). Image courtesy the Supernova Cosmology Project.

But dark energy is something different. During the 1990s the Hubble Space Telescope and new detector developments in astronomy allowed researchers to see further than ever before in the Universe. They were able to see supernovae almost at the edge of the visible Universe. By photographing a region of the Universe twice at an interval of a week using telescopes on Earth, astronomers discovered that within this period many stars had exploded in supernovae. They then used the Hubble Space Telescope and ground-based telescopes to see how the light from the exploded stars varied as they reached their maximum brightness and then faded away.

What was very unexpected about these observations was that the characteristic signature — the light curve as it's called — of the brightening and dimming of these supernovae was exactly the same as the signature of a big class of supernovae which we could see locally and understand quite well. This told us that these very distant supernovae are intrinsically the same as nearby ones. The only difference in their appearance is that they look dimmer, simply because they are far away.

This gave astronomers an exciting new way of determining the distance to far away objects: if you understand a supernova's workings, you know its intrinsic brightness and so if you measure its apparent brightness you can deduce how far away it is. You can therefore extend what's known as Hubble's law to a far greater distance than ever before. Hubble's law describes the fact that, in our expanding Universe, the speed with which galaxies and stars are receding away from the Earth is proportional to their distance from us.

This led to an extremely unexpected and revolutionary discovery. As you go out far enough, the expansion of the Universe starts to speed up. Different research groups scrutinised this result in great detail and discovered that the Universe's expansion was indeed accelerating. And to explain that acceleration, around 70% of the Universe has to be in a form of mass and energy that's able to create the acceleration.

That's odd, because to create accelerated expansion you need an anti-gravitational effect. Ordinary material that just attracts things creates a deceleration of expansion. That's just Newton's law, which says that the gravitational force of attraction between to masses is one over their distance squared. So what was discovered here is a new part to the gravitational force which comes into play at very great distances. Also, it's not inversely proportional, but directly proportional to distance. So as the Universe gets larger, things move further apart and feel a greater repulsive force. Over small distances, in the laboratory or in the solar system, this repulsive force is too small to be measured. Only on the largest astronomical scales does it exceed the Newtonian attraction and create visible repulsive effects.

There are also other astronomical observations, apart from expansion, which support the existence of a repulsive force. For example, without it, we can't understand the temperature fluctuations in the radiation of the Universe, as observed by satellites. So there is a nice cross check here — even if we didn't have the supernovae observations, there would still be evidence that this repulsive force exists.

Scalar energy fields

The 70% of mysterious stuff in the Universe that acts repulsively in response to gravity is called dark energy. We don't know what it is, in the sense that we can't identify it yet as being a particular type of matter that is predicted by some part of particle physics. But there are many possibilities for what it could be, perhaps too many.

A Hubble Space Telescope image of the supernova 1994D on the edge of Galaxy NGC 4526. Image courtesy NASA.

There are forms of energy in particle physics which are called scalar fields. These represent the simplest type of energy there is. The only feature of a scalar field is the magnitude of its energy at each point in space; there is no directionality involved. An example of a scalar field would be temperature: if we're describing the weather, then temperature is something that can change with time and from one position to another, but it does not come with directional information. This is in contrast to the wind velocity, which has a direction attached to it, as well as a magnitude: it is a vector field. But a temperature map is a scalar field. If the temperature were the same everywhere, then we'd get a constant scalar field.

Likewise, in particle physics, a scalar field describes a form of energy whose only property is the magnitude at each point in space and time. The only features of a scalar field are how fast it is changing over time — its kinetic energy — and how it is interacting with itself — its potential energy. It's been shown that if scalar fields change slowly enough, so their kinetic energy is very small, then they can contribute the repulsive effects that drive the acceleration of expansion. So dark energy can be thought of as a relatively simple energy field that permeates space — we don't need new physics to describe the acceleration of the Universe, but we need to identify the particles or energy fields that are responsible for what we see.

Einstein's "biggest blunder": the cosmological constant

Back in 1915 when Einstein created the theory of general relativity, the first thing he did was to apply it to the whole Universe, to try and find what kind of universes would satisfy his equations. He found that with only the conventional type of gravity, his solutions predicted a Universe that was either expanding or contracting. But no-one had ever thought that the Universe could be changing in this way, so Einstein tried to suppress this change. He noticed that his theory allowed another component of the force of gravity to exist and exploited this to force the solution to his equations to be static: he balanced Newton's force of gravitational attraction with a repulsive component of gravity. The two components exactly balanced, so the resulting Universe was static.

Later on it was discovered that Einstein's static Universe was a bit like a pin balancing on its tip. If you displace it in any way at all it will fall over — it's unstable. Similarly, if you wobble Einstein's static Universe, or move anything within it, it will start to expand or contract. This instability made the static Universe rather unlikely, and by 1929 Edwin Hubble had indeed gathered sufficient evidence to show that the Universe was not static, but expanding. By that time Einstein had already abandoned the whole idea of a repulsive force, calling it the "biggest blunder of his life". He missed the opportunity to predict that the Universe was expanding. That was done by Alexander Friedmann and Georges Lemaître.

This diagram shows the changes in the rate of expansion since the Universe's birth 13.7 billion years ago. The curve changes noticeably about 6.3 billion years ago when expansion accelerated. Image courtesy NASA.

Today we know that the expansion of the Universe is in fact accelerating, and Einstein's repulsive force gives us another way of explaining the dark energy which drives the acceleration. Einstein's repulsive force is a very simple type of scalar field which doesn't change: its potential energy is just a constant, which has become known as the cosmological constant. The ordinary Newtonian type of attractive gravity was dominant early in the history of the Universe and the expansion was decelerating. But when the Universe got to about three-quarters of its present size, the repulsive form of gravity took over and the expansion started to accelerate. Today we are in the phase of the accelerating Universe. (You can read more about the cosmological constant in the Plus article Lambda marks the spot)

Vacuum energy

Particle physicists have a very different way of thinking of the cosmological constant. In quantum mechanics one often thinks of the vacuum state of a physical system. This doesn't refer to a complete vacuum where there is nothing there. In physics, a vacuum state is the lowest energy state a system can be in. For example, if you drop a marble into a curved bowl, it will roll down to the bottom: the marble lying at rest at the bottom of the bowl corresponds the vacuum state. Similarly, we expect the Universe to have a lowest energy, or vacuum, state, and its energy there is not necessarily zero. Physicists believe that the cosmological constant tells us what this vacuum state of the Universe is.

Albert Einstein in 1921.

The big shock is that the value of the cosmological constant, viewed from the vacuum perspective, is very strange. There is a way to express the vacuum energy as a dimensionless number, that is one that has no units attached to it. If you use the observations to evaluate it, you get a value of around 10^-121 — that's about the smallest number anyone has ever thought about in science. But if you ask physicists to use their theories of particle physics to predict what the vacuum energy of the Universe should be, they will tell you that it's equal to one. So they get the wrong answer by a factor of ten followed by 120 zeroes — something is badly wrong somewhere!

Dark mysteries

Explaining why the dark energy field is present at all is a big challenge, but an even bigger one is explaining why it has such a strange value for its energy. Why did it come into play when the Universe was about three quarters of the way through its evolution to the present? If it had come into play minutely earlier, if the value had not been the observed 10^-121, but 10^-120, then we wouldn't be here. Once the Universe has started to accelerate, you can't make any galaxies anymore: material is pulled apart too quickly to condense and form galaxies. So we are rather fortunate beneficiaries of the fact that galaxies had formed before acceleration took over.

Of course it could all be a coincidence: there may not be deep reasons for why acceleration started when it did. There are theories of particle physics in which the creation of vacuum energy happens at random, rather like a needle balancing on its point: it falls in one direction or it falls in another direction, and whichever direction it happens to fall in determines the magnitude of the stress. Or there may be some deeper part of physics we don't know yet. String theory hasn't helped us in this respect so far, it hasn't told us anything useful yet about the cosmological constant. Trying to find a theory which can explain the dark energy is a big unsolved problem in physics.

How can we crack the dark energy mystery?

The strength of dark energy in our solar system or here on Earth is going to be fantastically small, so we have no possibility of carrying out experiments in the laboratory to detect its presence. But in astronomy there is the possibility of describing the nature of the dark energy scalar field in a little more detail. Is it a simple field, like Einstein imagined, where the density stays constant forever, or is it more complicated? By making more detailed observations of the effects of acceleration in the past, we might be able to draw up a historical development to see if the acceleration has changed over the last few billion years. Particle physicists don't seem as yet to have anything particularly new to say about dark energy and its origins within particle physics. Experiments planned at the Large Hadron Collider at CERN may help us in some unexpected way.

A representation of the evolution of the Universe over 13.7 billion years. The far left depicts the earliest moment we can now probe, when a period of inflation produced a burst of exponential growth. Image courtesy NASA/WMAP Science Team.

One tantalising possibility comes from the inflationary universe theory, which proposes that very early in the history of the Universe there was a period when its expansion accelerated. This caused all sorts of observable features. It's the same sort of phenomenon as the current acceleration, and the type of scalar fields you look to to explain early inflation, may also be creating this late acceleration effect. There could be a deep link here, for example the same scalar field, with periodic behaviour, could be responsible for both. There are many different theories for how inflation is implemented and there are quite a lot of theories as to how the present acceleration is driven and what dark energy is. So far there is no compelling way to link them together.

Is dark energy going to kill us?

The simple implication of the picture we see at the moment is that the Universe is going to expand forever. If the acceleration of the expansion continues, then things will be pulled apart more and more, no new stars and galaxies will form, old ones will die, and eventually all ways of processing information or creating complexity will be washed away — the Universe will become a rather uninteresting place. It's as though there is a habitable zone in time. You wouldn't expect life to be possible before the stars have formed because there would be no carbon, nitrogen or oxygen, and no planets. Similarly, you would be very hard-pushed to have life after all the stars have died. So we are in an interval in cosmic history in which the raw material is available to make complex biochemical organisms and in which there's a nice and stable thermodynamic environment provided by stars.

Of course there could be a big surprise: the dark energy may not be constant as Einstein thought, driving the acceleration forever. Joao Magueijo, Kate Land and I once developed a theory where the scalar field changed in a way that allowed the Universe to exit out of a period of acceleration. Or the scalar field could decay into something else at some stage, for example into ordinary radiation which has no repulsive gravitational effect. So the Universe could change gear again: expansion might decelerate a lot, causing the Universe to head back to a big crunch, or just a little, so that expansion continues forever. Either way, we see that until we understand the dark energy, we will not be able to predict the fate of the Universe with any confidence.