When driving on the motorway, have you ever wondered about (and cursed) the fact that cars in the other lane seem to be getting ahead faster than you? You might be inclined to account for this by invoking Murphy's Law ("If anything can go wrong, it will", discovered by Edward A. Murphy, Jr, in 1949). However, a recent paper in Nature by Redelmeier and Tibshirani seeks a deeper explanation.

According to the authors, drivers suffer from systematic illusions causing them to mistakenly think they would have been better off in the next lane. In this Plus article, we show that their argument fails to take into account an important observation selection effect - namely that cars in the next lane actually do go faster!

In their paper, Redelmeier and Tibshirani present some evidence that drivers on Canadian roadways think that the next lane is typically faster. (You can read their paper for more about this). They seek to explain the drivers' perceptions by appealing to a variety of psychological factors. For example:

• "A driver is more likely to glance at the next lane for comparison when he is relatively idle while moving slowly";

• "Differential surveillance can occur because drivers look forwards rather than backwards, so vehicles that are overtaken become invisible very quickly, whereas vehicles that overtake the index driver remain conspicuous for much longer"; and

• "Human psychology may make being overtaken (losing) seem more salient than the corresponding gains."

The authors recommend that drivers should be educated about these effects in order to reduce the temptation to switch lanes repeatedly. This would reduce the risk of accidents, which are often caused by poor lane changes.

While all these psychological illusions might indeed occur, there is a more straightforward explanation for the drivers' persistent suspicion that cars in the next lane are moving faster.

Observation Selection Effects

How big is the smallest fish in the pond? You take your wide-meshed fishing net and catch one hundred fishes, every one of which is greater than six inches long. Does this evidence support the hypothesis that no fish in the pond is much less than six inches long? Not if your wide-meshed net can't actually catch smaller fish.

The limitations of your data collection process affect the inferences you can draw from the data. In the case of the fish-size-estimation problem, a selection effect - the net's being able to sample only the big fish - invalidates any attempt to extrapolate from the catch to the population remaining in the water. Had your net had a finer mesh, allowing it to sample randomly from all the fish, then finding a hundred fishes all greater than a foot long would have been good evidence that few if any fish remaining were much smaller.

In the fish net example, a selection effect is introduced by the fact that the instrument you used to collect data sampled from only a subset of the target population. Analogously, there are selection effects that arise not from the limitations of the measuring device, but from the fact that all observations require the existence of an appropriately positioned observer. Such an observation selection effect can be seen when asking "For a randomly selected driver, are the cars in the lane next to them actually going faster?".

Cars in the next lane do go faster

One frequent cause of why a lane (or a section of a lane) is slow is that there are too many cars in it. Even if the ultimate cause is something else (for example, road works) there is nonetheless typically a negative correlation between the speed of a lane and how densely packed it is with vehicles. Cars travelling at higher speeds are usually separated by longer distances than slower cars, which tend to bunch up. This leads to a higher density of cars in lanes travelling at slow speeds, meaning that over a given stretch of road there are likely to be more cars in the slower lane(s) than in the faster lane(s). This in turn means that, when driving on a two lane (or more) road system, more of the average driver's time is spent in the slower lane(s).

An important idea in the theory of observation selection effects is that observers should reason as if they were a random sample from the set of all observers. This means that when making observations of the speed of cars in the next lane, it is important to consider yourself as a random observer, and think about what implications this has on your observation.

If you are driving on the motorway and think of your present observation as a random sample from all the observations made by all the drivers, then chances are that your observation will be made from the viewpoint that most drivers have, which is the viewpoint of the slow-moving lane (where more cars are likely to be). In other words, appearances are faithful: more often than not, the "next" lane is actually faster!

Furthermore, even when two lanes have the same average speed, it can be advantageous to switch lanes. This is because what affects a driver who wants to reach her destination as quickly as possible is not the average speed of the lane as a whole, but rather the speed of the section extending maybe a couple of miles forwards from her current position. More often than not, the next lane has a higher average speed at this scale than does her present lane. On average, there is therefore a benefit to switching lanes (which of course has to be balanced against the increased levels of effort and risk involved).

Adopting a thermodynamics perspective, it is easy to see that (at least in the ideal case) increasing the "diffusion rate" (that is, the probability of lane-switching) will speed the approach to "equilibrium" (where there are equal velocities in both lanes), thereby increasing the road's throughput and the number of vehicles that reach their destinations per unit time.

To summarise, in understanding this problem we must not ignore its inherent observation selection effect. This resides in the fact that when we randomly select a driver and ask her whether she thinks the next lane is faster, more often than not we will have selected a driver from the lane which is in fact slower and more densely packed. When we realize this, we see that no case has been made for recommending that drivers change lanes less frequently in order to speed up overall traffic flow.

Why are we here?

The theory of observation selection effects is a relatively recent development, and it may have implications in a number of other fields (see Nick's new book). One example is evolutionary biology, where observation selection effects must be taken into account when addressing questions such as the probability of intelligent life developing on any given earth-like planet.

We know that intelligent life evolved on Earth. Naively, one might think that this piece of evidence suggests that life is likely to evolve on most Earth-like planets, but that would overlook an observation selection effect. No matter how small the proportion of all Earth-like planets that evolve intelligent life, we must be from a planet that did (or we must be able to trace our origin to a planet that did, if we were born in a space colony) in order to be an observer ourselves.

Our evidence - that intelligent life arose on our planet - is therefore predicted equally well by the hypothesis that intelligent life is very improbable even on Earth-like planets, as it is by the hypothesis that intelligent life is highly probable on Earth-like planets. The evidence does not distinguish between the two hypotheses (provided that in both hypotheses intelligent life could have evolved somewhere).

Another example comes from cosmology, where observation selection effects are crucial considerations in deriving empirical predictions from so-called multiverse theories, according to which our universe is but one out of a vast ensemble of physically real universes out there.

Anthropic reasoning, the theory of how to detect, diagnose and cure the biases of observation selection effects, is a philosophical goldmine. Few fields are so rich in empirical implications, touch on so many important scientific questions, pose such intricate paradoxes, and contain such generous quantities of conceptual and methodological confusion that need to be sorted out. Working in this area is a lot of intellectual fun.

The mathematics used in this field, such as conditional probabilities and Bayes' Theorem, are covered by fairly elementary arithmetic and probability theory. The topic of observation selection effects is extremely complex, yet the difficulty lies not in the maths, but in grasping and analysing the underlying principles.

Read more

The website www.anthropic-principle.com features articles on observation selection effects and some puzzling applications.

References

Redelmeier, D. A., Tibshirani, R. J. Nature 401: 35 (1999).
Bostrom, N. Anthropic Bias: Observation Selection Effects in Science and Philosophy (Routledge, New York, 2002).

---

About the author

Nick Bostrom is a lecturer at Yale University. His research focuses on observation selection effects, foundations of probability theory, philosophy of science, and the ethical questions related to human enhancement-technologies and superintelligent machines. He has a homepage at www.nickbostrom.com.