Bad statistics can mislead, and who'd know this
better than mathematicians? It's ironic, then, that mathematics itself has fallen victim to the seductive lure
of crude numbers. Mathematicians' work is being measured, ranked and
judged on the basis of a single statistic: the
number of times research papers are being cited by others. And mathematicians
are not happy about it.
Is this good maths?
Like any other area in receipt of public money, mathematical
research needs to be accountable. A reasonable way to judge the quality of research is the impact it has on future research: ground-breaking work will be
heavily discussed and built upon, and mediocre work largely ignored. Traditionally, the reputations of
individual researchers, institutions, or research journals have
hinged on the opinions of experts in the field. The rationale behind
using citation statistics is that bare numbers can overcome
the inherent subjectivity of these judgments. In a competitive world it's
the bottom line that should do the talking.
Bottom lines are crude, however, and summary statistics open to
misuse. A whiff of scandal floated through this year's International Congress of Mathematicians, when the mathematician
Douglas N. Arnold (president of the Society for Industrial and Applied Mathematics)
exposed what appears like a blatant example of citation fraud. It involves the
International Journal of Nonlinear Science and Numerical
Simulations (IJNSNS) and a summary statistic called the impact
The impact factor of a journal measures the average number of
citations per article in the journal, but only taking into
account citations from the current year to articles that have appeared
in the previous two years. So old citations don't count and
neither do citations to articles that are older than three years.
IJNSNS has topped the impact factor chart for applied maths journals
for the last four years
by a massive margin. In 2009 its impact factor was more than double
that of the second in line, the esteemed Communications on Pure and
Applied Mathematics (CPAM). A panel of experts, however, had rated IJNSNS in
its second-to-last category: as having a "solid, though not outstanding
reputation". In the experts' opinion IJNSNS comes at best 75th in the
applied maths journal rankings, nowhere near the top.
There are some easy explanations for this mis-match between the impact factor chart and expert opinion. A closer look at citation statistics shows that 29% of the citations to IJNSNS (in
2008) came from the editor-in-chief of the journal and two colleagues
who sit on its editorial board. A massive 70% of citations to IJNSNS
that contributed to its impact factor
came from other publications over which editors of IJNSNS had
editorial influence. An example is the proceedings of a conference that
had been organised by IJNSNS's editor-in-chief Ji-Huan He. He
controlled the peer review process that srcutinised papers submitted
to the proceedings.
Another striking statistic is that 71.5% of citations to
IJNSNS just happened to cite articles that appeared in
the two-year window which counts towards the impact factor (the 71.5%
is out of citations from 2008 to articles that have appeared since 2000). That's
compared to 16% for CPAM. If you use a five-year citation window (from
2000 to 2005) to calculate the impact factor, IJNSNS's factor drops dramatically, from 8.91 to 1.27.
The conclusions from this are obvious: cite your own
journal as often as possible (with citations falling in the
two-year window) and make sure that authors who fall under your
editorial influence do the same, and you can propel your journal to the top of the rankings.
Libraries use impact factors to make purchasing decisions, but mathematicians are judged by them too.
What's worrying is that impact factors are not just being used
to rank journals, but also to assess the calibre of the researchers who
publish in them and the institutions that employ these researchers. "I've received letters
from [mathematicians] saying that their monthly salaries will depend
on the impact factors of the journals they publish in. Departments and universities are being judged by impact
factors," says Martin Grötschel, Secretary of the International
Mathematical Union, which published a highly critical report on
citation statistics in 2008.
Grötschel dismisses the blind use of impact factors as
"nonsense" and not just because they are open to manipulation. For
mathematics in particular, the two-year window that counts towards the
impact factor is simply too short. There are examples of seminal maths
papers that didn't get cited for decades. In fact, scouring 3
million recent citations in maths journals, the IMU found that
roughly 90% of all citations fall outside the two-year window and
therefore don't count towards the impact factor. This is in stark
contrast to faster moving sciences, for example biomedicine, so using
impact factors to compare disciplines presents mathematics in a truly
Another confounding factor is that papers may get cited for all the
wrong reasons. As Malcolm MacCallum, director of the Heilbronn Institute for Mathematical Research, pointed out at a round table
discussion at the ICM, one way of
bumping up your citation rates is to publish a result that's subtly
wrong, so others expose the holes in your proof, citing your paper
every time. Malicious intent aside, someone might cite a paper not
because it contains a ground-breaking result, but because it gives
a nice survey of existing results. If on the other hand your result is
so amazing that it becomes universally known, you might lose out on
citations altogether — few people bother to cite Einstein's original
paper containing the equation E=mc2 as the result and its originator are now part of common knowledge.
The list of impact factor misgivings goes on (you can read more in
report on citation statistics). The fact is that a single
number cannot reflect a complex picture. With respect to manipulation, Arnold points to Goodhart's law: "when
a measure becomes a target, it ceases to be a good measure". What's
more, no one knows exactly what the impact factor is supposed to
measure — what exactly does a citation mean? As a statistical quantity the impact factor is not sufficiently robust
to chance variation. As the IMU report points out, there's no sound
statistical model linking citation statistics to quality.
How, then, should mathematical quality be measured? Mathematicians
themselves aren't entirely in agreement on how big a role, if any, citation
statistics should play, or even whether things should be ranked at
all. Everyone agrees, however, that human judgment is
essential. "Impact factors — we
cannot ignore them, but we have to interpret them with great care,"
says Grötschel, the IMU Secretary. The IMU, together
International Council of Industrial and Applied Mathematics, has set
up a joint committee to come up with a way of ranking journals that
involve human judgment and statistics.
With their fight against the mindless use of statistics mathematicians will do a service not just to themselves.
"Some of [our work on this] has very broad
applications in other sciences," says Grötschel. "It's very
important that mathematicians are at the forefront of
For the latest episode of the Math/Maths podcast Peter Rowlett interviewed a very tired and exhausted Plus, reporting from the ICM in Hyderabad, India. It's nice to be on the other side of the microphone for a change! The podcast also explores Roberto Carlos' free kick; A New Kind of Baseball Math; More on P !=NP; The #mathgeek experiment; Clustered Networks; measuring physical
constants; testing string theory; Twitter Venn; Mangahigh; and more.
The Math/Maths Podcast: 5136 miles of mathematics is
a weekly conversation about mathematics between the UK and USA from Pulse-Project.org. Peter Rowlett in Nottingham calls Samuel Hansen in Las Vegas and the pair chat about math and maths that has been in the news, that they've noticed and that has happened to
Many things in life are more than the sum of their parts. Whether its the behaviour of crowds of people, flocking birds or shoaling fish, the unpredictable patterns of the weather or the complex structure of the Internet, it's often the interaction between things, rather than the things themselves, that generates complexity.
It's a challenge to science, whose traditional approach of taking things apart and looking at the individual bits doesn't work when faced with emergent complexity. But there are mathematical techniques to understand this phenomenon. The Living in a Complex World website, originally launched to accompany an exhibit at the Royal Society Summer Science exhibition, explores complexity in the real world and has some great factsheets looking at the maths used to understand it. It's well worth a look!
Well, maybe not quite ... it's a panel discussion on popularisation of maths. (Thanks to Jaime Carvalho e Silva for both of these photos.)
Plus with Cédric Villani.
3000 mathematicians trying to have dinner.
3000 mathematicians trying to have lunch.
3000 mathematicians trying to catch a hotel bus.
Plus with Christian Schlaga, Germany's acting ambassador to India. It's a long story, but basically Plus ended up with a sculpture of the Berlin bear (with a maths design) that had been presented to Schlaga at the German embassy's reception at the ICM.
What would you think if the nice café latte in your cup suddenly separated itself out into one half containing just milk and the other containing just coffee? Probably that you, or the world, have just gone crazy. There is, perhaps, a theoretical chance that after stirring the coffee all the swirling atoms in your cup just happen to find themselves in the right place for this to occur, but this chance is astronomically small.
Cédric Villani, Institut Henri Poincaré Fields medallist 2010.
The fact that such spontaneous separation never occurs in practice is an illustration of a deep physical law: it says that the entropy of a system, a measure of its complexity, almost always increases as time passes. When you first pour the milk into your coffee, for a split second milk and coffee will be neatly separated, but soon the milk disperses. The mixture of milk and coffee becomes more and more complex, until it reaches an equilibrium when both are completely mixed up and complexity is at its peak.
In the late 19th century the physicist Ludwig Boltzmann studied this phenomenon, looking at what happens when a gas is released into a room from a bottle. He came up with an equation describing the evolution of this process — the change over time — in terms of how individual atoms collide and other forces acting on the gas. His calculations showed that indeed entropy doesn't decrease. The atoms of gas start out in an ordered state — all sitting in the bottle — and end up in a state of maximum complexity, dispersed throughout the room.
Intriguingly, this result meant that there is what physicists call an arrow of time, something that isn't inherent in classical Newtonian physics. If someone showed you a movie of a billiard ball rolling across the table, then you wouldn't be able to tell if the movie was being played forwards or backwards: it's just as likely that the ball rolls one way as it is to roll in the opposite direction. If, however, someone showed you a movie of a coloured gas dispersed in a room suddenly entering a bottle, you'd know that something's wrong. The movie is being played backwards. Since the interaction of individual atoms was described in terms of Newtonian laws (which don't have a preferred direction of time), this emergence of an arrow of time created some headache for physicists. Debate on the arrow of time issue continues to this day. As the mathematician John von Neumann once put it, "nobody knows what entropy really is, so in a debate you will always have the advantage".
Villani, however, does understand was entropy is. One question that has until recently remained open was how fast the entropy of a system increases. Cédric Villani received his Fields Medal for developing rigorous mathematical techniques that provide an answer. They show that while entropy never decreases, it sometimes increases faster and sometimes slower. Based on this work, Villani developed a general theory, hypercoercivity, which applies to a broad set of situations.
Villani also used his understanding of entropy to explain a phenomenon that had puzzled physicists for 60 years. Back in the 1940s, the Soviet physicist Lev Davidovich Landau claimed that plasma, a form of matter similar to gas, spreads and reaches its equilibrium state without increasing its entropy. Landau argued that unlike gas, whose approach to equilibrium is driven by particle collisions (which also lead to a loss of order), plasma reaches equilibrium through a decay in its electric field. Landau made some progress on proving his claim, but despite hard work by many physicists over 60 years, it wasn't until Villani's arrival on the scene that Landau was proved right.
All this might sound somewhat esoteric, but Villani's deep understanding of entropy has direct consequences for real-life problems. In the 18th century the French mathematician Gaspard Monge started to think about how to transport goods to various places in an efficient way. For example, you might want to distribute a bunch of letters sitting in a post office to the various addresses in a way that minimises transport cost. Villani and his colleague Felix Otto made a crucial connection between this problem and gas diffusion, by drawing the following analogy: loosely speaking, the initial state (all letters in the post office) corresponds to the ordered state of gas sitting in a bottle, while the end state (letters delivered) corresponds to a state where the gas has dispersed. To any configuration of dispersed gas particles you can assign a cost by seeing how far the particles had to travel from the original ordered state. Using this analogy and their understanding of entropy, Villani and Otto made important contributions to optimal transport theory.
So next time you have a cup of milky coffee or receive a letter, stop to think that a deep understanding of either, or even better a combination of both, could have earned you a Fields medal.
You can find out more about Villani's work in this excellent description on the ICM website, on which this blog post was based. To find out more about the Boltzmann equation and optimal transport problems, read the Plus article Universal Pictures.
Suppose you throw an equal number of white and black balls into a rectangular box which is, say, 30 balls long, 10 balls wide and is now 5 layers deep in balls. What it the probability that you have a run of touching white balls from one end of the box to the other?
This question, asked all the way back in 1894 in the first issue of the American Mathematical Monthly, turned out be far from simple. In fact it appears to be the earliest reference to the rich mathematical field of percolation theory, according to Harry Keston, who told the International Congress of Mathematicians about Stanislav Smirnov's work in this area that lead to Smirnov winning the 2010 Fields Medal.
Just as the name conjures up the image of water percolating through soil or porous rock, percolation theory models this mathematically as a liquid flowing through a lattice of pipes. The points where the pipes join (mathematically known as the vertices of the lattice) are either blocked, stopping the flow of liquid, or open, allowing the liquid to flow through. You can imagine that if each vertex has a high probability p of being open (and the lattice is very porous like sand) then we can be fairly certain that the liquid will flow through the lattice of pipes. And if the probability p that each vertex is open is low (and the lattice is impermeable like hard clay) then we can be fairly sure the liquid is going to get stuck and not make it the whole way through.
It turns out that there is a particular critical probability for the vertices being open, pc, that determines exactly when a liquid can percolate across the lattice. If the probability that the vertices are open is below this critical probability, p < pc, then the liquid will never percolate through the lattice. When the probability of the vertices being open passes this critical point, and p > pc, the system flips behaviour and the liquid can trickle all the way through. (You can read an excellent more technical introduction in Percolation: Slipping through the cracks.)
Physicists are interested in this area as it is one of the simplest models to have a phase transition, where the behaviour of the system flips at a certain critical point. Phase transitions are an important part of physics, for example understanding the change between different phases of matter. Water immediately starts to boil once it is above a certain critical temperature (just under 100 °C at sea level) but will not boil at lower temperatures. What is particularly interesting is what happens at these critical points. How likely is it for the liquid to percolate across the lattice, called the crossing probability, when the vertices are open with the exact probability p = pc?
But the systems physicists are interested aren't neatly space lattices in regular shapes. Instead physicists are interested in what happens at the smallest possible scale when the spacing in the mesh of the lattice becomes finer and finer, which they call taking the scaling limit. For a given mesh size it is possible to calculate the crossing probability, and physicists were convinced from physical evidence that the crossing probability existed for the scaling limit. That is, they thought that as the mesh got finer and finer the crossing probabilities would get closer and closer to a final value that would be the crossing probability for the scaling limit. And the physicist John Cardy was even able to give a formula for calculating this final value. However, no one was able to prove mathematically either that this value would exist, or that the formula was correct.
In 2001 physicists and mathematicians alike breathed a sigh of relief when Smirnov proved that the crossing probability existed for the scaling limit for a two-dimensional triangular lattice, and that it was equal to the value calculated by Cardy's formula. Keston, himself a pioneer in the percolation theory, said that Smirnov's work would make statistical physicists very happy as it confirmed their assumptions and put the area on a solid mathematical foundation. And it is hoped that the novel techniques Smirnov used to prove this and related results will allow him and others to extend these results to any two-dimensional lattice (square, hexagonal, and so on), proving that Cardy's conjecture is universal and independent of the lattice being used.
At the end of Shmirnov's own presentation of his work at the ICM, he was asked if his work could be extended to three-dimensional lattices. Smirnov held up his hands, very little is known about how these models work in three-dimensions. Perhaps Smirnov, or a future Fields medallist, will take us there.