After four years of fieldwork, the largest ever experiment on genetically modified crops is coming to an end, with the results expected to be published in a scientific journal this summer. But environmentalists are voicing concerns, not only about the ecological effects of the trials, but also that the statistical analysis of the results will not spot ecologically significant effects.

The Farm-Scale Evaluations (FSE) is a programme of trials of crops that have been genetically modified to make them resistant to certain herbicides. There has been widespread concern that these GM crops would allow a greater use of herbicides, resulting in fewer plants for insects to live on, and consequently fewer insect prey for farmland birds. In the experiment, over sixty fields were planted with GM and conventional crops, and biodiversity indicators, such as the number of weeds, slugs, beetles and spiders, were measured for each crop.

Can the FSE detect ecologically significant change?

Pre-empting the publication of the results of the FSE, the environmental group Friends of the Earth has produced a report, "Science as a smokescreen?", claiming that the statistical design of the trials means they may not detect ecologically significant damage caused by the use of the GM crops.

To assess the difference between two statistical samples, scientists test the null hypothesis - the hypothesis that there is no difference between the two samples. In the FSE, one sample is the biodiversity data of the GM crop fields and the other is the biodiversity data of the conventional crop fields. Statistics can't prove that there is a difference between the samples (this is called the alternative hypothesis); it can only tell us if the evidence for a difference is significantly greater than what we would expect as a result of chance alone. When this happens, the null hypothesis is rejected and the alternative hypothesis may be true. If we can't reject the null hypothesis, this does not mean that the experiment has shown that it is true, only that the results do not provide enough evidence to reject it.

The FSE researchers are testing a specific null hypothesis for each of the varieties of crop planted - oilseed rape, sugar beet and maize - namely, "that there is no difference between the management of [GM] varieties and that of comparable conventional varieties, in their effect on the abundance and variety of arable plants and invertebrates". The alternative hypothesis is that there is a difference in abundance or variety - either an increase or a decrease. The experiment has been designed to detect any difference greater than 1.5-fold in the biodiversity indicators for weeds and insects - a difference in counts of, say, 100 to 150, or 67 to 100.

The statistical power of an experiment is the probability of correctly rejecting the null hypothesis when there is an actual difference in the samples. The FSE researchers have calculated that the power of the FSE is 80% - that is, they will be able to detect a 1.5-fold difference with 80% certainty.

In their report, Friends of the Earth express concern that the experiment is not sensitive enough for such a controversial subject. They say that much lower differences can be ecologically significant and would probably be missed by the study. "If there is an actual difference between the treatments but it is smaller than 1.5, the power of the test to detect that difference will be still lower," their report states. "It should be made clear that if the experiment fails to detect differences, they may still be present."

They also question the power calculation, which depends on many factors, including the random variation in the data (estimated by the researchers to be 50%). However, the random variation appears to be much higher than this for some of the biodiversity indicators, according to the researchers' own results from the pilot year of the FSE study; and Friends of the Earth question whether the experiment will be able to detect even a 1.5-fold difference for these indicators.

Les Firbank, who has been coordinating the FSE trials, defended the research, telling New Scientist that only when the data is published will the power of the experiment be known. In a jointly authored paper in the Journal of Applied Ecology, the researchers openly stated that the power will be larger for some indicators than for others. Firbank told New Scientist that "the interpretation comes not from looking at each species in isolation but from combining results from different species and looking for patterns."

Friends of the Earth have made it clear that they do not doubt the the integrity of the researchers. Rather, they are worried that the government will place more reliance on the study than is warranted, when deciding whether or not to allow commercial growing of GM crops. The researchers too have voiced their concerns in a recent paper: "Despite our efforts, we remain concerned that the findings will be overinterpreted, that they will be used as arguments for the widespread adoption, or rejection, of GM crops in general (depending on our findings)."

We can only hope that those in the government responsible for digesting these results will be numerate enough to interpret the statistics wisely, and scrupulous enough not to use science as a smokescreen.