Seeking truth with statistics

As if deciding on guilt or innocence wasn't a big enough burden for juries, but many now also have to struggle with complex scientific evidence. It is vital that the statistical arguments involved are presented in a meaningful way, and this difficult task was the subject of the latest session of the Royal Statistical Society's General Applications Section, on June 8.

Since the advent of DNA profiling, there has been an increased awareness amongst lawyers of the importance of accurate probability estimates and interpretation of the statistical evidence, said Colin Aitken, from Edinburgh University. But statistics has long been a part of legal proceedings, and their correct usage in specific trials was discussed as early as 1908 by mathematicians including Poincaré. In fact, statistical arguments may be necessary even when considering something as simple as witness statements about car identification, such as the famous taxicab problem.

Philip Dawid

This is not surprising, as statistics and the law are essentially the same subject, according to another speaker, Philip Dawid, from University College London. Despite coming from different backgrounds, both subjects deal with interpreting evidence, analysing data, testing hypotheses and making decisions under uncertainty. But recent high profile cases, such as the Sally Clark case and overturned convictions involving DNA evidence have highlighted the difficulties of presenting statistics in court cases.

Even when considering very rare events, like losing two children to SIDS (cot death), or that an innocent person's DNA matches that found at the crime scene, it is important not to mistake the small probability of these events happening with the probability that the defendant is innocent - the Prosecutor's Fallacy. Rare events do sometimes happen, and Bayes' Theorem must be used to assess the impact of this type of evidence.

Image from DHD Photo Gallery

For example, suppose the defendant's DNA matches the DNA profile of a blood stain left by the perpetrator at the crime scene. The expert witness explains that there is a 1 in 10 million chance that a random member of the population's DNA would match the profile of the blood stain - called the match probability. However this small probability, 1 in 10 million, is certainly not the probability that the defendant is not the source of the crime stain. In a population of 40 million people, say, there are likely to be 4 people who match, one of whom is the perpetrator and 3 of whom are innocent. So based on this evidence alone, the probability that the defendant is not the source of the blood could be argued to be 3/4.

As techniques improve, the match probability in DNA cases can now be as high as 1 in a billion. And although this level of precision - the "number of noughts" - affects the outcome of statistical assessment of the DNA evidence, said Dawid, it is still important that the same logical process is carried out. And in many more complex cases, such as those involving multiple perpetrators or crime stains, mixed DNA samples or mutation of DNA, the statistical community is still discussing how best to assess this evidence.

Ian Evett, from the Forensic Science Service, outlined rulings on past cases which mean that expert witnesses are not allowed to give their opinion on the strength of a proposition if there is a statistical assessment of the evidence available: only the statistic can be presented to the jury. So a forensic scientist giving evidence on a DNA profile match cannot say that in their opinion the evidence strongly supports the proposition that the defendant left the blood stain at the crime scene: they can only present the match probability from the DNA profiling process to the jury. However, experts in fields where there is no statistical data, such as handwriting identification or fingerprint matching, are allowed to give such an opinion.

Ian Evett

Evett hopes that all expert opinion will increasingly be based on sound scientific and statistical arguments, and agrees that it is right for the statistics to be presented as evidence to the jury. But he also feels it is vital to explore with the judiciary and legal profession ways of presenting statistical evidence that are balanced, robust and logical. Work has been done on this in the scientific community, but it appears that it has yet to influence the legal system.

It is clear that courts are not currently equipped to ensure that statistics are presented to the jury in a way that "best informs their decision-making", Evett said. It is in all our interests that the legal and statistical professions continue to work together to improve the use of statistics in the courts, no matter which side of the dock we find ourselves on.

Read more...

• about the Sally Clark case from Issue 21
• about presenting forensic evidence in court from Issue 19
• about DNA evidence from the Forensic Science Service
• about the use of Bayes Theorem in a DNA case