91av

PHYSICS and astronomy have always enjoyed a certain reverence among the sciences. They deal with crisp certainties, expressed as mathematical formulae that appear to leave no room for error.

But these days, physicists aren’t quite so sure of themselves. Today’s experiments designed to extend our understanding of the cosmos deal with wisps and ghosts. Often the evidence is so slender it can be easy to miss – or misinterpret. Stars flickering into existence at the dawn of the universe are so far away that their light is barely perceptible. And some particles interact so rarely that even the best detectors spot only a tiny fraction of them. The boldest claims about the universe rely on physicists sifting the truth from the millions of measurements they make.

Now physicists are starting to acknowledge that they have a lot to learn about handling uncertainties in their data. And they are getting help from a seemingly unlikely source: biologists and medical researchers, who deal with uncertainty all the time.

“For a billion dollars, we build fantastic telescopes with quantum-perfect detectors,” says Eric Feigelson, an astronomer at Pennsylvania State University in Philadelphia. “We use the best physical models we can, the best computers for processing data. We’d never be sloppy in classifying a star. But there’s one link in this steel chain that is made of paper: the one where we interpret the data using mathematics that’s 100 years old.”

Physicists like Feigelson are beginning to realise that instead of being a bane, uncertainty brings with it the potential for discovery. But to continue expanding our understanding of the cosmos they are going to have to get to grips with modern statistics just as biologists have. It has been a long time coming, says Paul Padley, a particle physicist at Rice University, Houston. “We tend to invent the wheel for ourselves. We’re just starting to realise that statisticians have a whole entourage of techniques that we can apply.”

As the following pages show, this realisation is starting to be put to good use. The newest and most direct confirmation of the ghostly dark energy that makes up almost three-quarters of the content of the universe has come via a seminar on the use of statistics in biology (see “Blown apart”, page 39).

And more of the missing universe may turn up as physicists catch up with statistical theory that other sciences have been using for decades (see “Dark matters”, page 40). Some researchers have deliberately begun importing techniques used in medical research. Physicists are also learning to recognise the role their own bias plays in shaping their science (see “We don’t believe the statistics!”, page 38).

These are just some of the examples of the surprises awaiting those who dare to take up modern statistical analysis, according to Feigelson. But, whatever their fears, it is time for physicists to treat their data with the respect it deserves. Hanging in the balance are scientific reputations, the fate of billion-dollar experiments, and Nobel prizes, no less.

1 “We don’t believe the statistics!”

How the Higgs wriggled free

Nothing better illustrates the crucial role of statistics in physics than the closing of the Large Electron Positron Collider (LEP) at the CERN particle research laboratory in Geneva, Switzerland, four years ago. The shutdown had long been scheduled for September 2000, and in the final months experimenters cranked up the power in the hope of discovering the last unconfirmed particle in the standard model of particle physics, the Higgs boson.

Tantalisingly, they saw something. A cluster of four massive particles with the predicted energy of the Higgs stood out against the background of ordinary particles. Yet it was too few to claim they had discovered the Higgs. Urgently, they pleaded for another year to gather results and improve the statistical analyses. But contracts had already been signed to tear LEP down and install the next big collider in the same tunnel, the $2.6 billion Large Hadron Collider. Huge amounts of money would be at risk if the LHC were delayed. Finally, the experimenters were granted one more month.

By November, the case was still not settled. The odds of the detections being a fluke were 2 in 1000 – technically dubbed a three-sigma result, where sigma is a measure of the error on the measurement. But physicists have had their fingers burnt many times by three-sigma discoveries that turned out to be mirages. They even have a saying: “half of all three-sigma results are wrong”.

CERN required the odds of a fluke to fall to 3 in 10 million (known as a five-sigma result) before claiming a new discovery. In contrast, medical and social sciences usually require a certainty of only three or even two-sigma (corresponding to about 1 in 20). “Why do we stick to five sigma?” asks Bill Murray of the Rutherford Appleton Laboratory in the UK, who worked on one of the teams that spotted the Higgs candidates. “It’s because we don’t believe the statistics!”

According to news reports, CERN’s governing committee was split down the middle on whether to continue the experiment. “At the end of the day,” says Louis Lyons, a particle physicist at the University of Oxford, “people made the decision based on feelings in their stomach.” On 8 November 2000, the director-general of CERN announced that they were pulling the plug.

What if the decision had gone differently? Like so much else in the world of statistics, the answer is fraught with uncertainty. But we will probably find out sometime after 2007, when the LHC starts up and either confirms the Higgs sighting or relegates it to the long line of unexplained anomalies.

The five-sigma requirement is an extravagant way to rule out statistical error, the chance of mistaking a fluke for the real thing. But other kinds of error aren’t so easy to dismiss. One is the possibility of systematic error – some unanticipated effect that distorts the results of an experiment. Another is experimenter bias – the possibility that researchers select results, intentionally or otherwise, that confirm their own beliefs. Here, too, physicists may have something to learn from biologists. Drug trials have long used the practice of “blinding” – hiding or disguising results from researchers while the experiment is in progress. Physicists have only recently begun to adopt this, though it is now standard at the Stanford Linear Accelerator in California.

Finally, physicists may need to re-evaluate the brand of statistics that they use. A long-running debate among statisticians pits “Bayesians” against “frequentists”. Bayesians interpret a probability statement as a degree of belief – the odds that you would give on a statement being true. Frequentists, on the other hand, interpret any probability as the fraction of times you would observe the statement being true if you repeated an experiment many times.

Medical researchers have had great success with Bayesian methods, but physicists have resisted them for years. “Particle physicists tend to be more frequentist than almost any group of scientists around,” says Lyons.

Their reluctance to go Bayesian is due to a hidden assumption that lies behind any Bayesian analysis. To work out the odds of a theory being correct in light of experimental results, you have to provide a value for the prior probability – your belief in a theory being correct before performing the experiment (91av, 13 March, p 39). And if there is no hard evidence to go on, there is only one way to do this: with an educated guess.

For years, physicists have struggled and squirmed against this incursion of subjectivity into their supposedly objective science. When they do use Bayesian methods, they use the most unbiased prior odds possible. Robert Nichol, an astronomer at Carnegie Mellon University in Pittsburgh, Pennsylvania, thinks this approach is misguided, because it throws away the great advantage of the Bayesian approach – the chance to learn from earlier theory or experiments. “You spent a lot of money on that information. You’d be stupid not to use it,” Nichol says.

Robert Cousins at the University of California, Los Angeles, is an unabashed supporter of Bayesian methods in the right circumstances. The situation the CERN managers found themselves in fits the bill, he says. There was an element of subjectivity in the decision they faced: they had to weigh the extra cost of running LEP against the possibility that the Higgs signal was real. This subjectivity had to be folded into the data, Cousins says. “There’s no way to get away from that. It’s why we have people making decisions instead of robots.”

2 Blown apart

Did anyone think to ask biologists about the biggest mystery in physics?

In 2001, statisticians Chris Genovese and Larry Wasserman of Carnegie Mellon University happened to attend a seminar on the use of statistics in biology. What they heard struck them as a possible solution to a stubborn problem that an astronomer colleague had been working on. The result? Our most direct evidence for the existence of dark energy.

Genovese and Wasserman knew that their colleague Robert Nichol’s work on galaxies was proving troublesome. According to cosmological theory, the large-scale structure of today’s universe – how many clusters of galaxies there are, how big they are – was set by sound waves that travelled through the universe in its infancy. The universe is like a drumhead with its oscillations frozen into place, and by analysing the shape of the drumhead astronomers can tell how it sounded.

Roughly speaking, they should hear three notes: one that reflects the curvature of the universe, one that depends on the density of baryons – particles of ordinary matter – and one that depends on the density of missing or “dark” matter. Nichol was looking for a wiggle in a plot of the galaxy distribution corresponding to the second note, the baryon density. But there was a lot of static in the recording and Nichol was having trouble getting rid of it without throwing away the note as well. The Carnegie Mellon biostatistics seminar was, effectively, about getting rid of static while keeping the good stuff – an analysis known as the false discovery rate.

When statistics give erroneous results, sometimes it’s not because the statistical procedure has failed, but because it has worked too well, picking up false positives. This is especially true when several hypotheses are being tested at the same time. Suppose medical researchers test 100 possible drugs in parallel. Even if none of the drugs is truly effective, chance variations will mean that at least one is likely to appear to have a powerful effect – it will be a false positive.

This sends a clear warning to scientists: if you do multiple tests, chances are you’ll find more false positives than you bargained for. One solution is to raise the standard for a successful test. But this approach leads to an unacceptable rate of false negatives – drugs that may genuinely work but fail to pass the extra-stringent screening.

In 1995, statisticians Yoav Benjamini and Yosef Hochberg at Tel Aviv University in Israel came up with a less strict criterion that makes good sense for exploratory studies. First you rank the drugs according to their statistical significance – the odds that their effectiveness is real, not a fluke – with the strongest drug first. Then you begin pooling the top ones, and as long as their combined statistical significance remains high, you class them all as effective.

For example, in a test of 100 drugs, you would class the most statistically significant drug as effective if the chance of its effects being down to chance were less than, say, 1 in 10,000. You would then look at the top two drugs together, and judge both as effective if the odds of the second-best drug being a fluke are less than 2 in 10,000. This lowers the bar for both drugs, but ensures that their combined statistical significance is still high, since they must both be above this threshold. The process continues until all the drugs are judged, and you end up with a pool of top performers. Benjamini and Hochberg showed that this false discovery rate (FDR) method strictly controls the number of false positives, even in multiple test experiments.

In astronomy, the “tests” are pixels in a digital image of space. Each pixel represents a separate experiment: does it contain a source of light such as a galaxy, or not? So a single image represents millions of simultaneous experiments. False positives are bright pixels that should be dark – they are the static in the image.

Following Genovese and Wasserman’s advice, Nichol realised that the FDR method was the best way to eliminate most of the static without turning down the contrast so much that the whole image ended up black. He detected the “baryon wiggle” where others had failed.

Last year Nichol, Wasserman and Genovese were part of a team that announced an even more dramatic discovery based on FDR analysis: the most direct evidence yet for the existence of the dark energy believed to account for the accelerating expansion of the universe. They detected a phenomenon called the Sachs-Wolfe effect, which predicts that in a universe that is being blown apart by dark energy, light that grazes the edge of a galaxy will become a little bit hotter.

Ironically, the FDR method probably will not appear in the published paper detailing this work, for reasons that illustrate the precarious path of innovation. “There were 32 authors on that paper,” Nichol says. “I was very much pressing for FDR to be mentioned. The lead author, Ryan Scranton, a young, dynamic cosmologist at the University of Pittsburgh, loved it as well. Then we farmed it out to the other 30 authors, and we started getting the comments back. ‘What is this FDR? I don’t really understand it.’ In the end we won them over, and it stayed in the paper.”

But then it went to the journal Physical Review Letters. “The referee didn’t really get it, and didn’t want it. To my sadness, the FDR will probably be removed,” Nichol says. Instead, the paper will cite a conventional statistical method that corroborated the FDR results. “That shows we’ve got work to be done to get FDR fully mainstream,” Nichol concludes. Whatever physicists borrow from biology, it’s hard to convince dyed-in-the-wool sceptics. “I’m not too disheartened, though, because I at least won over 30 people.”

3 Dark matters

How confident are we in the make-up of our universe?

Particle physics experiments smash subatomic particles together for several years, gathering data on millions of collisions in the hope of finding a precious handful that demonstrate the existence of a new, exotic particle. Unfortunately, known particles can sometimes masquerade as unknown particles if they decay in just the right way. By using existing theories, physicists can calculate how many such impostors they expect to see.

Imagine finding 13 cases where the particle tracks look like those of the exotic particle you are interested in. If theory says that three of these patterns are likely to have come from impostors, then the experimenters are in good shape. They can be reasonably sure that about 10 collisions did indeed produce the exotic particle, though they will never know which 10.

To quantify what “reasonably sure” means, they use decades-old statistical theory to set lower and upper limits on the number of exotic particles they would expect to see if they repeated the experiment. These limits are known as a confidence interval. The researchers announce that the exotic particle does exist and use the theory to claim with 90 per cent confidence that a repeat experiment would churn out between 4 and 16 of them.

But what if only three collisions look like the exotic particle when you expect three impostors? Then the experimenters cannot prove that the particle exists, but they can’t be sure that it doesn’t. Erring on the side of caution, they assume they saw nothing. But they can use the same statistical theory to set another confidence limit – this time, an upper bound on the number of exotic particles that might show up if they repeated the experiment. Given the no-show in this first trial, the researchers can conclude that a repeat experiment would find fewer than eight exotic particles.

That might sound reasonable, but it didn’t sit well with Gary Feldman and Robert Cousins, particle physicists at Harvard University and the University of California, Los Angeles, respectively. Eight years ago, they became bothered by this ad hoc approach. They say that deciding after the fact whether to present the result as a successful detection or as a failure to spot anything might seem harmless, but it changes the statistical procedure – and thus invalidates the interpretation of the results. They even gave this method the derisive name “flip-flopping”.

Instead of experimenters changing their minds as they go along, Feldman and Cousins believed that a better solution would be to decide in advance whether to present results as successes or failures. And so they developed a “unified” method for navigating the grey zone between probable no-shows and probable discoveries. This involves using theory to decide the confidence interval for each possible result before the experiment begins.

But just before they published their technique, Feldman paid a visit to the Harvard statistics department. There he received a shock: statisticians had known about this unified technique for decades. Eventually Feldman found the idea had been discovered in 1937 by statistician Jerzy Neyman.

Cousins was delighted to have been scooped by six decades. The endorsement from statisticians, he believes, has smoothed the path to widespread use of the technique – now called the Feldman-Cousins method, although Cousins is uncomfortable getting credit for a technique someone else had invented earlier.

The rejuvenated technique was almost immediately engulfed in controversy, however. In 1998, an Anglo-German experiment called KARMEN reported its results on neutrino oscillations – where neutrinos change their mass as they travel. Other experiments have shown that neutrinos from the sun do alter their mass en route to Earth, but these neutrinos only oscillate after travelling thousands of kilometres. In contrast, KARMEN looked at neutrinos produced just tens of metres away, so theorists predicted the experiment would see nothing except the typical noise of about three impostors.

No one expected what they actually found: zilch. Not only had the oscillating neutrinos failed to show up, neither had the noise. Conventional statistics are a disaster for making sense of this. They would force the experimenters to conclude with 90 per cent confidence that they had observed fewer than zero neutrino oscillations. However, the Feldman-Cousins method avoids such nonsensical outcomes, so the KARMEN team used the technique to calculate the behaviour of neutrinos with meaningful confidence limits.

Unfortunately, the result contradicted a similar experiment at the Los Alamos National Laboratory in New Mexico called LSND (liquid scintillator neutrino detector), which claimed to have detected short-range neutrino oscillations. Some people blamed the Feldman-Cousins method for setting confidence limits that are too tight. And even today, no one knows which experiment is right.

Much is at stake here. If the LSND experiment turns out to be right, it will certainly upset a few apple carts. Neutrinos would be heavier than expected, and could thus become a major constituent of the missing dark matter in our universe. It would mean rethinking the history of the cosmos.

In the end, Feldman and Cousins’s nifty approach did not end the debate, but it greatly clarified just how badly the two experiments conflict with each other. Encouragingly, a more sophisticated version has identified a sliver of ground on which they might be able to agree. If physicists only knew the right confidence trick, it might change our view of the universe.