91av

WE’VE all been there. Everything’s set for a great barbecue but when the day
arrives the weather forecast turns out to be as reliable as a mafia tax
return—and down comes the rain.

Still, what can you expect? We all know weather forecasts aren’t 100 per cent
reliable and we all know why. It’s the butterfly effect, isn’t it? That
notorious spin-off of chaos theory which says that the way the weather unfolds
is so sensitive to tiny fluctuations that a butterfly flapping its wings in
Kansas could trigger a typhoon in Singapore—or a downpour on your summer
party. With the weather being that sensitive to tiny changes, what hope have
forecasters got?

But now a small team of researchers is challenging this idea. David Orrell, a
mathematician at University College London, says it’s time to break the
butterfly’s spell and rouse meteorologists to a bright, new truth: weather
forecasts can be far, far better than they are right now. In fact, Orrell is
confident that we should be able to make near-perfect forecasts for at least 3
days ahead—perhaps even for more than a week. At the moment, forecasters
struggle to predict much beyond 24 hours with any accuracy.

No one is suggesting that chaos theory is wrong, or that forecasting is a
doddle. Just that the butterfly effect is not where the trouble lies. For
short-term forecasts—even up to a week—the butterfly effect hardly
figures, say Orrell and his colleagues. The real problem, they say, is errors in
the meteorologist’s own mathematical models—the vast array of equations
they use to capture the physics behind the weather. It’s these errors that
dominate the picture and ruin the forecasts.

It’s a brave assertion, and it won’t go down well with many forecasters. Some
cynics suggest that the butterfly effect has served as a fig leaf for
meteorologists, conveniently hiding the embarrassing flaws in their models. For
everyone else, though, it’s wonderful news—because, unlike the butterfly
effect, a faulty mathematical model is something you can fix.

With help from Orrell’s new mathematical analysis, meteorologists are already
starting to pin down exactly where the problems lie and are devising clever ways
to fix them. Soon we may finally see the accurate predictions we’ve always
longed for—a truly reliable forecast that farmers, pilots, sailors and
even the financial markets can trust. Something you can even stake your barbecue
on.

To most people, the notion that mathematical models are to blame for dodgy
forecasts may seem less than revolutionary, especially when you consider the
task that meteorologists face. To create any kind of forecast, they have to
build hugely complex models of the atmosphere, crammed with equations describing
the way the whole thing evolves. They plug in the starting values—the
current weather conditions—and set the model running to calculate what the
next few days or weeks will bring.

Meteorologists would certainly be the last to claim that their models are
perfect. Even with today’s supercomputers, they admit they’re nowhere near
nailing every detail of the 5 million billion tonnes of air swirling over our
heads. But what they have come to believe is that any errors arising from their
equations are tiny compared to the humungous inaccuracies thrown up by the
butterfly effect. Chaos, they believe, always amplifies small errors in the data
into huge ones, rapidly making the forecasts wildly inaccurate.

Forecasters haven’t always thought that way. During the 1960s and 1970s,
people assumed that it was the models that were the source of forecast errors.
But then came Ed Lorenz, a meteorologist at the Massachusetts Institute of
Technology. He was the first to highlight the role of chaos in weather
forecasting—and coined the term “butterfly effect”. “Chaos theory caused a
major change in thinking,” says Orrell. “People then somehow started to think
that this was the major source of the trouble.”

It fitted right in with the zeitgeist. During the 1980s, newspapers and
magazines began carrying stories on chaos theory. James Gleick’s Chaos:
Making a new science was riding high in the best-sellers lists. And
according to Britain’s Meteorological Office, it was the butterfly effect that
was ultimately responsible for its failure to predict the Great Storm of 1987,
which struck southern England with winds of over 150 kilometres per hour.

Since chaos hit the scene, the vast majority of meteorology research has gone
into trying to gauge its impact on weather forecasts. It has led to the
emergence of the “ensemble technique” for predicting the weather, in which a
computer model generates families of 50 or more forecasts, each starting with
slightly different figures for temperature, pressure and the like. How strongly
these forecasts diverge from each other over time gives scientists a handle on
the current strength of the butterfly effect. It tells them how much confidence
they should place in the computer’s predictions, and what direction the weather
is most likely to take.

Oddly, however, the uncertainty estimate culled from ensemble techniques
still frequently fails to include what actually happens, even on three-day
forecasts. A day of sun and scattered showers, for example, might turn out to be
just a steady downpour instead. It’s as if there’s something else throwing the
forecasts off-track.

According to Orrell, that something is not chaos, it’s the model itself. And
he presents his evidence in a forthcoming paper in Nonlinear Processes in
Geophysics, co-authored by Lenny Smith of Oxford University and Tim Palmer
and Jan Barkmeijer, two meteorologists from the European Centre for Medium Range
Weather Forecasts near Reading.

The paper is the result of three years of research carried out by Orrell and
Smith at Oxford. Together, they developed a technique to reveal exactly where
the errors in the forecasts creep in. There were two main hurdles, says Orrell.
First, you have to separate the total error in the forecast into its constituent
parts—that due to the model itself, and that caused by uncertainty in the
starting conditions plugged into the model. “Second, you have to show how the
model error evolves with time, so that you can measure its relative importance,”
he says.

The difference between model error and starting-condition error—which
is what the butterfly effect feeds on—reveals itself in a phenomenon
called “shadowing”. Imagine that meteorologists have a perfect model of the
weather, capturing its every nuance, and they use it to produce a family of
forecasts. As these forecasts fall prey to the butterfly effect, their
predictions would diverge from the real weather in a very distinctive way,
spreading out but with the cloud of uncertainty still clustering around the
actual weather. In other words, the model’s family of forecasts would “shadow”
the real weather conditions in the short term.

In reality, of course, even the best models have some flaws. And this, Orrell
and Smith found, makes forecast errors behave in a fundamentally different way.
Although the errors grow under the influence of the butterfly effect, the family
of forecasts no longer clusters around the true weather. The flaws in the model
mean the “shadow” drifts away from its expected position. Before long, there is
no overlap at all between what the model predicts and the real weather.

Orrell and Smith quickly realised that patterns of shadowing could test their
suspicion that model errors rival—if not surpass—the butterfly
effect. Orrell went further, and proved a mathematical theorem that shows
precisely how errors due to the model itself grow with time. This amounts to a
fingerprint that tells you when the model, rather than the butterfly, is to
blame for inaccurate forecasts.

As any textbook on chaos theory will tell you, the errors due to the
butterfly effect grow exponentially with time. Orrell showed that errors caused
by the model follow a completely different law: they will increase with the
square root of time—growing relatively rapidly at first, before slowing
down to a much more sedate pace after a day or so.

As part of a long-standing collaboration, Orrell and Smith teamed up with
Palmer and Barkmeijer to look for the telltale sign of drift in forecasts
produced by the ECMWF Operational Model—widely regarded as one of the best
weather models in the world. If model error was dominating, it would betray
itself through forecasting errors that grow very rapidly early on before
levelling off. Then the error curve should start to bend upwards again as the
exponentially increasing butterfly effect begins to dominate.

When the team plotted its forecast errors against time, the curve followed
Orrell’s square-root law with amazing accuracy
(see Diagram): “It certainly
didn’t look anything like the exponential curve expected for the butterfly
effect,” says Orrell.

Further analysis revealed a remarkable fact. Around 90 per cent of all the
forecast errors for the first 72 hours could be pinned on flaws in the model
rather than the butterfly effect: “It seems that the model errors account for
the large majority of the forecast error even as far out as five days.”

The team was taken aback by the speed with which the flaws in the Operational
Model drove it off-target. “We estimate that its forecasts shadow reality for
only three to four hours,” says Orrell. In other words, the flaws in the model
start to have an effect almost immediately. This, Orrell says, has important
implications for the ensemble techniques now widely used by meteorological
services. “The assumption is that the basic model is pretty much perfect, so
only the initial-condition errors are important,” says Orrell. “Our results
suggest this isn’t the case. That would at least explain why these ensemble
methods seem to over-estimate the reliability of the final forecasts.”

Their conclusion, Orrell believes, may be extremely unwelcome in some
quarters of the meteorological community, but everyone else should be pleased.
Despite what chaos theorists have been telling us for years, the butterfly
effect needn’t crush all hopes of major improvements in forecasting—both
in reliability and range: “The weather may be chaotic,” says Orrell, “but it’s
not that chaotic.”

Over timescales of at least several days, flaws in the model are by far the
bigger source of forecasting error. This means that fixing them could bring big
payoffs. Get the models right and, according to Orrell, you could see almost
perfect three-day forecasts. It should even push the limits of reasonably
reliable forecasting well beyond the current limit of around 10 days or so. “If
model error introduces large errors over the first few days, then the butterfly
effect will amplify those errors at longer times,” says Orrell. “So improving
the model will help nip those errors in the bud too.”

It’s a message that is already going down well with those with a big stake in
getting forecasts right such as Joseph Hrgovcic, head of research into
weather-risk management at Houston-based commodity broker Enron. “The paper is
brilliantly done,” he says. “The implications of his results are wide-ranging.”

But sorting out the problems Orrell has uncovered is a daunting task. “It is
not easy to deal with model error,” says meteorologist Peter Houtekamer of
Canada’s environment ministry. “After all, we do not know what’s wrong with the
model.” And ensemble forecasting remains crucial in order to see the effects of
uncertainty, says Smith, “even if it is blind to the effects of model error.”

Palmer insists that the ECMWF has been tackling model error for some time,
but he is adamant that getting more accurate data to feed into the models is
just as important. A 10-year international research programme entitled THORPEX
should help achieve that. This will use devices such as robot planes to gather
up-to-the-minute data from inaccessible regions of the globe.

Finding which of the possible model errors are most significant will be far
harder. Many atmospheric interactions are difficult to model—like the
way the surface of the Earth creates turbulence in the atmosphere. Complex
things like cloud formation and dissipation are also vital, says Orrell. “But
they are one of the hardest to model.” Here NASA’s GIFTS satellite could help.
Due for launch in 2004, it should provide high-quality atmospheric measurements.
“My own bet,” says Orrell, “is that errors will be biggest at low altitudes,
where there is plenty of data but the equations are hard to get right.”

Meteorologists must also rely on their equations to account for what the
weather is doing at scales much smaller than 40 kilometres. One solution, Palmer
suggests, may be to include a random-noise component in the equations to help
offset this.

David Stensrud from the National Severe Storms Lab in Norman, Oklahoma, has
already begun to use Orrell’s technique. He has found that errors seem to grow
fastest in the lower atmosphere, where turbulence is significant, just as Orrell
predicts. Stensrud now plans to configure a number of different models and use
the same techniques to pick out the versions that perform best.

Yet not all meteorologists are ready to wake from the butterfly’s spell. When
Orrell and his colleagues present their work at conferences, some are reluctant
to accept that there’s scope for dramatic improvements in forecasting. “There’s
still something of a consensus that model error is small, and it has proved hard
to convince some meteorologists otherwise,” says Orrell.

Perhaps the community will slowly come around to the idea. Houtekamer already
sees signs that the butterfly is loosening its grip. “I think we’re all now
discovering that model error is what limits the quality of our forecasts
and that we have to address it in order to make further progress.”

Orrell hopes that his findings will reinvigorate a field whose veterans put
up with endless criticism from a public that barely comprehends the scale of the
problem they tackle. “Modelling the Earth’s atmosphere has to be one of the most
difficult computational tasks ever attempted,” he says. So having flawed models
is hardly something to be surprised at—or ashamed of.