Uncharted Territory

March 26, 2013

Forecasting and Philosophy

I noted yesterday that:

“Weather forecasting (and climate prediction) is not just about computer power. Deep philosophical ideas also come into play.”

I fear I may have under-delivered on the philosophy.

I intended to suggest that all forecasts, such as of weather, are necessarily and systematically inaccurate.

To recap, my main point yesterday was that running an inaccurate forecasting model numerous times doesn’t solve all the inherent problems:

“All ensemble forecasters know is that a certain proportion of computer model runs produce a given outcome. This might help identify possible weather events, but doesn’t tell you real-world probabilities. If there is some factor that the computer model doesn’t take account of then running the computer model 50 times is in effect to make the same mistake 50 times.”

Let me elaborate.

We can dismiss the normal explanation for forecasting difficulties.  Forecasters normally plead “chaos”.  Perfect forecasts are impossible, they say, because the flap of a butterfly’s wings can cause a hurricane.  Small changes in initial conditions can have dramatic consequences.

I don’t accept this excuse for minute.

It may well be the case that computer models suffer badly from the chaos problem.  In fact, the ensemble modelling approach relies on it.  I suspect the real world is much less susceptible.  Besides, given enough computer power I could model the butterfly down to the last molecule and predict its wing-flapping in precise detail.

No, the real-world is determined. That is, there is only one possible outcome.  Given enough information and processing power you could, in principle, predict the future with complete accuracy.

Of course, there are insurmountable practical problems that prevent perfect forecasting:

  • The most fundamental difficulty is that no computer can exceed the computing capabilities of the universe itself.  Although the future is written, it is in principle impossible to read it.
  • You might try to get round the computing capacity problem by taking part of the universe as a closed system and building a huge computer to model what is going on in that relatively small part.  The difficulty then is that the entire universe is interconnected.  Every part of it is open, not closed.  If the small part you were modelling were the Earth, say, then you’d have to also model all celestial events, not just those which might have a physical effect, but all those which might be detectable by humans and therefore able to affect thought-processes and decision-making.  And, since our telescopes can see galaxies billions of light-years away, there’s a lot to include in your model.  That’s not all, though.  You’d also need to model every cosmic ray that might disrupt a molecule, most dramatically of germ-line DNA – though a change to any molecule is of consequence – and even every photon that might warm a specific electron, contribute to photosynthesis or allow a creature to see just that bit better when hunting…
  • Then there are problems of what George Soros terms reflexivity.  That is, people’s behaviour is modified by knowing the predicted future.  They might act to deliberately avoid the modelled outcome, for example by deflecting an asteroid away from its path towards the Earth, which we might term strong reflexivity.  Or they might change their behaviour in a way that unintentionally affects the future, for example by cancelling travel plans in light of a weather forecast – weak reflexivity.  With enough computer power, some such problems could conceivably be overcome.  One might predict the response to an inbound asteroid, for example.  But it’s not immediately apparent how a model would handle the infinitely recursive nature of the general problem.

In practice, of course, these would be nice problems to have, because computer simulations of the weather system are grossly simplified.   They must therefore be systematically biased in their forecasting of any phenomena that rely on the complexity absent from the models.  As I noted yesterday, all runs in an ensemble forecast will suffer from any underlying bias in the model.

Two categories of simplification are problematic:

  • Models divide the real-world into chunks, for example layers of the atmosphere (or of the ocean).
  • And models necessarily represent closed systems – since the only truly closed system is the universe as a whole.  Anything not included in the model can affect the forecast.  For example, volcanic eruptions will invalidate any sufficiently long-term (and on occasion short-term) weather forecast.  Worse, weather models may be atmosphere only or include only a crude simplification of the oceans.  That is, they may represent the oceans in insufficient detail, and furthermore fail to include the effect of the forecast on the oceans, which in turn affects the forecast later on.

The good news, of course, is that it is possible to improve our weather forecasting almost indefinitely.

Perhaps those presenting weather forecasts should reflect on the fact that, as computer models improve, ensemble forecast ranges will narrow.  The 5-day forecast today is as good as the 3-day forecast was a decade or two ago. The probability of specific real-world conditions will not have changed.  That has always been and always will be precisely one: certainty.

It makes no sense to say “the probability of snow over Easter is x%”, when x depends merely on how big a computer you are using.

No, forecasters need to say instead that “x% of our forecasts predict snow over Easter”, which is not the same thing at all.


July 31, 2011

Uncertain about Risk and Uncertainty

Filed under: Complex decisions, Global warming, Philosophy of science, Reflections, Science — Tim Joslin @ 6:57 pm

Here’s an interesting – nay, potentially iconic – figure from a paper, Greenhouse-gas emission targets for limiting global warming to 2C, Meinshausen et al, Nature vol.458, p.1158, 30th April 2009:

It represents the results of a heroic assemblage of climate modelling data. The horizontal axis gives the emissions from 2000-49 (belying the paper’s title, incidentally) in GtCO2 for each set of data plotted and the vertical axis – careful wording alert – the probability of the given model output set predicting a global mean temperature increase of 2C or greater over pre-industrial levels before 2100. The dots and swathe of colour are the outputs of all this modelling; the solid black line some kind of best fit (I know not how determined) or “illustrative default”; the dotted line incidentally is the outcome based on a set of models including carbon cycle feedbacks, which implies less likelihood (carefully chosen word) of hitting 2C for given fossil fuel and land-use change emissions.

The good news is that if this modelling exercise represented a set of real world possibilities, we could, for example, emit around 1500GtC02 and still have a 50% chance of avoiding “dangerous climate change” defined as the 2C temperature increase. The bad news is that the grey area bottom left of the figure represents the 234GtC02 of actual emissions from 2000-06 (7 rather than 6 years, I believe, though the paper is scandalously ambiguous on this point) – basically we have to slow way down.

You could probably write a dissertation about the diagram – for example, why are we discussing scenarios for the future such as the IPCC’s A1F1 (top right) which apparently emit more carbon before 2050 than is stored in the world’s fossil fuel reserves (and even more in the second half of the century)? In fact, why are we discussing more than one of the IPCC scenarios, since even the best of them, B1, is unlikely to lead to less than 2C warming? There is surely no longer any need to nuance the basic point that unmitigated emissions will lead to dangerous climate change.

But all I want to cover just now is essentially one word. The word “probability” on the vertical axis.

What we have here are not probabilities in any real-world sense. There will only be one outcome. We will be at one position on the horizontal axis, depending on (in this case) our emissions before 2050 (though different pathways may give different outcomes for the same 2000-49 emissions). The distribution about the illustrative default (a vertical slice through the diagram) is an indication of our state of knowledge (as encapsulated by the models used) as to what will happen to the global mean temperature as a result. It is not, in a strict sense, a probability.

Using the term probability in this context is unfortunate. In fact, I might even go so far as to say it is symptomatic of a pathology: the same pathology evident when the idea of carrying out climate “experiments” is discussed. Guys, we are using models, not instances of the real world. If parallel universes do exist, we do not have access to them.

My personal issue with all this is that when I read the word “probability” I assume we’re talking about risk, when in fact the topic is uncertainty. I simply can’t help it.

It is extremely unfortunate that the climate modelling community (and perhaps a wider group) has chosen to use the word “probability” to refer to uncertainty as well as risk when they could simply have used “likelihood”. This has been extended to probability distribution functions (PDFs) which are often nothing of the sort (I say “often”, because, confusingly, the horizontal axis may be a probability function and the vertical axis what I would call a likelihood function). These figures should be renamed likelihood distribution functions, with the added advantage of a less overused TLA, LDF rather than PDF.

Apart from confusing models with the real world, the inappropriate use of the term “probability” has caused another problem: too much weight is being given to quantifiable knowledge (included in models) and too little to that which has not been quantified. The term “likelihood” would instead force people to focus on the right questions. What’s happened is that the climate science community is pretending it can somehow answer epistemological questions – those about the state of knowledge – in a scientific, quantifiable way. Presenting the information in precise terms – “my belief in the actual increase in the global mean temperature we can expect for 1500GtC02 emissions is represented by this graph” – doesn’t alter the fact that all we are discussing is the state of our knowledge. And when we let machines produce the graph we automatically lose anything not input to the calculation.

I thought I’d try to clarify the problem with a simple analogy.

If I toss a coin and call “heads” the probabilityrisk – of losing is approximately 50%. I can make decisions on this basis.

But I said “approximately”, because there is some uncertainty – the coin may have a bias.

Now, any estimate of uncertainty in this case depends entirely on my state of knowledge. For example, I might have tested the coin – suspecting it might be weighted – before the crucial coin toss and found that of hundreds of attempts 55% came up heads. This (and my knowledge of statistical confidence testing) may then lead me to believe the risk of losing to be not 50% but 45%. But this would be exceptional. For a randomly chosen coin I would normally be prepared to say it’s likely – an expression of certainty – that the risk of losing when calling heads is, for all practical purposes, 50%.

If asked to quantify how sure I am that the risk of losing is 50%, I might say 99.99%, because I believe the vast majority of the coins in circulation are equally likely to come down heads as tails. But this 99.99% is what I would term a likelihood – a judgement of the uncertainty – not a probability.

On the other hand I may be asked to pick a coin from a bag I’m told contains 50 normal coins and 50 weighted so as to come down heads only 25% of the time. Now it becomes a probability as to how likely it is that the coin is true. I can even calculate the overall probability of tossing a head (37.5%). There may still be uncertainty, though – whoever told me 50% of the coins are weighted might have been lying.

What I hope this analogy conveys is the importance of being clear about what we know and what we don’t know. We should only talk about risk and probabilities within a defined theoretical framework. When making judgements as to the state of our knowledge we should be discussing uncertainty and (I suggest) likelihood.

Let’s turn back to the iconic figure from Meinshausen et al and that vertical axis labelled “probability”.

Are we really talking about probability or likelihood?

Remember where the numbers come from – a series of computer model runs. Here’s a philosophical point: no scientific model is the same as the real world. A simple scientific law such as F=ma or E=mc2 may make very good predictions, but has different characteristics to the real world. And these simple models give the same answer every time (or rather, are not sensitive to small variations in the initial conditions). When we use complex computer-based models such as of climate, we find that quite different answers result from multiple runs with minimally perturbed initial conditions.

What we don’t know is how much of the variation in model runs depends on a lack of determinism of the real world and how much is a result of the characteristics of the models. Remember the real world is air, water, sunlight, clouds and so on. The model is numbers representing big chunks of real stuff in a computer. Maybe there is a “butterfly effect” and the magnitude of global warming in a century’s time will depend on minor unpredictable events that happen today. Somehow I doubt it – weather may be affected by small events, but not climate – though the issue could be discussed long into the night.

What is less disputable, though, is that we don’t know how much of the variation in model predictions of global warming is due to different real world possibilities and how much is due to the characteristics of the models.

By failing to distinguish probability from what I term likelihood, and labelling their vertical axis as “probability”, papers such as Meinshausen et al are perhaps implicitly asserting that what is represented is entirely due to real world variability.

It seems to me far wiser to do the opposite. We should assume that the variability of model outputs represents uncertainty. We are best off considering that if the models were perfect they’d give the same answer every time.

But Meinshausen et al might argue that all they’ve done is use the word “probability” loosely. They might agree with my argument. Now, though, we have another problem. We are giving much more weight to model variability than to other forms of uncertainty. The models only capture some of the known unknowns. There are known real-world phenomena that are not included in the models – poorly understood phenomena, for example, carbon-cycle feedbacks, such as methane release from tundra. And then there are unknown unknowns.

Here’s my biggest problem – if we are to present some forms of uncertainty in numeric form then surely we are obliged to present all forms of uncertainty in the same way. It is misleading to simply list the things we haven’t taken account of. We have to make judgements about the known unknowns and unknown unknowns and adjust the uncertainty distribution accordingly. It’s not all bad news: we might argue that the models overstate the uncertainty in outcome and narrow the distribution in figures such as that by Meinshausen et al.

Policymakers need a considered view of the state of climate knowledge, not diagrams that present dubious “probabilities” and a set of provisos.

Create a free website or blog at WordPress.com.