Monday, 2 October 2017

Statistical variance and global warming

The idea of global warming is a matter of meteorological record, not personal opinion. For the past quarter-century, the world's weather has been very different to the preceding half century, and this has been noted by weather bureaus around the globe.

For example, the town where I live, Uppsala, has one of the longest continuous weather records in the world, starting in 1722. The recording has been carried out by Uppsala University, and the averaged data are available from its Institutionen för geovetenskaper. This graph shows the variation in average yearly temperature during those recordings, as calculated by the Swedish weather bureau (SMHI — Sveriges meteorologiska och hydrologiska institut) — red represents an above-average year and blue below-average. As you can see, below-average years have not been recorded since 1988, which is the longest run of red on record.

Uppsala temperatures over three centuries

The consequences of this weather change have been particularly noted in agriculture, because the growth of plants is very much dependent on sunshine and rainfall — a change in either characteristic will almost certainly lead to changes in harvest quantity and quality, as well as harvest timing.

Grape harvests

Grape harvests have been of particular interest for economic reasons, given the importance of the wine industry to many countries. However, they have also been of interest because there are many long-term harvest records for Europe, and so the changing of the harvests in response to weather conditions over several centuries has been recorded, and can be studied. I have discussed this in a post on my other blog — Grape harvest dates as proxies for global warming; and this may be of interest to you, so check it out.

I conclude that we should be in no doubt that the recent change in the weather has had a big effect on wine production, and that we can reasonably expect that this will continue while ever the current weather patterns continue.

What seems to be more contentious, however, is assessing the causes of these weather patterns, and how the people of this planet might respond, if at all. For example, Steve Heimoff, over at the Fermenattion blog, has recently discussed this issue (Inconvenient timing for a climate-change heretic).

One of the important issues here is the concept of statistical variance; and this is what I will discuss in the rest of this post.

Statistical variance

"Statistical variance" refers to the variation that occurs due to random processes and stochastic events. For example, we do not expect the average yearly temperature to be the same from year to year, which is what would happen if there was no statistical variance. Instead, we have observed that each year is somewhat different, with some years being above average and some below. In the graph above, the years varied from 8°C below the long-term average to 8°C above the long-term average — these numbers quantify the amount of statistical variance that has occurred in Uppsala's weather over the past three centuries.

We also do not expect regular patterns in the statistical variance. For instance, we should be very surprised if the years always alternated between above-average and below-average temperatures. Instead, we expect runs of several years above or below, without any necessary pattern to how long those runs will be. This is precisely what is shown in the graph above, where there are runs of anywhere from 1 to 9 consecutive years with similar weather.


Human beings need to understand this concept of statistical variance in order to work out whether anything unusual is happening around them. For example, a business person needs to work out whether a run of several months of poor economic performance is simply statistical variance, or whether it indicates that something has gone wrong (and needs to be corrected). Alternatively, a run of several months of good economic performance may also be simply statistical variance, and not at all an indication that the company is being well run!

This seems to be a very hard thing for people to grasp. Runs of events, whether good or bad, are often interpreted as being non-random; and runs of apparent bad luck can be very depressing, while runs of good luck can lead to over-confidence (which is well known to come before a fall).

This topic has been discussed in a number of books. One of the better known of these is Leonard Mlodinow's 2008 book The Drunkard's Walk: How Randomness Rules Our Lives. This has a detailed discussion of "the role of randomness in everyday events, and the cognitive biases that lead people to misinterpret random events and stochastic processes." His particular message is that people who don't grasp the idea of statistical variance can be led astray by randomness — a run of bad luck does not necessarily make you a failure, nor does a run of good luck necessarily make you a success. You can watch a video presentation by him on Youtube (thanks to Bob Henry for alerting me to this).

Why we should address statistical variance

Like Mlodinow, I am particularly interested in how people respond to statistical variation.

In this context I will mention a simple example, called the Gambler's Fallacy, which is very relevant. Gambler's often think that in a game of 50:50 chance they will break even in the long term, because they will eventually win and lose the same amounts of money. However, mathematicians have shown that, due to statistical variance, this can only be guaranteed in practice if the gambler has the resources to allow for infinite gains and infinite losses (ie. they can sustain an infinitely long winning or losing streak). Such long runs of wins and losses do not result from expertise or lack of it — they will happen anyway, just by random chance. However, in practice, the gambler will stop playing when their bankroll reaches zero (from too many consecutive losses) or when they bust the casino (from too many consecutive wins). So, there is no way to guarantee to break even in the long term — either you or the casino may go bust before that happens.

The importance of this example is that it emphasizes how we deal with statistical variance, in practice. For example, in practice it does not really matter whether current climate change is a permanent change (perhaps caused by modern industrial societies) or the result of statistical variance. It will affect us either way — metaphorically, it is just as possible for either we or the casino can go bust due to statistical variance, as going bust from any other possible causes. The practical question is: what are we going to do about it? We are sentient beings (that's what our scientific name Homo sapiens means), and we thus have the ability to recognize what is happening, and to potentially do something about it. We need to decide whether we want to do something, or not.

Several decades ago, when it was pointed out that there was a hole in the ozone layer, a possible cause was identified (use of CFCs), and a potential response was outlined (stop using CFCs, because there are alternatives). We decided to respond, globally; and the latest reports show that the hole is now shrinking. Maybe the increase and decrease in the hole are simply the result of statistical variance; but maybe we are actually smarter than the skeptics think. We keep records (we describe the world), we think about the patterns observed in those records (we explain the world), and we work out how we might respond (we try to forecast the future).

However, being sentient doesn't necessarily make us intelligent. Some people are skeptics because that is how they are built; others are skeptics because they have their own agenda (often to do with them making money, and lots of it), and to hell with everything else. Intelligence requires more than skepticism; and this applies to global warming as much as anything else.

So, the skeptics are right when they point out that rapid changes in long-term weather have occurred before in our recorded history; and I will discuss the data in a future post. However, this fact is irrelevant. Our response cannot be determined by these past patterns, because the current effects are occurring now, irrespective of whether they also occurred back then. Furthermore, even if any particular climate change is "natural" doesn't mean that we will be unaffected. We are going to look like complete fools if we (metaphorically) go bankrupt while attributing it to statistical variance. This is like leaping into a deep hole while yelling "look at me, I'm falling!"

Common ways to deal with statistical variance

By way of finishing, I will mention a couple of ways that people have developed to address the effects of statistical variance. You might like to think about whether any of these can be applied to global warming.

The basic idea is to reduce the statistical variance. That is, we try to prevent long runs of positive and negative changes from happening — we reduce the extent of both the upswings and the downswings. Sadly, there is no known way to reduce the negative changes (runs of bad luck) without also reducing the positive changes (runs of good luck).

In economics and gambling, one way to do this is by hedge bets. This involves investing most of our money in one way while simultaneously investing a smaller amount in the opposite way. So, we might bet most of our money on one particular team winning the game while also placing a smaller bet on the other team. This will reduce or losses if we have put most of our money on the losing team (because we will still win the smaller bet), although it will also reduce our winnings if we have put most of our money on the winning team (because we will still lose the smaller bet). So, hedge betting reduces our possible wins and losses — that is, it reduces the statistical variance. In economics, so-called Hedge Funds operate in precisely this manner; and they seem to be quite successful financially.

A somewhat different approach is taken in card games like poker. Professional online poker players usually play hands at multiple tables simultaneously. That is, they are placing multiple bets at the same time. Each table is potentially subject to wide statistical variance, but the average across all of the tables will usually have much less variance. Across any one betting session, the losses at one table will be counter-balanced by the wins at other tables, thus reducing the statistical variance for the poker player. This is an important component of being a professional in any field — the effect of random processes (good luck and bad luck) needs to be minimized.

It might strike you as a bit odd that I am talking about gambling in terms of dealing with statistical variance, but the same principles apply to all circumstances. In practice, a poker player betting at multiple tables is mathematically no different from an insurance company having lots of policy holders — you win some and you lose some, but you will reduce the extremes of winning and losing by being involved in multiple events. Most of our understanding of the mathematics of probability has come from studying both gambling and insurance.

Mind you, often the optimal strategy in poker is to go all-in with a good hand, which means that you will immediately either double your money or go bankrupt. This is not a recommended strategy when dealing with the world as a whole!


  1. Chapter 7 of Leonard Mlodinow's book "The Drunkard's Walk: How Randomness Rules Our Lives" discusses (and I quote from the table of contents) "the bell curve and wine ratings."

    1. It seems very unlikely that wine ratings ever follow a bell curve, as this curve assumes that variation results solely from stochastic processes acting on a mean value. However, as shown in some of my previous posts, there are distinct biases in wine scores. Stochastic variation is important, but so are biases.

  2. Excerpt from Stanford University News
    (June 30, 2011):

    "Global warming could significantly alter the U.S. premium wine industry within 30 years, say Stanford [University] scientists"

    By Sascha Zubryd

    "Higher temperatures could hurt California and other premium winegrowing regions of the United States in the next 30 years, according to a new study led by Stanford University climate scientists.

    "Writing in the June 30 [2011] edition of 'Environmental Research Letters,' the scientists report that by 2040, the amount of land suitable for cultivating premium wine grapes in high-value areas of Northern California could shrink by 50 percent because of global warming. . . ."

  3. Excerpt from Napa Valley Register
    (May 4, 2013):

    "Napa wine industry warned of future climate threat;
    Local growers confident of ability to adapt"

    By Howard Yune

    "A study published last month in the journal 'Proceedings of the National Academy of Sciences' forecasts temperature increases triggering the loss of two-thirds or more of the Napa Valley’s current grape output by 2050, with similar losses projected in France and other prime winemaking regions.

    "California’s territory suitable for wine grapes is predicted to shrink by about 70 percent by midcentury, with an even steeper 85 percent loss forecast for France, Italy and the rest of Mediterranean Europe.

    “ 'What the report says is that using current grape varieties and current techniques, those areas would become not very good for producing wine,' said Lee Hannah, the report’s lead author and a senior research fellow for Conservation International."

  4. See this TED talk:

    "The Connection Between Climate Change and Wine"

    By Gregory Jones, Ph.D.
    [Director of the Center for Wine Education and is Professor of Environmental Studies at Linfield College in McMinnville, Oregon]