Statistical variance and global warming

The idea of global warming is a matter of meteorological record, not personal opinion. For the past quarter-century, the world's weather has been very different to the preceding half century, and this has been noted by weather bureaus around the globe.

For example, the town where I live, Uppsala, has one of the longest continuous weather records in the world, starting in 1722. The recording has been carried out by Uppsala University, and the averaged data are available from its Institutionen för geovetenskaper. This graph shows the variation in average yearly temperature during those recordings, as calculated by the Swedish weather bureau (SMHI — Sveriges meteorologiska och hydrologiska institut) — red represents an above-average year and blue below-average. As you can see, below-average years have not been recorded since 1988, which is the longest run of red on record.

The consequences of this weather change have been particularly noted in agriculture, because the growth of plants is very much dependent on sunshine and rainfall — a change in either characteristic will almost certainly lead to changes in harvest quantity and quality, as well as harvest timing. (See Climate change: field reports from leading winemakers. 2016. Journal of Wine Economics 11: 5-47.)

Grape harvests

Grape harvests have been of particular interest for economic reasons, given the importance of the wine industry to many countries. However, they have also been of interest because there are many long-term harvest records for Europe, and so the changing of the harvests in response to weather conditions over several centuries has been recorded, and can be studied. I have discussed this in a post on my other blog — Grape harvest dates as proxies for global warming; and this may be of interest to you, so check it out.

I conclude that we should be in no doubt that the recent change in the weather has had a big effect on wine production, and that we can reasonably expect that this will continue while ever the current weather patterns continue.

What seems to be more contentious, however, is assessing the causes of these weather patterns, and how the people of this planet might respond, if at all. For example, Steve Heimoff, over at the Fermentation blog, has recently discussed this issue (Inconvenient timing for a climate-change heretic).

One of the important issues here is the concept of statistical variance; and this is what I will discuss in the rest of this post.

Statistical variance

"Statistical variance" refers to the variation that occurs due to random processes and stochastic events. For example, we do not expect the average yearly temperature to be the same from year to year, which is what would happen if there was no statistical variance. Instead, we have observed that each year is somewhat different, with some years being above average and some below. In the graph above, the years varied from 8°C below the long-term average to 8°C above the long-term average — these numbers quantify the amount of statistical variance that has occurred in Uppsala's weather over the past three centuries.

We also do not expect regular patterns in the statistical variance. For instance, we should be very surprised if the years always alternated between above-average and below-average temperatures. Instead, we expect runs of several years above or below, without any necessary pattern to how long those runs will be. This is precisely what is shown in the graph above, where there are runs of anywhere from 1 to 9 consecutive years with similar weather.

Importance

Human beings need to understand this concept of statistical variance in order to work out whether anything unusual is happening around them. For example, a business person needs to work out whether a run of several months of poor economic performance is simply statistical variance, or whether it indicates that something has gone wrong (and needs to be corrected). Alternatively, a run of several months of good economic performance may also be simply statistical variance, and not at all an indication that the company is being well run!

This seems to be a very hard thing for people to grasp. Runs of events, whether good or bad, are often interpreted as being non-random; and runs of apparent bad luck can be very depressing, while runs of good luck can lead to over-confidence (which is well known to come before a fall).

This topic has been discussed in a number of books. One of the better known of these is Leonard Mlodinow's 2008 book The Drunkard's Walk: How Randomness Rules Our Lives. This has a detailed discussion of "the role of randomness in everyday events, and the cognitive biases that lead people to misinterpret random events and stochastic processes." His particular message is that people who don't grasp the idea of statistical variance can be led astray by randomness — a run of bad luck does not necessarily make you a failure, nor does a run of good luck necessarily make you a success. You can watch a video presentation by him on Youtube (thanks to Bob Henry for alerting me to this).

Why we should address statistical variance

Like Mlodinow, I am particularly interested in how people respond to statistical variation.

In this context I will mention a simple example, called the Gambler's Fallacy, which is very relevant. Gambler's often think that in a game of 50:50 chance they will break even in the long term, because they will eventually win and lose the same amounts of money. However, mathematicians have shown that, due to statistical variance, this can only be guaranteed in practice if the gambler has the resources to allow for infinite gains and infinite losses (ie. they can sustain an infinitely long winning or losing streak). Such long runs of wins and losses do not result from expertise or lack of it — they will happen anyway, just by random chance. However, in practice, the gambler will stop playing when their bankroll reaches zero (from too many consecutive losses) or when they bust the casino (from too many consecutive wins). So, there is no way to guarantee to break even in the long term — either you or the casino may go bust before that happens.

The importance of this example is that it emphasizes how we deal with statistical variance, in practice. For example, in practice it does not really matter whether current climate change is a permanent change (perhaps caused by modern industrial societies) or the result of statistical variance. It will affect us either way — metaphorically, it is just as possible for either we or the casino can go bust due to statistical variance, as going bust from any other possible causes. The practical question is: what are we going to do about it? We are sentient beings (that's what our scientific name Homo sapiens means), and we thus have the ability to recognize what is happening, and to potentially do something about it. We need to decide whether we want to do something, or not.

Several decades ago, when it was pointed out that there was a hole in the ozone layer, a possible cause was identified (use of CFCs), and a potential response was outlined (stop using CFCs, because there are alternatives). We decided to respond, globally; and the latest reports show that the hole is now shrinking. Maybe the increase and decrease in the hole are simply the result of statistical variance; but maybe we are actually smarter than the skeptics think. We keep records (we describe the world), we think about the patterns observed in those records (we explain the world), and we work out how we might respond (we try to forecast the future).

However, being sentient doesn't necessarily make us intelligent. Some people are skeptics because that is how they are built; others are skeptics because they have their own agenda (often to do with them making money, and lots of it), and to hell with everything else. Intelligence requires more than skepticism; and this applies to global warming as much as anything else.

So, the skeptics are right when they point out that rapid changes in long-term weather have occurred before in our recorded history; and I will discuss the data in a future post. However, this fact is irrelevant. Our response cannot be determined by these past patterns, because the current effects are occurring now, irrespective of whether they also occurred back then. Furthermore, even if any particular climate change is "natural" doesn't mean that we will be unaffected. We are going to look like complete fools if we (metaphorically) go bankrupt while attributing it to statistical variance. This is like leaping into a deep hole while yelling "look at me, I'm falling!"

Common ways to deal with statistical variance

By way of finishing, I will mention a couple of ways that people have developed to address the effects of statistical variance. You might like to think about whether any of these can be applied to global warming.

The basic idea is to reduce the statistical variance. That is, we try to prevent long runs of positive and negative changes from happening — we reduce the extent of both the upswings and the downswings. Sadly, there is no known way to reduce the negative changes (runs of bad luck) without also reducing the positive changes (runs of good luck).

In economics and gambling, one way to do this is by hedge bets. This involves investing most of our money in one way while simultaneously investing a smaller amount in the opposite way. So, we might bet most of our money on one particular team winning the game while also placing a smaller bet on the other team. This will reduce or losses if we have put most of our money on the losing team (because we will still win the smaller bet), although it will also reduce our winnings if we have put most of our money on the winning team (because we will still lose the smaller bet). So, hedge betting reduces our possible wins and losses — that is, it reduces the statistical variance. In economics, so-called Hedge Funds operate in precisely this manner; and they seem to be quite successful financially.

A somewhat different approach is taken in card games like poker. Professional online poker players usually play hands at multiple tables simultaneously. That is, they are placing multiple bets at the same time. Each table is potentially subject to wide statistical variance, but the average across all of the tables will usually have much less variance. Across any one betting session, the losses at one table will be counter-balanced by the wins at other tables, thus reducing the statistical variance for the poker player. This is an important component of being a professional in any field — the effect of random processes (good luck and bad luck) needs to be minimized.

It might strike you as a bit odd that I am talking about gambling in terms of dealing with statistical variance, but the same principles apply to all circumstances. In practice, a poker player betting at multiple tables is mathematically no different from an insurance company having lots of policy holders — you win some and you lose some, but you will reduce the extremes of winning and losing by being involved in multiple events. Most of our understanding of the mathematics of probability has come from studying both gambling and insurance.

Mind you, often the optimal strategy in poker is to go all-in with a good hand, which means that you will immediately either double your money or go bankrupt. This is not a recommended strategy when dealing with the world as a whole!