Wine scores and Top 100 lists

I have noted before that Quality scores changed the wine industry, and created confusion. The fundamental characteristic of quality scores is that they are an opinion, and yet they often get treated as a piece of mathematics. Until this issue is resolved (which it may never be), confusion will continue (along with lots of discussion), which cannot be good for the wine industry; or, indeed, any other industry that asks people to score their customer experience (movies, accommodation, etc).

In normal human language, opinions are usually expressed using some sort of adjectives (good, acidic, watery, sweet, etc), but wine scores express the opinion as a number. In one sense, this can help make the opinion clear (a 95-point wine is claimed to be better, in some sense, than a 90-point wine). However, contradicting this is the tendency to treat a number as having a strict mathematical property. This makes little obvious sense, because opinions, and adjectives, are not mathematical.

In a blog post (The fundamental problem with wine scores) I noted this issue explicitly: the single wine-quality number is trying to do too many things all at once (formally: wine scores represent multidimensional properties that have been summarized as a single point in one dimension). With words, we can describe anything we want to, because we can use as many words as we need; but a wine-quality score is one number only.

This issue becomes obvious when we try to compare wine-quality scores across different wine types or terroirs, and, especially, between different tasters (see The sources of wine quality-score variation). The basic issue is that we cannot tell what any numerical similarity or difference of scores actually means (Why comparing wine-quality scores might make no sense). In particular, it is not obvious what calculating an average of quality scores might actually mean, although some well-known web sites do this regularly; and it was also done for the infamous Judgment of Paris (see: A mathematical analysis of the Judgment of Paris).

We would all, I presume, prefer it if any one commentator, or set of commentators from a single publication, is consistent. If this is so, then we could at least search for value-for-money wines by comparing scores to prices (The relationship of price to wine-quality scores). This is possibly the most valuable use of a wine score!

However, one thing that still needs addressing is the combining of wine-quality scores to produce a "best of" list. The fact that the scores are numbers does not mean that you can simply pool them from various sources. Indeed, as I have noted, the annual Wine Spectator Top-100 lists from 1988 to 2019 showed considerable variation through time.

To emphasize this point, we can look at the recent Top 100 Wines of 2021 from James Suckling and his colleagues. This is not simply a compilation of the best–scoring wines from this year, based on the 11 individual Top-100 lists from that year (Argentina, Australia, Austria, Chile, China, France, Germany, Italy, New Zealand, Spain, USA). Such a thing would not be very useful, on its own — this would be an aggregation of numbers, not a synthesis of opinions.

In this case, there are at least nine opinions to be synthesized: “Tasting team members James Suckling, Claire Nesbitt and Kevin Davy held down the fort in Hong Kong, while Jo Cooke, Stuart Pigott, Zekun Shuai, William McIlhenny, Nathan Slone and Nick Stock tasted and rated thousands of more bottles on the road in Europe and the United States as well as South Australia and China.”

James notes that: “The main criteria that went into selecting this year’s list of the best 100 wines of the world was quality ... Then we looked at pricing, and finally what we call the ‘wow’ factor. The latter are wines that excite us emotionally as well as technically as tasters.” The graph below lists all of the 100 wines in rank order, horizontally, with their assigned score shown by the vertical bar (one bar per wine).

As you can see, the Top 100 wines all scored between 98 and 100 points. However, you will note that even scoring 98 could still get a wine into the top 20 (the best of them is ranked 20th). Moreover, scoring 100 did not even guarantee that a wine got into the top 60 (the last is 63rd). Indeed, according to the website, there were 38 100-point wines (of the almost 25,000 wines tasted), but only 25 made it into the Top 100 (some of the others had only a small production, and were thus disqualified). Top scoring is apparently a tricky business for a wine (see: How many 100-point scores do critics really give?).

So, some wine commentators really do know that there is more to wine-quality assessment than simply giving the wine a number. Indeed, Robert Parker, who was heavily involved in popularizing the scoring of wine, actually did note, way back in 1989 in the Wine Times: “[My] scoring system was always meant to be an accessory to the written reviews, tasting notes. That’s why I use sentences and try and make it interesting ... There’s a certain segment of my readers who only look at numbers, but I think it is a much smaller segment than most wine writers would like to believe.” I am not quite so sure about that last bit, which is why I have written this blog post.

Alternatively, James Halliday, from Australia, once commented: “Points are as subjective as the words in the tasting notes, but are a separate way of expressing the taster’s opinion, to be assimilated along with the description of the wine in the context of the particular tasting. All this may frustrate some consumers, but the ultimate reality is that Australia can never make Champagne, a Burgundy or a Bordeaux, so direct points comparison is fraught with contradictions and qualifications.”

We should not, of course, conclude from this that points are pointless. But we might conclude that the sometimes-heard argument that numbers are more precise than words is not really relevant in the case of assessments of wine quality. There is more to wine than can be expressed by a number; and, moreover, there are other ways to express quality (see: If not scores or words to describe wine quality, then what?).

By the way, how many of you know that Jancis Robinson has a degree in mathematics (and philosophy) from the University of Oxford? More to the point, how many wine scores have you ever seen from her? This should make you pause, and think.