Monday, 19 March 2018

Laube versus Suckling — their scores differ, but what does that mean for us?

There seem to be two general attitudes toward professional wine-quality scores. First, they can be seen as the sum of assessments of various sensory "components" of the wine. The classic example of this is the UCDavis 20-point score, which was originally designed to train students in detecting wine faults. This approach has been perhaps taken to its logical extreme in the fascinating book by Clive S. Michelson, Tasting and Grading Wine (2005. JAC International).

The alternative view is that the scores are expert, but subjective, opinions about the quality of the wine. For example, on March 15 1994, in response to a reader query, the Editor of the Wine Spectator magazine noted:
In brief, our editors do not assign specific values to certain properties of a wine when we score it. We grade it for overall quality as a professor grades an essay test. We look, smell and taste for many different attributes and flaws, then we assign a score based on how much we like the wine overall.
This seems to be the approach adopted by most of the professional media, especially when they use the 100-point scale. Some of them claim to be considering wine components individually (eg. complexity, concentration, balance, texture, length, overall elegance), but there is little evidence of this in their final scores.

James LaubeJames Suckling

I have shown in several blog posts that professional wine commentators do not necessarily provide comparable wine-quality scores when tasting the same wine. This can happen for many reasons, including different expertise, different personal preferences, different wine bottles, and different tasting conditions. This is why we seem to both love and hate wine critics. Let's look at this issue in more detail.

An interesting exercise

To look at variation in wine-quality scores, it is of interest to eliminate the last two factors listed above (bottles and tasting conditions), by having the scores be produced from the same bottle at the same time. This, of course, is what happens at most group wine tastings; but rarely do we see published the scores from several people at a single tasting, to make the direct comparison.

However, one pair of commentators where we can do this is James Laube and James Suckling who, at various times, have both provided wine-quality scores to Wine Spectator magazine regarding Cabernet wines, with Laube as the California expert and Suckling as the Bordeaux expert. Suckling has subsequently parted company with the magazine, but Laube remains as their California correspondent.

The dataset I will use here is from the "Cabernet Challenge" of 1996 (see Wine Spectator for September 15, 1996, pp. 32–48), in which the two James tasted 10 California Cabernet blends and 10 Bordeaux red wines from both the 1985 and 1990 vintages. This gives us 40 bottles of wine with which to compare their scores.

The data are shown in the first graph, with Laube's scores vertically and Suckling's horizontally. Each point represents one of the 40 bottles.

Suckling vs. Laube for 1985 and 1990 cabernets

I don't know about you, but this does not look too good, to me, in spite of the fact that Marvin Shanken, as the Editor of the article, claimed: "For the most part, our two critics found themselves in much agreement". To me, there is a wide spread of points in the graph — the scores differ by up to 9 points, with 5 of the bottles differing by more than 6 points. Furthermore, the mathematical correlation indicates only 29% agreement between the two sets of scores.

However, it is worth noting that the average scores from the two critics are almost identical (90.5), with very similar maximum (100 vs. 98) and minimum (both 82) scores. On average, Laube gave slightly higher scores to the California wines than to the Bordeaux wines; and Suckling gave lightly higher scores to the Bordeaux wines than to the California wines.

Now, let's look at what we might expect from critics who do agree. This next graph shows what perfect agreement would look like (the solid line) — for bottles whose points are on this line, the two James perfectly agreed with each other. Clearly, this is only 5 out of the 40 bottles. The Laube score is > the Suckling score 18 times, and 17 times it is the other way around.

Suckling vs. Laube for 1985 and 1990 cabernets

The two dashed lines in the graph show us ±2 points from perfect agreement — for bottles between the two lines, the two James' point scores were within 2 points of each other. This allows for the approximate nature of expert opinions — technically, we are allowing for the fact that the scores are presented with 1-point precision (eg. 88 vs. 89 points) but the experts cannot actually be 1-point accurate in their assessment.

There are only 23 of the 40 bottles (58%) between the dashed lines. So, even when we allow for the approximate nature of expert opinions, there is not much more agreement here than there is disagreement.

Another way of dealing with the approximate nature of expert scores is to greatly reduce the number of score categories, so that all the experts need to do to agree is pick the same category. This is the reasoning behind using star scores instead of points (eg. 3 or 5 stars), or word descriptions instead of numbers. The Wine Spectator does it this way:
95 – 100
90 – 94
85 – 89
80 – 84
75 – 79
50 – 74
 Classic: a great wine
 Outstanding: a wine of superior character and style
 Very good: a wine with special qualities
 Good: a solid, well-made wine
 Mediocre: a drinkable wine that may have minor flaws
 Not recommended

So, I have shown this scheme in the third graph. For bottles within the boxes, the two James' point scores agree as to the word categories of wine quality. Once again, this is only 25 of the 40 wines (63%). So, even this broad-brush approach to wine quality assessment provides only two-thirds agreement between the two critics.

Suckling vs. Laube for 1985 and 1990 cabernets

As an aside, it is worth noting the overall low scores given to the wines. Only 17 of the wines scored >90 points, even though they are all quite expensive. The only one of the 40 wines that I have tasted is the 1985 Château Mouton-Rothschild, and I was no more impressed by it than was either of the two James (85 vs. 89 points).

What does this mean for us?

The magazine is presenting their scores as representing some sort of Wine Spectator standard of quality, but clearly this is not an objective standard of quality. The scores are personal (but expert) judgments by their individual critics, who may have very little in common. At issue here is whether quality is an intrinsic property of wine, or whether it is mainly context dependent (see Jamie Goode).

The formal explanation for the degree of disagreement is this: the tasters are not using the same scoring scheme to make their assessments, even though they are expressing those assessments using the same scale. This is not just a minor semantic distinction, but is instead a fundamental and important property of anything expressed mathematically. As an example, it means that when two tasters produce a score of 85 it does not necessarily imply that they have a similar opinion about the wine; and if one produces 85 points and the other 90 then they do not necessarily differ in their opinion.

This situation is potentially a serious problem for all wine-quality assessments, when the scores represent expert, but subjective, opinions. Scores will look the same because they are written using the same scale, and people will inevitably try to compare them. But, if the scale does not have the same meaning for any given pair of people, then the numbers cannot be validly compared, because they have different meanings.** Not only would we be comparing apples and oranges, we would be comparing different (but unknown) numbers of apples and oranges. What is the point of that?

I will look at the mathematical consequences of this topic in a future post, illustrating the issue with a well-known data set.

Finally, one practical consequence of this mathematical characteristic is clearly being exploited by wine marketers. When looking at these scores on the web, it quickly became obvious that the wine stores are simply choosing to report the higher of the two critics' scores, when advertising any of the 40 wines, almost never producing both scores. This is an interesting example of "cherry picking".

Reproduced from Robert Dwyer at Palate Press

Thanks to Bob Henry for all of his help with this post — he has long championed the use of standardized wine-quality scoring schemes, often in vain.

** As a specific example, here are quotes from each of the two critics. James Suckling: "I was more concerned with the texture and aftertaste of the wines than with their aromatic qualities or flavor characteristics." James Laube: "I like my wines young, rich, highly concentrated and loaded with fruit flavors."

Monday, 12 March 2018

The rise and rise of New Zealand wine

Wine from New Zealand has very much become flavor of the month in many parts of the world. According to the recent book by Warren Moran (New Zealand Wine: The Land, The Vines, The People. 2017), the New Zealand wine industry only really got going 50 years ago, and as recently as 1990 Müller-Thurgau was it’s principal grape variety. However, the NZ vineyard area has been growing rapidly for the past 20 years (see the first graph), since it started to focus on Sauvignon blanc, instead. And yet, even the Wine Advocate did not have its first comprehensive report on New Zealand until 2008.

Wine exports have continued to increase along with production, particularly at the premium end of the market. I thought that a good way to explore this might be by comparing recent exports trends with those of their nearest geographical and socio-political rivals, Australia.

The wine export data come from Wine by Numbers (Il Corriere Vinicolo n. 23, July 2017), which covers the years from 2005 to 2016, inclusive, plus the recent Wine Export Report for 2017. This next graph illustrates the annual trends for both countries, using Australian dollars for the wine export value.

Annual wine exports from Australia and New Zealand

As I noted in a previous post (Getting the question right), Australia has had its ups and downs in the export world this century. Australian wine was itself the flavor of the month back in the 1990s, when good wine was available from Australia more cheaply than from many other countries. Times have changed since then; but a comeback of sorts appears to be in progress, starting from 2014.

By comparison, the rise and rise of New Zealand wine exports has been inexorable. They haven't caught Australia's value yet, nor do they look like they will do so any time soon; but this may depend on exactly what the exports markets consist of. Apparently, the New Zealand wine industry has a target of NZ$2 billion of exports by 2020 (New Zealand Winegrowers Annual Report 2017).

Annual wine exports from Australia

So, let's take a look at Australia, first, for comparison, before we turn to New Zealand in more detail. The third graph shows Australian wine exports (to 2016) by general world region.

As you can see, the North America (mainly the USA) and Europe (mainly the UK) patterns run in parallel. The peak export period coincides with the initial rise of the Yellow Tail brand. The decline, on the other hand, at least partly reflects the decline of what was then Australia's largest wine company, Southcorp. This decline occurred because they were involved mainly in the budget end of the wine spectrum, where volume sales are essential, margins are tight, and sustainable returns are not likely. This conglomerate was bought (in 2005) by a different conglomerate, the Fosters Group (Australia's biggest beer company), which later (2011) split it's wine-making arm off as what is now Treasury Wine Estates (currently Australia's largest wine producer by total revenue).

The sharp rise in the pattern for exports to Asia after 2014 reflects the fact that the Australian wine industry is now specifically targeting the premium wine market in China (the world’s fastest-growing major wine market). Notably, in 2012 China became the biggest destination for Australian bottled exports sold at AU$7.50 per liter and above; and currently China accounts for more than half of Australia’s exports of wine at AU$10 or more per liter. China has become a premium market.

Annual wine exports from New Zealand

Turning now to New Zealand, the same three geographical regions all show a different pattern from that of the Australian exports. Exports to the Asia/Oceania region principally reflect sales to Australia (not China, yet). These exports have been dominated by Sauvignon blanc, of course, which apparently now accounts for 75% of all NZ wine exports; and since 2008 this has actually been the biggest selling white wine in Australia. This market seems to have been saturated since 2012, with no further increase in sales.

Exports to the European region are principally to the UK, as they are for Australia. The graph shows that this market may also be becoming saturated for NZ wine, with a recent plateau in sales. This means that the North America market is now the biggest one for New Zealand exports, mainly to the USA; and it is this market that has shown an inexorable rise since 2011.

This suggests that we should look at the data in a bit more detail, by country rather than region. The data below also come from Wine by Numbers, this time the annual edition for 2016, which covers the year 2015 only.

By volume, almost exactly two-thirds of New Zealand's 210 million liters of wine exports for 2015 were bottled wine, as opposed to the remaining exported in bulk. The three biggest destinations for bottled wine were the USA (28%), the UK (24%) and Australia (24%), while those for bulk wine were the UK (36%), the USA (30%) and Australia (24%). So, most of the NZ wine (81%) went to just three countries (from where it may have been re-exported, of course), with the USA now their global biggest market.

However, more revealing is the premiumization of new Zealand wine. We can see this by looking at the dollar value per liter of the bottled wines being exported to various of the top countries, as shown in the table.

Hong Kong
United Arab Emirates
NZ$ / liter

This indicates that the Asian market takes principally premium bottled wines, whereas the North American and European markets also take some of the cheaper stuff. Indeed, none of the bulk wine actually goes directly to Asia, while France takes slightly more expensive bulk wine than does anywhere else. Premiumization is an important topic in the world of wine marketing, and the New Zealanders seem to have learned the lesson — sometimes, less is more, financially.

Unfortunately, the combined bulk and bottled export figures do seem to show that the New Zealanders are dumping their cheap wine principally in Australia, a country that already has enough cheap wine of its own. The only export countries where the price per liter of NZ bottled wines was cheaper than Australia were: the Netherlands, Ireland and Germany; and combined these countries imported only 28% of the volume sent to Australia. Similarly, the only bulk-export country where the price per liter was cheaper than Australia was Canada; and it imported only 3% of the volume sent to Australia.

The Australia — New Zealand rivalry clearly continues!

Monday, 5 March 2018

When did California become a "red wine" state?

The short answer is: in 1998.

California is often perceived as a state that specializes in the making of red wines. However, that image seems to be set mainly by the wines of the Napa Valley, where Cabernet sauvignon wines are king. Indeed, the "cult" wines of the USA principally come from this region, enhancing the state's reputation, which is often the focus of wine writers.

However, the most widely planted grape variety is actually Chardonnay, not Cabernet, with 91,333 bearing acres in 2016 versus 84,584 acres, respectively (California Grape Acreage Report 2016 Crop). Furthermore, Chardonnay comprised 14.5% of the 2017 grape-harvest crush, versus 14.2% for Cabernet sauvignon (California Grape Crush Report Preliminary 2017).

Indeed, the California Department of Food and Agriculture's data show that it was not all that long ago that the white-wine grape crush exceed that of red-wine grapes. The data in the first graph are from the Crush Report for 2017, and show the crush size for each of the years from 1989 to 2017, inclusive, categorized by grape type. Eating grapes include both Table (fresh) grapes and Raisin (dried) grapes.

Crush tonnage of grapes in California through time

Obviously, before 1996 more white grapes were crushed than red, while the two crushes were almost identical in 1996 and 1997. Since then, the red crush has progressively outstripped the white. Note, also, that in 1992 the red-wine grape crush was even less than that for the edible grapes!

So, red-wine grapes have been the most important in California only for the past 20 years. This is not necessarily surprising, as the Bordeaux wine region of south-western France, globally recognized for its red wines, only moved from mostly white wines to mostly reds in the 1970s.

The most likely reason for this change in emphasis is also revealed in the same Crush Report. The following graph, taken directly from that report, shows the price paid for the various grape types during the past 10 years.

California grape price through time

The graph makes the interesting point that the prices of Table grapes, Raisin grapes, and White-wine grapes have not changed much over the past decade. However, the price or Red-wine grapes has shown an increase of c. 50%. The has been distinctly so in the Napa Valley, of course.

The report also shows that in 2017 the Cabernet sauvignon production was up 17% above the 5-year average, while the Chardonnay production was down 13% below the 5-year average. Indeed, the Chardonnay crop was the smallest since 2011, although it remained just ahead of Cabernet as the biggest percentage of the crush (as noted above).