Did California wine-tasters agree with the results of the Judgment of Paris?

The short answer is: "No".

In May 1976, Steven Spurrier and Patricia Gallagher organized a wine tasting that has become known as the Judgment of Paris. Here, wines from France were tasted alongside some wines from California, and the latter acquitted themselves very well in the opinions of the tasters.

These sorts of comparative tastings had been conducted before, but mostly within the USA, whereas the Judgment took place in France itself with French judges; and, more importantly, it occurred in conjunction with the US Bicentennial celebrations. It therefore attracted much more media attention than any of the previous tastings. Indeed, it may well be the third most important event in the social and economic history of wine in the USA, after the imposition and then repeal of Prohibition.

However, the results of the Judgment were very variable among the tasters. Hardly any of them agreed closely with each other about the quality scoring of the wines, and especially about which wines were the best among the 10 reds (bordeaux grapes) and the 10 whites (chardonnays). This raises the question as to what other people thought about the relative quality of those same wines, at that same time.

This question is answerable to some extent by looking at the tastings of the Vintners Club, based in San Francisco. This club was formed in 1971 to organize weekly wine tastings (usually 12 wines). This club is still extant, although tastings are now monthly, instead of weekly. For our purposes here, the early tastings are reported in the book Vintners Club: Fourteen Years of Wine Tastings 1973-1987 (edited by Mary-Ellen McNeil-Draper. 1988).

One of the Club's tastings was an attempt to to evaluate the results of the Judgment tasting nearly 2 years afterwards (January 1978), as reported in a previous blog post (Was the Judgment of Paris repeatable?). However, the Club also tasted the individual wines before May 1976, usually in comparisons including other California wines (ie. those not chosen by Spurrier for the Judgment). Indeed, the success of the wines at these tastings seems to have played some part in establishing their respective reputations, leading to Spurrier choosing them to take part in the Judgment.

So, we can compare the Judgment wines quite independently of the Judgment itself, but in the same time period. This is an interesting exercise; and it emphasizes the point made at the time by Frank J. Prial (New York Times June 16 1976, p. 39), about the variability of wine assessments: "One would be foolish to take Mr Spurrier's little tasting as definitive."

Comparisons

Here, I will focus on the six California cabernets and six California chardonnays, each of which was tasted at the Vintners Club at least once before and once after the Judgment. All four of the French Bordeaux reds were also tasted at least twice at the Club, but not always both before and after the Judgment; and only one of the four French Burgundy whites was ever tasted at the Club.

Immediately preceding the Judgment, four of the six California chardonnays were tasted at the Chardonnay Taste-Off (February 1976), and four of the six California cabernets were tasted at the Cabernet Sauvignon Taste-Off (March 1976). These comparative Taste-Offs at the Vintners Club are explained in an earlier post (Wine tastings: should we assess wines by quality points or rank order of preference?).

The results of the various tastings are shown in the two graphs. The scores for each wine are the average from those tasters present on each occasion, based on the standard UC Davis scoring system, as used by the Vintners Club. The dates of the tastings are shown relative to the Judgment of Paris (May 24 1976).

In order to facilitate comparisons, the wines are listed in the graph legends in the order of their results at the Judgment itself (ie. highest score to lowest).

Among the cabernets, the Stag's Leap, Ridge, and Heitz wines were pretty much equal at every tasting. These wines were consistently rated as superior to the other three red wines, but we should not see any one of these three as being better than the other two. Interestingly, the only occasions on which the Stag's Leap wine was judged to be "best" was at the Judgment itself and again at its re-enactment. Also, note that the results for the Mayacamas wine were rather erratic, especially given its third-place result at the Judgment.

Among the chardonnays, the Chateau Montelena wine did not do well. Indeed, the Chalone wine consistently scored better than the Montelena, except at the Judgment of Paris itself. Indeed, most of the white wines scored better than than the Montelena except at the Judgment and its re-enactment. Also, for the whites it was the David Bruce wine that received particularly erratic results, on at least one occasion performing very well.

Finally, it is worth noting that 9 of the 12 wines received much higher scores at the 1978 re-enactment of the Judgment than they had before the Judgment tasting. It is hard not to see a subjective post-hoc bias in this result.

Conclusions

Clearly, the unique accolades heaped on the Judgment's two "winning" wines were not justified by the Vintners Club comparisons of the 12 wines. The Montelena white, in particular, was usually bested by the Chalone wine; and the Stag's Leap red was never better than those from Ridge or Heitz. This emphasizes the unreliability of single tastings for assessing wines — the outcomes depend too strongly on the circumstances, particularly the tasters present. Furthermore, some wines obviously received very variable assessments, sometimes being rated much more highly than on other occasions — either these wines were showing bottle variation or they were in an unusual style (as has been noted for the Bruce wine).

Getting the question right

I have written quite a few posts in which I analyzed a dataset using some particular mathematical model. Obviously, the model chosen is of some importance here — different models might give different outcomes (although, hopefully not). However, the choice of model is actually determined by the original question being asked of the data — we need to match the question and the appropriate model.

This raises the important issue of getting the question right. This is especially true if we are trying to relate causes and effects. For example: is the causal factor the presence of some something, or the absence of something else? Sherlock Holmes is famous for drawing Inspector Lestrade's attention to "the curious incident of the dog in the night-time." It turned out that the important thing was that the dog did nothing, under circumstances when a guard dog should clearly have done something. Holmes solved the crime by asking a question about an absence, not a presence.

As an example from the wine world, consider the following graph. It shows the recent time-course of the percentage each of five countries has had of the global wine export market. The data are taken from Kym Anderson & Nanda R. Aryal (2015) Growth and Cycles in Australia’s Wine Industry: a Statistical Compendium, 1843 to 2013, with additions listed by the AAWE.

We could ask any number of questions about these data. For example, we could ask about the general increase across the five countries since 1990, and whether it can be sustained. However, the most obvious question is likely to be about the time-course pattern for Australia, which seems to be dramatically different to the other four countries. But should that question be about the sudden increase that occurs from 2000 onwards, or the sudden decrease that occurs after 2005? Which pattern do we try to explain?

The second question (which seems to be the one that the Australian wine industry has been asking) would ask about why the "good times" suddenly crashed in 2005, and what the industry might do about it. On the other hand, the first question might ask about why the increase occurred in the first place, assuming that the subsequent decrease is simply a "return to normal" after a short-term aberration.

Let's look at how we might analyze Question 1. This next graph shows the Australian data compared to the average time-course of the other four wine-exporting countries (ie. excluding Australia).

The red line shows a very straightforward increase in export percentage through time. We might treat this line as a possible model of the "expected" pattern of growth, and then try to explain why the pattern for Australia does not fit in with it. This would be one way of answering Question 1. What we would do would be to apply a mathematical model to the red line, and then see how that model compares to the Australian data.

The next graph shows the fit of a simple Polynomial model to the average data, as indicated by the red dashed line. This model fits the data extremely well, as it accounts for 98% of the variation in the Average data.

We can, of course, now use this model to explore possible forecasts for future export growth. For example, the model forecasts that the Average export percentage will peak at 4.2%, which will occur in c. 2024. This might be a reasonable goal for an exporting country, to capture 4-5% of the market, and to consider themselves to have done well if they exceed this level.

More to the point, we can compare this model to the Australia data, as shown in the next graph. The blue dashed line is simply the red dashed line raised by 1.2 percentage points (which is the best fit to the Australia data). This reveals that from 2013 onwards the Australian exports were exactly where we would forecast them to be, based on the 1990-1995 data.

So, answering Question 1 would quite a reasonable way to tackle these data — the data do support the idea that the decrease in Australian export percentage may well be simply a return to "normal" after a short-term aberration. The downwards trend can be seen, not as a crisis, but merely as a correction. These are two quite different interpretations.

Getting the question right is crucial. Data analysis often suffers from what is called confirmation bias, in which we simply try to confirm the assumed answer to our initial question. That is, we look for what the dog did in the night-time, instead of looking for what it did not do — and we often find something that the dog did, no matter how irrelevant it may be!

Why lionize winemakers but not viticulturists?

It is widely noted that viticulturists can have as much influence on the quality of the final wine as do winemakers, and yet it is still the winemakers whose names are most widely known, because they are the ones who most commonly appear in the wine press. So, the people in the winery get the media attention more than those in the vineyard, even though the location of that vineyard is acknowledged to be of prime importance.

To counteract this trend, in this post I discuss one example, from Australia, where the viticulturist often gets almost as much press as the winemaker.

Wynns Coonawarra Estate is by far the biggest winery in the Coonawarra region of Australia, a region that has an international reputation for the quality of its cabernet sauvignon wines (although the shiraz wines are not too shabby, either). Wynns consistently project three people as being their "team", as listed in the first photo below. [Note: Ben Harris, the Vineyard Manager, tends to go missing from most of the press; see the photo at the bottom of the post.]

What is more important for our purposes here, the media actively go along with Wynns' attitude. I have listed a few press reports at the end of this post, as a small sample of what the wine media have to say. The two people titled "winemaker" do get more press than the viticulturist, although much of their personal press does tend to emphasize them as females in a male-dominated profession. Indeed, Sue Hodder and Sarah Pidgeon were jointly named the 'Winemaker of the Year' at the 2016 Australian Society of Viticulture and Oenology (ASVO) Awards for Excellence.

However, back in 2010, when Sue Hodder was named Australian Gourmet Traveller WINE’s 'Winemaker of the Year', a new award was introduced for Allen Jenkins: 'Viticulturist of the Year'. Part of the reason for acknowledging the importance of the viticulturist at Wynns has been his role in rejuvenating the vineyards over the past 15 years, and the clear effect that this has had on the quality of the wines.

L to R:  Sarah Pidgeon (Winemaker)
Sue Hodder (Chief Winemaker)
Allen Jenkins (Regional Vineyard Manager)

The rejuvenation program

Sue Hodder joined Wynns just prior to the 1993 vintage; and she was then appointed Chief Winemaker in 1998, at which point Sarah Pidgeon became Winemaker. Allen Jenkins arrived as the viticulturist in 2001-2, at least partly because Hodder and Pidgeon had realized that the vineyards needed extensive treatment, if the wines were to be improved.

For example, during the 1990s it was noted that the vines were building up too much dead wood, as a result of 20 years of (minimal) mechanical pruning. Indeed, the vines were reported to be so low yielding that they were hard to pick. The rejuvenation started in 2000, and was accelerated in 2002. It was expected to take eight years to complete; and the change in the wines was reported widely in the media starting from 2010.

The process involved large-scale vine regeneration by heavy chainsaw pruning of very old vines (shiraz up to 120 years old, cabernet sauvignon up to 60 years old), removing the dense clusters of dead wood, and thus bringing the vines back to a new physiological balance. Tired or diseased vines were grubbed out, along with the removal of lesser varieties. These were all replaced by new clones and rootstocks of cabernet and shiraz, for which the winery developed a heritage nursery, based on cuttings from time-proven vines. There was re-trellising, along with changed canopy management and new pruning techniques. The vineyards were also converted from sprinkler to drip irrigation.

Along with all of this, the winery was also modified to focus more on small-batch vinification, from 2008 onwards. This allows the grapes to be picked at perfect physiological ripeness, as even a large vineyard block can now be processed in many small batches instead of a few large ones. This takes advantage of the increased grape quality in the vineyard. The oak maturation of the wines has also been re-visited, resulting in a lighter handling, which now produces softer, more elegant wines. Indeed, the latter approach is a return to the style from the 1960s, rather than the heavier style favored in the 1980s and 1990s.

The flagship Wynns wines are the John Riddoch Cabernet Sauvignon and the Michael Shiraz, which are made only in years when grapes of very high quality are available. Production was stopped on both of these wines during the 2000-2002 part of the rejuvenation period. So, to see the effects of the rejuvenation on the quality of the Wynns wines, we need to look at a different product from the winery.

Black Label Cabernet Sauvignon

Within the Wynns range, the Black Label Cabernet Sauvignon holds a special place, even though it is marketed as the "basic" wine from the winery, with an average annual production of roughly 40,000 cases. The wine is currently blended from about 20 different small parcels of grapes, out of up to 80 that are contenders each year. The vines were planted mainly in the 1960s, 70s and 80s.

The flagship John Riddoch cabernet is always denser, more powerful and oaky than the cheaper Black Label, but the latter is always better value for money, selling for less than one third of the top wine's price (and often being aggressively discounted by retailers). Indeed, it has been repeatedly shown that the Black Label can age for decades, making it "possibly the most important cellaring wine in Australia", and forming "the backbone of many Australian cellars for over 50 years". This makes it "one of the most important wines in Australia’s wine history". Myself, I think that it is the best value-for-money cabernet wine that Australia produces.

There have been a number of retrospective tastings of this wine organized by Wynns, which go all the way back to the first vintage, in 1954. For example, there was an important vertical tasting covering the 50 years from 1954- 2004, which Hodder has described as the catalyst for the winemakers changing the style away from the heavier style of the 1990s. There was also a 60-year vertical tasting earlier this year.

However, the published reports from these tastings are somewhat sporadic. So, for an evaluation of the effects of the vineyard rejuvenation it will be simpler to cover a shorter period. The graph above shows the weighted average scores from the Wine-Searcher database, covering the vintages from 1990 (ten years before the rejuvenation started) to 2014, inclusive.

Note that for almost every year since 2004 the wine has been scored 91 or higher, whereas before that 91 was the rare top score. There is no doubt that the wine is in the best form it’s been in for years. And the viticultural team can take most of the credit.

Buy yourself a bottle. Put it away for ten years. Then drink it. You will see what I mean about value for money.

Bibliography

Who says New World wines don't develop? — Michael Apstein

Gourmet Traveller | Viticulturist of the year — Susanne Bell

Who dares Wynns — James Halliday

Wynns Coonawarra: a revolution many years in the making — James Halliday

Wynns wine legend turns 60 — Huon Hooke

A 17-year winemaking partnership — Cathy Howard

Interview with Sue Hodder —Jeannie Cho Lee

Wynns unleashes Coonawarra’s diversity — Chris Shanahan

Wynns Coonawarra — great winemaking but the marketing sucks — Chris Shanahan

How Sue Hodder’s history lesson improved Wynns’ Coonawarra reds — Chris Shanahan

Profile: Sue Hodder — Tyson Stelzer

Increase in US wine consumption over 10 years

Recently, the American Association of Wine Economists published on their Facebook page some data (here and here) showing the annual wine consumption per capita in the USA, both in 2005 and in 2014. Recalculating these data shows that there was, on average, a 13% increase in wine consumption per person between these two years, for the country as a whole.

However, this is only part of the picture, as we all know that wine consumption will not have increased equally among all of the states. So, I have plotted the state-by-state data in the graph below. Here, each of the points represents a particular state, with its location on the axes representing consumption (liters per person) in 2005 (horizontally) and 2014 (vertically). If the consumption per capita was the same in both years then the points would lie along the pink line; and if they are above the line then the consumption was greater in 2014 than in 2005.

The graph shows that the biggest boozers are in the District of Columbia. Indeed, the per person consumption in DC is more than double that of fully 37 of the states. This may explain some of the decisions that come out of Washington.

DC is followed a long way back by New Hampshire, followed even further back by Vermont and Massachusetts. For comparison, a couple sharing a bottle of wine per week would consume 20 liters per adult per year, which is equivalent to 15 liters per capita (given that 25% of the population is below drinking age). The graph shows that only DC, NH, VT and MA exceed this annual level (ie. the top four points in the graph).

For 20 of the states the annual wine consumption increased between 2005 and 2014 by more than 1 liter per person — the dotted lines on the graph indicate plus/minus 1 liter per person. The biggest increases were in Vermont, followed by Massachusetts, New Jersey and New Hampshire. Of these, only Vermont was a long way above the average increase.

None of the states had a decrease in annual consumption of more than 1 liter per person. However, three of the states were a long way below the average increase (ie. much less than +13%): Delaware, Colorado and Nevada.

These results seem to be quite good for the wine industry. As Charles Olken says: "Phew. Thank goodness" (Americans are turning away from wine ~~ No, they are not. Yes, they are). Nevertheless, almost all Americans drink much less than a bottle of wine per week; so there is much room for improvement.