Monday, June 5, 2017

How many 100-point wine-quality scales are there?

In the previous post (How many wine-quality scales are there?) I discussed the range of ratings systems for describing wine quality that use 20 points. However, perhaps of more direct practical relevance to most wine drinkers in the USA is the range of systems that use 100 points (or, more correctly, 50-100 points).

The 100-point scale is used by the most popular sources of wine-quality scores, including the Wine Spectator, Wine Advocate and Wine Enthusiast; and so wine purchasers encounter their scores almost every time they try to purchase a bottle of wine. But how do these scores relate to each other? Using the metaphor introduced in the previous post, how similar are their languages? And what do we have to do to translate between languages?


All three of these popular scoring systems have been publicly described, although I contend that it might be a bit tricky for any of the rest of us to duplicate the scores for ourselves. However, there are plenty of other wine commentators who provide scores without any explicit indication of how they derive those scores. This means that some simple comparison of a few of the different systems is in order.

As explained in the last post, in order to standardize the various scales for direct comparison, we need to translate the different languages into a common language. I will do this in the same manner as last time, by converting the different scales to a single 100-point scale, as used by the Wine Advocate. I will also compare the quality scales based on their scores for the five First Growth red wines of the Left Bank of Bordeaux, as I did last time.

The scales for nine different scoring systems are shown in the graph. The original scores are shown on the horizontal axis, while the standardized score is shown vertically. The vertical axis represents the score that the Wine Advocate would give a wine of the same quality. If the critics were all speaking the same language to express their opinions about wine quality, then the lines would be sitting on top of each other; and the further apart they are, the more different are the languages.

Nine different 100-point wine-scoring systems

There are lots of different lines here, which indicates that each source of scores uses a different scheme, and thus is speaking a different language. Many of the lines are fairly close, however, and thus many of the languages are not all that different from each other. Fortunately for us, they are most similar to each other in the range 85-95 points.

First, note that the line for the Wine Spectator lies exactly along the diagonal of the graph. This indicates that the Wine Advocate and the Wine Spectator are using exactly the same scoring system — they are speaking the same language. In other words, a 95-point wine from either source means exactly the same thing. If they give different scores to a particular wine, then they are disagreeing only about the quality of the wine — this is not true for any other pair of commentators, because in their case a different score may simply reflect the difference in language.

It is worth noting that almost all of the Wine Advocate scores came from Robert Parker, while most of the Wine Spectator's were from James Suckling, along with a few from Thomas Matthews, James Molesworth and Harvey Steiman (who have all reviewed the red wines of Bordeaux for that magazine), plus some that were unattributed.

Second, the line for the Wine Enthusiast always lies below the diagonal of the graph. This indicates that the Wine Enthusiast scores are slightly greater than those of the Wine Advocate (and Wine Spectator) for an equivalent wine. For example, if the Enthusiast gives a score of 80 then Parker would give (in the Advocate) 78-79 points for a wine of the same quality. This situation has been noted in Steve De Long's comparison of wine scoring systems, although it is nowhere near as extreme as he suggests.

Third, the line for Stephen Tanzer always lies above the diagonal of the graph, indicating that his scores are usually slightly less than those of the Wine Advocate (and Wine Spectator). Indeed, a 100-point Parker wine would get only 98-99 points from Tanzer.

All of the other lines cross the diagonal at some point. This indicates that sometimes their scores are above those of the Advocate and sometimes they are below. Interestingly, most of these systems converge at roughly 91 points, as indicated by the dashed line on the graph. So, a 91-point wine means more-or-less the same thing for most of these commentators (except Tanzer and the Enthusiast) — it is the only common "word" in most of the languages!

The most different of the scoring schemes is that of James Suckling, followed by those of Jeannie Cho Lee and Richard Jennings (which are surprisingly similar). Suckling is a former editor of Wine Spectator, and he actually provided most of the scores used here for that magazine — this makes his strong difference in scoring system on his own web site particularly notable, as it implies that he has changed language since departing from the Spectator.

Finally, it is important to recognize that all I have done here I have evaluate the similarity of the different scoring systems. Whether the scores actually represent wine quality in any way is not something I can test, although I presume that the scores do represent something about the characteristics of the wines. Nor can I evaluate whether the scores reflect wines that any particular consumer might like to drink, or whether they can be used to make purchasing decisions. Nor can I be sure exactly what would happen if I chose a different set of wines for my comparisons.

Conclusions

The short answer to the question posed in the title is: pretty much one for each commentator, although some of them are quite similar. Indeed, the Wine Spectator and the Wine Advocate seem to use their scores to mean almost the same thing as each other, while the Wine Enthusiast gives a slightly higher score for a wine of equivalent quality.

While there are not as many wine-quality rating systems as there are languages, the idea of translating among them is just as necessary in both cases, if we are to get any meaning. That is, every time a wine retailer plies us with a combination of critics' scores, we have to translate those scores into a common language, in order to work out whether the critics are agreeing with each other or not. Different scores may simply reflect differences in scoring systems not differences in wine quality; and similarity of scores does not necessarily represent agreement on quality.

Averaging the scores from the different critics, as is sometimes done, notably by Wine-Searcher and 90plus Wines, is unlikely to be a valid thing, mathematically. Given the results from this and the previous post (How many wine-quality scales are there?), calculating a mathematical average score would be like trying to calculate a mathematically average language. Jean-Marie Cardebat and Emmanuel Paroissien (American Association of Wine Economists Working Paper No. 180. 2015) have correctly pointed out that the different scoring systems need to be converted to a common score (ie. a common language) before any mathematics can be validly applied to them.

6 comments:

  1. "Indeed, a 100-point Parker wine would get only 98-99 points from Tanzer."

    Parker has awarded 100-point scores to over one hundred wines. (I don't know the actual number.)

    Whereas Tanzer has awarded a 100-point score to just ONE wine:

    2010 Egon Muller Scharzhofberger Riesling Trockenbeerenauslese, Mosel, Germany

    Source: http://www.wine-searcher.com/critics-11-stephen+tanzer

    Price? $16,926.14

    Source: http://www.wine-searcher.com/find/egon+muller+scharzhofberger+riesling+tba+wiltingen+saar+mosel+germany/2010

    ReplyDelete
  2. From The Drinks Business Online
    (May 7, 2015):

    "PARKER: NOT AWARDING 100 POINTS 'IRRESPONSIBLE' "

    Link: https://www.thedrinksbusiness.com/2015/05/parker-not-awarding-100-points-irresponsible/

    By Patrick Schmitt

    During an interview with the drinks business earlier this year, Parker – who developed the 100-point rating system – expressed his urge to award full marks to great wines, and his dismay at those who don’t.

    “When, in your mind, the wine is the best example you have ever tasted of this particular wine, you have an obligation to give it a perfect score,” he told db.

    On the other hand, he branded those who are incapable of awarding a perfect score “irresponsible”.

    “I think the person who can’t give 100 is really dodging responsibility, because there’s no way they haven’t tasted a wine that is the best example they have tasted from this producer, the best example they could ever think of.”

    He then stated, “I think it’s irresponsible not to give a perfect score if you think the wine is perfect.”

    ReplyDelete
  3. Personally, I still prefer the UC Davis system, why? Because it rates the quality of the wine in various degrees to come up with an overall quality rating. I have used it with novices and it works even with them. The problem with the others is subjectivity.

    Under the Davis system, there are just two subjective points, under Parker:
    • Color and appearance have 5 points.
    • The aroma and bouquet receive up to 15 points.
    • The passage of mouth and aftertaste receive up to 20 points.
    • Finally, the overall quality level and the potential for future evolution and improvement receive up to 10 points.
     
    I have been in tastings with other 100 point systems that can have 20-25 subjective points. That is the problem: we aren't judging wine quality, we are letting the taster tell us what he/she likes and frankly, who cares? I certainly don't since I differ greatly in what Parker looks for in a wine and have been disappointed at times when I buy (used to) based on his or anyone's rating
    I would be interested in your opinions and will end with a study that was done showing point escalation or 'rounding up' to 90 which this article illustrates, although not stated, to me.

    For my 50th birthday, we had a party on the Napa Wine Train. Waiting to board I saw this in a gift shop:
    Man in tasting room: This wine is awful...worst I have ever tasted!
    Pourer: Really? Parker gave it a 90.
    Man: I'll take two cases!

    Just as Sideways tarnished Merlot, and embellished Pinot's (the joke being that Mile's favorite wine was Cheval Blanc which is predominantly Merlot), ratings are leaving a lot of good wines sitting on retailers shelves.

    While I applaud Parker for creating a system that has improved quality of wine (in his early publications there were many wines with mid-80's scores and until the '82 Bordeaux, I can't recall any wine with a rating of 99 or higher), and Michel Rolland's efforts (although I fear all wines would taste alike if everyone made them to his standards), it is time for the consumer to take control and have the confidence to drink and pour for guests, wines they feel are high quality. The same goes for vintages: there are winners and losers in each one and terroir does make a difference.

    That's how I see it.
    Trader Bill (from my wine blog: www.traderbillonwine.com)

    ReplyDelete
    Replies
    1. The UCDavis scale certainly tries to reduce the subjective component, which is a desirable goal. The main problem is, as I discuss in a later post, that in order for the sum to have any mathematical meaning then each of the points added must be equivalent to one another. This is rather hard to implement when we are considering all of the very different organoleptic qualities of wine.

      cheers, David

      Delete
    2. That's the point. All we NEED to know is if the wine is well made. I don't care if all the experts agree to a 't' but it doesn't match YOUR taste profile. Why don't we simply trust our own pallets? Because we are Americans!

      Delete
    3. William:

      Here is the wine cartoon you allude to:

      https://mir-s3-cdn-cf.behance.net/project_modules/disp/db950c3989818.5601cbb9befe6.jpg

      Pacific Wine Company was a leading wine merchant in its day in San Francisco.

      Illustrator Bob Johnson was commissioned to create original artwork for their circa 1980s era newsletter.

      More cartoons:

      https://www.behance.net/gallery/686573/SF-PACIFIC-WINE-COMPANY-Wine-Cartoons

      Enjoy!

      ~~ Bob

      Delete