Monday, March 6, 2017

Are the quality scores from repeat tastings correlated? Sometimes!

Wine quality scores from commentators tend to be set in stone while the wines are still available commercially. That is, a single score is produced when the wine is released, and it stands indefinitely. Sometimes, wines are tasted several times, and a consensus score is then produced, but not often. Sometimes, only the latest score is the one presented, without reference to any previous scores — Wine-Searcher does this, for example, when compiling critics' scores.

One thing that is of interest to the consumer is how these repeat scores relate to each other. It would be nice if repeated tastings of a particular wine produced the same score, because then we could have some confidence in it. However, there will be some component of the scores that is due to different bottles of the wines, and so we cannot expect perfect repeatability. More importantly, this issue will be confounded by possible changes in the wines themselves as they age, so that any one bottle varies through time.

However, a lot of the variation in scores will be due to what is technically called intra-individual variation in the scorer, which you might call within-taster variation — the same wine tasted on repeat occasions by the same person does not receive the same score no matter how similar it is. The assessment of quality is the result of a taster’s previous experiences as well as their personal conceptions; and even experienced wine tasters have been shown to incorporate their own preferences in their judgments. In addition to this, the environment of the tasting is also known to affect quality judgments.

This issue has only occasionally been studied in the professional literature; and I have included a list of relevant published papers at the end of this post.

What I will do in this post is look at some particular examples of scores from repeat tastings by five different commentators. Some of these tastings come from retrospective vertical comparisons of a single wine, where many of the previous vintages are tasted on a single occasion, or horizontal tastings of a number of wines from the same region and vintage — these new scores can be compared to the scores previously assigned to those same wines.

Some examples

Most of these examples are restricted to wines where there are many vintages to compare, and where the producer actively provides retrospective vertical tastings of their products. Furthermore, my expertise in this regard is in Australian wine. This creates a distinct bias in which wines I can use as examples.

For my first example, I will use some scores from a book by the Australian commentator Jeremy Oliver, The Australian Wine Handbook, of which there were at least three editions: 1993, 1994, 1996. I introduced this book in a previous blog post (Wine writing, and wine books). The wine that I will look at is Penfolds Grange Bin 95, of which there are now more than 60 vintages, and which I also introduced in a previous post (Poor correlation among critics' quality scores).

The first graph compares the quality scores for 38 vintages of this wine in the 2nd and 3rd editions of the book, at which time Oliver was using a 10-point quality scale. Each point represents a single vintage; and if the scores were identical in the two editions, then the points would all lie along the pink line.

There are only 21 vintages (55%) for which the scores are the same, 8 (21%) that decrease from 1994 to 1996, and 9 (24%) that increase. The maximum decrease is 2 points, and the maximum increase is 3 points. The book does not make clear what the circumstances were that lead to the two sets of scores, but there is obviously considerable variation in the opinions about quality. Nevertheless, there is no evidence of any bias in Oliver's opinions about the wines,

Now let's consider an example where the second tasting involved a retrospective vertical comparison of the wines. This example involves the same wine and commentator. The first set of scores comes from The Onwine Australian Wine Annual, published in 2000, at which time Oliver was using a 20-point scale. These scores are not exactly the same as those from 1996. The second set of scores comes from a retrospective tasting, Making it a Date with Grange, published in 2004. There are 45 vintages included in this next graph.

Once again, there is no evidence of any bias in Oliver's opinions about the wines, although the scores change by a maximum of 1.8, both up and down. The correlation between the scores shows that they share approximately half (51%) of the variation, which is not particularly high, given that they are the same wines tasted only 4 years apart.

As an alternative example of a retrospective vertical tasting, we can look at another Australian commentator, James Halliday, and the wine Cullen Cabernet Sauvignon Merlot (now called Diana Madeline) from the Margaret River region of Western Australia. The first set of scores comes from various editions of The Australian Wine Companion, using a 100-point scale. The second set of scores comes from a retrospective vertical, Tasting an Icon, published in the Halliday Wine Companion Magazine for Feb/Mar 2014. There are 19 vintages included in the next graph.

This time we see several vintages that have very different scores in the two datasets. Two of the vintages have their scores reduced by 5-6 points (bottom of the graph), and one gets a 5-point increase (top-left). Furthermore, the scores are not highly correlated even if we exclude these three vintages, with only 20% of the variation being shared between the two datasets.

Moving on, the wines of Bordeaux often have repeated scores from single sources that cover many vintages. For example, the American magazine Wine Spectator published two retrospective vertical tastings of the top wine from Château Lafite-Rothschild, one on 15 December 1988 and one on 30 November 1991. There are 34 vintages included in the next graph.

Once again, we see several vintages that have very different scores in the two datasets; these vintages are labeled in the graph. Even if we exclude these three vintages, then the scores are still not correlated with each other, as only 8% of the variation is shared between the two datasets. This is a very poor correlation, given that they refer to repeat tastings of the same wines tasted only 3 years apart.

As an alternative approach, we could try comparing a range of wines from the same region in the same vintage year — that is, a horizontal tasting rather than using a vertical one. To do this, let's look at the American commentator James Laube, and the 1986 vintage California Cabernets. The first set of scores comes from the 1988 book California's Great Cabernets: The Wine Spectator's Ultimate Guide for Consumers, Collectors and Investors. The second set of scores comes from a retrospective horizontal tasting, 10 Years After, published in Wine Spectator for December 15 1996 (pp. 67-70). (Thanks to Bob Henry for compiling these two datasets.) There are 45 different cabernet wines included in the next graph.

As is obvious, there was a major re-assessment of the wines at the second tasting, as almost all of the points lie below the pink line, rather than being scattered around the line (as in the above graphs). This vintage did not turn out to be as good as originally expected!

On release, the wines were assessed generally in the score range 88-96, but 10 years later these same wines scored only 86-94 points, with an average score reduction of 2.6 points per wine. This seems to relate to the idea that there are bonus quality points available for wines based on their expected longevity (see the post What's all this fuss about red versus white wine quality scores?). After 10 years, it was obvious that the 1986 cabernets were not going to last as long as expected, and so their bonus points disappeared.

It is clear that the consumer should pay attention to the Wine Spectator and the Wine Advocate, both of which are known to conduct reviews of California Cabernets at the 10th, 20th, 30th and sometimes even 40th anniversaries of the vintages.

Finally, we can return to the vertical tastings, and look at a British commentator, Jancis Robinson. We can also return to Australian wines, this time the Henschke Hill of Grace Shiraz, a wine from the Eden Valley region of South Australia, and with almost as long and distinguished a pedigree as the Penfolds Grange discussed above. Robinson took part in two retrospective vertical tastings of this wine, one on 18 March 2003 and the other on 12 March 2013, which marked the 40th and 50th anniversaries, respectively, of the first vintage. There are 23 vintages included in the next graph, scored on a 20-point quality scale.

Once again, there was apparently a major re-assessment of the wines at the second tasting, as almost all of the points lie above the pink line, rather than being scattered around the line. Note that this is a comparison of two retrospective tastings, unlike the above graphs.

The age of the wines at the first tasting was 5-28 years, and the scores ranged from 15-18.5, with an average of 16.7 points. The age of the same wines at the second tasting was 15-38 years, and the scores ranged from 16-19, with an average of 17.7 points, for an average increase of 1 point. Obviously, either the wines changed or Robinson's perception of them did. That is, is this score inflation, or did most of the wines get better through time?

To work this out, we can plot all of the scores from the two retrospective tastings, not just those tasted on both occasions. This final graph shows the two sets of scores for each vintage tasted, in different colors. (Note: the missing vintages in the second tasting are ones in which the wine was not considered good enough to release.)

Clearly, all of the scores on the second occasion (in green) are consistent, and so there is no evidence that the quality of the wines improved over time. Instead, we must conclude that Robinson simply scored the wines 1 point higher on the second occasion. This may reflect her better familiarity with this style of wine, or it may reflect her well-known dislike of assigning wine scores in the first place.


Wine quality scores are usually presented as though they are inviolate, and represent a critic's opinion that does not change. That is, we are given one number, which does not get changed. This may be valid if the commentator tastes each wine once, and once only. However, the commentators have been known to reevaluate wines at different times, especially if they are invited to a retrospective tasting, either vertical or horizontal. In this case, the evaluations at different times can be really quite different.

Research Literature

Robert H. Ashton (2012) Reliability and consensus of experienced wine judges: expertise within and between? Journal of Wine Economics 7:70-87.

Chris J. Brien, P. May, Oliver Mayo (1987) Analysis of judge performance in wine-quality evaluations. Journal of Food Science 52:1273-1279.

Richard Gawel, Peter W. Godden (2008) Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research 14:1-8.

Richard Gawel, Tony Royal, Peter Leske (2002) The effect of different oak types on the sensory properties of chardonnay. Australian and New Zealand Wine Industry Journal 17:14-20.

Robert T. Hodgson (2008) An examination of judge reliability at a major U.S. wine competition. Journal of Wine Economics 3:105-113.

Robert T. Hodgson (2009) How expert are "expert" wine judges? Journal of Wine Economics 4:233-241.

Harry Lawless, Yen-Fei Liu, Craig Goldwyn (1997) Evaluation of wine quality using a small-panel hedonic scaling method. Journal of Sensory Studies 12:317-332.


  1. With David's indulgence, I proffer two l-o-n-g comments.

    Part One . . .

    Excerpts from Jancis Robinson, MW Website
    (circa 2002):

    “How to Score Wine”


    . . .

    I would be much happier in my professional life if I were never required to assign a score to a wine. I know so well how subjective the whole business of wine appreciation is and, perhaps more importantly, how much the same wine can change from bottle to bottle and week to week, if not day to day. I frequently find myself re-tasting a wine at the same stage in its life. So far I have rarely marked more than 0.5 points out of 20 differently on the two occasions, but it wouldn't surprise me at all if I did.
    And as for tasting the same wine at different stages in its life, this is even less likely to yield identical scores. Quite apart from bottle variation there are differences in tasters' moods and vast differences in how wines mature in bottle.

    Even I have to admit, however, that scores have their uses. The most obvious is to help the reader-in-a-hurry . . .

    I like the five-star system used by Michael Broadbent and Decanter magazine. Wines that taste wonderful now get five stars. Those that will be great may be given three stars with two in brackets for their potential. But Brits being as polite, or just plain cowardly, as we are, almost all the wines get between three and five stars in Decanter so it's not an especially nuanced scoring system -- although I have been known to use it for wines likely to be very close together in quality such as de luxe Champagnes or mature vintage Ports.

    When even I have to admit that I really need a numerical scoring system is when tasting a wide range of wines of the same sort when readers, or subscribers to, need a shorthand reference to my favourite wines. . . .

    I know that Americans are used to points out of 100 from their school system so that now they, and an increasing number of wine drinkers around the world, use points out of 100 to assess wines. Like many Brits, I find this system difficult to cope with, having no cultural reference for it.

    So, I limp along with points and half-points out of 20, which means that the great majority of wines (though by no means all) are scored somewhere between 15 and 18.5, which admittedly gives me only eight possible scores for non-exceptional wines -- an improvement on the five star system but not much of one. (I try when tasting young wines to give a likely period when the wine will be drinking best, so I do cover the aspect of its potential for development.)

    But, perhaps strangely for someone who studied mathematics at Oxford, I'm not a great fan of the conjunction of numbers and wine. Once numbers are involved, it is all too easy to reduce wine to a financial commodity rather than keep its precious status as a uniquely stimulating source of sensual pleasure and conviviality.

  2. Also with David's indulgence.

    Part Two (with a Part Three and Part Four as text length necessitates)

    From Jancis Robinson, MW Website
    (Posted September 1998):

    Notes From Attending an Yquem Vertical Tasting

    Link: no longer available

    1784 (President Thomas Jefferson’s collection bottle, 1 tasted by Michael Broadbent, H R [host Hardy Rodenstock] and German cronies at Wiesbaden in 1985 soon after H R's acquisition of this Thomas Jefferson collection from a mysterious "bricked up cellar in Paris" before another was auctioned at Christie's in 1986. The bottle tasted in 1998 was much darker than that described in 1985.)

    Very dark brown syrup with copper coloured rim. Bottle stink immediately after pouring. After 5-10 minutes a very beguiling bouquet of dried roses emerged and the wine was lively, aromatic, fragrant for a good 40 to 50 minutes. On the palate the wine was very gentle, very delicate, very feminine to the 1787's more aggressive appeal, and the sweet fruit was lovely and very, very long before fading (earlier than the 1787). A marvel of a relic rather than unmitigated pleasure.

    1787 (another dark Thomas Jefferson bottle, engraved not labelled, with a deep punt).

    Deep, deep brown with a greenish rim and, like the 1784, smelt slighty mouldy at first. There was definite life here, however, in a wine that was slightly treacly, extremely lively with marked but not unpeasant acidity. On the palate a burnt sugar start, dry finish, no great persistence. After 40 minutes there was an intense nose of chestnuts, autumnal and briary. More robust and concentrated but less charming than the 1784. Powerful, chunky.

    1811 (the year of the comet)
    A quite amazing wine, served blind with 1831, 1911 and 1931 it was the most intense, yet least evolved of the lot.

    Deep amber with green gold rim. So vibrant and multilayered on the nose, it smelt as though it was just starting to unfold, yet was utterly convincing about the treasures it had yet to give up. Spicy and rich and so, so piercingly clean. Racy, long piercing essence of cream and spice. Very, very powerful, long and complete. After 40 minutes in the glass it took on a hint of rum toffees which is not a flavour I happen to like (c.f. the greater delicacy of the 1847) but that is the only criticism I could possibly muster. This is presumably a one-off and probably deserves an even higher ranking than the 1847. 25 points [Bob Henry comment: Her 25 point score exceeds her own 20 point scale.] and still a great deal to give. I hope very much to have a chance to taste it again before I die.

    Deep amber.
    Nose not quite knit, slightly volatile. Dry finish. Less intense than the other wine vaguely in this style, the 1899 (as well it might be). 18 points and going downhill slowly.

    Wide, pale rim with a heart of deep amber. Very very intense yet subtle nose with nots of nuts and cream. A superb wine with layers and layers of flavour and richness. Angelo Gaja suggested baby powder and roasted hazelnuts. Wonderfully smooth texture. Its effect on this jaded palate was medicinal in the best possible way: a quite delicious pick-me-up. So long, yet delicate. A great, great wine that happened to be served with one or two even greater ones. 24 points [Bob Henry comment: Her point score exceeds her own 20 point scale.] and probably at its peak.

  3. Part Three.

    From Jancis Robinson, MW Website
    (Posted September 1998):

    Notes From Attending an Yquem Vertical Tasting (Continued)

    The big issue of the day was whether this of the 1811 was 'better'. (Both were absolutely extraordinary. The 1847 gave me more pure tasting pleasure, but apparently this wonderfully pure scent of raspberries and vanilla cream had been apparent on the 1858 and the 1869 tasted previously, whereas there is nothing quite like the 1811 for intensity and youthfulness.) Relatively light tawny-amber. Extraordinary nose, at first perfectly ripe, warm raspberries and then heady vanilla cream. Beautifully balanced. Gentle. Delicate. Perfect texture. Nothing could be finer. 26 points [Bob Henry comment: Her point score exceeds her own 20 point scale.] and probably still climbing, although the 1811 will outlast it.

    Extraordinary in every way. Looked almost like black syrup, a PX, with gamboge rime. Smelt of treacle toffee and tea and moved like a thick treacle too. Very very sweet and concentrated. Certainly not fine but, amazingly, well balanced. A one-off. 23 points [Bob Henry comment: Her point score exceeds her own 20 point scale.] and nearly at its peak.

    1893 (recorked 1996)
    Very very deep brownish mahogany; looks thick and treacly. Correct nose of sturdy deep richness. Intense flavour of a much more conventionally massive build than the 1899. Lots of ripeness and length and potential. 19 points and still a long way to go.

    1899 (recorked 1994)
    Layered mahogany. No nose to begin with but delicate and somehow convincing. Lovely dancing delicate texture on the palate. Great sweetness counterbalanced by acidity. Not one of the pinnacles of this tasting but a gorgeous and extremely satisfying wine. 19 points and still improving.

    1900 (recorked 1900)
    Fox red of only medium intensity and a yellow-green rim. Sweet and heady with a slight hint of estufa on the nose. Light in weight and sweetness with a slightly dry end. 16 points and fading.

    1911 (recorked 1996)
    Hint of dark brown (as opposed to rich mahogany) in slightly lacklustre hue. Initially slightly mouldy but underneath a gorgeous bouquet of steeped raisins. Very, very sweet at first with notable acid at the end of the palate. Slightly spindly c.f. the 1811 and 1831 it was served with. NB recorking. 19 points and ready

    1931 (recorked 1997)
    Very clear, pale amber. Pure, clean, sharp but not especially intense nose. Quite lean and light with almost madeira-like acidity. With its less-than-usual charge of sweetness and exceptionally palate-rinsing-like crispness, this was the only wine that might have been difficult to recognise immediately as great Sauternes. It could almost have been a very old, light fortified wine. 17 points and on the way down

    Very very deep mahogany, extremely viscous. Yellow/green rim. Essence of rose petals on nose with something almost suggestive of oak. Rich. complex nose. Very intense flavour, extremely sweet - fuller and rounder than 1947 or 1949. Perfect texture, balancing acidity, and so much more than just sweet. 20 points and ready.

    As deep a mahogany as 1945 with similar development at rim. Smells creamy with hint of something vegetal and a floral topnote. Not as overwhelming sweet as either 1945 or 1949 but extremely youthful, lively and crisp. Could be great with nuts; less so with anything very sweet. Those who know the wine better than me were slightly disappointed by this bottle. 18 points and still considerable evolution to come.

    Deep tawny/amber with pale yellow rim. Scent of raisins, not as subtle a nose as the 1945 or 1947 with only medium intensity but.. on the palate a great thwack of purest raisin cream with great length of flavour. 19 points and not yet at peak.

  4. Part Four.

    From Jancis Robinson, MW Website
    (Posted September 1998):

    Notes From Attending an Yquem Vertical Tasting (Continued)

    Looks much less viscous and much paler than the three vintages above; deep gold with some amber highlights. Relatively lightweight on the nose but definite creme brulee. A hint of something not 100 per cent clean about this bottle. Creamy and sweet on the palate, very refreshing, long, could give enormous pleasure served in isolation; next to the heavyweights of the 1940s it looked very slightly lean. 17 points and still evolving.

    Lively, deep orange and tawny. Both grass and sweetness on the very intense nose. Very compete palate. Extremely long and complex with many reverberations. 18 points and still climbing.

    Deep tawny with brown notes. Intense nose with strong floral notes on Christmas pudding flavours. Full, round, rich, long but slightly brawny and drying out at the end. Not fine, rather aggressive and old, hint of maderisation. 17 points and going downhill.

    Deep tawny marmelade colour. Very slightly mousey to begin with on the nose. Palate very rich and extremely long, but a bit of dryness at the end. Not complete; a bit jagged. 16 points and probably near its peak.

    Lively colour of a ginger cat. Looks more like an Australian stickie than an Yquem. Smelt of ginger Edinburgh rock. Very unsubtle. 13 points; can't imagine evolution.

    Deep butterscotch colour. Rich creme brulee scent. Very very ful flavoured, quite brutal impact on the palate. This wine could become something splendid but for the moment is about 16 points.

    Pale tawny. Relatively simple, sugary nose. Lots of unresolved acidity on the palate. The wine may well improve in bottle but is an awkward. 14 points at the moment.

    1983 (imperiale)
    Deep apricot colour. Exotic nose of dried tropical fruit - mango? Gorgeous, full bodied, delightfully middle aged, between youthful and embryonic and fully blown. Long and powerful though a a drying hint of dried apple peel on the finish. 20 points and still a long way to go.

    1988 (double magnum)
    Very pale straw. Lovely pure botrytis notes. Youthful reminder of quite what a transformation bottle age is. Still relatively simple but very pleasurable. Sweet, uncomplicated, beguiling. 18.5 points with decades to go.

    1990 (magnum)
    Light youthful gold. Peachy smell redolent of botrytis. Also a hint of fine polished wood on the nose. Very long, firm, sleek, confident. Big and rich. 20 points and a long long way to go.

    Paler than 1990. Pale gold. Smells of very ripe pears. relatively simple and unevolved. Clean quite tart palate. Noticveably lighter bodied than 1990 -- not as pure and tingling either. 17 points and developing but not a great Yquem.

  5. Citing Robinson's reproduced reviews of older Yquem, what's more perfect than "perfect" (i.e., 20 points)?

    Robert Parker exhibited a different scoring gaffe when he granted an interview to Wine Times magazine (later renamed Wine Enthusiast magazine) in 1989.

    The subject was "glass ceilings" on scores awarded to varietals that -- in his opinion -- don't have the ability to improve with bottle age.

    See excerpts from a transcript of that interview below.

  6. WINE TIMES: How is your scoring system different from The Wine Spectator’s?

    PARKER: Theirs is really a different animal than mine, though if someone just looks at both of them, they are, quote, two 100-point systems. Theirs, in fact, is advertised as a 100-point system; mine from the very beginning is a 50-point system. If you start at 50 and go to 100, it is clear it’s a 50-point system, and it has always been clear. Mine is basically two 20-point systems with a 10-point cushion on top for wines that have the ability to age. …

    … The newsletter was always meant to be a guide, one person’s opinion. The scoring system was always meant to be an accessory to the written reviews, tasting notes. That’s why I use sentences and try and make it interesting. Reading is a lost skill in America. There’s a certain segment of my readers who only look at numbers, but I think it is a much smaller segment than most wine writers would like to believe. The tasting notes are one thing, but in order to communicate effectively and quickly where a wine placed vis-à-vis its peer group, a numerical scale was necessary. If I didn’t do that, it would have been a sort of cop-out.

    I thought one of the jokes of the 20-point systems is that everyone uses half points, so it’s really a 40-point system — which no one will acknowledge — and mine is a 50-point system, and in most cases a 40-point system.

    WINE TIMES: But how do you split the hairs between an 81 and an 83?

    PARKER: It’s a fairly methodical system. The wine gets up to 5 points on color, up to 15 on bouquet and aroma, and up to 20 points on flavor, harmony and length. And that gets you 40 points right there. And then the [ balance of ] 10 points are … simply awarded to wines that have the ability to improve in the bottle. This is sort of arbitrary and gets me into trouble.

    WINE TIMES: You mean when you are in the cellars of Burgundy, you look at a wine and say this is a 4 for color, a 14 for bouquet, and so on [ ? ]

    PARKER: Yes, most of the times. What happens is that I’ve done so many wines by now that I know virtually right away that it’s, say, upper 80s, and you sort of start working backwards. And color now is sort of an academic issue. The technology of color is refined and most color is fine. My system applies best to young wines because older wines, once they’ve passed their prime, end up getting lower scores.

    . . .

    WINE TIMES: Do you have a bias toward red wines? Why aren’t white wines getting as many scores in the upper 90s? Is it you or is it the wine?

    PARKER: Because of that 10-point cushion. Points are assigned to the overall quality but also to the potential period of time that wine can provide pleasure. And white Burgundies today have a lifespan of, at most, a decade with rare exceptions. Most top red wines can last 15 years and most top Bordeaux can last 20, 25 years. It’s a sign of the system that a great 1985 Morgon [ BEAUJOLAIS ] is not going to get 100 points because it’s not fair to the reader to equate a BEAUJOLAIS with a 1982 Mouton-Rothschild. You only have three or four years to drink the BEAUJOLAIS.

    WINE TIMES: In your system, what would be the highest rated BEAUJOLAIS?

    PARKER: 90. That would be a perfect BEAUJOLAIS, and I’ve never given one. I have given a lot of 87s and 88s.

    [Bob Henry's comment : In 1990, Parker awarded a score of 92 points to the 1989 vintage Georges Duboeuf "Jean Descombes" Morgon BEAUJOLAIS, contradicting his then year-old statement above.

    Fast forward to 2011: the stellar 2009 vintage cru BEAUJOLAISes garnered scores in the 91 to 94 point range from Wine Advocate.]

    WINE TIMES: So it’s the aging potential that is the key factor that gets a wine into the 90s.

    PARKER: Yes. And it goes back to how I evaluate vintages in general. To me the greatness of a vintage is assessed two ways: 1) its ability to provide great pleasure — wine provides, above all, pleasure; 2) the time period over which it can provide that pleasure.

    . . .

  7. Adding to David's bibliography . . .

    From The Wall Street Journal “Weekend” Section
    (November 20, 2009, Page W6):

    “A Hint of Hype, A Taste of Illusion;
    They pour, sip and, with passion and snobbery, glorify or doom wines.
    But studies say the wine-rating system is badly flawed.
    How the experts fare against a coin toss.”


    Essay by Leonard Mlodinow
    Lecturer at Caltech