When critics disagree

Many people have presented themselves as assessors of wine quality, whether they be wine professionals, competition judges, or interested amateurs (with a blog). In general, it is usually accepted that wine assessments are highly correlated, so that high-scoring wines for one assessor are also high-scoring wines for other assessors. This is a topic that I will address in a future post.

In the meantime, it is interesting to consider situations where the assessors disagree. Some of these disagreements are legendary, while others are more prosaic. They usually involve a single vintage of a single wine — for example, the well-known disagreement between Jancis Robinson and Robert Parker over the 2003 Château Pavie (Bordeaux).

However, disagreements can be larger than this, and the example I am presenting here is one of the more thorough ones that I have encountered.

The wine in question is Palari Faro, from Sicily. The Palari winery was at the forefront of the renaissance of Sicilian wine-making in the 1990s; and their top wine, Faro, is considered by several commentators to be the best Sicilian red wine.

A few years ago (June 2013, to be precise) I was considering buying a bottle or two, and I wondered which vintage I should buy. So, I looked at what the online commentators had to suggest. I found two sites that seemed to be in thorough disagreement: that of Walter Speller and that of Doctor Wine.

Their scores for the 14 vintages from 1996 to 2009, inclusive, are shown in the first graph. Each point represents one vintage, located based on the two assessments that they each received.

The graph shows almost no relationship between the two sets of scores — they neither increase in synchrony nor decrease in contradiction. Mathematically, the correlation coefficient is –0.043, which is as close to zero as makes no odds. (For the cognoscenti, the coefficient of determination is a miserable 0.2%.)

Since the two assessors used different scales (20-point versus 100-point), the next graph adds a line that shows the nominal relationship between the two scales. If the two critics agreed exactly, then the points would all lie nearly on the line.

This shows that only 2 of the 14 vintages have been given comparable scores by the assessors (ie. they lie near the line). Indeed, the Doctor Wine scores often seem to be somewhat higher than the Speller scores (ie. they deviate from the line further to the right than to the left). Only one wine is rated highly by both assessors (in the top-right of the graph) — it turns out to be the 1998 vintage.

I do not attempt to explain this situation. However, you can imagine that I did not find it helpful for choosing a wine to buy. The only other vintage scores readily available for this wine are from the Wine Enthusiast and CellarTracker websites, but their scores cover only the second half of the time period. During that time, both of these score sets are moderately correlated with the Doctor Wine scores, rather than with the Speller scores.

Finally, it is worth noting that neither of the two web sites linked above seems to have their Faro reviews available any more. I happened to download them at a unique time in internet history, apparently. Most of the scores are still available from the Wine-Searcher website, with their Speller scores coming from the Jancis Robinson website.