Differing opinions of amateurs at the same wine tasting

I have previously written about a direct comparison of two professional wine writers tasting the same wines at the same time (Laube versus Suckling — their scores differ, but what does that mean for us?). This begs the question of what happens at wine tastings attended by several people, especially if they are interested amateurs rather than wine professionals. This is a trickier question to address, because of the paucity of published data. However, I will look at one possible dataset here.

On 20 January 2014, at the Ripple Restaurant, in Washington DC, there was a vertical tasting of wines from Château Calon Ségur (located in Bordeaux), for 16 vintages from 1982-2010. This was attended by a number of people, three of whom have put their quality score for each wine online:
  Panos Kakaviatos (Calon Segur 1982-2010: first ever promotional tasting in the US)
  Aaron Nix-Gomez (The Calon Segur vertical 2010-1982)
  Kevin Shin (dcwino on CellarTracker)
We can try to compare these scores.

Furthermore, we can also try to compare these scores to those from a professional wine writer. On 6 November 2013, there was another tasting of many of the same wines, at the Carré des Feuillants restaurant, in Paris. From this tasting, Jane Anson has also put her quality score for each wine online (Chateau Calon Ségur: retrospective 1995-2011). These can be added to our comparison, given that the tastings were only a few weeks apart.

As I have done in previous blog posts (eg. How large is between-critic variation in quality scores?), we can quantify the relationships among the scores using correlation analysis between pairs of tasters. This measures the percentage of the data variation that is shared in common between those tasters — the larger the percentage then the better the agreement there is among the quality scores of the wines tasted. This is shown in the table for all six possible pairs.

There is quite a reasonable degree of agreement among the three wine-interested amateurs, especially Kevin Shin and Aaron Nix-Gomez. Indeed, these percentages are higher than the level I observed among the professional critics in the post cited above (often only 10-40%). Perhaps amateurs are less determinedly different from each other than are professionals?

Notably, the comparisons between these amateurs and the professional (Jane Anson) show much lower agreement. This may reflect the bottles opened at the two different tastings; but the values are certainly in accord with those found among other professionals.

It might be also be useful to look at a picture of these data, rather than a table of numbers. To do this, we can employ a network, as I used in the post on professionals cited above. This is shown in the graph.

The important point for interpreting the graph is that the length of the lines represents how similar are the wine-quality scores. The lines in the center represent the shared similarity among the scores, while the lines leading to the names represent the variation in the scores that is unique to that person.

What this network says is that very little of the variation in quality scores is agreed upon by the four people, and that they each have their own personal opinion, which differs notably. In this case, there is little consensus on the quality of the different vintages.

So, amateurs may be somewhat different from professionals, but they still go their own way. Wine quality is apparently not a shared experience.