Having multiple expert wine assessments is important

One thing for which the wine industry has often been looked down upon is having Experts, who tell us which wines are Good. Indeed, we even have recognized International Experts, whose pronouncements can have a big influence on wine sales (eg. in my lifetime, Jancis Robinson in the UK, or Robert Parker in the USA). However, this idea hits a brick wall when we try to do factual research experiments in the wine industry.

Scientists insist upon what they call Replication, which means attempting to measure the same thing several times independently. That last word is important. It means that, if a wine is tasted in a scientific experiment, then multiple people must taste it, and the experimental result will be a combination of all of these tastings, not just the pronouncement of any one person.

This leads me to ask: which has the biggest affect on the experimental outcome of assessing wines — differences among the people or differences among the wines? Let’s look at a specific example.

The example that I will look at here was published as: The scent of the fly (Journal of Chemical Ecology 44: 431–435, 2018). This looks at the idea that a container of wine can be spoiled by even a single vinegar fly (Drosophila melanogaster). Six different experiments were conducted, each comparing two different “treatments” that might affect the smell of a fly in a wine glass (including the chemical associated with the flies, called Z4-11Al).

To address this topic, the people involved in assessing the wine were:

Eight members (comprised of two women and six men) of the sensory panel for organoleptic tests for the wine-growing area of Baden [Germany] evaluated the odor of D. melanogaster and synthetic Z4-11Al. Members of this panel have been trained and selected for the official quality assessment of wines produced in Baden, at the Federal Institute for Viticulture, Freiburg, Germany. Each test comprised three glasses, a control and two treatments, which were presented in random order. The panel was asked to score odor intensity, ranging from 1 (weak, silent) to 9 (strong, loud), and to comment on odor quality.

So, in this case we can have a look at how much variation there was in the final set of smell scores due to both: (i) the different experimental treatments, and (ii) the different people doing the sniffing. I won’t bore you with the details, but the following graph shows you the Maximum / Average / Minimum difference in scores for the people when each of them compared the same two treatments for each experiment.

Note that for Experiments 4 and 5 there is a big range of scores between the panelists, compared to Experiments 1, 2 and 3, for example, and especially compared to Experiment 6. Apparently, there can be quite some differences between what the people smelled, each time. So, replication of the people was clearly important in this experiment.

However, what we are actually most interested in is whether there was any consistency as to which panelists differed from the others. That is, did any one panelist, for example, consistently give higher or lower scores than the other panelists?

The answer here is: No. Once again, I will not bore you with the details, but a Two-Factor Analysis of Variance (with no replication)* shows that 85.2% of the experimental variation is associated with the variation between the experimental treatments, and only 6.5% is associated with any variation between the panelists. This is the sort of result that scientists like (ie. consistency)! **

I chose this experiment as my example simply because it came along recently in my reading. Nevertheless, it does show that replicate people are important to have in experiments, because they are different from each other, but that they are not always different in any consistent way. This applies just as much to experiments in the wine industry as to anywhere else.

We might all like it if the same could be said of so-called Wine Experts, in general. To me, there is nothing more useless than wine advice from someone whose tastes consistently do not match my own. So, since I have a science background, I always insist that at least two Experts must pronounce a wine to be good value-for-money before I will consider purchasing it (see my post: Calculating value for money wines).

---

* See LibreTexts Statistics for an explanation.

** The authors’ final conclusion is that that their experiments: “corroborate the observation that a glass of wine is spoilt by a single D. melanogaster fly falling into it, which we here show is caused by Z4-11Al.”