Monday, March 1, 2021

Do different wine critics agree on value for money?

There are only two things I really need to know about a bottle of wine that I am thinking of buying: 1. Is it good value for money? 2. When should I drink it? Let's look at the first question in this post. (Other features, such as possible food pairings, I can work out later.)

I have written several times about evaluating value-for-money, when searching for wines (see "Value for money" under the "Labels For Posts" list at the right of this page. One simple idea is to compare the current bottle price with a quality score from one or more wine commentators.

We all know that different wine tasters can come up with very different opinions (and scores).* However, a somewhat different question is whether they agree on which wines are value for money, irrespective of the actual score they give. This involves the relationship between their points and the cost of the wine — high points for low cost = good value for money.

It therefore becomes relevant to ask whether we would get the same outcome depending on which commentator we choose. The degree of between-commentator variation is another topic that I have discussed before (eg. Are wine scores from different reviewers correlated with each other?  Does the relationship between critics' scores differ by wine type?).

So, it seems reasonable to wonder whether the assessment of value-for-money also differs between commentators. Note that value-for-money is not directly related to price, as expensive wines often get high scores from both professionals and amateurs — these wines may or may not be worth that expense.

To answer this question, I will be comparing two sources of wine scores available here in Sweden, which I have used in posts before: Jack Jakobsson at BKWine Magazine, and Johan Edström at Vinbanken. These commentators are discussed in the post on: Are wine scores from different reviewers correlated with each other?

Here, I will be using the procedure that I outlined in the post on: Calculating value for money wines. For new wines, both of our commentators produce scores out of 20, representing increasing quality, and also report the (standard) bottle prices (in Swedish kronor, SEK). We can simply plot price versus score; and we can calculate the financial cost of each quality point (ie. divide kronor by points, to get a kronor-per-point value).

So, I have gathered the data for last year (2020) from both sources. That particular year was a bit disturbed, of course, resulting in a few data anomalies (see my post How close are repeated wine-quality scores?). However, this should not affect my analysis (when different scores appeared, I simply averaged them).

This process yielded data for 589 wines from Edström and 512 wines from Jakobsson, with 487 of the wines being in common (the commentators could not get to taste all of the released wines, due to Covid-19 restrictions). We could start our analysis by simply looking at the relationship between the scores, for the shared wines. This is shown in the first graph, with each point representing one wine. The pink line would represent equal scores for each wine, while the black line shows the actual relationship.

Score comparison

This shows that the Edström scores are generally higher than the Jakobsson scores, for the same wines. Furthermore, only two-fifths (39%) of the scores are related to each other. This emphasizes that the quality scores themselves do differ between the two critics.

However, our question is about value for money, not the scores themselves. This is plotted in the next graph, encompassing all of the wines. Each point still represents one wine, showing its price (vertically, logarithm scale) and its score (horizontally). The Jakobsson scores are in blue and the Edström scores are in pink. The lines represent the two exponential relationships between price and score. Last year, $US 1 ≈ 9 SEK.

Value for money comparison

As expected, the Edström points lie to the right of the Jakobsson ones, due to the higher scores. However, the relationship between price and score is not that different between the two critics — about half of the variation in price is related to the score (higher scores generate higher prices). So, it seems that the two critics do have roughly the same opinions in terms of the wines' value for money.

We can look at this directly by calculating the number of kronor that each wine cost per quality point assigned. This is plotted in the next graph, for the shared wines. As usual, each point represents one wine, with the pink line representing equal assessment of value for money by the two critics, and the black line showing the actual relationship.

Value for money direct comparison

As you can see, the two critics do, indeed, have almost equal assessments. That is, in spite of their differences in scoring, their assessment of value for money is very similar.

This is very good for me, of course, as a consumer. I can use either critic to identify wines for purchase, if I am interested in trying new wines that are identified as being of particularly good value. This is, of course what I do do — I buy wines that I know and like, and I also buy new wines to try. As noted in the previous post, my preferred "wine recommendation system" is actually people, not a computer (Wine recommenders, and the tyranny of choice). In this case, it will make little difference which person I choose (I just need to allow for the difference in scoring). The best-value wines are simply those in the bottom-left corner of the final graph.

* Consider these two reviews, for the Château la Brande Tradition Cabernet Franc 2018 (Fronsac):
Christian Stojkovic:
Full-bodied, great aroma with ripe fruit of cherries, black currants, plums, herbs, oak and coffee, and a good, long aftertaste. In a really good vintage like 2018, Bordeaux can deliver such a good wine for relatively little money. 17 points.
Jack Jakobsson:
Slightly wet dog, young, floral with a little hyacinth, acidic. 14.5 points.


  1. As David demonstrated in two earlier posts . . .

    "Laube versus Suckling -- their scores differ, but what does that mean for us?"


    -- and --

    "How much difference can there be between critics?"


    . . . wine reviewers can "agree to disagree."

    However, if these reviewers are faithfully using the same scoring/rating system (above: Wine Spectator magazine's 100-point scale), then one would "think" those disagreements should be infrequent in occasion and narrow in scope.

    Another example of when two reviewers from the same publication "agreed to disagree" occurred between Neal Martin and Robert Parker.

    "Liv-ex Interviews Neal Martin, Part One"



    (Preface:) Neal Martin founded the independent website Wine-Journal in 2003, before joining Robert Parker’s The Wine Advocate three years later. He reviews wines for regions including Burgundy, South Africa and Sauternes, and in 2015 he took over the role of reviewing Bordeaux En Primeur from Parker.

    Question: "How do you expect a consumer to react to a wine that has two very different tasting notes and scores -- for example Larcis Ducasse 2005 [with scores of 87 points from Neal Martin and 100 points from Robert Parker]?"

    Answer: "It’s up for a consumer to decide. It could be the other way round -- I like Grand Puy Lacoste 2005 more. Consumers want somebody to be consistent. If I had to score like someone else then people would see right through that straight away. They know if I really like a wine where it’s going to be, just the same as Bob, Lisa and everybody else. The important thing is for each reviewer to be consistent. But there’s not a single critic in the world who is so consistent that every time they taste a wine they’re going to give it exactly the same score. People say wine is subjective: I completely agree. Anyone who says, 'Yes, I tasted it twenty times and every single time it was exactly the same' -- that’s not reality, thank God. Part of the reason why we love wine is that we have our favourites and we never quite know what’s going to happen when we pull the cork."

    Bob Henry's observation: that is a huge difference between a wine that scores 100 point ("Perfect"), and one that scores 87 points ("Good: a solid, well-made wine.").

    Those contested 13 points (between 87 and 100) are nearly equal to all of The Wine Advocate's rating system's 15 points assigned to bouquet and aroma.

    Quoting from a 1989 interview that Parker granted Wine Times magazine (later rebranded Wine Enthusiast magazine):

    WINE TIMES: "But how do you split the hairs between an 81 and an 83 [points]?"

    PARKER: "It's a fairly methodical system. The wine gets up to 5 points on color, up to 15 on bouquet and aroma, and up to 20 points on flavor, harmony and length. And that gets you 40 points right there. And then the [balance of] 10 points are . . . simply awarded to wines that have the ability to improve in the bottle. This is sort of arbitrary and gets me into trouble."

    WINE TIMES: "You mean when you are in the cellars of Burgundy, you look at a wine and say this is a 4 for color, a 14 for bouquet, and so on [?]"

    PARKER: "Yes, most of the times. What happens is that I've done so many wines by now that I know virtually right away that it's, say, upper 80s, and you sort of start working backwards. . . ."

  2. David:

    An interesting calculation would be to see the minimum and maximum number of kronors it takes to procure a bottle whose score is:

    17 points

    18 points

    19 points

    (There are no 20 point scored wines in the graph above.  It appears that 19.5 was the highest score attained.)

    ~~ Bob

    1. The minimum and maximum could be any price. But the 5% and 95% prices are:

      17 points 200 kr 550 kr
      18 points 280 kr 780 kr
      19 points 400 kr 1100 kr

      No-one in Sweden gives 20 points!