Quantifying value-for-money wines - part 2

This is the second of a four-part set of blog posts looking at how we might identify value-for-money wines. The topic is not as simple as we might like.

Quantifying value-for-money wines - part 1
   — issues with quantifying value for money
Quantifying value-for-money wines - part 2
   — empirically comparing wines within a specified wine group
Quantifying value-for-money wines - part 3
   — formulae for assessing individual wines against a baseline wine
Quantifying value-for-money wines - part 4
   — empirically comparing wines across wine groups

Empirically comparing wines within a specified wine group

In the previous post I considered the basic issues with trying to quantify value for money as related to wine, which is usually expressed as a quality to price ratio (QPR). As Kim Brebach at the Best Wines Under $20 site has noted: "A $10 wine that scores 91 is a serious bargain, while a $50 wine that scores 91 would not get a recommendation from us. It is the relationship between score and price that is crucial."

Any QPR measure will only ever be an approximation, although hopefully it will be a useful one in practice. There are fundamental problems with quantifying wine quality, and I discussed how these can be overcome. However, the relationship between the price and quality of wines is non-linear, and therefore any simple QPR will be inappropriate.

There are two different approaches to dealing with this issue of non-linearity, which I discuss in this and the next post.

In this post I consider how to quantify QPR when we are looking at a dataset consisting of a group of wines of a similar type; for example, red Burgundy, California chardonnay, or Australian shiraz. The data will consist of values of both the quality and the price of each wine in the set. We are interested in identifying those wines within the dataset that have a good QPR score.

As a specific example, the first graph shows the data for 303 red Bordeaux wines from the 2000 vintage. Each point represents a single wine, located according to its quality score (horizontally) and its price (vertically). Are there any good QPR wines among these 303?

The data are taken from the September 2004 issue of the QPRwines newsletter, produced by Neil Monnens. This was the premier issue; the newsletter was later revamped as The Wine Blue Book, in 2007; and it sadly ceased publication in 2012.

The QPRwines index

Each issue of the QPRwines newsletter consisted of a compilation of QPR scores for a large set of wines; the contents of each issue are still listed at the Wine Lovers Page web site. Here I will describe the derivation of these QPR scores. [Note: Even though this newsletter is no longer published, the QPR method outlined here is still used by, for example, the Wine Peeps blog.]

The wine prices used in the QPRwines calculations were compiled from the Wine-Searcher web site, as explained in an interview Monnens gave to the Napa Valley Register. In the same interview, the compilation of the quality scores is described like this:

[Monnens] checks eight web sites that use the 100-point scoring method (he has subscriptions so he can have access to all the scores, but declines to reveal the sites). Since not all sites review the same wines, he must have a minimum of two scores for each wine in order to include that wine.

This is a robust method for dealing with the inherent variability of wine quality scores, as I discussed in the previous post (Part 1). However, I also noted that the scores need to be standardized for inter-assessor variability, before calculating their average, which may not have been done.

In the original press release for the newsletter (still available at the Vinography web site), the QPRwines index is described in this manner:

A wine's QPR is how much more or less it costs compared to the average price of similarly scored wines (critics' average scores). For example, the average price for a 93-rated 2000 Bordeaux is $119. But the 2000 Chateau Pontet-Canet from Pauillac costs only $50 — a number that is 42% of the average price for a 93-point wine. Pontet-Canet thus has a QPR of 42%.

So, the QPRwines index works in reverse, with lower values indicating good value for money.

This procedure is illustrated in the next graph for the Bordeaux wines, where I have added (in pink) the average wine price for each quality score. Wines below the average have good QPR (within that score group), while wines above it have poor QPR.

This method for quantifying QPR seems to be effective in general. In particular, it works well when there are a lot of wines in each quality-score group. However, it is important to note the QPR values are calculated separately for each group, and not all groups have many wines, particularly for the highest quality scores.

Thus, the graph makes clear the method's essential limitation — the price averages are not necessarily comparable between quality scores. That is, the average price for any given score may be higher or lower than for neighboring scores, so that there is no concept of a general relationship between price and quality. Each measurement of QPR thus depends solely on what other wines have been given the same score. It thus becomes possible to have every inconsistent results between adjacent categories (increasing the quality can actually decrease the QPR!).

This issue becomes particularly obvious when we consider the effect of the two luxury wines that are evident in the dataset (see the post on Luxury wines and the relationship of quality to price). These two wines (Château Petrus and Château Lafleur) have outrageous prices that are not comparable to any of the other wines. Importantly, they distort the calculation of the average price for their respective quality-score groups, by artificially raising the average price.

In turn, this has the effect of increasing the number of wines that will be identified as having a good QPR (ie. they will be a long way below the inflated average). For example, the cheapest wine in the 96-point group (Château Léoville Barton) is identified by the QPRwines index as having "Great value", and the second cheapest (Château L'Église Clinet) as having "Value", solely because the average price for their score-group is artificially high (due to the price of the luxury wine Château Lafleur).

An improved method

This issue can be dealt with by calculating the "average" prices simultaneously using all of the quality scores, not just the prices for each score separately. I have outlined this method in a previous blog post (The relationship of wine quality to price).

The basic idea is to recognize that the relationship between wine price and quality fits an exponential (or log-linear) model. This is a standard model in economics, which represents the idea that the prices are multiplied as the quality increases, and that the multiplier also increases with quality.

So, all we have to do is fit the exponential model to the dataset of wines, and we will have a line on the graph that simultaneously represents the average price for all quality scores (ie. the line represents QPR=1). This is shown in the next graph.

First, however, it is important to note that I have deleted the two luxury wines from the dataset. As I explained in the earlier post on the topic (Luxury wines and the relationship of quality to price), the price-quality relationship needs to be modeled separately for the luxury and non-luxury wines. (With only two wines, we cannot calculate QPR for the luxury wines here.)

So, wines below the pink dashes have high QPR under the QPRwines method, while wines below the solid line have high QPR for my version.

As you can see, the wines identified as being good value for money by the QPRwines method and my own differ only for those scores where a very high-priced wine has artificially raised the average price (eg. scores 93, 96, 98). The exponential model smooths out the averages, and allows a more equitable assessment of QPR across the quality range.

For example, the Château L'Église Clinet wine mentioned above as having QPR "Value" is actually above the line for the exponential-model average price, rather than below it — it has low QPR not high! My method thus identifies fewer wines as having a high QPR, but it does so in a more consistent manner, and is thus an improvement.

Where to from here?

Clearly, the empirical approach outlined here can only be used if you happen to have access to a database of prices and quality scores. This works if you are publishing a newsletter or a blog; but it is not very convenient if you are standing in a wine shop wondering which wine to buy.

In the next post I will discuss some QPR approaches that allow a single wine to be assessed based on very little information.