Fun With Statistics – Wine Ratings Edition

By Kevin Meyer

I consider myself lucky in many respects, not the least of which is the fact that I live in one of California's premier wine-producing regions with over 220 wineries within 30 miles.  Over the last several years I've been able to partake of many of their products and have refined my taste down to a very small subset.  Those of you that follow us on Twitter occasionally get to hear of my new favorites.

One tool that many vinophiles use is the 100 point wine scale, especially the one developed by Robert Parker.  The scale has an illustrious history.

It was in this climate that in the 1970s a lawyer-turned-wine-critic
named Robert M. Parker Jr. decided to aid consumers by assigning wines
a grade on a 100-point scale. Today, critics like Mr. Parker exert
enormous influence. The medals won at the 29 major U.S. wine
competitions medals are considered so influential that wineries spend
well over $1 million each year in entry fees. According to a 2001 study
of Bordeaux wines, a one-point bump in Robert Parker's wine ratings
averages equates to a 7% increase in price, and the price difference
can be much greater at the high end.

Pretty impressive how much power that dude has.  But I'm sure many of our more statistically inclined readers are already starting to wonder how something as subjective as taste and bouquet can be accurately determined, repeatably, to single percentage points. 

But what if the successive judgments of the same wine, by the same wine
expert, vary so widely that the ratings and medals on which wines base
their reputations are merely a powerful illusion? That is the
conclusion reached in two recent papers in the Journal of Wine
Economics.

Houston (or Napa), we have a problem.  Who is this heathen that dared question Robert Parker and his ilk?

Both articles were authored by the same man, a unique blend of
winemaker, scientist and statistician. The unlikely revolutionary is a
soft-spoken fellow named Robert Hodgson, a retired professor who taught
statistics at Humboldt State University. Since 1976, Mr. Hodgson has
also been the proprietor of Fieldbrook Winery, a small operation that
puts out about 10 wines each year, selling 1,500 cases.

He started off with a statistical analysis of judges and the judging process.

In his first study, each year, for four years, Mr. Hodgson served
actual panels of California State Fair Wine Competition judges—some 70
judges each year—about 100 wines over a two-day period. He employed the
same blind tasting process as the actual competition. In Mr. Hodgson's
study, however, every wine was presented to each judge three different
times, each time drawn from the same bottle.

The results astonished Mr. Hodgson. The judges' wine ratings
typically varied by ±4 points on a standard ratings scale running from
80 to 100.  Mr. Hodgson also found that the judges whose ratings were most
consistent in any given year landed in the middle of the pack in other
years, suggesting that their consistent performance that year had
simply been due to chance.

Uh oh.  So then what?  He took on the medal process.

This time,
from a private newsletter called The California Grapevine, he obtained
the complete records of wine competitions, listing not only which wines
won medals, but which did not. Mr. Hodgson told me that when he started
playing with the data he "noticed that the probability that a wine
which won a gold medal in one competition would win nothing in others
was high." The medals seemed to be spread around at random, with each
wine having about a 9% chance of winning a gold medal in any given
competition.

To test that idea, Mr. Hodgson restricted his attention to wines
entering a certain number of competitions, say five. Then he made a bar
graph of the number of wines winning 0, 1, 2, etc. gold medals in those
competitions. The graph was nearly identical to the one you'd get if
you simply made five flips of a coin weighted to land on heads with a
probability of 9%. The distribution of medals, he wrote, "mirrors what
might be expected should a gold medal be awarded by chance alone."

Now that a substantial part of the framework of our polite society has come crashing down, what should we do?

One answer would be to do more experimenting, and to be more
price-sensitive, refusing to pay for medals and ratings points. Another
tack is to continue to rely on the medals and ratings, adopting an
approach often attributed to physicist Neils Bohr, who was said to have
had a horseshoe hanging over his office door for good luck. When asked
how a physicist could believe in such things, he said, "I am told it
works even if you don't believe in it." Or you could just shrug and
embrace the attitude of Julia Child, who, when asked what was her
favorite wine, replied "gin."

As for me, I have always believed in the advice given by famed food
critic Waverly Root, who recommended that one simply "Drink wine every
day, at lunch and dinner, and the rest will take care of itself."

Bottom line: just try some wines.  Preferably a lot of different wines.  And figure out what you like and dislike.  Then buy more of it.  Who cares about points and medals and especially the supposed value correlation with high price?  Just enjoy.