My March 21 post on this topic generated an amazing amount of interest. I guess it didn’t hurt that a blurb about that post made the front page of Slashdot – and thus conferring near-god-like status on me according to my most geeky friends (“Chris – you’ve been slashdotted!”). The Slashdot post generated over 130 comments, and the post on my blog hit a record 35 comments as of today. Wow. Who’d of thought that a challenge to the beer-drinking habits of scientists would generate such interest?

To review, Czech ornithologist Tomas Grim published a paper showing a negative correlation between social beer drinking and scientific output. I then posted a critique of that study explaining why I thought that the data did not support his conclusions.

Many of the comments I received were fun, some were silly, and still others were just plain weird. Some comments made suggestions for future work (yes, I am considering carefully whether I should conduct my own study), while others criticized my analysis. I think some of these critiques are worth responding to.

Some people pointed out the obvious cause-and-effect relationship between drunkenness and poor thinking skills. Isn’t it obvious that drinking would lead to bad science? However, the paper gives enough information to conclude that the average rate of beer drinking among studied subjects is not much different from the Czech average – less than one pint (0.5 liters) per day. Although the raw data is not presented (to protect the identities of the subjects, according to the author), it seems unlikely that the heaviest drinkers in the study rose to the level of alcohol abusers. And since the survey only asked about average weekly beers consumed, there is no data on binging or even if any of the heaviest beer drinkers got a buzz!

Several people caught my error where I referred to R-squared as the “correlation coefficient”. I definitely deserve a reduction in my beer rations for that bit of sloppiness. R-squared (or R^2) is often called the “coefficient of determination” and, for the case of least-squares linear regression, R-squared is equal to the square of the correlation coefficient (r).

Several people jumped on my case of claiming that the five lowest scientific output data points were outliers that should be removed from the analysis. I never used the term outlier, and I never said they should be removed from the analysis. Instead, I claimed that these data points were extremely influential (without them the correlation nearly disappears). Properly dealing with outliers is extremely tricky and I’d be surprised if 1 in a 100 scientists could do it well. But understanding how to handle highly influential data points (and what that means for the interpretation of the fitting results) is even harder, and is almost always ignored. Pointing out that five highly influential beer guzzlers controlled the fit is very important when trying to draw conclusions that apply to tens of thousands of scientists.

Which brings me back to the main point of my critique of Dr. Grim’s study – the low number of subjects. I mentioned that there were 34 data points – but this is actually an exaggeration. Grim made two surveys of beer drinking habits, one in 2002 with 18 responses and a second one 2006 with 16 responses. Thus, there were 34 data points. But the people being surveyed were the same! Thus, the total number of people involved in the study looks to be 18. (I wonder which data point(s) belong to Dr. Grim.) The analysis of the 2002 data showed an R^2 of 0.34, while the 2006 survey showed an R^2 of 0.52. The correlation coefficients are just barely large enough to claim with 95% confidence that the results are statistically different from an R^2 of 0 (no correlation) – assuming all confounding factors have been properly eliminated.

But that’s the rub. There is no way to know whether all confounding factors (read systematic errors) have been eliminated. And it seems unlikely, as I mentioned before, that one can plausibly claim that the data points are independent, since it seems likely that at least some of this small group of Czech ornithologists socialized together. Thus, a low R^2 for a study with a small number of data points on a subject of great complexity is unlikely to be very revealing despite the most rigorous of statistical treatments.

Interestingly, the most revealing thing I learned in the past week about this study came from Dr. Grim’s website. The page on his beer study is labeled “Beer vs science – first laugh, then think (what to drink:-)”, which makes me suspect he is angling for an Ig Noble award. He posts his now-infamous paper for download, as well as giving links to many of the press reports about his work. Then come a collection of pictures showing that Dr. Grim’s data point would not lie on the low beer consumption side of the graph. If nothing else, it seems that Dr. Grim is having fun. Good for him!

By the way, I did not drink a single beer while I wrote this post. Only scotch.