In my last two posts on this blog, I commented on a recent scientific publication:

Tomáš Grim, “A possible role of social activity to explain differences in publication output among ecologists”, Oikos, OnlineEarly Articles, 8-Feb-2008.

The article found a correlation between the weekly drinking habits of Czech avian ecologists and their scientific publication output. I was pleased to see that Dr. Grim, the author of this study, commented on both of my posts. Here, I will address most of his comments and give a further critique of his paper.

I complained in my first post that correlation does not imply causation. Dr. Grim correctly pointed out that the word “‘causes’ or ‘causation’ does not appear in my paper. I am not that stupid.” He then suggested that I “read more carefully, please.” But I have read his paper very carefully. And while Dr. Grim is most definitely smart and refrained from using the term ‘causation’, his theory of causation is evident throughout the paper. He states in his paper that “human cognitive performance during and after drinking is decreased”, that it “significantly decrease[s] cooperativeness” and the “effects of alcohol use are well known to decrease mental and working performance in general”. Dr. Grim offers no other theory to explain the observed correlation (even though there are many), and in fact his idea of causation is evident from the title of his paper. His recent comments on this blog also confirm his adherence to one causation theory, adding that hangovers impede productivity (even though his survey provided no evidence of drunkenness or hangovers on the part of the participants). Thus, my complaint that Dr. Grim’s observed correlation is insufficient evidence to support his theory of causation is entirely relevant, despite Dr. Grim’s protests.

Dr. Grim criticized my speculations about the nature of the raw data for his study, saying that I “just do not know” and that my speculations were wrong. I accept these criticisms. A bigger complaint, however, is that I must speculate at all. Dr. Grim does not provide any of the raw data for his study, instead showing only transformed data (without publishing transformation parameters). He provides virtually no descriptions of the data either – no means or ranges of either publication rates or drinking habits. As a result, it is impossible to determine the importance of the resulting correlation: does drinking at twice the mean rate correlate with a 5% decrease in scientific output, or a 50% decrease? In other words, how big is the effect? To my mind, that would seem important – and in fact the whole point. Unfortunately, readers are given no choice but to speculate (and possibly to hear Dr. Grim tell them how much they don’t know). Dr. Grim’s defense to this criticism – that a promise of confidentially requires that he keep the data secret – simply points out a basic flaw in the design of his study. His choice to survey only his fellow Czech avian ecologists may have made for good barroom banter among his buddies, but did nothing to benefit science.

I thank Dr. Grim for clearing up my misunderstanding about the number of participants in the 2006 study (34 rather than 16). Figure 1 in his paper labels 18 data points as “included in the first survey 2002” and as “past”, while 16 data points are labeled “included in 2006” and “present”. From such labeling it is easy to misinterpret the relationship between the data points in the two surveys. In fact, I’m still confused. Are the 34 data points in Figure 1 all from the 2006 survey? What does “past” with n = 18 and “present” with n = 16 refer to? Are the 34 total data points for 34 different people, but some surveyed in 2002 and others in 2006? I’m trying, but I just can’t “[s]ee the bloody Fig. 1 properly, man!” Perhaps a more complete explanation would have made the paper clearer.

I described the correlation coefficients for his 2002 and 2006 survey results as “barely large enough to claim with 95% confidence that the results are statistically different from an R^2 of 0 (no correlation)”, though my confusion as to the number of data points in the 2006 study would seem to invalidate my analysis for that correlation coefficient. In somewhat colorful language, Dr. Grim called into question my statistical abilities. His advice to me: “PLEASE, read some statistical textbook first”. While I certainly don’t consider myself a leading expert in statistical analysis, I have read a statistical textbook or two over the years. In fact, I read quite a few in 2006 when I taught a graduate-level statistics course at the University of Notre Dame. I recall teaching my students about the Student’s t-test to compare the means of two normally distributed populations each having the same estimated variance, and about Fisher’s z-transform to transform a distribution of correlation coefficients into something approximately normal, so that the Student’s t-test can be used. For a sample size of 18, the R^2 must be greater than 0.25 to be different from 0 with 95% confidence. The 2002 study showed values of R^2 just barely larger than this, as I had stated previously.

I also recall teaching my students that statistics are often the last refuge of the mediocre scientist. When systematic error (bias in the data) is greater than random error, the use of statistics is a waste of time. Worse, in that situation the use of statistical analysis can give a false sense that one’s conclusions are scientifically rigorous. So while an R^2 of 0.5 and a sample size of 34 may be statistically significant in the absence of bias in the data, one must do more than enter numbers into JMP if one wishes to understand whether the correlation is scientifically significant. So let’s look at Dr. Grim’s study a little more closely to see whether it is likely to be free of significant bias.

For a study such as Dr. Grim’s, there are a number of opportunities to introduce bias into the data. Here I’ll address the three most important:

1. Survey bias. As anyone who has seen political parties trot out competing polls showing significantly different results knows, it is easy to bias survey results by the way surveys are written or administered. Does Dr. Grim have experience in the proper design and administration of human behavior surveys? Was his survey vetted by other scientists with such experience? Considering the fact that many (if not most) of the survey participants were likely friends or colleagues of Dr. Grim, and that the subject matter of the survey (social drinking) is a sensitive topic on both personal and professional levels, the possibility for bias in the responses is extreme. It doesn’t take an expert in human behavioral research to see that his survey does not pass the smell test. (As an aside, a typical ethical requirement for human behavioral studies would prohibit the use of subjects with any significant personal or professional ties to the researchers, especially if those ties are not disclosed.)
2. Sampling bias. When one cannot measure an entire population, one must sample that population and draw inferences about the population from the sample. If the sample is not representative of the population, sampling bias can invalidate any conclusions drawn from the study. So what was the population under study by Dr. Grim? In his comments on this blog, Dr. Grim claims that the population under study were Czech avian ecologists (of which, according to Dr. Grim, there are 38). If this were the case, then Dr. Grim’s study would be interesting to, oh, about 38 people. Hardly the kind of thing that gets coverage in the New York Times. In fact, Dr. Grim’s paper implies a much greater population is at play. The title of the paper claims application to all ecologists, and the text of the paper contains more than a dozen references to science, scientists, and scientific productivity unqualified by the Czech avian ecologist subgroup. The abstract of the paper discusses science and scientists in general, without reference to a narrower population. His hypothesis under test is clearly stated in the abstract: “I predicted negative correlations between beer consumption and several measures of scientific performance.” The penultimate paragraph of the paper is filled with wide-ranging conjectures about the implications of this study to the lives and careers of scientists (including the bizarre speculation that social drinking may impact the “biological success” of scientists as well). Dr. Grim’s statement that the population under study is limited to Czech researchers studying avian evolutionary biology and behavioral ecology is simply disingenuous. His sample was limited to Czech avian ecologists, and is thus horrendously biased compared to the population of all ecologists or all scientists. Simply put, if his population is all Czech avian ecologists, then his study is essentially worthless to the greater population of scientists, and if his population is the greater population of scientists, then his study is essentially worthless due to extreme sampling bias. Either way, the conclusion is the same.
3. Confounding factors. When factors outside of the control of the researcher have significant influence on the resulting correlation, even a statistically justified correlation may lead to no insight as to cause and effect (and in fact may mislead). Did Dr. Grim’s study identify all of the significant confounding factors? This is always one of the most difficult questions facing the researcher of human behavior.

There are other criticisms of this study as well, such as the lack of independence of the data points (thus invalidating essentially all of Dr. Grim’s statistical analysis), and of course the use of a very small sample size to draw sweeping conclusions about human behavior. Dr. Grim’s defense that a “sample of that magnitude (34) is quite good for an ecological study” is of little importance since this was not an ecological study but a human behavior study, involving extremely complex human behavior at that. His final fallback, that “[e]ven just stating the logic of a hypothesis without any empirical data is worth publishing”, is more than a stretch. I think a strategy of trying to publish hypotheses without any empirical data would have a bigger impact on a scientist’s journal publication rate than beer drinking ever could.

But all of these criticisms boil down to one simple fact: Dr. Grim surveyed a small number of fellow Czech ornithologists in what was probably great fun, but definitely bad science. Dr. Grim himself described his paper as “half-joke-half-study”. Now, I’m the first person to appreciate a good joke, especially at the expense of us scientists. However, Dr. Grim failed to mention the “half-joke” part of the equation in his published paper, preferring instead to pass off his bit of fun as real science. As such, publication of his paper in a scientific journal borders on the unethical. Fortunately for Dr. Grim, he chose to publish his paper in an ecology journal, rather than one specializing in human behavioral studies, where it is likely that his reviewers would have been less kind. Unfortunately for Dr. Grim, he must now own up to his joke or face digging an even deeper hole for himself.

My March 21 post on this topic generated an amazing amount of interest. I guess it didn’t hurt that a blurb about that post made the front page of Slashdot – and thus conferring near-god-like status on me according to my most geeky friends (“Chris – you’ve been slashdotted!”). The Slashdot post generated over 130 comments, and the post on my blog hit a record 35 comments as of today. Wow. Who’d of thought that a challenge to the beer-drinking habits of scientists would generate such interest?

To review, Czech ornithologist Tomas Grim published a paper showing a negative correlation between social beer drinking and scientific output. I then posted a critique of that study explaining why I thought that the data did not support his conclusions.

Many of the comments I received were fun, some were silly, and still others were just plain weird. Some comments made suggestions for future work (yes, I am considering carefully whether I should conduct my own study), while others criticized my analysis. I think some of these critiques are worth responding to.

Some people pointed out the obvious cause-and-effect relationship between drunkenness and poor thinking skills. Isn’t it obvious that drinking would lead to bad science? However, the paper gives enough information to conclude that the average rate of beer drinking among studied subjects is not much different from the Czech average – less than one pint (0.5 liters) per day. Although the raw data is not presented (to protect the identities of the subjects, according to the author), it seems unlikely that the heaviest drinkers in the study rose to the level of alcohol abusers. And since the survey only asked about average weekly beers consumed, there is no data on binging or even if any of the heaviest beer drinkers got a buzz!

Several people caught my error where I referred to R-squared as the “correlation coefficient”. I definitely deserve a reduction in my beer rations for that bit of sloppiness. R-squared (or R^2) is often called the “coefficient of determination” and, for the case of least-squares linear regression, R-squared is equal to the square of the correlation coefficient (r).

Several people jumped on my case of claiming that the five lowest scientific output data points were outliers that should be removed from the analysis. I never used the term outlier, and I never said they should be removed from the analysis. Instead, I claimed that these data points were extremely influential (without them the correlation nearly disappears). Properly dealing with outliers is extremely tricky and I’d be surprised if 1 in a 100 scientists could do it well. But understanding how to handle highly influential data points (and what that means for the interpretation of the fitting results) is even harder, and is almost always ignored. Pointing out that five highly influential beer guzzlers controlled the fit is very important when trying to draw conclusions that apply to tens of thousands of scientists.

Which brings me back to the main point of my critique of Dr. Grim’s study – the low number of subjects. I mentioned that there were 34 data points – but this is actually an exaggeration. Grim made two surveys of beer drinking habits, one in 2002 with 18 responses and a second one 2006 with 16 responses. Thus, there were 34 data points. But the people being surveyed were the same! Thus, the total number of people involved in the study looks to be 18. (I wonder which data point(s) belong to Dr. Grim.) The analysis of the 2002 data showed an R^2 of 0.34, while the 2006 survey showed an R^2 of 0.52. The correlation coefficients are just barely large enough to claim with 95% confidence that the results are statistically different from an R^2 of 0 (no correlation) – assuming all confounding factors have been properly eliminated.

But that’s the rub. There is no way to know whether all confounding factors (read systematic errors) have been eliminated. And it seems unlikely, as I mentioned before, that one can plausibly claim that the data points are independent, since it seems likely that at least some of this small group of Czech ornithologists socialized together. Thus, a low R^2 for a study with a small number of data points on a subject of great complexity is unlikely to be very revealing despite the most rigorous of statistical treatments.

Interestingly, the most revealing thing I learned in the past week about this study came from Dr. Grim’s website. The page on his beer study is labeled “Beer vs science – first laugh, then think (what to drink:-)”, which makes me suspect he is angling for an Ig Noble award. He posts his now-infamous paper for download, as well as giving links to many of the press reports about his work. Then come a collection of pictures showing that Dr. Grim’s data point would not lie on the low beer consumption side of the graph. If nothing else, it seems that Dr. Grim is having fun. Good for him!

By the way, I did not drink a single beer while I wrote this post. Only scotch.

I read an interesting story in the New York Times on Tuesday – interesting, but perplexing. It seems that a Czech ornithologist (more specifically, an avian evolutionary biologist and behavioral ecologist) surveyed other Czech ornithologists (more specifically, other avian evolutionary biologists and behavioral ecologists) on their beer drinking habits. He then correlated their scientific output (as measured by publications/year and citations/paper) with their annual beer consumption. The result was counterintuitive – higher beer consumption led to lower scientific output.

My first thought was to scoff at the study – after all, I drink a lot of beer, and my scientific output has been pretty good. Further, I hang out with quite of few other prolific scientists who also drink their fair share of man’s greatest beverage. There must be something strange about those Czech bird watchers.

But as I began to think further on the subject (and enjoy a fine Pale Ale to settle me down), I realized I was making two cardinal mistakes in my approach to this startling scientific development: 1) I trusted my limited anecdotal evidence over a statistically valid scientific study, and 2) I based my understanding of the science on a journalist’s description of a technical paper. Recognizing my initial flaws, I moved on to a smooth and especially bitter IPA and got on the internet. After a few minutes I had located the original paper in the biology journal Oikos. Here is the citation:

Tomáš Grim, “A possible role of social activity to explain differences in publication output among ecologists”, Oikos, OnlineEarly Articles, 8-Feb-2008.

The paper is only three pages long, so it was a quick read. It was also fairly easy to find the defects in the work. First, there was the common mistake of confusing correlation with causation. The author implied that increased beer drinking caused reduced scientific output. An equally likely explanation is that poor performance in one’s chosen career (in this case ornithology) led to increased beer drinking (and after all, the subjects live in a country with the world’s highest per capita beer consumption). Alternatively, a third, unmeasured factor could be leading to both poor job performance and higher beer consumption (a nagging spouse, for example).

As I looked more carefully at the data, I found a much more significant problem. The total number of data points (as these bird-watching scientists had been reduced to) was 34. This is not an exceptionally large number of subjects when one wishes to draw conclusions about all beer-drinking scientists. The discovered linear relationship between beer consumption and scientific output had a correlation coefficient (R-squared) of only about 0.5 – not very high by my standards, though I suspect many biologists would be happy to get one that high in their work.

But it was while I was switching to a magnificent Pacific Northwest microbrew porter that I saw the real problem. Looking at the graph of the 34 data points, it was clear that the entire correlation was caused by the five lowest-output scientists. Without those five data points, the remaining 29 – showing a wide range of scientific output and beer consumption habits – exhibited absolutely no correlation. Thus, the entire study came down to only one conclusion: the five worst ornithologists in the Czech Republic drank a lot of beer.

Other significant problems were also evident. Standard linear regression, with all the fanciest statistics one can muster, still makes the assumption that each data point is independent. But this study was specifically looking for the impact of social habits on scientific output. Isn’t it likely that some, or many, of these scientists socialized together? After all, the Czech avian evolutionary biology community is not that large. I know that much (possibly most) of my beer drinking is done with fellow lithographers. For all we know, the five lowest-output scientists that created this whole controversy were all part of a drinking club – they’re probably enjoying a fine pilsner and having a fine joke at our expense right now!

In the end, though, I was pleased to see that careful reading and analysis of the original published work led to an easy debunking of the silly notion reported in the press that somehow beer drinking was bad for scientific performance. With the reputation of beer-loving scientists restored to its rightful glory, I sat back and sipped my double-chocolate stout. Ah, the life of a Gentleman Scientist.

After graduating with my bachelors degrees in 1982, I spent the summer working in an optics lab at the National Security Agency and then went off to CalTech to work towards a degree in applied physics. But a funny thing happened on my way to a PhD. I got married (not a particularly wise decision for me at the age of 22) and realized I needed a break from school after four intense years as an undergrad. One semester at CalTech was enough – I dropped out. But now I needed a job.

Since I had spent the previous summer at NSA, I decided to call someone I knew in their HR/Recruiting office about the possibility of a permanent job. After a few phone interviews, I got a job offer from a brand new group – the Microelectronics Research Lab. I hopped on a plane to start a new career and a new life in Maryland.

There is something important to know about working for the NSA – it requires a Top Secret Special Intelligence security clearance. Such a clearance is not trivial to get. One takes a battery of psychological exams, personality tests, and a particularly unpleasant lie detector test. A very thorough background check is done, including interviews with friends, neighbors, teachers, etc. The whole process takes at least nine months, and typically one year. Fortunately, I had just gone through this ordeal in order to get my temporary job the previous summer. Thus, I already had a clearance. When I arrived at the beginning of February, 1983, they even let me skip the two week orientation class and I went straight to my new boss’s office.

Why is all of this important? My new boss didn’t realize that I already had a clearance, and so was expecting me to show up for work in about a year. She had not even begun to think about what I was supposed to do and how I would fit into the group. She gave me some busy work while she pondered my fate. In the meantime, another young engineer in the group noticed my boredom and took pity on me. He was trying to work on etch and deposition (though we were in a very crude lab – our clean room would take a few years to build), and had recently ordered a very small, very manual contact printer (almost a toy, really) so that he could make himself some test patterns. The contact printer arrived the week that I showed up, and to give me something to do, he pointed me to the box. Even though I couldn’t spell it, that week I become a lithographer.

I often wonder what might have happened to me and my life if a different piece of equipment had shown up that week – an electrical prober, or a wafer cleaner, maybe. In hindsight, it seems that lithography was ideal for my educational background and my temperament – something that could have been a perfect plan rather than a perfect fluke. And while my marriage (the thing that sent me into this job) did not last but a few years, lithography has stuck with me for 25 years. Go figure.

By the way, while I was waiting for our clean room to be built (don’t expect things to move fast in the government), I decided the best way to learn about lithography was through simulation. I read Rick Dill’s 1975 papers and fell in love with the idea of lithography simulation. I started to write my own simulator that summer. As they say, the rest is history.

Believe it or not, I started my first lithography company while I was in high school. My parents had moved our family to Texas in order to start a business, so the idea of starting my own business just seemed natural to me. After giving up on my first idea of a used book store, I settled on printing T-shirts. The silk-screen process begins with using contact printing on a photographic emulsion on the screen. Both resolution for fine lines and overlay for four-color printing were important. Still, I spent most of my time worrying about defects (the emulsion getting beat up during screening) and turn-around time (customers can be so demanding). In the end, lithographic quality didn’t matter much as my business acumen was insufficient to allow my survival. It didn’t occur to me that this was my first lithography job until many, many years later, since I certainly didn’t use the word “lithography” (or even know what it meant) until after I got into the semiconductor industry.

I suppose that the failure of my first business was inevitable, since I was soon bound for college anyway (though I was able to make some extra money in college by printing T-shirts and hats for various campus groups). In high school I was a good student, but it was in college, at Rose-Hulman Institute of Technology, that I found my groove. I graduated four years later with four bachelor degrees (physics, chemistry, electrical engineering, and chemical engineering).

Next Time: my failure in graduate school, and how it led to a career in semiconductor lithography

I didn’t grow up saying I wanted to be a lithographer – does anybody? So, like most of us lithographers, I came to my profession the old fashion way – by accident. The story of how I became a lithographer is a relatively short one, so I’ll make it long by adding lots of extraneous details.

My first real job, at age 16, was working for my father in the construction business. He gave me all the dirtiest jobs: digging ditches, laying tie-rod for concrete, running a jack hammer, doing demo (demolition) work. That summer in Dallas saw 41 days in a row above 100F, and I never saw my dad slow down. It was a relief when school started again in the fall, and I decided that when I got to college I was going to work really hard! I few years ago I told this story to my dad. His only response: “It worked”.

During this same time, my mother and her sister-in-law had started a fabric store (the early seeds of an entrepreneurial spirit?). I helped out a bit there, so that by the end of that year I was the only kid I knew who could run a jack hammer and make his own shirts.

Next Time: my first “lithography” job