Sometimes we should not avoid Type I errors

In any introductory statistics course you will learn how to control the probability of making a Type I error. This is the error of concluding that the alternative hypothesis is true, when in fact it is not. But sometimes that kind of error is much, much better than the opposite.
Scott Field first got me thinking about this more than 20 years ago in the context of deciding to protect a Koala population or not. The hypothesis in question was whether a particular population of Koala was declining or not. In traditional statistical analysis, the goal is to avoid concluding that a decline is present when in fact the population is doing just fine. We limit the probability of making this error by stating a priori how likely the observed decline is if the null was in fact true. The usual "bar" is to say that the observed decline must occur less than 1 time in 20 just by chance if the null hypothesis is true. Where things get interesting is that the probability of the other type of error depends on how high the bar is set for avoiding a Type I error. A Type II error, concluding that the population is safe when in fact it is not, is more likely when you restrict the probability of a Type I error.

In most cases that is the correct trade off when your goal is to build reliable knowledge about reality. But what if your goal is to protect an endangered species, or avoid some gruesome risk to your health? In that case, we need to consider the relative costs of the two types of errors. In the Koala case it turned out that the Type II error was so expensive that it didn't make any sense to risk it at all! Economically, the best strategy was to simply protect the species, and not spend any funds  at all on monitoring.

I started thinking about this again in the context of diet choice. In particular, avoiding gluten and carbohydrate is becoming a bit of a fad diet with no scientific support, at least according to some. I've written about some of these studies here and here. I agree that the studies suggesting that grain rots your brain are correlation not causation, have small sample sizes, and are therefore speculative hypotheses at best. But. What should my goal be here? Reliable knowledge? Or avoiding Brain Rot. Let's see ...

	Conclusion
Truth	Eat Gluten	Do not eat Gluten
Gluten is safe	Correct!	Type I Error
Gluten rots your brain	Type II Error	Correct!

So the science is working on the left hand column, and we have two outcomes; gluten is either safe to eat, or it rots your brain. In response, we can make one of two choices, eat gluten or not eat gluten. Right now, Dr. Perlmutter says that grain rots your brain, asserting that the alternative hypothesis is true. If he is right, and you don't eat gluten, then you have done the correct thing. If he's wrong, and you ignore him and continue to eat gluten, you're also doing the correct thing. At this point however, we don't really know for sure if he is right or wrong. Science™ focuses on avoiding the upper right corner of the table, the dreaded Type I error, at the cost of increasing the probability of making a Type II error. Again, if your goal is reliable knowledge, this is the correct weighting.

However, for me personally, the costs of a Type I error are very different from a Type II error. If I make a Type I error, I give up gluten when I didn't need to. OK, I don't eat bread (and cake and pasta) that I really, really like. I avoid beer. In contrast, the consequences of a Type II Error are an increased risk of decades of mental decay, institutionalization, and premature death. The Type II error sounds much worse to me.

How much should I adjust my probability of a Type I error? Is an error of 1 in 10 acceptable? 1 in 5? Well, that depends on how much worse the consequences are, and how "powerful" the science could be. Power in this case refers to the ability of a given scientific analysis to distinguish the alternative hypothesis from the null when the alternative is true. Unfortunately I have no control over the latter; it depends on what studies are proposed, funded, and published. Moreover, really conclusively testing this idea would require enrolling a large number of people (maybe thousands of people), and then getting them to agree to follow a particular diet for many years, even decades. So, whatever science is done will have low power (a low probability of avoiding a Type II error) regardless of how I set the Type I error probability.

When I add in the fact that avoiding gluten also helps me control my blood sugar, it's a no brainer. No bread.