Would you bet $25 to win $100?
We would gladly take the bet if the chance of winning were 96 percent. We certainly would not change our minds if the odds dropped to 94 percent. However, many scientists make different decisions on the basis of the p values inherent in this example—whether p = 0.04 or p = 0.06.
Most scientists utilize Frequentists statistics to prevent abuse of data. Frequentists require that you specify your hypothesis, statistical test, and the criteria of success in advance. Often this criterion is a “significant” p value, especially in medical studies. While p ≤ 0.05 is often the criterion of choice, one may also choose 0.01/0.001 for “definitive studies” or 0.10/0.15 for “exploratory studies.”
Much has been written about why this is a flawed strategy. Most statisticians admit the strategy is flawed in hallway conversations or even in articles such as this one. Clients of statisticians will state that they know this is only part of the conclusions from a data analysis. Yet, at the end of the day, p = 0.08 is deemed a failure and p = 0.04 is a success.
In this article, we will informally present the Frequentists dilemma and describe differences between the Frequentists and Bayesian schools of statistical thought.
Leach P. Why Can’t You Just Give Me the Number? An Executive’s Guide to Using Probabilistic Thinking to Manage Risk and to Make Better Decisions. Sugar Land (TX): Probabilistic Publishing; 2006.
Radding A. Give me a number—introducing the DIST. Big Fat Finance Blog. http://bigfatfinanceblog.com/2009/08/04/give-me-a-number-introducing-the-dist. Posted August 4, 2009. Accessed January 25, 2010.
Thall PF, Wathen JK. Practical Bayesian adaptive randomization in clinical trials. Eur J Cancer 2007; 43:859–866.