The Ruinous Impact of Our Obsession with Statistical Significance on Science

Over a hundred years ago, two unusual yet domestic challenges helped define what modern science considers "real": a Guinness brewer tasked with quality control and a British woman claiming she could tell whether milk or tea was poured first. While quaint, these stories set the stage for methods that now determine which findings are published, recognized, and trustedand which are dismissed as "not significant." Fields like economics and medicine have rigidly adhered to statistical significance, often with harmful consequences.

In recent decades, the fixation on statistical significance has contributed to excessive prescription of antidepressants and a headache medication with fatal side effects. Another path was possible. Sir Ronald Fisher, a century ago, made statistical significance a central part of scientific research. For decades, scientists have warned that strict adherence to his methods has misdirected scientific practice. Today, many disciplines face a crisis of false-positive results and biased findings due to this rigid focus.

The Rise of Small-Sample Statistics

At the start of the 20th century, statistics as a science was blossoming. One innovation was small-sample statistics, designed to work with limited data. William S. Gosset championed this approach, but it was overshadowed by Fisher's methods, limiting our ability to draw accurate conclusions. Reviving Gosset's methodology for experimentation and estimation could benefit modern research.

Fisher's framework of "statistical significance" involves collecting data to test a hypothesis, calculating a p-value under the null hypothesis (the probability of observing the data if no effect exists), and comparing it to a threshold, usually 0.05. If the p-value falls below this level, the effect is deemed present. While Fisher introduced many statistical tools still used today, he could not foresee their misuse in the era of big data and computing power.

Gosset and Industrial Statistics

William Gosset, educated in mathematics and chemistry at Oxford, joined Guinness in Dublin to help improve beer quality on a massive scale. Sampling each barrel by taste was impractical, so he developed a small-sample formula to determine how many samples were needed for accurate estimates. Using everyday work with malt sugar levels and even data from British prisoners, Gosset created formulas that allowed precise industrial quality control.

Publishing under the pseudonym "Student," Gosset introduced his formula in 1908. This approach proved useful for many real-world applications where measurements are approximately normally distributed. Fisher later recognized the potential and mathematically validated Gosset's work, while Gosset himself prioritized accurate estimation over significance testing, a philosophy now called "estimation culture."

The Tea-Tasting Experiment and Fisherian Significance

In his 1936 book The Design of Experiments, Fisher described a woman claiming to distinguish the order of milk and tea. To test her, Fisher proposed eight trials, computing the probability of her success under the null hypothesis. He suggested that a p-value under 0.05 indicated statistical significance. This example illustrates how Fisherian significance revolves around binary yes-or-no questions rather than measuring effect magnitude.

This binary approach can mislead research. A groundbreaking clinical trial might appear "insignificant" if p-values exceed 0.05, even if the effect is meaningful. Experiments are intended to expand knowledge, not just validate a null hypothesis. Historical experiments, such as Millikan's oil-drop experiment to measure the electron's charge, exemplify estimation cultureaiming to quantify nature rather than confirm a hypothesis.

Consequences of Statistical Significance

Significance culture has led to publication bias. Research indicates that statistically significant results are much more likely to be published. In medicine, this distortion has serious consequences: for instance, Vioxx was reported as having no statistically significant risk of heart complications, despite a fivefold increase in affected patients. Millions of prescriptions were issued before the risk was widely recognized.

In the era of big data, false-positive findings are increasingly common. More studies and larger sample sizes create a flood of statistically significant but trivial results. This trend threatens the credibility of scientific research and the reliability of published findings.

Embracing Estimation Culture

Gosset valued estimating real-world quantities, quantifying confidence, and evaluating costs and benefits. Modern industry data scientists, working for companies like Google and Microsoft, naturally follow this estimation culture, focusing on accuracy rather than publication. Academics and journalists could adopt similar practices: start with estimates, assess their reliability, consider relevance, and weigh potential outcomes.

The lesson is clear: science can advance more reliably by moving away from rigid statistical significance and toward an approach that prioritizes estimation and context, just as Gosset demonstrated at Guinness over a century ago.

Analysis: Why Statistical Significance Is No Longer Enough

From my perspective, the historical contrast between Ronald Fisher and William Gosset explains many of today’s scientific failures. Fisher’s framework, built around p-values and fixed thresholds, was designed for a different era. It offered a clear rule for decision-making, but over time it narrowed research questions into binary outcomes that often ignore effect size, uncertainty, and real-world relevance.

The dominance of statistical significance has shaped publication incentives and research behavior. Results that cross the 0.05 threshold are rewarded, while equally informative findings that do not are often excluded. This has contributed to biased literatures in economics and medicine, where statistically “non-significant” outcomes can still carry substantial practical consequences.

Gosset’s approach, developed for industrial quality control, focused on estimation and precision under real constraints. His methods were designed to answer how large an effect is and how reliable an estimate may be. This estimation-centered logic aligns more closely with how modern data-driven industries evaluate evidence and risk.

In the context of large datasets and automated analysis, strict adherence to significance testing increases the likelihood of false positives and trivial discoveries. Revisiting estimation culture is not a rejection of statistics, but a correction of its priorities. For science to remain credible, conclusions must be grounded in magnitude, uncertainty, and context—not solely in whether a threshold is crossed.

Sources:

• Our Obsession With Statistical Significance Is Ruining Science