Two opposite ways a statistical test — or a model's decision rule — can be wrong, and why you can't just dial one of them down for free.
Every statistical test, and every model that has to decide "is this real or not," can fail in exactly two directions. It can cry wolf — declaring an effect that isn't actually there. Or it can miss a wolf that's genuinely present — declaring nothing's there when something is. These aren't the same mistake wearing different clothes. They're opposite failure modes, they're driven by different mechanics, and — critically — reducing one, on its own, tends to make the other one worse.
A Type I error (a false positive) means incorrectly rejecting a null hypothesis that was actually true — concluding there is an effect, a difference, or a signal when there really isn't one. Concluding a new drug works when it actually doesn't. A spam filter flagging a perfectly legitimate email as spam. The probability of making a Type I error is called the significance level, α (commonly set at 0.05) — and unlike most error rates, this one is a threshold the analyst deliberately chooses beforerunning the test, as the maximum false-alarm rate they're willing to tolerate.
A Type II error (a false negative) is the mirror image: incorrectly failing to reject a null hypothesis that was actually false — concluding there's no effect when one genuinely exists. Concluding a drug doesn't work when it actually does. A spam filter letting real spam sail through to the inbox. The probability of a Type II error is denoted β, and its complement — statistical power, 1 − β— is the probability of correctly detecting a true effect when one is actually there. High power is what you want; it means real signals don't slip past the test.
Formally:
Picture the "no effect" world and the "real effect" world as two overlapping distributions of possible test statistics, sliced by one decision threshold. Everything past the threshold gets called "effect detected"; everything short of it gets called "no effect." Since both α and β are defined relative to that same threshold, sliding it in either direction mechanically shrinks one region while growing the other — there is no way to move a single cut point and shrink both tails at once. The only lever that reduces α and β together, at a fixed threshold, is collecting more data: more data narrows the spread of both distributions, shrinking their overlap, which raises power without loosening the significance threshold at all.
Plot every possible test outcome under "no real effect" as one bell curve, and every possible outcome under "a real effect actually exists" as a second bell curve shifted to the right. A test has to draw one vertical line — the decision threshold — and call everything past it "effect detected." Wherever that line falls, it slices through both curves at once.
False, or at best dangerously incomplete. Lowering α specifically reduces Type I errors — the test now demands stronger evidence before declaring anything significant, so false alarms drop. But that same stricter bar simultaneously increases Type II errors: real effects that would have cleared the old, looser threshold now fail to clear the new, stricter one, so more genuine signals get missed. A stricter threshold isn't "more accurate" in some absolute sense — it's a different point along the exact same α/β tradeoff curve, trading missed detections for fewer false alarms. Reducing both error types together generally requires more data (which raises statistical power at a given significance level), not simply relocating the threshold.
Explains the two opposite ways a statistical test (or any model making a real-vs-not decision) can be wrong: a Type I error (false positive) — incorrectly rejecting a true null hypothesis, controlled by the significance level α — and a Type II error (false negative) — incorrectly failing to reject a false null hypothesis, with probability β, whose complement (1 − β) is statistical power. Illustrated with a 2x2 outcomes matrix and overlapping-distribution diagrams showing why tightening one error rate loosens the other, and why more data — not a moved threshold — is the only way to shrink both together.
It is tempting to treat a test's error rate as one dial to turn down, but a hypothesis test actually has two independent, opposing error rates operating over the same decision boundary. Tightening the boundary to catch fewer false positives necessarily lets more false negatives through, because both are defined relative to the identical threshold slicing through two overlapping possibilities: "no real effect" and "a real effect exists." Treating a stricter significance level as unambiguously "better" or "more accurate" ignores which of the two error types just got worse.
The significance level α is the probability of a Type I error: rejecting the null hypothesis (concluding an effect is real) when the null hypothesis is actually true. It is chosen by the analyst before the test is run — commonly 0.05, sometimes 0.01 for higher-stakes decisions — as the maximum false-alarm rate they are willing to accept. β is the probability of a Type II error: failing to reject the null hypothesis when it is actually false, i.e., missing a real effect. Statistical power, defined as 1 − β, is the probability of correctly detecting a true effect when one exists, and is the quantity researchers actually want to be high.
Power depends jointly on the significance level (α), the sample size, and the true effect size: for a fixed α and effect size, more data increases power (reduces β). For a fixed sample size, lowering α (to reduce Type I errors) reduces power (increases Type II errors), and vice versa. This is why power analysis — calculating the sample size needed to detect a given effect size at a given α with acceptable power, typically 0.80 or higher — is standard practice before running an experiment, rather than after seeing a disappointing result.
Picture the distribution of possible test statistics under "no real effect" (the null distribution) and under "a real effect exists" (the true-effect distribution) as two overlapping bell curves, with the true-effect curve shifted to the right. A test works by drawing one vertical decision threshold and calling everything past it "effect detected." The null distribution's tail that falls past the threshold is the Type I error region — real occurrences of "no effect" that the test still calls significant. The true-effect distribution's portion that falls before the threshold is the Type II error region — real effects the test still calls insignificant.
Because one threshold cuts through both curves simultaneously, sliding it toward the true-effect distribution (a stricter, lower α) shrinks the Type I region but grows the Type II region, and sliding it toward the null distribution (a looser, higher α) does the reverse. The only way to shrink both error regions at once, without simply trading one for the other, is to reduce how much the two distributions overlap in the first place — which more data (a larger sample size) does, by narrowing the spread of both curves around their respective means.
Neither is universally worse — it depends entirely on the real-world cost of each mistake. Approving an ineffective drug (Type I) can be far more dangerous than delaying an effective one a bit longer (Type II), which is why clinical trials favor low α. But in early-stage cancer screening, missing a real tumor (Type II) is typically judged far worse than a false alarm that triggers further, cheaper testing (Type I), so screening tests are often tuned toward high sensitivity (low β) even at the cost of more false positives.
It is a historical convention (popularized by statistician Ronald Fisher in the 1920s), not a law of nature. 0.05 means accepting a 5% chance of a false positive on any single test where the null hypothesis is actually true. Fields with higher stakes or many simultaneous comparisons often use stricter thresholds (like 0.01 or 0.001, or corrections such as Bonferroni), accepting more Type II risk in exchange for fewer false discoveries.
Sample size does not directly change α, which the analyst sets independently. But for a fixed α and a fixed true effect size, a larger sample narrows the spread (standard error) of the sampling distribution, which shrinks the overlap between the null and true-effect distributions and thereby reduces β — increasing power. This is the one lever that improves the false-negative rate without touching the false-positive rate at all.
Yes, under different names but the identical logic. A binary classifier's false positive rate is exactly a Type I error rate, and its false negative rate is exactly a Type II error rate — the classifier's decision threshold on a predicted probability plays the same role as α does in hypothesis testing, and moving that threshold trades precision-side errors against recall-side errors in the same way. The confusion matrix used to evaluate classifiers is the same 2x2 structure shown in the diagram above.
Yes, but not by adjusting the decision threshold alone — that only trades one against the other. Achieving both low α and low β (high power) simultaneously requires either a larger sample size, a larger true effect size (which isn't under the analyst's control), or a less noisy measurement — anything that narrows the overlap between the null and true-effect distributions rather than just sliding the cut point between them.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.