Hypothesis testing is a formal procedure in statistics for deciding whether sample data provide enough evidence to reject a null hypothesis in favor of an alternative hypothesis, while keeping the probability of a false alarm at a chosen level. It turns a question such as “did the new process change the average defect rate?” into a test statistic, a p-value and a decision.
The method does not prove hypotheses true. It quantifies how surprising the observed data would be if the null hypothesis held, and it controls two kinds of error over many repetitions. Its correct use depends on the study design, the assumptions of the chosen test and an honest reading of effect size alongside significance.
The steps of a hypothesis test
Every hypothesis test follows the same sequence of steps, regardless of the specific test. Fixing these steps before looking at the data is what keeps the error rates valid.
- State the null hypothesis H0, usually “no effect” or “no difference”, and the alternative H1.
- Choose the significance level alpha, most often 0.05, sometimes 0.01 or lower.
- Select a test that matches the data type, design and assumptions.
- Compute the test statistic from the sample, such as t, z, chi-square or F.
- Obtain the p-value, the probability of a result at least as extreme under H0.
- Decide and report: reject H0 if p is at most alpha, together with the effect size and a confidence interval.
The alternative can be two-sided, testing for a difference in either direction, or one-sided, testing a single direction. A one-sided test is only justified when the opposite direction is genuinely irrelevant and the choice is made before seeing the data.
Errors, significance and power
A hypothesis test can be wrong in two distinct ways, and the design of a study is largely a matter of balancing them. The table shows the four possible outcomes.
| Decision | Null hypothesis true | Null hypothesis false |
|---|---|---|
| Reject H0 | Type I error, probability alpha | Correct detection, probability equals power |
| Do not reject H0 | Correct non-rejection | Type II error, probability beta |
The significance level alpha caps the long-run rate of false positives. Statistical power, equal to 1 minus beta, is the probability of detecting a real effect of a given size; studies are commonly planned for 80% or 90% power. Power rises with sample size and effect size and falls with data variability and stricter alpha.
The p-value is often misread. It is not the probability that H0 is true and not the probability that the result arose by chance alone. It is a statement about the data under the assumption that H0 holds, so a small p-value means the data are unusual under the null, nothing more.
Common tests and when to use them
The right test depends on the type of outcome variable, the number of groups and whether observations are paired. The following table covers the tests used most often in practice.
| Test | Question answered | Key assumptions |
|---|---|---|
| One-sample or two-sample t-test | Do means differ? | Independent observations, roughly normal means |
| Welch t-test | Do means differ with unequal variances? | No equal-variance assumption |
| Paired t-test | Did the mean change within subjects? | Paired measurements |
| Chi-square test | Are two categorical variables associated? | Expected counts of about 5 or more |
| ANOVA | Do means differ across three or more groups? | Similar variances, independence |
| Mann-Whitney U test | Do two distributions differ in location? | Ordinal or skewed data |
Parametric tests such as the t-test rely on the sampling distribution of the mean, which is close to a normal distribution for moderate sample sizes even when the data are not. With small, heavily skewed samples, nonparametric tests or resampling methods such as permutation tests and the bootstrap are safer.
Frequentist testing compared with related approaches
Classical hypothesis testing is a frequentist method, defined by error rates over hypothetical repetitions. Other approaches answer related but different questions.
| Approach | Main output | Question it answers |
|---|---|---|
| Hypothesis testing | p-value and reject or not | Is the evidence against H0 strong enough? |
| Confidence interval | Range of plausible values | How large could the effect be? |
| Bayesian statistics | Posterior probability | How probable is each hypothesis given the data? |
| Equivalence testing | Rejection of a meaningful difference | Are two treatments practically the same? |
A confidence interval contains the same information as a two-sided test and more: a 95% interval that excludes zero corresponds to rejection at alpha 0.05, and its width shows the precision of the estimate. For this reason, many fields now require intervals and effect sizes in addition to p-values.
Pitfalls and misuse
Most problems with hypothesis testing arise from how tests are used, not from the mathematics. The mistakes below are common in both research and business analysis.
| Mistake | Consequence | Safeguard |
|---|---|---|
| Multiple comparisons | Inflated false-positive rate | Bonferroni or false discovery rate control |
| P-hacking | Spurious significant findings | Pre-registered analysis plan |
| Optional stopping | Alpha exceeded in repeated looks | Fixed sample size or sequential methods |
| Significance as importance | Trivial effects overstated | Report effect sizes |
| Absence of evidence as evidence | Real effects dismissed | Check power, use equivalence tests |
Large samples make tiny differences significant, while small samples can miss important ones. Biased data also undermine any test: sampling bias or a lack of randomization produces a precise answer to the wrong question, which no p-value can correct.
The idea in brief
Hypothesis testing works like a courtroom. The starting assumption is innocence, which means nothing has changed. The data are the evidence. Only when the evidence would be very unlikely under that assumption does the verdict switch to “something changed”.
A verdict of “not proven” does not mean innocence was proven. It may simply mean there was too little evidence. The test measures the strength of evidence, not the truth itself.
A practical scenario
A bottling plant introduces a new filling nozzle and wants to know whether it changes the average fill volume, which should be 500 ml. The quality team sets H0 as no change in mean volume, chooses alpha 0.05 and plans a sample large enough for 90% power to detect a shift of 1 ml.
They measure 60 bottles from the old nozzle and 60 from the new one. A Welch two-sample t-test gives a mean difference of 0.8 ml with a 95% confidence interval from 0.3 to 1.3 ml and a p-value of about 0.002.
The team rejects the null hypothesis but also notes that the shift is small relative to the legal tolerance. They recalibrate the nozzle by 0.8 ml instead of replacing it, which keeps the improvement in speed while restoring the target volume.