What is hypothesis testing?
A hypothesis test turns a question such as "did the new supplier change the strength?" into a decision with a known risk of being wrong. The significance level α is the risk of a false alarm (concluding there is a difference when there is none). The confidence interval shows the range of plausible values for the difference, which is often more useful than the p-value alone.
Which test to use
| Question | Data | Test |
|---|---|---|
| Is the mean on target? | One sample of measurements | 1-sample t-test |
| Do two groups differ? | Two independent samples | 2-sample t-test (Welch) |
| Did a change affect the same items? | Before and after on the same items | Paired t-test |
| Is a yield or defect rate on target? | Events out of n | 1-proportion test |
| Do two rates differ? | Events out of n for two groups | 2-proportion test |
| Do three or more groups differ? | Several samples | ANOVA |
Hypothesis test example
Tensile strength was measured on 20 samples from each of two suppliers. Supplier A averages 52.0 MPa and supplier B 50.6 MPa. The Welch 2-sample t-test gives t = 2.11 and p = 0.042, so at α = 0.05 the difference is significant. The 95% confidence interval for the difference is 0.05 to 2.80 MPa: real, but possibly very small, and a medium effect size (Cohen's d = 0.67). Before switching supplier, check whether a 1.4 MPa difference matters against the specification. The tool includes examples for every test type.
Common pitfalls
- Statistical is not practical significance: with large samples, tiny differences become significant.
- Paired data analysed as independent: you lose power. Use the paired test when the same items are measured twice.
- Testing many things at once: with 20 tests at α = 0.05, one false alarm is expected.
- Ignoring the data: look at the plot. Outliers and mixed populations can drive the result.
How to use this Hypothesis Testing tool
- Choose the test. Pick the test that matches your question and data: t-tests for means, proportion tests for rates.
- Set the hypotheses. Choose two-sided or one-sided, the significance level and the hypothesized value.
- Enter the data. Paste the samples from Excel, or type the counts and sample sizes for proportions.
- Read the result. See the p-value, the confidence interval, the effect size and a plain-language conclusion.
- Check the assumptions. Review the data plot for outliers and skew, especially with small samples.
- Export. Export the result and charts to PDF or Excel for your report or change record.
Frequently asked questions
What is hypothesis testing?
Hypothesis testing is a way to decide, from sample data, whether an observed difference is real or could be due to chance. You state a null hypothesis (no difference, H0) and an alternative (H1), calculate a test statistic, and compare its p-value with the significance level α, usually 0.05.
What does the p-value mean?
The p-value is the probability of seeing a result at least as extreme as yours if the null hypothesis were true. A small p-value (below α) means the data would be unusual under H0, so you reject H0. It is not the probability that H0 is true, and it says nothing about how large or important the effect is.
Which test should I use?
Use a 1-sample t-test to compare a mean with a target, a 2-sample t-test to compare two independent groups, and a paired t-test when the same items are measured twice (before and after). For yields or defect rates, use the 1-proportion or 2-proportion test. For three or more groups, use ANOVA.
Should I assume equal variances in the 2-sample t-test?
Usually not. The Welch test (unequal variances, the default here) is reliable whether or not the variances are equal, and loses very little power when they are. Use the pooled test only when you have good reason to believe the variances are the same.
One-sided or two-sided?
Use a two-sided test unless you decided before collecting the data that only one direction matters (for example, a yield below target). A one-sided test has more power in that direction, but cannot detect an effect in the other direction.
What does "not significant" mean?
It means the data do not provide enough evidence of a difference: either there is none, or the sample was too small to detect it. It does not prove that the groups are equal. Look at the confidence interval: if it is wide, collect more data.