What is a before-and-after analysis?
After an improvement, someone will ask: did it work? A before-and-after analysis answers with evidence. It compares the measure in the period before the change with the period after, quantifies the difference, and tests whether it is larger than normal variation would explain. It is the Check in PDCA and the verification step in DMAIC, 8D and CAPA effectiveness checks.
The statistics
| Result | Question it answers |
|---|---|
| Change in mean and % | How big is the change? |
| Welch's or paired t-test | Could the change in the mean be chance? |
| 95% confidence interval | What is the plausible range of the true change? |
| Mann–Whitney U test | Same question without assuming normal data |
| F-test on variances | Did the variation change? |
| Cohen's d | Is the change small, medium or large relative to the spread? |
Before-and-after example
A QC laboratory introduced review by exception in its LIMS in June. Release lead time for 25 batches from January to May averaged 7.2 days; for 20 batches from July to October it averaged 4.5 days, 38% lower. Welch's t-test gives p < 0.001 and the true reduction is between 1.9 and 3.6 days with 95% confidence; the Mann–Whitney test agrees and the effect size is large (d = −1.9). The standard deviation also fell by 19%, but the F-test (p = 0.35) cannot confirm that the variation really changed. The lab records the improvement as verified and keeps monitoring the variation.
Pitfalls
- Other changes at the same time (new product mix, staffing, season) can cause or hide the effect.
- Too short an after-period may show a temporary effect; check that the after values are stable in the time-order plot.
- Regression to the mean: if the change was triggered by an unusually bad period, some improvement would have happened anyway.
- For data over many periods, a run chart with the change marked shows whether the shift is sustained; for a formal test of two groups see hypothesis testing.
How to use this Before-and-After Analysis tool
- Describe the measure. Name the measure and unit, choose whether lower or higher is better, and describe the change that was made.
- Paste the data. Paste the values from before the change in the left box and from after the change in the right box.
- Independent or paired. Choose paired only if the same items were measured before and after, in the same order.
- Read the conclusion. The verdict states whether the change is a significant improvement, with the size of the change and its confidence interval.
- Check the chart. The time-order plot shows whether the change is stable; the box plots compare the distributions.
- Export. Export the chart, statistics and data to PDF or Excel as evidence for the change record.
Frequently asked questions
What is a before-and-after analysis?
A before-and-after analysis compares a measure before and after a change, such as a new procedure, a machine adjustment or a training programme, to decide whether the change made a real difference. It combines the size of the change with a statistical test that shows whether the difference could be chance.
Which test should I use?
For independent samples (different batches, orders or patients before and after), use Welch's t-test, which does not assume equal variation. If the same items are measured twice, use the paired t-test. If the data is skewed or has outliers, check the Mann–Whitney test, which does not assume normal data. This tool runs them all and flags when they disagree.
What does the p-value mean?
The p-value is the probability of seeing a difference at least this large if the change had no effect. A p-value below 0.05 is the usual threshold for "statistically significant". It does not tell you how big or how important the change is; look at the difference and its confidence interval for that.
What is the effect size (Cohen's d)?
Cohen's d is the difference in means divided by the pooled standard deviation. As a rule of thumb, 0.2 is small, 0.5 medium and 0.8 large. It shows whether a significant change is also a meaningful one.
Why check the variation as well as the mean?
Many improvements reduce variation rather than moving the average, and customers feel variation. The F-test compares the variances before and after, and the box plots show the spread.
How much data do I need?
More is better, but 15 to 30 values in each period usually gives a clear answer for a meaningful change. Make sure the before and after periods are comparable (same product mix, season and measurement method).