Fill in these first
- What was tested: [e.g. new checkout button text]
- Primary metric: [e.g. purchase conversion rate]
- Variant A, control: visitors [number], conversions [number]
- Variant B, and any others: visitors [number], conversions [number]
- Test dates and duration: [start date, end date]
- Significance level agreed before the test: [e.g. 95 percent]
- Minimum lift worth shipping: [e.g. 3 percent relative]
- Secondary metrics, optional: [e.g. average order value, refund rate]
Task
Analyse the test and write a readout a product manager can act on in five minutes.
Output format
- Results table:
| Variant | Visitors | Conversions | Conversion rate | Relative lift vs A | 95 percent confidence interval of lift | p-value |
- Method: one line naming the test used, such as a two-proportion z-test, and why it fits.
- Verdict: one of significant winner, significant loser, or not significant.
- Practical check: whether the lift clears the minimum worth shipping.
- Risks to the result: sample ratio mismatch, short duration, novelty effect, peeking.
- Recommendation: ship, stop or extend, with the extra sample size needed if extending.
Quality rules
- Show the formulas used and round rates to two decimal places.
- Check that traffic split matches the intended split; flag a gap above 1 percentage point.
- Never call a result significant because it looks large; use the agreed level only.
- Separate statistical significance from business value in plain words.
- If the test ran under one full week, say that weekday effects may not be covered.
Example
| B | 10,240 | 532 | 5.20% | +8.3% | +1.1% to +15.9% | 0.021 |






