Apple's Product Page Optimization lets you test up to three treatments against your original product page. Their documentation is specific on the parameters: up to three treatments, a maximum test duration of 90 days, a recommendation to wait for at least 90% confidence, and conversion rate as the metric.
Read those together and there is a problem sitting in plain sight. Ninety percent confidence is a statement about one comparison. Running three treatments means making three comparisons against one baseline, and the probability that at least one of them looks like a winner by chance alone is not 10%.
It is 27%.
The arithmetic
If each comparison has a 10% chance of a false positive, and the comparisons are independent, the chance that all three come back clean is 0.9 × 0.9 × 0.9 = 0.729. So the chance that at least one does not:
1 - 0.9³ = 1 - 0.729 = 0.271
Just over one test in four, run with three treatments and read at Apple's recommended threshold, will hand you a winner that is noise. Not one in ten. The 10% figure was never a statement about the experiment; it was a statement about a single arm of it.
This is the multiple-comparisons problem, and it is not exotic — it is the reason clinical trials pre-register their endpoints. What makes it worth writing about here is that Apple's interface hands you three treatments and Apple's guidance hands you a confidence level, and nothing in between tells you that using all of one changes the meaning of the other.
The correction, and what it costs
The simplest fix is a Bonferroni correction: divide the family-wide error rate across the comparisons. At a 10% family-wide rate over three treatments, each comparison is judged at 3.33%, or 96.67% confidence per comparison.
That is a real cost, and it should be stated rather than glossed. Holding 80% power and a two-sided test, the corrected threshold needs roughly 43% more traffic per arm than the uncorrected one — the required sample scales with (z<sub>α/2</sub> + z<sub>β</sub>)², and correcting moves z<sub>α/2</sub> from 1.645 to 2.128.
Bonferroni is conservative and it is not the only option; Holm and Šidák are less blunt. But the comparison worth making is not between corrections. It is between correcting and not, because an uncorrected reading of a three-treatment test is how a noise result gets shipped as a redesign.
| Uncorrected | Corrected for 3 | |
|---|---|---|
| Confidence needed per comparison | 90.00% | 96.67% |
| Chance of at least one false winner | 27.1% | 10% |
| z<sub>α/2</sub> | 1.645 | 2.128 |
| Traffic needed per arm (80% power) | 1.00× | 1.43× |
What this looks like on a real result
Verified against the calculator in Appstro's product page test: a baseline converting at 3.00% against a treatment at 3.45%, reaching 92.8% confidence.
Read as a single comparison, that clears Apple's 90% bar. It is a winner. Ship it.
Read as one of three, it does not clear 96.67%, and the confidence interval on the difference crosses zero. The honest verdict is not conclusive — which is a different thing from "the treatment lost".
Same data, same arithmetic, two opposite decisions. The only input that changed was whether you counted how many comparisons you were making.
Two details that are easy to get backwards
If you are building or checking this yourself, two choices in the maths are deliberately asymmetric:
Sample size uses the unpooled variance form. The pooled shortcut understates the requirement when the two rates differ — which is precisely the direction that causes harm, because it tells you to stop collecting data earlier than you should.
The test statistic and the confidence interval use different standard errors. The test statistic is evaluated under the null hypothesis, where the two rates are assumed equal, so the pooled estimate belongs there. The confidence interval estimates a real difference between two rates that are not assumed equal, so it uses the unpooled form. They are not interchangeable and a calculator that uses one for both is wrong somewhere.
None of this was eyeballed. The implementation was validated by Monte Carlo simulation across four configurations: realised power came out at 79.3%, 78.7%, 79.5% and 89.6% against targets of 80/80/80/90, and the realised false-positive rate matched the corrected alpha in each. The normal CDF and its inverse round-trip to within 1e-6.
The most useful output is often "this cannot resolve"
Apple caps a test at 90 days. Feed a low-traffic app's numbers into a corrected sample-size calculation and the required duration frequently exceeds that cap.
When it does, the right answer is not a number. It is: this test cannot resolve within Apple's limit. Printing "you need 214 days" invites someone to run it for 90 and read the result anyway, which is the worst of the available outcomes — the cost of the test with none of the certainty.
If that is where you land, the options are to test a bigger change (a larger true effect needs less traffic to detect), to test fewer treatments (two comparisons instead of three loosens the correction), or to accept that this decision will be made on judgement rather than on data. All three are legitimate. Running an underpowered test and believing the result is not.
What this cannot tell you
- Not a typical App Store conversion rate. There is no published benchmark worth repeating. Your own baseline is the only baseline.
- Not whether a winning treatment brings worse users. PPO measures impressions to downloads. A screenshot set that wins on installs and loses on day-30 retention is a real outcome, and it is visible only in your own analytics.
- Not whether the effect will persist. A test measures a window. Novelty and seasonality live inside that window.
- Not what to test. The statistics tell you whether a difference is real. They have nothing to say about which idea was worth trying.
The work
Before you start, run the numbers in the test calculator: put in your baseline conversion rate, the smallest lift that would actually change your decision, and the number of treatments you intend to run. If the required duration comes back under Apple's 90-day cap, the test is worth running. If it does not, you have learned something more valuable than the test would have told you.