Almost every company I work with has run A/B tests. Very few have an experiment programme, and the gap between those two things is most of the value.
A test is an event. A programme is a rhythm that produces compounding knowledge about your own customers. Here is what separates them.
You need enough traffic, and you need to admit it
Start here, because it kills a lot of plans and it is better killed early.
A/B testing needs volume to reach statistical significance. If your key flow sees a few hundred visitors a month, you cannot run a meaningful test on it — not with a clever tool, not with a longer runtime, not with a Bayesian model. You will get a number, and the number will be noise.
Low traffic does not mean no optimisation. It means qualitative research, session recordings, heuristic review and shipping the obvious fixes without pretending you measured them. That is a legitimate approach. Dressing it up as testing is not.
Quantitative to find it, qualitative to understand it
The sequence that works: analytics tells you where, recordings tell you why.
Start with the funnel and find the step with the drop. Then watch sessions at that step in something like Hotjar until you can describe what is going wrong in a sentence — a form field people retype, an unexplained charge, a button below the fold on the most common screen size.
A hypothesis built from both is testable. A hypothesis built from a team brainstorm is a preference.
Prioritise with ICE, and be honest about the C
I score every candidate on Impact, Confidence and Ease. It is a common framework and it works, provided you are honest about Confidence.
Confidence is not how much you like the idea. It is how much evidence you have that the problem is real: a drop-off you can see, sessions you have watched, a survey response, a support ticket theme. An idea with no evidence behind it scores low on Confidence no matter how good it sounds in a meeting — and most brainstormed ideas belong there.
Instrument the experiment before you run it
The experiment needs its own measurement, and it needs to exist before traffic is split.
The paper “A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments” with my co-authors @alexdeng and @lukasvermeer will appear in KDD 2022. See https://t.co/ixgLhxdp4T #abtest #intuitionbusters #experimentguide
Ronny Kohavi ran experimentation at Microsoft, Amazon and Airbnb, and that paper is a list of the ways smart teams fool themselves. Peeking at results and calling a test early is number one on it.
That means the variant is a property on the events, so every downstream step can be broken down by arm. Otherwise you learn that the variant got more clicks, without learning whether those clicks became revenue — and a test that lifts clicks while depressing purchases is a test you will ship by mistake.
Decide the primary metric and the runtime before you start, and write them down. Calling a test when it happens to look good is the most common way a testing programme produces results that do not replicate.
Write up the losers
This is the part that makes a programme compound, and it is the part everybody skips.
A losing test that is documented is knowledge: you now know something about your customers that you did not know last month, and nobody will run it again in eighteen months when the team has turned over. A losing test that is quietly closed is a cost with no asset.
My write-ups are short. What we thought, what we changed, what happened, what we believe now, and what it means for the next one. Three paragraphs, in a playbook the client keeps.
After a year of that, the playbook is worth more than any individual win in it — and it is the thing that is still working when I am no longer there.
Roll the winners out properly
A win that stays live only on the tested page is half a win. The pattern usually applies elsewhere: the same form on a different flow, the same reassurance copy at a different step, the same removed field across the whole product.
Rolling winners out across related surfaces is the cheapest growth work there is, and it is routinely left undone because the test is marked complete and everyone moves on.
What good looks like after a year
- A continuously running stream, not a burst followed by silence
- A backlog scored on evidence rather than enthusiasm
- Every experiment instrumented before it starts, with the metric agreed in advance
- Losers written up and kept
- Winners rolled out everywhere the pattern applies
- A playbook that means your next hire inherits what you learnt
None of that is clever. All of it is rhythm — which is why it works, and also why most programmes never get there.
one stream, always running — beats four in bursts
Questions people ask
How much traffic do I need to A/B test?
How many experiments should run at once?
What do I write up for a test that lost?
Written September 2026. Platform facts checked at the time of writing; if you are reading this much later, check them again.