Most companies test new channels the way people buy lottery tickets: occasionally, without a plan, and with a vague sense that this one feels promising. Then the result is interpreted after the fact, in whichever direction suits whoever is presenting it.
The fix is not more rigour in the analysis. It is writing down four things before any money moves — because a test that was unreadable at launch cannot be rescued by clever analysis afterwards.
The four things, written down first
1. The hypothesis, stated so it can be wrong
"We should try TikTok" is not a hypothesis. It contains no claim that could turn out to be false, which means no result can contradict it.
A usable version: "Our buyers under 35 discover products in short-form video, and we can acquire them there below $80 CAC — within 20% of our current blended cost." That is falsifiable. It names an audience, a mechanism and a number, and any of the three can be shown wrong.
2. The threshold, agreed before launch
What number would make you scale this? Decide now, while nobody is emotionally invested and no money has been spent.
Thresholds set afterwards are set to justify whatever happened. Everyone does this and almost nobody notices they are doing it. The whole value of writing it down first is that it removes the option.
3. The measurement plan
How, specifically, will you read the result? The honest answer for many channels is "not cleanly", and it is far better to know that going in.
- Geo holdout — run the channel in some regions and not others, then compare. The cleanest option available to most businesses, and underused.
- Matched-market test — pair similar regions and run the channel in one of each pair. Useful when you cannot hold out a large enough area.
- Clean last-touch — acceptable for direct-response channels with an unambiguous click path. Actively misleading for awareness channels.
- Post-purchase survey — crude, self-reported, and often the only signal available for offline and word-of-mouth channels. Better than nothing, worse than a holdout.
If none of these can work for a given channel, the test may still be worth running — but you must accept in advance that the result will be directional rather than conclusive, and size the bet accordingly.
4. The kill criteria
Under what conditions do you stop early? Name them: cost per acquisition above some multiple of target after some spend, or zero conversions after a defined volume, or a hard budget cap reached.
Kill criteria protect you in both directions. They stop a genuinely dead channel from consuming budget out of stubbornness, and — more usefully — they stop a live channel from being killed on day four by someone impatient, because everyone agreed up front what the stopping conditions were and "it looks slow" was not among them.
The one-page test brief
Hypothesis · audience · offer · budget cap · duration · success threshold · measurement method · kill criteria · who decides. If it does not fit on one page it is not a test, it is a launch.
The mistakes that ruin most tests
Testing too small to read
A budget that produces four conversions cannot distinguish a good channel from a bad one. Work backwards from the volume you need to see a difference, and if you cannot afford that, do not run the test — genuinely. An unreadable result is worse than no result, because it feels like evidence and gets cited for years.
Reading it too early
Every channel has a learning period during which performance is unrepresentative — algorithms calibrating, creative finding an audience, your own team getting competent at a new format. Reading on day three measures the learning period, not the channel.
Changing things mid-flight
Halfway through, someone adjusts the targeting, swaps the creative and raises the budget. Performance improves. What did you learn? Nothing you can act on, because you now have one result and four possible causes.
Confusing a channel test with a creative test
Bad creative will kill a good channel and get the channel blamed. If your test produces poor numbers, the honest follow-up question is whether you tested the channel or merely tested one execution on it. Budget for at least a few creative variants inside a channel test, or accept that a negative result is provisional.
Expect most of them to fail
Roughly one in four tests clears the threshold. That ratio is the programme working, not a bad quarter — and the arithmetic holds because failures are capped at a known amount while winners are not capped at all.
This is why we run channel development as a standing portfolio rather than an occasional project. Three to five small bets running at any time, funded from a ring-fenced line at roughly 10–15% of acquisition budget. Below that the tests are too small to read; above it you are gambling with money that should be buying customers today.
Write down the failures too
The single most under-rated output of a testing programme is the record of what did not work.
Without it, institutional memory walks out whenever someone leaves. Two years later the same idea resurfaces, gets tried again, fails the same way, and nobody remembers the first attempt. A channel ledger — one line per test, what was tried, what it cost, what happened, why — is a genuine asset, and it takes about ten minutes per test to maintain.