A/B testing is the one method that answers the causation question cleanly. Split your audience randomly, show each group a different version, compare. Because assignment is random, the groups are comparable on everything — including factors you never thought to measure.
That is the theory. In practice most business A/B tests are run in ways that guarantee unreliable answers, and the failure modes are consistent enough to be worth naming.
Decide the sample size before you start
This is the step teams skip, and skipping it causes most of the rest of the problems.
Before running anything you need three inputs:
- Your current baseline. If conversion is 3%, that is your starting point.
- The smallest lift worth detecting. Not the lift you hope for — the smallest one that would change your decision. If a 0.2% improvement would not alter what you do, do not design a test to detect it.
- Your tolerance for being wrong. Conventionally 95% confidence and 80% power.
Those give you a required sample size. The number is frequently larger than teams expect — detecting a modest lift on a 3% baseline can require tens of thousands of visitors per variant.
That is genuinely useful information, even when unwelcome. If you cannot reach the required sample in a reasonable window, you have learned something important before spending anything: this test cannot answer this question at your traffic level. Better to test a bolder change with a larger expected effect than to run an underpowered test that produces a coin flip dressed as evidence.
The peeking problem
The most common way business tests go wrong.
You launch on Monday. Wednesday, variant B is ahead and the result reads "significant." You stop and declare a winner.
The problem: if you check repeatedly and stop the moment you see significance, you will find it eventually even when the variants are identical. Random fluctuation crosses the threshold sooner or later. Checking daily and stopping at the first significant result produces false positives at a rate far above the 5% the statistics assume.
Two legitimate fixes: fix the duration and sample size in advance and do not stop early regardless of what you see; or use a sequential testing method explicitly designed for continuous monitoring, which adjusts thresholds to account for repeated looks. What is not legitimate is peeking daily with fixed-horizon statistics and stopping when the number looks good.
Why peeking manufactures winners
Run for whole business cycles
Weekday and weekend behaviour differ. Payday weeks differ from mid-month. A test running Tuesday to Friday captures one slice of behaviour and generalises poorly.
Run for at least one full week, ideally two. In Malaysia, be deliberate about festive periods: a test spanning Hari Raya or Chinese New Year measures behaviour during an atypical period, and the result may not hold in an ordinary month.
One change at a time — usually
Change the headline, the image, the button colour and the price simultaneously, and a winning result tells you the bundle won without telling you which element did the work. You cannot carry that learning forward.
The honest exception: if you are testing a fundamentally different approach rather than optimising an existing one, testing the whole concept is legitimate. Just be clear which you are doing, because the two produce different kinds of knowledge.
Statistical significance is not business significance
The ASA's statement on p-values is blunt about this distinction, and it matters commercially.
With enough traffic, a 0.05% conversion improvement can be statistically significant — meaning the difference is probably real — while being commercially irrelevant once you account for the cost of implementing and maintaining it.
Always ask for the effect size and its range alongside the significance verdict. "B beat A by 0.4 percentage points, 95% interval 0.1 to 0.7" is a decision-ready result. "B won, p < 0.05" is not.
Read the effect size, not just the verdict
What to do with a losing or flat test
Most tests do not produce winners, and organisations that treat this as failure stop testing.
A flat result is genuinely informative: it says this change does not matter, so stop investing in that direction. A losing result is more valuable still — you avoided shipping something harmful, which is exactly what the test was for.
The failure mode to guard against is re-running a flat test with slightly different segmentation until something looks significant. Slice a null result enough ways and one slice will appear to win by chance alone.
Where AI helps
- Sample-size calculation from a plain-English description of your baseline and target.
- Generating variants worth testing — headlines, copy, layouts.
- Analysing results correctly, including effect sizes and intervals rather than a bare verdict.
- Flagging design problems — ask it to critique your test design before launch and it will often catch an underpowered setup or a confound.
The caution mirrors the one elsewhere in analytics: AI will summarise whatever result you give it in confident language. If the test was underpowered or stopped early, the summary will still read persuasively. The design discipline is human work.
A workable checklist
- Write the hypothesis and the smallest lift that would change your decision.
- Calculate the required sample size. If unreachable, redesign the test.
- Fix duration in advance — at least one full week.
- Randomise properly. Not "morning versus afternoon," not "KL versus Penang."
- Do not peek and stop early unless using a sequential method.
- Report effect size with a range, not just significance.
- Record the result either way, including flat ones, so the organisation accumulates knowledge.
Our AI Analytics programme covers experiment design, sample sizing and result interpretation for business teams — HRD Corp SBL-KHAS claimable for eligible Malaysian employers.
