What are the rules of A/B testing?

Feb. 4, 2019, 1:08 p.m.

1 Answer

Feb. 24, 2019, 5:01 p.m.

When the A / B test is started and the traffic has gone, nothing depends on you anymore. Will the experiment prove useful, or will it be a waste of time and money? It depends on what you put into it before launching - what elements, hypotheses, ideas.


Rule one: the default control option is correct

If this is not about creating a website from scratch, but about testing an already operating site, then the initial version of the page should always be in priority. A kind of "presumption of efficiency." And so on until it turns out that one of the test options shows higher rates.

This approach ensures the continued development of the site, in which each new version is superior to the previous ones.


Rule two: always wait a week

In seven days everything can change. Yesterday you had a winner with a margin of 240%, but then his performance began to deteriorate sharply. Continue the tests. Visitors can behave differently on different days of the week, time of day, on weekends and on weekdays. It is more reasonable to spend an extra day on testing than to fix errors long and hard.


Rule three: Use Options with 99% Statistical Significance

If the test goes with a sufficient level of traffic, the statistical reliability of its results increases every day. What mark is it worth staying at? Many marketers consider the “gold standard” level to be 95%. This is about one “false positive” result for every 20 tests. Is it a lot or a little? You decide.


Now, such tactics will not help you achieve success now or in the future. Today, you need to resort to web analytics to get information based on user feedback in real time. This data can be used to optimize any area of your site since it will be based on the real opinions of real customers.