How long should you run your A/B test?

Feb. 1, 2019, 11:37 a.m.

1 Answer

Feb. 23, 2019, 5:21 p.m.

While conducting advertising campaigns, we are constantly focused on improving the main metrics: CTR, CPC, CPA, CV, etc. One of the methods that help us achieve our goals and which we actively use in our work is A / B testing (Split testing).

A / B testing is a powerful marketing tool to increase the efficiency of your online resource. With the help of A / B tests, they improve the conversion of landing pages, select the optimal ad headers in advertising networks, and improve the quality of search. To run an A / B test, you need to create two versions of one element, such as a landing page or ad, with a change in one variable. Then you need to show these versions to two identical-sized audiences and, after the data set, analyze which version worked better.

The smaller the sample, the more influence each result has. You have already made a decision, and then the action of the new client completely changes the picture, and another action seems more logical. In order to judge the results of the A / B test with 100% certainty, it is necessary to conduct it with all the people on earth. Naturally, this is impossible, and not worth it. Moreover, it is not necessary to conduct an A / B test on all of your clients. It is enough to choose the optimal audience size, the increase of which will affect the results slightly. This is the subject of statistics. To understand that your result is not random, but amenable to statistical dependencies, you need to calculate the size of the sample, which should work in each variant.