No one has that kind of time

How many times have you gotten excited to run an a/b test and calculated the required sample size only to discover that your test will need to run for months? Often at times, we are quick to mumble to ourselves, “If only we had more traffic.” While it’s true, your daily traffic does play a key role in how long your experiment will run, don’t be so quick to point fingers. The blame sometimes lies with the person in the mirror.

Minimum Detectable Effect

Imagine that one day you are tasked with spotting all the animals in the forest. You open your eyes and you see the scene below:

The human eye is a funny thing. When it is presented with similar stimulus for a prolonged period of time, it tends to ignore details. Because the forest is mostly green, grey and brown, your eyes can easily miss the squirrel in the middle of the photo?—?as well as my handsome face hidden in the tree trunk.

The reason you (probably) missed my face was that you didn’t know to look for anything that small. If you had known, you probably would have taken more time to examine the photo. This, in essence, is the concept of Minimum Detectable Effect (MDE)?—?i.e. the smallest thing you should look for.

In the example above, had you were told to look for squirrel sized animals, you would have completed your task quickly. Had you were told to look for a quarter-sized Rommil, you would have taken much longer.

When running experiments you have to decide how big an effect to detect. Do you care about a 1% difference? a 5% difference? or a 20% difference? The bigger the difference you are trying to spot, the smaller the sample size you need.

But I don’t think I’ll see a large effect?!

Depending on your business, you many not have a baseline visit-to-order conversion rate any higher than 2–3%. It would be unrealistic to expect any experiment to yield enormous differences here.

In these cases, you should focus on micro-conversions. Breakdown your conversion rate into smaller steps and somewhere along the funnel you should find a conversion rate where a huge change is possible. Make moving that micro-conversion rate your experiment’s KPI and you will dramatically shorten your experiment’s flight time?—?sometimes from months to days.

Below, is a framework to help you understand the effect MDE and KPI on experimentation flight-time. As you can see, selecting a larger MDE has a strong effect in shortening your experimentation time than the KPI you select.

On a related note, this is another reason why you should always test extreme variants. Not only do they help you find the global maximum, those tests run faster and in turn, help you learn faster.

A world of caution

While your experiment will indeed run faster, don’t go crazy and select overly large MDEs. Remember your experiment has to have enough power to detect something meaningful.

(Special thanks to Georgi Georgiev and John Meakin for contributing and making sure the stats are solid!)

Come back this space often?—?I’ll be sharing even more Experimentation tips in the future!

UPDATE?—?June 25, 2020

Or you could use a Sequential or Bayesian method. Just saying =)

What do you do to increase your learning velocity? Feel free to reach out! Always interested in hearing how others tackle experimentation challenges.