Avoid making these common mistakes when running multiple variants Experiment Nation

Avoid making these common mistakes when running multiple variants

If you aren’t careful you might select a less-than-optimum option

Often at times, we want to test multiple variants against a control at the same time. There are many reasons for this — sometimes we don’t want to wait to run each variant against a control in series, sometimes we’re indecisive, sometimes we don’t want to create another ticket for a future sprint (I don’t judge). Whatever the reason is — here we are, with multiple variants running at once. At least we’re not running an MVT. (Joking aside — there are some justifiable reasons to run MVTs but I’d only do so after running a series of split tests. The up-front work to set up an MVT is worth the trouble in my opinion. Don’t @ me.)

One common mistake is not to account for multiple comparisons. Simply put, the idea is that if you dig hard enough, you’ll eventually find something that’s statistically significant — which is most likely a false positive. A common procedure to control for multiple comparisons is to use something like a Bonferroni correction (remember kids, Wikipedia is not a reliable reference). This correction, in essence, decreases the alpha of your test — i.e., making it harder for a variant to be determined as different by making the threshold harder to reach. For those familiar with Adobe Target, you can see an option to correct for this on their sample size calculator below (I personally don’t love that it’s a bit hard to see.)

However, a more common mistake is to pick the variant that produces the highest lift without checking if it’s statistically different from the other variants. Unless you check the difference between the variants you risk actually selecting a less performant variant!

What other gotchas do you watch out for when running multiple variants? I’d love to hear them!


Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

The ROI of brand building

In our data-driven and data-informed world, we are increasingly being asked to prove that everything we do is delivering value Read more

Exploring the suitability of t-tests for A/B tests involving RPV with integrate.ai’s Summer Hu
Photo of Summer

A conversation with integrate.ai’s Summer Hu about Experimentation I recently had the pleasure of catching up with Summer, one of Read more

The how, what, and why of copy testing with Green Light Copy’s Eden Bidani

A conversation with Green Light Copy’s Eden Bidani about Experimentation I recently spoke to Eden about what conversion copywriting is, Read more

TechTarget’s Renee Thompson on how to effectively test on B2B customer properties
Renee Thompson

A conversation with TechTarget’s Renee Thompson about experimentation Experimenting on properties trafficked by B2B customers can be quite different than Read more

Rommil Santiago
Cart
  • No products in the cart.