Ask the Experiment Nation - Experiment Nation

Ask the Experiment Nation

Q: Can you run tests on sites with little traffic?

In a recent post, Magnet Monster posted an interview with Jeremy Horowitz about what to do when you don’t have enough data. You can check out the post here.

Jeremy goes on to say that we aren’t in a science lab and we don’t have to wait for statistical significance. This, as expected, poked the CRO community the wrong way. However, resisting my knee-jerk reaction, I have to admit there is a lot of truth to what Jeremy says, though perhaps it could have been framed a bit better. (Or perhaps it was phrased perfectly since the CRO community engaged with the post).

Here’s my take based on my years of experience running hundreds of online Experiments (as well as some offline ones) for companies like Loblaw Digital (Canada’s largest retailer), Bell Canada, Autodesk, Ritual, 500px, Flipp, and theScore.

From a pure experimentation perspective – one absolutely can forego using statistics and just see what happens. We we can believe the dictionary, the dictionary definition of an experiment is: “a scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact.” A scientific procedure (aka the scientific method) states nothing about having to use a statistical approach for analysis.

That said, without statistical rigour (and, I’m not saying that only randomized control trials will allow for statistical rigour because, surprise, Experimentation is bigger than A/B testing on landing pages), the data isn’t very trustworthy and definitely increases the chances of making a wrong decision and drawing the wrong conclusions.

Now before the angry CROs claim victory, if one has the time, the funding, the agility, and the energy to go down potentially incorrect paths and rollback once it’s discovered that they are down the wrong path – one can do so. It’s a pretty expensive path, one I wouldn’t recommend in most cases, but it is a path.

In cases where you don’t have a lot of traffic, you might be better served in spending your time doing research and understanding the customer. But even with that done, you eventually have to try something on real people – so you’re back to our initial problem, not having enough traffic.

So assuming you have a good amount of research, and you need to try something, I’d personally lean towards finding several statistical approaches that worked with the available traffic and comparing the results to see if they aligned so that I could make a more trustworthy decision. For example, one could take a Bayesian approach (which performs well with low sample sizes), or take bootstrap with replacement approach, or run repeated low powered tests and run a meta analysis, or even, in the interest of beating the already dead horse and writing run-on sentences, a roll out with a pre/post analysis (which is a form of pseudo-experiment where one could perform a paired-t analysis). Each of these approaches are statistical – just with varying levels of bias.

So to make a long story short, there are always statistical approaches one can take and one should explore them before potentially throwing away money. But at the end of the day, it comes down to choosing what kind of Experiment you want to run.

You’ll have to do a cost-benefit analysis (and understand your organizational’s core competencies) and choose between a more trustworthy “science-y” approach or the see what happens route.

Which kind of Experiment would you run?

See you in 2 weeks,

Rommil Santiago
Founder, Experiment Nation


Download our eBook, “Learning Together”

With intros from Shiva Manjunath, Andra Baragan, and Tracy Laranjo, CAPM, our new eBook compiles over 60 learnings from 2 years' worth of Read more

Ask the Experiment Nation

Q: How do I change people's attitudes towards Experimentation? Of course, there are a million ways to answer this question Read more

Ask the Experiment Nation

Q: How do you respond to someone who says, "Let’s not experiment, it just slows us down. Can’t we just Read more

Question of the week: When is it OK to run 2 A/B tests simultaneously on a site? Is it ever OK?

Q: When is it OK to run 2 A/B tests simultaneously on a site? Is it ever OK? We recently Read more