Ask the Experiment Nation - Experiment Nation

Ask the Experiment Nation

Q: How do I change people’s attitudes towards Experimentation?

Of course, there are a million ways to answer this question that range from demonstrating wins, being inclusive, and evangelism. This week, I thought of sharing a couple of the little things you can do that, if done consistently over time, can start to realize gains in the long run.

Words matter

Experimenters and CROs often say that learning matters, but at the same time, when seeing a test that didn’t go the way they wanted, say something like “At least we learned something.” Let’s take a second to reflect on that sentence. “At least” implies something minimally acceptable, like a consolation prize. I implore you to think about putting learning at the forefront of how you speak to your clients and stakeholders. Something like, “Now we know” is far more powerful. It implies that experimentation, regardless of the outcome, is a step forward. Running a test is in of itself progress.

Framing can help remove bias

Georgi Georgiev recently posted about the importance of “blinding” – where you essentially don’t share which treatment was the control and which was the challenger. They are rather presented equally as variants. While this won’t work in all cases, there is great value in doing this. More often than not, it helps to remove biases in interpreting results. One should not underestimate an audience’s attachment to solutions – especially those they were involved in building. Blinding helps you facilitate the focus on the proper interpretation of the results – which helps you and your place of work get that much closer to making more evidence-based decisions.

See you in 2 weeks,

Rommil Santiago
Founder, Experiment Nation


Download our eBook, “Learning Together”

With intros from Shiva Manjunath, Andra Baragan, and Tracy Laranjo, CAPM, our new eBook compiles over 60 learnings from 2 years' worth of Read more

Ask the Experiment Nation

Q: How do you respond to someone who says, "Let’s not experiment, it just slows us down. Can’t we just Read more

Ask the Experiment Nation

Q: Can you run tests on sites with little traffic? In a recent post, Magnet Monster posted an interview with Read more

Question of the week: When is it OK to run 2 A/B tests simultaneously on a site? Is it ever OK?

Q: When is it OK to run 2 A/B tests simultaneously on a site? Is it ever OK? We recently Read more