Elise Maile on the importance of an Experimentation Roadmap and getting buy-in from developers Experiment Nation

A Conversion Conversation with Swiftmonkey’s Elise Maile

Experimenters face a lot of challenges. Some of these challenges are expected, like technology issues, Experiment design hurdles, and resource shortages. But there are a lot of overlooked challenges we face as well such as the friction from developers who weren’t consulted on a vendor-selection process and the frustration of convincing leaders that ignore your recommendations. I recently chatted with Elise, a 6-year veteran in this field, to explore these topics.

Rommil: Hi Elise, how’s it going? Thank you for taking the time to chat with me today!

Elise: Very well thank you, and thank you for asking me!

Could you tell our readers a bit about yourself and how you got into Experimentation?

Sure. I fell into experimentation about 6 years ago. I’d been working as a self-employed web designer and developer and one of my contracts was doing a little bit of A/B testing. They were a B2B company so had limited traffic levels meaning that experiences were running for months, but despite that, I could see the potential. My next role was within a dedicated Optimisation team at Thomas Cook, building A/B and MVT tests. I’ve never looked back.

Having worked with so many clients over the years?—?what would you describe as an Experimentation culture?

Essentially it’s about embracing experimentation; using data and research to drive decision making. And that may look different depending on the company?—?for some businesses that could be testing every single feature, but for many companies, this isn’t realistic, in which case simply using research (whether its qualitative or quantitative) to support decision making is a step in the right direction. They may not have the bandwidth to test everything, but so long as they’re asking the question, that’s the beginning of an Experimentation culture.

What are some of the telltale signs that a company hasn’t embraced Experimentation?

Investing in tools but not people or process is a big one, and when results and recommendations are ignored by management in favour of their own subjective opinions. It is frustratingly common.

What are some of the hurdles for companies in adopting such a culture and what are some of the most effective things people can do to nurture one?

There seems to be an assumption that a ‘culture of experimentation’ is when everyone in the business is empowered to run their own tests, but I don’t believe that is a solution that is suitable for every business. In my experience, having a centralized team or person to manage experimentation is the best way to begin to build that culture, someone should continue to own and monitor experimentation across a business because there can be a lot of conflicts, repeat experiments or overlapping.

In terms of nurturing this kind of culture, it always comes down to communication. Finding ways to get different teams involved in experimentation, educating them and getting them excited about what you’re learning. That could be via workshops, presenting your findings on a regular basis, sharing results and just being visible and approachable. If you do it well you’ll find that others will start seeking you out with their own ideas and research. I think building that excitement is the hardest part but having everyone on board is a big part of an Experimentation culture.

Changing gears. Let’s talk about roadmaps. A lot of Experimenters, especially during the early stages of an Experimentation program run pretty random experiments. In your opinion, what is, and what is the value of, and Experimentation roadmap?

An experimentation roadmap is essentially a plan of what experiments you’re going to run over a set period of time (commonly 3 months) that have been prioritized for maximum impact. By planning ahead you can ensure that you’re experimenting on a variety of areas on your website or app that are more likely to improve conversion or user experience. It’s a useful exercise to do because similar to a project plan, it gives the business visibility of what is going to be tested and you can ensure experiments are aligned with campaigns.

How are the key elements of an Experimentation roadmap?

I present mine like a Gannt chart, cascading days/weeks and clearly marking build time, run time and reporting time. But roadmaps need to take into consideration a couple of elements; test clash (tests that run at the same time on the same page), marketing campaigns or peaks which usually run to strict timelines, how complex the experiment is to get set up and the estimated traffic or conversions are for where the test is running?—?it might require longer to build or to run.

What should NOT be in an Experimentation roadmap?

I’d say ‘fixes’?—?those last-minute bugs that experimenters are asked to run at 100% on their testing tool because there’s no dev time available. But honestly, I actually build in time to allow for reactionary work, fixes included, but also those campaigns that someone forgot to mention three months ago.

Now, as Experimenters start running more Experiments?—?they start running into each other. How do you suggest people handle that, and what is it important to address?

The majority of testing tools claim to mitigate this, either by not allowing a user to fall into multiple tests or through their stats engine. I believe it’s best to try to avoid running tests on the same page at the same time and to track different KPIs for tests that do run in parallel. It’s a good idea to also segment your audience; device, channel etc.

Experimenters are a rare breed according to many. What is your take on that statement and why do you think that is?

CRO and experimentation is still a relatively new discipline, in the sense that the concept has been around for a long time, but the tools and best practices have not. Experimenters are rare because it’s not a traditional career path; we’re marketers, developers and product owners who’ve stumbled across CRO and decided to change paths. Alongside the multitude of skills required (attention to detail, analytical mindset, curiosity of human behaviour, creativity) it’s a unique person who fits into the role of experimenter.

Finally, it’s time for the Lightning Round!

Have you ever A/B tested something in your own life?

No; there are too many variables to get statistical significance, I’m only one user. But I do know which is the fastest tunnel to exit my local tube station.

What is the one thing you wished your clients did that would make your life easier?

Nurtured their developers and engineers?—?they’re responsible for installing the tools and implementing ideas. Get their buy-in before you even buy a tool or invest in an experimenter, otherwise, I’ll spend much of my time building bridges or defending my profession.

If you couldn’t work in Experimentation what would you do?

I might have continued in front end development, or become a PA as I’m extremely organized.

Bayesian or Frequentist?

Bayesian

What is your favourite Experimentation platform?

I don’t mind so long as it’s installed correctly.

Describe Elise in 7 words?

In true CRO form, I did some research and asked my friends & peers. The most common words were: “Calm, honest, loyal, thoughtful, smart, independent & resilient.”

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.