Experimenting in an Agile Environment

Magnolia Carvajal

A chat with Journey Further’s Magnolia Carvajal about Experimentation

I recently chatted with Magnolia to get her take on whether we should avoid overlapping tests and how to kickstart a culture of Experimentation.


Rommil: Hi Magnolia, thank you for taking time away from your busy schedule to chat with me today. How are you?

Magnolia: Hello, no worries at all thanks for reaching out, I am very well, happy to be back in the office after Covid and finally enjoying the warmer weather 🙂

Can’t complain about more sunshine, that’s for sure. So Magnolia, I’d love to hear about how you got into Experimentation?

It was really random actually, I studied marketing with Psychology at university and at my first job at Tesco and I was lucky enough to be invited to a discussion around reducing average handling time on the websites; from that meeting I asked if they would put me on a digital marketing course through Tesco Academy and they did! I learnt a lot and then I got put to work on the Tesco Clubcard website testing to reduce the calls coming into the call centre — it was all CRO from there on.

What are your most and least favourite things about Experimentation?

Most Favourite thing: When you have done all the research, got all the data behind your test, you run the test and it LOSES. Why??

Least favourite thing: As above.


Advertisement


Changing gears to something more tactical — What’s your view on deciding what to test?

Analyse, test, analyse, repeat. Also, insights taken from previous tests is a real driver of what to test.

A common question I hear is, “How many changes should be made to comprise a variant?” What’s your take?

Very much depending. I generally will only make one change unless it is a split URL or landing page test. Otherwise — how do you know what it was that worked/didn’t work?

How do you go about selecting a success metric, or a set of metrics?

Depends on the hypothesis and what we want to get from the test.

Companies that catch the testing bug always want to run more tests. What’s your opinion on concurrent testing? How do you account for potential interactions?

My opinion is that there is no way to not do it in an agile environment, but this does not mean running 2 tests on the same page as that is a direct crossover. However, running tests on the homepage and then further down the customer journey like the checkout is necessary in an agile environment. We account for potential interactions by segmenting properly when doing post test research and examining the interactions between any 2 sets of A/B tests

How do you know if a company has a culture of Experimentation?

You would know a company has a culture of experimentation if they had a data driven web development process and not just changing stuff from the top guy.

What steps can you take to help a company adopt one?

Keep testing — show them the results of failed tests including the money saved from dev work, show them that if you had released the failed test on to the website without testing it, it would have actually hurt the company’s conversion rate as well as losing money paying for dev work and therefore losing revenue.

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

Bayesian

Favourite experimentation tool.

My dev really likes SiteSpect

If you couldn’t work in Experimentation — what would you do?

Something outdoors, only experimentation will keep me inside!

Describe Magnolia in 5 words or less.

Analytical, Methodical, Creative, Innovative, Extrovert

Thank you!

And thank you 🙂



You may also like

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s