Fresh Egg’s Sebastian Larsson: Experimentation roadmaps begin with research
A Conversion Conversation with Fresh Egg’s Sebastian Larsson
We hear stories of how some companies run tens of thousands of tests every year, and how they test anything and everything. While testing a wide array of ideas that range from optimization and exploration is good— something that the best Experimenters often encourage is having an Experimentation roadmap. An Experimentation roadmap is, as the name implies, a plan of upcoming Experiments that is rooted in research and goal alignment. I recently spoke with Sebastian to understand how his agency works with clients to create them.
Rommil: Hi Sebastian, how are you? Thank you for taking the time to chat with me today!
Sebastian: Hey Rommil, I’m great thanks. Kids half term holiday here in the UK so have had a few days off work to look after my two girls.
Can you share with our readers what the Fresh Egg Digital Marketing Agency is about and what you do for them?
Sure, Fresh Egg is a digital agency located on the south coast of England in a town called Worthing. We pride ourselves on our customers-first approach to digital marketing. We always start any client relationship by understanding our client’s customers and their experience with the brand. My role is Senior Conversion Strategist in our Conversion Services team.
How does Fresh Egg leverage Experimentation internally?
If I understand your question correctly, it’s something I admit we could be better at. We run experimentation for multiple clients, but we are not as good at using it on our own website. It’s usually a time issue as client work is always prioritized.
When working with clients, how do you go about helping them develop their Experimentation roadmaps and strategy?
We always start with a CX Discovery. This involves qualitative and quantitative research. We do everything from internal stakeholder interviews and Audience Empathy Mapping sessions to user testing, user interviews, heuristic website analysis, competitor analysis and of course a thorough check of their quantitative data. All this information then informs the creation of an experimentation roadmap. We do everything from running the whole testing programme for clients to training up internal teams to run their own CRO.
One of the biggest hurdles for Experimentation is the sharing of learnings. Because you work with so many clients, how do you go about ensuring everyone shares and absorbs the learnings?
We are a small team, four of us, and we all sit together so it’s natural for us to discuss learnings or issues that come up daily. We also have a document where we keep combined test data for all clients.
What are some of the challenges you face when trying to launch experiments for clients and how have you overcome them?
The main challenge is usually technical. For newer clients, we don’t know their website as well as they do, so having a thorough technical handover before starting the work is crucial. We also try and get sign off during the design stage. That way we don’t end up making changes once the variations have been built as this usually takes a lot more time and ends up delaying test launch. Basically, agreeing to the process with the client before starting the work saves a lot of time for everyone.
So every client, understandably, wants to understand the return on their investment with regards to Experimentation. Often at times, the observed lifts you see from Experiments, don’t materialize when they are rolled out. How do you explain this to clients?
This is a tricky one. We often work with smaller clients who might not be used to running experimentation programmes. I believe that it’s best to have a discussion upfront that test results are only valid for that test period. We can say to a decent degree of certainty that for the duration of the test we have increased revenue/leads by a certain amount. Leaving a holdout group running in control or rerunning the same test down the line to see if the same results still stand is one way of seeing if the initial result holds up. I’m not saying that we have the perfect solution to this as it’s hard when clients want to see that ROI and it tends to be short term wins and not a long-term strategy that the client is after. We are constantly working on better ways of explaining the value of experimentation to our clients that is not just instant revenue uplifts but long term strategy that leads to customer satisfaction.
Finally, it’s time for the Lightning Round! Describe experimentation in only 3 words.
Research, Testing & LEARNING
Bayesian or Frequentist?
On the fence so far, I think there is room for both if you understand how and when to use each method but I’m leaning towards Frequentist until someone convinces me otherwise.
What is the Swedish word for Conversion Strategist?
Ah, it’s funny because I have only worked with CRO in the UK. I had to look it up as I’ve lived here for over 14 years now and it didn’t exist as such when I left Sweden. I now know it’s Konverteringsexpert
Sebastian, thanks for joining the conversation!
Thanks for having me. Really enjoy reading the interviews on Experiment Nation so keep up the good work!