Specsavers' Melanie Kyrklund on How to run a high-impact optimization program
Rommil Santiago 0:00
Hi, I'm Rommil.
Tracy Laranjo 0:02
and this is Tracy and you're listening to Experiment Nation: The Podcast.
Rommil Santiago 0:06
On this show we interview experimenters from around the world who share their stories, their lessons, and advice with you, our listeners.
Welcome everyone to a new episode of Experiment Nation and today on the episode we have Melanie Kyrklund from Specsavers. Hi Melanie.
Melanie Kyrklund 0:49
Thanks for having me
Rommil Santiago 0:50
Yeah, no problem. Thanks thanks for being here. How have you been?
Melanie Kyrklund 0:54
Yes all good. We're getting to the end of a long winter here yeah locked down winter as well but we're seeing the first glimpses of spring so looking forward to some sunny days
Rommil Santiago 1:07
Yeah for this for our audience could you tell us a little bit about yourself you know where you're at because clearly you have an accent so i'm sure our audience would be very interested in that so yeah we'd love to learn more about you
Melanie Kyrklund 1:20
Yeah sure so I'll address the accent first so I'm half British half Finnish and i grew up in Italy and i now live in Netherlands so I guess my accent is a mix of all of these influences in my life and all these places.
Rommil Santiago 1:39
So not not to start any wars or anything but which one's your favorite?
Melanie Kyrklund 1:46
Yeah well i think you'd have to be Italy right? Just the food lifestyles weather. It's a beautiful place I don't know if you've ever been.
Rommil Santiago 1:54
I've never been to Italy I've been to the Netherlands and England and I have to say I'm very jealous of your background that sounds like a great way to grow up.
Melanie Kyrklund 2:05
Yeah indeed, it has its advantages in that you kind of fit in everywhere but then you also grow up without a strong national identity as well so it's like a double edged sword.
Rommil Santiago 2:19
That's fair and so okay so now you've you've come from all these countries where are you at now and what do you do?
Melanie Kyrklund 2:25
So I'm based in Amsterdam Netherlands and I'm working for Specsavers at the moment which is a very large global optical retail chain and I work on the ecommerce websites working on contact lenses and online prescription glasses which is a relatively new and exciting market to be in. Looking at my career in digital I've been around for a long time so I started out in marketing but the past 11 years have really been focused on optimization at ecommerce companies so I've worked at booking.com at Staples for example which are well known companies and optimization experimentation has really been the common thread throughout my career the past decade or so there have been moments where I focus more on analytics of product management but yes experimentations always been there. You never get stalled.
Rommil Santiago 3:25
Coming from marketing how did you end up in optimization?
Melanie Kyrklund 3:28
At the time I was talking to a company a large broadband and tv provider in Europe I didn't get that particular job but a few months down the line they had a consulting budget available to set up the optimization program so yes I sort of landed in optimization of the back of that.
Rommil Santiago 3:53
What kept you there? You've been there for quite a few years now.
Melanie Kyrklund 3:56
Well it's just such a multifaceted discipline I think that's that's what really sparks my interest in it. You have to understand the commercial dimension you have to understand the user you have to understand data and also have to translate insights into features that can be tested. So i think the whole experiment design parts very interesting as well then you're there's a psychology element on top of that so you're constantly learning and this whole space has been evolving quite rapidly over the past 10 years as well it's becoming more and more embedded into organizations and i think generally speaking it's being done a lot better than it was become a lot more mature.
Rommil Santiago 4:40
So I've seen you know this kind of this debate about optimization where there are folks that feel that should be separate to product and marketing like its own thing or the other side of the coin where folks believe that should be embedded in all the functions do you have a feel of which way it should go I've seen both. I'm curious to hear what you think.
Melanie Kyrklund 5:02
Yeah, good question. And I think it all depends on the company you're in, and the maturity you're at within the organization. I think it's better if experimentation is integrated into operations more. So within marketing and product, I think that helps foster the culture of experimentation. I think sometimes it makes sense to have an experimentation function, which is spun off. But that's more when you're working in the innovation space, where it might be beneficial to have a separate team who are removed from day to day operations in order to be able to foster a bit more of that innovative thinking, so looking five, seven years ahead.
Rommil Santiago 5:49
Cool. So let's change gears a bit. Now, before all my interviews, I like to do a little bit of Googling on our guests. You know, I try to come up with questions. And I stumbled on a picture. I think you're looking at it right now. Where you're holding something, I believe it's a brain. I'd love to understand, you know, what this picture is about? And why did you choose to hold a brain?
Melanie Kyrklund 6:15
Well, firstly, if that's if that's the worst thing you could find out about me, then, you know, I'm kind of relieved. So that was when I was working for a digital agency a couple years ago. So actually, that that whole pose wasn't my doing. It was part of sort of internal branding they were doing for some Medium articles we were writing where the the people contributing had to have to hold a brain.
Rommil Santiago 6:49
Okay. Okay. So I was kind of hoping there's something more sinister back there. But yeah, that that sounds very logical.
Melanie Kyrklund 6:56
No, there wasn't any, like, weird dissection going on, or anything like that.
Rommil Santiago 7:02
Or strange e commerce company that sells body parts.
Melanie Kyrklund 7:06
Yeah a good one. Think out on the dark web, you can find that.
Rommil Santiago 7:11
Exactly how to even know what optimization looks like. And that's probably very blackhat. Okay, so a section I like to do with all our guests is something in the news. . And to kick it off, I like to play this little sound. And you got to bear with me because I try to be professional.
In the news. So there's this article that I found in Harvard, HBR, Harvard Business Review, and it was an article by Thomas, sorry, I can't remember his name..it's Stefan Thomke . I'm probably butchering his name. And if he hears me, I'm sorry, but I did buy his book. So at least maybe you take comfort in that. So in this article, he talks about the culture of experimentation. And that it's it's more than just tools that are important. There's been so much talk about how to build an experiment culture at companies these days. I was curious, in your opinion, what does a good experimentation culture look like? And why is it important?
Melanie Kyrklund 8:10
Yeah, good question. So good, can look different depending on the organization. I think when I reviewed that, that article, I think what frustrated me is that booking.com is also always touted as being sort of the flagship, experimentation culture.
Rommil Santiago 8:29
Yeah, oh, this interview just got really interesting. I was interested before, but now I'm really interested.
Melanie Kyrklund 8:37
Basically, booking is also essentially, a marketplace, they don't really have any decentralized operations when it comes to e commerce. So it's like one domain. But you can imagine if you're running separate web domains in different markets, you have decentralized operations, you don't have a lot of traffic necessarily in all in all markets. If you have stock issues. In our case, we also have retail partners that we need to think about. So some ways that that picture becomes a lot more complex, then it's hard to really be able to aim for what booking.com is doing. Yeah. Now, I think what they do do well, is Well, apart from the whole, like experimentation platform and the centralization of the tooling and the best practice from a data perspective, I think that's, that's definitely a foundational element of a good experimentation culture. What I noticed when I was working there is that senior leaders decided that there was a common way of working and tackling problems through experimentation. So indeed, everyone who walked through that door was trained in that way of working and thinking.
Rommil Santiago 9:49
Kind of like this is how we were going to work and it includes experimentation. Yeah. If you don't, if you're not along with it, then you're going to have a rough time. Exactly.
Melanie Kyrklund 9:56
That makes sense. Yeah. I think that that decision from, let's say, the highest level to work that way, and then actually making that filter down into the training was very, very consistent.
"...it's really about creating that sort of pocket of excellence within that organization and showing the value of experimentation. And then, I think, bottom-up, once you've proven the value. Then you're able to partner with other areas of the business." - Melanie Kyrklund
Rommil Santiago 10:10
I really liked the way that you've described that where experimentation is clearly part of their culture. And I guess I had a question of, maybe it's self evident. But what comes first? Is it the culture or the program?
Melanie Kyrklund 10:22
Well, the program has to start first. So if you're stepping into an organization that's not familiar with this way of working, then it's really about creating that sort of pocket of excellence within that organization and showing the value of experimentation. And then I think, bottom up, once you've proven the value, then you're able to partner with other areas of the business. So being able to solve problems together by experimentation.
Rommil Santiago 10:54
So like leading, but leading by example, essentially.
Melanie Kyrklund 10:56
Exactly, exactly. So if you think about it, you might start on the e commerce journey, and working with some product owners, and aligning the experimentation with the product roadmap, and introducing a new way of working that through experimentation. But then I think further down the line, especially if you have to solve customer problems. For example, if you need to fix acquisition, then it's not only about the on site experience, then you have to bring experimentation further out in the funnel. And we'll be tackled as journeys cohesively with marketing. So they have the opportunity to start bringing marketing and multidisciplinary teams together to tackle some of these business issues and customer issues.
Rommil Santiago 11:44
On a tactical level, and you touched upon working with product, I'm always wondering how people tackle this problem in the sense that, how do you ensure that your experimentation program aligns with what products trying to do their their vision? And like, how do you always ensure that that you can support them?
Melanie Kyrklund 12:02
I think with experimentation, one of the key goals of the program should be to create a common understanding of what should be worked on. And why. Because if optimization and product teams aren't on the same page about that, then probably you're not making best use of your resources. So I think a key goal of experiment is to really become sort of facilitators within the business to create this common understanding. And the way you do that is by using user research, quantitative techniques, in order to get to a very clear picture of who the user is, what their needs are, and how the features align to those needs. And then well basically is a case of really aligning the testing and the product development with needs, and the features that align to those. In the absence of that, then testing can become quite fragmented, rather than being focused on well thought out solutions and reliable body of knowledge to support a certain customer need.
Rommil Santiago 13:07
So to reflect on that it sounds like to partner with with product, you have to have this common context in terms of what the situation is with the customer their needs. And then partnering them to to understand what good looks like and to to co develop these features slash experiments. Is that do I have that right?
Melanie Kyrklund 13:28
Yeah, absolutely. And sometimes testing will take the lead in certain areas. And in other areas, the product owners might be taking the lead on what's worked on.
Rommil Santiago 13:40
So let's say you have this alignment, you know, you have these main areas that you feel like you should work on. Yeah, time and resources are limited. How do you pick the effort or the initiative to pursue?
Melanie Kyrklund 13:55
So there's, there's a few few factors that sort of play into this. One is getting an understanding of what the bigger themes are to tackle. And typically, you'll see that in your user research, as well as in customer feedback, if you're running post purchase polls, for example. So those areas will will become quite apparent about what the biggest pain points are looking at those and in a bit more detail, you can look at what your potential impact is in those areas. So based on potential revenue from a certain uplift, yeah, so quantifying those opportunities as well as important and then you need to also factor in development time for each hypothesis, deployment time and test durations, you get a sense of where your quickest wins are and biggest wins. I think also, if you're stepping into a company where maybe optimization is in its infancy, obviously there's a lot to tackle across the whole user experience. Something that's helped me is also past experience can help as well. So you'll probably have a good feeling of which lines of testing are most likely to have impact. Because you probably want to show some results pretty fast. Yeah, if it's a new optimization program, so for example, in my case, when I started at Specsavers, one of the thing, first things I tackled was the offer journey. Because customers are looking for value online. It's probably one of the primary drivers to online buying across a variety of verticals. And so I spotted some definite opportunities of making the signposting over offers much clearer as well as the basket interaction with offers. And that was based on what I'd known had worked repeatedly in the past. So I think as you gain more experience, you're also able to bring this dimension in.
Tracy Laranjo 15:55
So Melanie. When you go about the process of identifying these different areas of opportunity, do you have a favorite prioritization framework that you use? Or do you just kind of use, you know, your past expertise and go from there?
Melanie Kyrklund 16:13
I use a scoring template. I wouldn't say it's a standardized one. But I bring into the dimensions already sort of covered off. So the potential revenue, uplift the time to development, the test duration, also the time it would take to deploy the feature if it proves successful.
Tracy Laranjo 16:33
Awesome. So is this a template?
Melanie Kyrklund 16:36
A Google Doc.
Tracy Laranjo 16:37
So this is a resource that someone could hypothetically just kind of pull to start on their own if they're new to the experimentation world.
Melanie Kyrklund 16:48
Yeah, absolutely. And we have the revenue up lifts per all the different funnel areas and in different markets already defined. So that's a supporting data sheets, we were able to just look back at that when doing your revenue estimates, rather than having to pull the data off which experiment.
Tracy Laranjo 17:45
Issues of traffic volume aside, yeah, do you think a business can be too small to benefit from an experimentation program?
Melanie Kyrklund 17:53
They could never be too small, because the different techniques that you can pull in in order to learn. So I've worked on products where which were non existent even. So new product development, taught me a lot. Because previously, I always thought, with user research, it's only 10 people, it's only 20 people, I'd rather put an experiment out there and get the volume behind the hypothesis. So when working on smaller companies and newer products, I had to I didn't have that traffic available to me. So I had to rely on different insight techniques and testing techniques. And I found them very illuminating , the user research. And whenever I tried to bolster my learnings from the user research with maybe some quantitative surveys, I often saw those learnings converge. I suppose I was able to trust user research a lot more after that. And I must say, at Specsavers over the past year, some of the biggest wins have been off the back of moderated user research as opposed to quantitative analysis.
Tracy Laranjo 19:02
So in terms of the user research, do you mean surveys, heat maps, like what's your go to for this kind of situation where you don't have traffic on your side?
Melanie Kyrklund 19:12
That'd be user interviews, and potentially asking users to conduct tasks as well on competitor sites, even if you don't have your own front end experience to put in front of them. So yes, the interviewing and task based studies are definitely very illuminating.
Tracy Laranjo 19:33
Great. Now, how often would you say you're surprised by your experiment results?
Melanie Kyrklund 19:37
Oh, all the time, all the time. And especially at the moment that I'm working on. There's a newer market, which is online prescription glasses. So not necessarily that established I'm, I'm constantly surprised by the experiment results, so it's still very fulfilling area to work in.
Rommil Santiago 20:01
Great, yeah, I've been in a space for a little bit too. And you you never guess correctly all the time. How do you keep your ego intact? When you're when your guesses end up being wrong?
Melanie Kyrklund 20:14
Oh no, I went through this process years ago. I shed my ego a long, long time ago.
Rommil Santiago 20:23
It feels like something that all CROs or optimization experimenters out or you want to call them, they kind of have to check the ego at the door. Yeah, and going, Oh, I could be wrong, I probably will be wrong.
Melanie Kyrklund 20:34
It's absolutely humbling. So actually, I I've run an experiment just recently, which I thought would be a big improvement. And it's actually bombing the negative downturn. But that also shows me apart from the fact that the learning off the back of it will be very valuable. But also having a negative results like a conclusively negative result also shows me that I'm testing bold ideas, right. And there's a lot of learning to come off the back of those.
Rommil Santiago 21:05
I think my time in this industry, when when folks come to me from whatever department saying this is totally going to give us a 15% lift in so and so. Yeah, I've become very skeptical. I was like, I'm sure it will. Let's give that a go. Let's see the test says. So it a little bit of a change of gears. I have the section I like to call the lightning round. And because it's a section of course it has some music. These are not questions to make you think very hard. You know this, your first reaction would be great. Yeah. So here they come. Be ready? Yeah. Bayesian or frequentist?
Melanie Kyrklund 21:43
Frequentist.
Rommil Santiago 21:44
Okay. Why are folks in the Netherlands so tall?
Melanie Kyrklund 21:49
The Dutch like to put it down to their diets. They eat a lot of bread and cheese and milk. Okay. So yeah, I know, I think in most countries, that wouldn't constitute a healthy diet. It seems to serve its purpose over here.
Rommil Santiago 22:07
Yeah. I was there many years ago. And, like, I'm six feet tall, and I never felt so short. Where I was dwarfed by, like, everyone, not just men, but women. Old people, everyone. Yeah.
Tracy Laranjo 22:20
Yeah, absolutely.
Rommil Santiago 22:22
I've never seen up someone's nose before. Okay, so according to Trafalgar, I'm probably pronouncing that wrong. The Dutch love licorice. I don't know if this is fact. I just read it recently. So is black licorice, the worst candy ever? Or is it just terrible?
Melanie Kyrklund 22:39
So this is a polarizing topic. Yeah. Right. Because some people some people love it. Some people hate it. I quite like it, to be honest. So. So I would say it's just terrible and interesting. Because if I think of the worst candy ever, like as images are conjured in my mind, like something like aniseed flavor. I think that would be far worse than a bit of like salted licorice.
Rommil Santiago 23:04
Oh, yeah. Interesting. Yeah, I am not a huge black licorice fan. My wife is I there's got to be something good too. But I don't know. It just never never resonated with me. Okay. Describe yourself in five words or less?
Melanie Kyrklund 23:23
Yeah, I think when you when, when you interviewed me, for Experiment Nation in the past, I said it was impossible. So I'll stick to that. Oh, yeah. Well, you know, I think as as one gets older, you sort of realize that, you know, that the so many sort of extremes in one's character, like no one is completely one thing or completely another. So for example, you can be sensitive, but you can also be very strong, and you can be extroverted, but also, you know, like some downtime, so it's quite hard for me to to pick some constant traits.
Rommil Santiago 24:07
That that's the deepest answer I've ever gotten.
Melanie Kyrklund 24:10
Okay, good.
Rommil Santiago 24:13
It didn't hit the five word limitation, but I really like
Melanie Kyrklund 24:16
Yeah, no, no, nowhere close. Never close. Sorry about that.
Rommil Santiago 25:29
I just wanted to thank you for joining the conversation and I hope you have a good day.
Melanie Kyrklund 25:34
Thanks to you. Rommil and Tracy was really nice chatting with you today. I appreciate you creating space for me. Wish you all a great day.
Rommil Santiago 25:43
This episode was edited by Jin Yang. She is the co-founder of Mindset Masters. Mindset Masters works with professionals in all industries to encourage high school and university students to develop a growth mindset. If you are interested in speaking in one of their online workshops, please reach out at any time at contact.mindsetmasters@gmail.com.
Transcribed by https://otter.ai