The top CRO Myths that experimenters have to bust ft. Lucas Vos

AI-Generated Summary

• 00:00 Developers in the product organization are mostly focused on releasing features, facing high pressure from top management to deliver promised features and bonuses.
• 09:52 One of the myths regarding experimentation is that it is dangerous for the technical stability of your platform. This myth persisted due to a past experience where a poorly planned experiment caused a breakdown of the subscription process. However, this was a result of a lack of checks and proper testing. Generalizing this isolated incident to all experiments created unnecessary fear.
• 19:29 Another common myth is that experimentation is regarded as extra work, especially in a high-pressure environment focused on releasing features. Developers often face top management pressure to deliver features and assume they will work without validation. However, experimentation can be positioned as a means of validation, contributing to important projects and creating space for necessary changes.
• 28:50 Another common myth in experimentation is the assumption that testing obvious changes is unnecessary. Some believe that if a change seems evident, it doesn’t require testing. To counter this myth, we intentionally showcase unexpected negative results in our presentations to demonstrate the importance of testing even seemingly obvious changes.
• 38:36 In conclusion, we’ve discussed three common myths in the world of experimentation. It’s essential to address these myths and approach testing with a willingness to explore, ask questions, and challenge assumptions. Testing shouldn’t focus solely on winners; it’s equally important to uncover ineffective or negative outcomes to make informed decisions and drive progress.

Video

Audio

AI-Generated Transcript

Lucas Vos 0:00
developers in a product organization or something happens to be the how to organize. But they’re mostly focused on getting release features out features, release features, basically being a bit of a feature factory.

Charlotte April Bomford 1:02
Welcome to the experimentation podcast. I’m your host, Charlotte Bomford are going to chat live with this boss. Thank you for joining us today with us. And he’s currently working as a senior conversion specialist for RTL. Netherlands. Am I correct, Lucas? Yeah, that’s correct.

Lucas Vos 0:16
Move high pressure also from the top management because they don’t want to have that fancy feature. That’s they have been promised. They only assume that it’s going to work. And yeah, that produces a bit of maybe a bit negative boop, boop negative, but I think people can relate to that. At some point, we have also designated kind of pressure also, within you have two really important projects. thing everyone has had, it was always a really important project that everyone’s working on. And And yeah, if you have experiments that are not related to that, yeah, good luck with that getting gets privatized.

Almost for for Janna. For you for this major broadcast. For those who don’t know LCL battlelands. Yeah. Thanks for introduction. I work for RTL Netherlands for four years now as a serial, cine Shiro specialist. So we have besides also the TV broadcasting channel, so we have it a four to five, to seven, and so on. We have also a streaming platform video land, which is pretty much competing with Netflix is number one. Everywhere, and maybe the number one. So yeah, we’re, we’re second. So amazing. Yeah. Competing with the Disney and HBO max. And that’s weird, at least help I have over here. The screen guys what was called Video loads, maybe, if you have heard of it, some of the listeners might be. Yeah, and for as a cover specialist, I focus mainly on the video platform, because it’s a major digital product, of course, onboarding, like getting new customers here, but also to get some experience on the platform itself. And also, outside of it, we’re starting to have a bit of a fix for our weather platform by it’s called major weather app. And for those who note a bit of the Netherlands, you can have full four seasons in one day. So you really want to have a weather app, so to check when the drain is falling down. So that’s also very good to know. Yeah, that’s a bit and yeah, that’s amazing.

Charlotte April Bomford 3:00
Thanks for that introduction. Yes, that’s a good introduction, Lucas. Yes. So what we’re going to discuss today is about certain myths regarding experimentation, that sometimes block or show resistance towards price I’m actually really interested in with what we’re gonna say, Lucas. But before we delve into the nitty gritty of things, I would, I would want to know more about yourself and your journey into becoming Yes. Or going into the CRO world.

Lucas Vos 3:32
Give me a bit of a long journey. When to start? Maybe good to know I’ve 1112 years of working experience when I graduated from university to University of Twente from the east of the Netherlands, in marketing communication, when I started it with a large energy company here as a trainee, but yeah, already experienced that Dennis wasn’t of my company, somehow, and about a digital environment, like setting up web shops, which I already did during my time at university with some friends, so just try to sell stuff. That’s triggered me always. Yeah. I fought a let’s continue with that all first. Then I became an SEO consultant for one and a half, two years for a consulting company here in the Netherlands. Which gave me a bit of opportunity because then I run into a vacancy within the media industry. And RC that’s a major quality newspaper over here. I think it’s fair maybe comparable to a Dutch sized New York Times or a bit of the guardian in the UK which they were searching for an online marketeer basically good manage the whole, fit so on for the whole of acquisition of subscribers to manage all the channels that they are they are. And meanwhile they were also switching and it’s very important for me to register to get into C row is that we’re also switching from a very separate model, like I have a new newspaper a really long paper back to those days, years ago, and separate west sides, which was run by a very small proportion of the editorial departments. Have, you’ve been gone to an FFT editorial editors, and only five were mainly discussing the whole nine things, and the rest was only focused on the newspaper for us, and they had already learned to switch that to more a story first, and then choose in general to bring your story to life like, Hey, do you want to have it on newspaper? Or do you want to bring it online? That’s the major transformation that day. We’re starting. And one of the things was also creating a website which had a paywall. So if subscriber sites which has not done too much traffic got a major boost in traffic, because we were introducing one of the first Bibles in the middle of so Okay, back, you had Babel, and you have to start optimizing that. Yeah, that’s true. And that’s where it all began for me. So basically, we had a background on that Babel. And first AV testing that I did is just remove the background, so which not so too much HTML skills, I just removed it out. First, you just live with it held off. And there was also a bit of coincidence. Back in those days, our office of MSC was situated one floor below the European head office of Optimizely. No, European art art, it was at least to touch one big so I could take the elevator to when I got some questions. So it was to do so when I got into something like that. And that’s where it all started. So this was my very first one. I think it was eight years ago. And so

Charlotte April Bomford 7:33
did you guys like choose Optimizely as your A B testing platform back then to or?

Lucas Vos 7:40
We had we had it running even before I got there. So you ever think of like having a bit of an expensive car to the garage? And people not knowing that there isn’t any Oh, wow. So

Charlotte April Bomford 7:54
they didn’t know that. My son was just running in the background.

Lucas Vos 7:58
So it was running in the background. Okay, let’s use it. Sometimes felt like that, like Alice in Wonderland. Oh, hey, this screen district? Let’s do it. Let’s try it. Let’s try it. Let’s try. Before had notices I forgot what was already 10 to 15. Maybe it is. And then we got a bit of stretching like, hey, we know a bit what’s kind of worked. Very limited, though. But so yeah, that’s where I started my CRO journey. And so I think eight years ago in media three and a half to three years before, okay, hey, I was done. They’re like, Okay, I like to company I like to people, but there was not much of a challenge for me at some point. So I moved on. For started. That wasn’t my fit. There wasn’t a direct fit can happen. And then it all came along. And it was really Yeah, I think almost a year before COVID hits enough. That’s a big one in streaming for the past years. But yeah, was the female I looked for like a to the large numbers that you get have to test with and really to ramp up and experimentation program because also good to know. When I started almost four years ago, it was done by an agency who got who was in one day a week. I think 90% of the people didn’t know who those guys were setting up the betas releasing it and went out and then after two weeks, you saw them again. Hey, we did have this IP test and this is what it did before Okay, nice. It’s show left or not. Okay. And we’ve talked about the work so it was no impediment of the culture or something. So more or less the first A to full time CRO specialists with RTL. And yeah, we’ll look at havior has lots of potential over here. So let’s start wrapping up and put down a gas pedal. And then yeah, it wasn’t like nice rollercoaster ride until now because now we’re like, above 100 experiments a year. Wow. So and also not only running because then back in those days, we were only running on certain kinds of pages, which were only meant for getting subscribers in, and how they’re also running experiments to optimize the discovery journey, the discovery cover journey of finding new content, that’s a major thing like, Hey, I think everyone knows who has a streaming subscription somehow, that you want to download catalog for 40 minutes, and anything I don’t know, still not know what to choose cannot have too little time left to produce to really watch something. So I close again, that sounds like me.

Charlotte April Bomford 11:08
That’s what I do sometimes, too.

Lucas Vos 11:10
Yeah. And I think it’s very relatable for a lot of people. Because we see, we we see those also. And it’s also hard, because how do you what’s the perfect mix, and that’s where we have started a few months ago, I really gigantic got into that kind of journey. And also, a thing that we also do is trying to optimize for excellence. That’s the thing that we’re really that we have done it a bit in past. And after we’ve got both of it. But now we’re here. We’re back into it. And we’re already finding the perfect balance for because we have also a subscription model, this is a bit hybrid. So you pay a small proportion of a subscription fee, and you get some ads also, on top of it, but it’s a bit like a you’ve seen ads, so you’ve got a bit of discount to your normal subscription fee, figure.

Charlotte April Bomford 12:06
So kinda like you’re passionate about the job that you’re in the company, I can tell that you’re really passionate about it. Okay, so I think like, I’ve actually am interested in the things that you’re gonna say about the myths that you’re about to bust, you know, you have you want to share the first one and then expand into it.

Lucas Vos 12:28
Let’s talk about maybe it first, the myths that we have encountered in beauty past years. One of them is the myth that experimentation is dangerous for the technical stability of your platform. And the other one is an experiment that running great experiments is a lot of additional work we don’t have time for another one is also experiment. running experiments is like, only finding confirmation for the obvious, obvious fix. I think maybe maybe limited to two or three because there you can think maybe those myths that can be dealt in your organization that people really think that it is true, that that’s the truth. And that could hinder your progress and getting your your experimentation philosophy going or as you get get going into something new things like no. So that’s got down to the first one I had to date your spots. Get that myth was pretty persistent, I think for maybe one and a half, two years. And I have to admit in that area that I have then it was guilty also, albeit with causing it’s because how we had a one of the experiments that we want wanted to launch was just basically technically connecting to kind of products like okay, hey, we want to move over a certain piece of data that you put in and then move it over to that. But it’s broke. A bit of it broke the whole onboarding class. People weren’t able to subscribe. To detail was just that’s that experiment was running during one of the big moments of Oh, no. So I got a call during a Friday night, like hey, why are we running a BTS on this very moment. The whole thing of whole onboarding is now broken. How you have costed expectation is dangerous. It was angry and it was kind of unfair. You can also a bunch of developers were already, I think, for 30 minutes looking for the kind of a call that was broken. And their solution, the temporary solution for a Friday night because they wanted to drink their beers or something, wants to basically remove the whole testing solutions out of it. So really doing like bitumen, open heart surgery and getting the hearts out, just like me, so it’s out, and then we stitch it back together. And then the warning still works. And good luck with your testing solution. It’s now removed.

Charlotte April Bomford 15:36
Yeah,

Lucas Vos 15:36
thank you very much. We move on. So that’s what’s a bit of the gods. And that thing persisted, like, hey, whoever one time got a breakdown of Todoist subscriptions helped me do it must be very dangerous to do run experiments, because otherwise it won’t happen. It was a bit of a lag of checks, before we launch a very single experiment, a specific one. But it was extrapolated to a bowl kind of experiments, that it’s almost dangerous. And, yeah, that’s a bit of that situation that the myth was there. And that was, it still was there for quite a long time. We showed a lot of other experiments when we got the solution bank gain. And then still, when someone has a big title was announced that, yeah, we got to launch a big title on dates and don’t run experiments. Other days, if you have that five or six hours a year, and it’s gonna hinder you, and

Charlotte April Bomford 16:47
the things that that happens, you know, like, they’re even if you go through the list of checks, there are things that you want, see, or you won’t be able to predict. And then something happens. So really just basing saying that experiments can be dangerous because of one thing, and then generalizing it. Yeah, that’s a bit.

Lucas Vos 17:14
It was already a generalized. So how do we tackle that because it was Henry, people who don’t didn’t want to have. So I got into some conversations with some developers, we got also another thing going on, and it did, it was outside of our influence, or it just happened that it’s talking about something, your Lightning Lightning strike. And then we had, we had a very serious hacker attend on all of our systems. So I took Neverland, and also, we had some, some servers running most of our oversight in the US there be a bit of the data in that area. So we had to also we rethink of some kind of procedures to make something more safe. And then also the exploitation solutions came along. And then we’re in that process, I got into more contacts with developers. And it basically came down a day or two things happening. Basically, they didn’t know all the details about our structure of the district solution that was outside of their scope. They knew everything off, or it was at least someone, a technical guy, and most of you must know, guys wouldn’t do everything. So they were just saying, Okay, we know that he knows everything. So it’s, it should be good. But that wasn’t what they were feeling what they would say about the solution that we have. And one other thing was also all got all releases, were talking through a certain kind of releasing procedure. And Vijay Singh was not done in that procedure. So he had the very first one. That’s something key developers were very vocal once were involved in rethinking and reinstalling a bit of also our testing solution that needed to be dealt with. So that hacker attempts, because, yeah, we had to make it more safe. And another thing was also, like really, more showing what we do, and also, sometimes do a code check with more internal people, because most of the experiments that we run, especially in your first days were developed by external people saw it, it was also a bit of a challenge there. So it’s really getting into more contacts with SOC key developers and Talk about him and listen to their feelings about it because he has sometimes it’s can grow in like that. So there was a bit of the whole thing and then afterwards Yeah. That’s all all the developers are really fed of it. That’s yeah, it’s like okay, yeah. Now now they’re indifferent sometimes. Oh, it’s that’s better than being hostile. Yeah. We’re already hostile to us IV testing, because it’s dangerous is not according to the processes it says.

Charlotte April Bomford 20:33
Different from from things and they have to do more things. Yeah, like adding to their workload sometimes are Yeah, but it’s Yeah.

Lucas Vos 20:43
Exactly. So that’s, that was one of the one of the first myths we had to bus and then afterwards, yeah, it’s, it’s more stable, and also more trustworthy. And not every time that people saying, hey, no, don’t run tests, like, Hey, what are you going to blame for running tests? They’re always asking good on certain kind of tense schedules, like, Hey, we’re gonna do some some major titles or something. But now, it’s more like, okay. Oh, you know, you know what you’re doing? Okay. It’s more trust. So that’s it. Let’s go. Yeah, let’s

Charlotte April Bomford 21:19
move on to the next one. Why, like, that’s very interesting. And I actually like, kind of, like, relate to that in a bit, because, because I’ve done agency work before. And then now I’m in house. So there are some companies back then who would be like, you know, I’m not sure if we should run a B testing during the times that there’s promotions. Because they would think that the promotions, the AV testing would hinder them from getting more when it’s actually to be honest, the best time to run AV testing, because there’s more traffic. Yeah, I’m interested to see what your what you’re gonna say next.

Lucas Vos 21:56
Yeah, maybe to jump in on it. So it’s bit like, sometimes you have to maybe step stepped out a bit. Because sometimes, if you really have a beat, that’s gonna also screw us the results, like, peak season, that’s maybe different behavior. It’s balancing also, it’s a balancing act.

Charlotte April Bomford 22:22
I agree. I agree. That’s actually a good point. Yeah. Next rotation is regarded

Lucas Vos 22:26
as having extra work. So. And I think a lot of people can relate to the fact that developers and product organization or something happens to be the house organized, but they’re mostly focused on getting release features out features, release features, basically being a bit of a feature factory. Move high pressure, also from the top management, because they don’t want to have that fancy feature that they have been promised. They only assume that it’s going to work. And yeah, that produces a bit of maybe a bit negative boop, boop negative, but I think people can relate to that. At some point, we have also designated kind of pressure also, within you have to really important projects. thing everyone has had, it was always a really important project that everyone’s working on. And And yeah, if you have experiments to do or not related to that, yeah, good luck with that, and getting gets prioritized. What we did, especially during the last, I think 12 to 18 months is to get more into really important projects like Hey, okay, hey, you want to do that? Sounds great. But maybe we have to validate more. And we got more about we did there two things. One is to basically say like, Hey, it’s not about experimentation. This was validation. So you want to validate if you do do the right fix. If you can fill it in with a, let’s say, a bit of a pre experiment, like hey, getting some mock ups as they use tools like usability hub, and then already say, hey, oh, this kind of design could work on events. And then afterwards also do the final validation in an AP test and also Yeah. And do some pre AP testing so get key components okay. Hey, you want to Oh, you want to do this? That you also fake of hey, that we do should do this. Maybe we can prepare that in setting and creating space. Like, hey, if you’re on homepage on a mall, I don’t have that much space and everyone wants to be dealt with buttons, etc. So yeah, sometimes You need to create space over there. And what we did is basically first run the experiment. So create that space, just remove stuff. So can you create the space that’s necessary or not? And then present those results. So then you got some fun buying, okay, hey, you’re, you’re contributing to the really important project. And that’s where you showed it, also the added value, and not having too much in the tournament, okay, it’s not actual Oh, now he can do really, you can really come to it on the important things, we think that’s what to do. So

Charlotte April Bomford 25:39
that’s really interesting. They do like pre testing first, and then do the actual, like, big testing kind of thing.

Lucas Vos 25:47
Yeah, mostly the both. We now also in a one, and now we did first brief validation with some designs in a usability study, then we have also done the Make space experiments. We’ve done that as well. And now I think, one of the first after launch experiments is running. So we get, hey, do the assumption that check check the file on assumptions, we have key assumptions that were there for a project. So before we get into, hey, after a month for one and a half months that people think okay, now the results are really more negative than I thought it was hidden people. Yeah, try to do big changes to kind of designs because it’s not what we’re doing. And then people are already focused on getting just kind of results. Yeah.

Charlotte April Bomford 26:51
Testing. Usually, there’s kind of like a sequence to though, like, it depends on like, you start with something that starts the whole sequence of tests, which is really interesting when you Yeah, yep. So let’s, is that your second one? Or third? One?

Lucas Vos 27:09
There was a second one did additional work? Which one?

Charlotte April Bomford 27:16
So how would you tackle that if in case, like, for example, because, you know, I’ve been to companies as well, where like, the ABX, testing, the results are really, really good. But it will take a while before it becomes a permanent change. Because we have like, like you said, like, we have to launch these features, we have to watch this features. How can you? Yeah, how can you tell the developers to prioritize it to be as a permanent change, ASAP, to get the benefits?

Lucas Vos 27:53
Basically, we have now for one of the major floods, we have one of the product, the deal that every sprint we do at least one implementation if there is any. So we just have, yeah, he creates a bit of space. And mostly, most of them are pretty small. So sometimes, you know, so developers need to work on a really important project. And it’s also really big stories, sometimes is they really want to do, they’re not in the mood for that. So they want to do something else, which is also sort of that was good that you are also sometimes I’m not going to do something else, which is also which also lies already long time over there, which is pretty small. So I can feel a bit of satisfied that I’ve done that, because then I have finished at least some job. And we try to create this kind of stories for that. Yeah.

Charlotte April Bomford 28:49
I think one of the things that I tried to like cuz I had like a few conversations about permanent changes. And then you know, that back and forth within myself and the developer, the first thing I say is that, you know, like if we let it because what I do is if in case wins, I put it up as a permanent change in Google Optimize. As soon as I wait for the developer to be free to make a permanent change, you know. And so the problem is, this is something that I tell the developers because they do ask me all the time, like, well, we can let it live and Google Optimize until we’re ready and stuff and and I tell them like we’ll get good, but if we have like let’s say five or 10 personalizations, then there’s a lot of problems that would arise most likely, the website will be slow. There will be a lot of like scripts that will be popping up and yeah, I kind of like try to say that in that way. Just to get one of my permanent changes in Yeah, I’m not sure how you how you do it, because but I’m pretty sure you had that conversation as well.

Lucas Vos 30:06
Not in that sense, because we don’t do 100% things. Yeah, we don’t do that. And for a few reasons, one of them is the technical stuff. I talked about experimentation being regarded as a b2b attack yet as such. For us, this means, if we do that, then we get a perception or in and it’s also I think, it’s gonna really break down some things. I’ve got a lot of stories that people are gonna hate. Oh, the church was there for ages, because it was a for ages are at 100% of Google Optimize W Oh, yeah, you don’t want to have that. It’s terrible for loading speeds, etc, and lots of other stuff. So we basically say, it’s like, Hey, you’re, you’re losing money to implement this, we’re not going to do that with your testing tool, because you don’t want it you want. You don’t need to create your technical debt. Because that’s basically what you do. And technical debt is also the last one in life. Because it’s, we have to do it, we have to refactor that. But we need to also do that really important project feature. And, yeah, the maintenance of your calls is being prioritized again and again. That’s if you strongly believe if you do that 100% of the carbonization, I think you do have the right risks that you end up being regarded as all the technical solutions. So yeah, we have to do it. It’s already there are 100%. So it’s doing the job already. So okay. Yeah. And we tried to stay away from that. So okay, hey, if you implement that as not running now, because we just put, we have switched off the tests, we’re just numbering new ones. Only let us know when you’ve got to work on that. Because you might think I screw up currently running tests. But then you implement a winner, and then you see the others. And so yeah, that’s creating more pressure to improve the implementation phase. Yeah, exactly.

Charlotte April Bomford 32:31
Yeah. That’s a good one. What’s the third one? Lucas was the third.

Lucas Vos 32:37
I talked about the extra work I talked about the dangerous part. And I talked about Oh, yeah. Extra rotation. We were gonna just get an obvious Yeah, some. It’s I’m also from the development side, some people think, yeah, well, you present results, and you show winner then they say, yeah, that’s all of yours. Hey, why don’t we just say this? Everyone could figure out that a lot of people can relate to that as someone who is resilient Yeah. So every that we had as friends, and still some people think that it’s Yeah, I agree with you don’t don’t think of have a baby do listeners don’t think that every myth is totally busted in on our side, we have reduced it. And it’s also like reducing it at some point that is not a common myth, like the common parent or in your company. More or less minorities. That’s it again. And to be realistic, in a sense, back to the obvious things like hey, sometimes people think that and what we did nothing for the last two years try to show people that also, we sometimes are really wrong in our assumptions. So we try to play around with the expectation so hey, when we present results how we have a month almost almost every month we can present the results in a big product meeting where a lot of developers are a product owners, etc. And we big song experiments and always try to we always try to do big the unexpected months. So unexpected. We Yeah, if so, obvious things yeah. Sure, but in fact getting their data attention, but if you can surprise the audience like okay, hey, you, we follow you we all thought that this could work right? Yeah. Yeah, at least Yeah, and that’s why that’s, and then people start asking questions. Why is that? And if you do this quiet, repeat this kind of things like that and say, hey, if people there, we also notice we are also right and 25 to 30% of the time and then yeah. It’s that’s a common fix your own people know, but outside of our profession, it’s not that don’t yet. Well, we try to educate people for that that we as well were wrong.

Charlotte April Bomford 35:35
Oh, oh, it didn’t work. It

Lucas Vos 35:37
didn’t work was not the results are even negative. I really liked that we expect balsa that you have negative. And also, let’s really also it also showed the negative results like Hey, Sam something to check, as we see see that it was negative. So we have a void at a loss. We’ve talked to losers, because it’s a pretty negative thing, but loss aversion thing. And then people say, oh, yeah, that’s, that’s cool that we did test that. And we didn’t move on to that. Because at some point in the getting people getting a mythbuster that you only test the obvious that’s showing the negative and the unexpected results. So much more influence or that they are just having a winner show. Let’s put it that way. And it’s nice to show winners, sometimes, hey, you have to do that, as well. Well, if you only show winners, then it’s then I think not that powerful. And at least from our experience, not that powerful if you show really the other way. Yeah, yeah. And also inclusive ones. Like, hey, we’re worth quite a lot of time, or there’s no effect. So maybe we have to stop. And that’s thinking that sense and think of other topics

Charlotte April Bomford 37:09
as exactly, I think like, especially if it’s a high impact or high effort type of it’s kind of like oh, you know, it’s an obvious fit fix, quote, unquote, and then it’s high effort at the same time. If we did a B tests that then that’s a lot of like, time wasted money wasted, if that was just moved into the permanent change, fix. You know what I mean? So, yeah, yeah, definitely, definitely. I’ve heard those. I’ve heard those comments before. How have you guys like, tackled that conversation? So if someone tells you look, as this is an obvious fix, what are you gonna say?

Lucas Vos 37:59
I start asking questions. And there’s also what I’ve learned to do is just ask questions. Okay. Why is it obvious? Okay. What’s, why do you think it’s obvious? It can be obvious. sounds somewhat logical, but I just want to know, more background bobike or whatever. They have evidence. So they don’t have evidence. Yeah, just everyone does it. Right. Yeah. Yeah. See?

Charlotte April Bomford 38:28
I agree. Sometimes the best practice from another company doesn’t apply to you as well. You know what I mean? So it is different from from Abby, every other industry, they have their own best practices for a reason, but doesn’t mean that it applies to the problems that you’re trying to solve in your own company.

Lucas Vos 38:45
Yeah, exactly. It’s also a bit like picking your battles. And in that sense, and sometimes you’ve already think, okay, hey, there’s also a bit of when, where everyone knows, okay, head it’s really an obvious thing. Like, okay, let’s move on. Yeah,

Charlotte April Bomford 39:01
I

Lucas Vos 39:02
agree. If it’s something like using the goods be a chance that we are wrong in this and that’s good be sometimes well, yeah. So hey, do not say, Hey, do you want to do you want to have the risk that is unexpectedly got worse numbers than ever before? Probably a good question. It’s about risk. Yeah,

Charlotte April Bomford 39:30
probably a good question is what is the definition of, of the the obvious fix? How would you define Agus fix?

Lucas Vos 39:45
So I think a very good one. I think there is not really Yeah, what is really, I think bug fixes are obvious fixes. Like people have been showing arrows, buttons, not working, etc. Outside of it. Gig If you really have like, bug fixes or things that show that show like it’s not working, but it’s still working, but it’s not showing anything like, hey, a bit of like a hidden button, or people don’t know where to claim. Those kind of things are pretty obvious. Outside of it, I would, I would be very hesitant to call things obvious because I’ve seen all sorts of things that people, yeah, that’s obvious. Okay, but let’s do an experiment, I think. Let’s see if it’s obvious. Really move on. And also say, you have evidence that it’s obvious. So you ask questions, and don’t think it’s obvious. You have to evidence and you don’t want? And what it will say is, I won’t try to also say something about a risk appetite. Okay. Do we want to have the risk that’s that you’re, you’re bullish for? What’s your opinion, are pretty obvious and could be, but it shouldn’t? Or it can be it’s not? You want to hit the reset? It’s not? And most of them pick, okay, hey, maybe it’s good to check. In. And also, most, most things are pretty, pretty easy to test. And then if you prioritize it, it’s also on the testing backlog, like hey, why don’t we move on also tools to do that for a very often pretty quickly, and you get also to buy it? Unless people are okay, hey, I fit people comfortably. I think it’s obvious. necessity, or at least parts of it. And then yeah, and then move on, as you put it into your into the schedule. Okay. Very good that you come to us. And also now we have also like, people say to one another that okay, it’s obvious. I’m not 100% sure that necessity, and then they gotta come to us. Like, again. Hey, guys. So we had a bit of a discussion not everyone was agreeing on yeah, having is like, Oh, yes. So it should be there. So that’s the feeling of, yeah.

Charlotte April Bomford 42:18
There are, I think our CRO specialists who would relate to companies as well, who are saying, oh, that’s an obvious fix. How would How would you like, you know, you’re very, very experienced? How would you? Or what is your advice to CRO specialists or CRO managers as an agency, and companies are saying like, Oh, that’s an obvious fix, let’s not test that.

Lucas Vos 42:50
It’s so important if you have the opportunity, and the ability to just be variables. And just the obvious thing. Still, like just tries to set it up. Especially, it’s pretty easy. You something and if you add also us also your own experience, okay, if you may be in doubt about for us obvious, but for me is really isn’t when it’s that much of balance, and try to pick something that you think okay, this is this might be opportunity to do that. And if and also, if you are wrong, it was obvious. And so hey, I’ve tested it’s going after I still tested, you’re where you are, right? I was wrong. But most of the times you also think something, Hey, I see something unexpected. Maybe it’s side effects or something. Oh, okay, distance. Yeah, that’s why we do something like that. So it’s always sometimes you really have to show, you have to have that breakthrough, because I’ve also seen a quite a lot of internal discussions here that’s going back and forth, back and forth, back and forth. It’s only you only can move on. If you have additional evidence onto the table, then you can move on to the next stage, at least have the discussion and it’s not too many times that it’s fully done, but at least you’re getting to the next stage and people are not discussing all of it anymore, but there may be certain aspects. So you have you have made progress. So it’s also a bit of, yeah, how to you can approach it. Yeah,

Charlotte April Bomford 44:44
that’s good advice. That’s really good advice. Do you want to conclude like what’s our conclusion on this one, Lucas?

Lucas Vos 44:52
I think you have heard Freemius and I think maybe people People listening have animus and I really persistent message that that’s hindering them is holding them back to make progress. I think one of the red lines between like, hey, try to get in to get into the face like hey, what’s causing it where it is coming from? Whoo. Yeah, that’s the spirit of the whole thing where it’s come across. And then there’s gossip, like are really ask questions. Okay, why? Why do you think they are? Or hey, where it’s coming from? Try to explore that. And then afterwards say you can also see. Yeah, five common grounds, most of the times like, hey, oh, let’s do that. And also for most, it can be a bit full for most zero specialist to only show that you’re focusing on winners after the show on on hey, let’s audience Pacific’s really think of hey, that you’re not testing because it’s obviously now you’re testing because you want to not do the things that are not having effects or even negative effects, and shoulders, all sorts of people. Because that’s that for. I hope for everyone that they have also the disk space and the environment is like a shell. Okay, hey, people, it’s, it’s not well, we have expected that and it wasn’t not intended, but we should we still find a result that we’re not happy with. At least we worked and that they can give me a bit of the Yeah, this can outside of all the showing winners and having a new rave has felt but more of a thing of, okay, hey, show learning’s show do to failure educate. More in that sense?

Charlotte April Bomford 47:01
That’s a really good conclusion. Should companies be afraid of CRO?

Lucas Vos 47:06
No, not at all? I think. I don’t know if you’re not on listening. Maybe some of the listeners do? I think so. And he says, Sometimes I see a row is not about showing fancy fancy tricks on websites. And then you go you’re boosting converter with extraordinary numbers. No, it’s not. It’s about we do. We are in the area of decision making. He says I agree. Risk management. So we reducing risk. And I think senior managers are older that they do not know. But they’re really neat. They need us to reduce the risks in their decision making. And we can help with that. If you are in that sense, then I think in that area you are seen also basically that’s why then I think you have made some major steps. Awesome.

Charlotte April Bomford 48:07
Thank you so much, Lucas, for your time today. And yeah, we’re so glad to have you and experimentation

If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

---