Get Executive buy-in for your CRO program with Jonny Longden

Video

Audio

Transcript

Jonny Longden 0:00
Let’s face it, experimentation is completely misunderstood. The power and the benefit and potential of it is to innovate and develop a business. And when you think about it, that’s what experimentation has always been in relation to the scientific method. If you think about spaceflight, medicine, anything great that’s happened in the history of modern civilization, this has all happened as a result of the careful application of the scientific method.

Jonny Longden 0:41
If you ask anybody who runs experimentation, the biggest problem that they face, more often than not, they’ll tell you that they want it to be taken more seriously. So you have people doing CRO and experimentation in businesses, who feel like the senior leadership of that business, don’t take it seriously. And don’t understand the power and the value that it can have. And that is really important. It’s really important to have people understand it and to take it seriously. And let’s face it, experimentation is completely misunderstood. The power and the benefit and potential of it is to innovate and develop a business. And when you think about it, that’s what experimentation has always been in relation to the scientific method. If you think about spaceflight, medicine, anything great that’s happened in the history of modern civilization, this has all happened as a result of the careful application of the scientific method. And that’s what businesses should be doing. Businesses should be using the scientific method to innovate their business. But they don’t do that business is more often run using opinion and guesswork. And that’s the problem. If we want experimentation to be taken more seriously, we have to try and persuade senior people to use experimentation to make decisions as opposed to using opinion and guesswork. But that’s very hard. It’s very easy for people to try and come up with ideas by themselves. And actually, a lot of the way that business runs in terms of its hierarchical structure and how people are hired and remunerated doesn’t help the problem, because people are employed for their past experience and for their opinions and for their ability to use that past experience. So of course, they come into a business, and they need to be able to demonstrate the use of that opinion in order to justify their salary. But it doesn’t mean we can’t try and change it, it doesn’t mean we can’t try and figure out ways to change that system. And that’s what I’m talking to you about today. So how do we get senior business leaders to pay more attention to experimentation to take it seriously and to understand the value that it can provide to innovating the business? Well, the answer for me is about strategic questions about using experimentation to answer bigger strategic problems and questions that the business has. Senior Business Leaders don’t care about whether your call to action is above the fold. They don’t care about whether your call to action or your header is sticky, are about nuanced details of the user experience of the site where things are laid out. That’s all information that they think somebody else needs to deal with, and that they’re right, because somebody else shouldn’t be dealing with that. But senior business leaders do care about very big problems and big questions. And those might be things like, should we launch into new markets? How do we appeal to a younger audience? How do we win against this new disruptive competitor, which has just entered the market? How do we generate more profit next year without damaging customer experience? These are big problems that senior people grapple with. And the important point is that experimentation has the power to answer these questions if approached properly, that is the beauty of it. That’s how it should work. And so that’s what we have to try and understand how to do is how do we use experimentation to tackle these bigger things? So answering big questions, helps the business innovate and grow. But it also helps you as an experimenter to deliver value beyond just the website or beyond just the things that you’re testing on. Because by answering those questions, you’re helping answer questions that have wider ramifications. So if you’re trying to understand how to improve profitability, or how to launch into new markets or something like that, then you might be experimenting on an aspect of the website. But then the implication of that is wider. The benefits of it may be far reaching into media into other kinds of things into products etc. So by answering those questions, you’re helping the business do things on a wider level, and you’re showing more value than just the small things that experimenters often do. There’s actually two key ways that experimentation as we know it today goes wrong in this respect.

Rommil Santiago 5:15
This is Rommil. Santiago from experimentation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing, it helps us a bunch. Now back to the episode.

Jonny Longden 5:29
The first one is that it’s basically about moving a metric about transforming a metric. So that might be about increasing conversion rate. Or it could be a bit deeper than that, it could be about increasing revenue. Or if you’re really advanced, you might be trying to increase profitability. But at the end of the day, that is increasing a metric. And a metric is not a strategy. It’s a metric. So from a business leaders point of view, if they want to increase their conversion rate, how they do that is simply a cost, they would know that they just need to assign some money, some budget, in order to try and do something to increase the conversion rate, the same way that increasing traffic to a website or any other metric like that is a cost. It just needs budget assigning to it. And then that comes into the problem of is that going to deliver return? Can I trust that that’s going to deliver return? And you know, then everybody starts to debate on is PPC, the right thing to do? Is this the right thing to do? And it’s all just a case of what’s the right cost? What’s the right return on investment. And you’re not actually answering any big questions. Because exactly as I said, a metric is not a strategy. It’s just a metric. The second problem is that CRO when experimentation tends to be a spray and pray endeavor. So typically, everybody sits around in a room, looks at a website and comes up with hundreds of ideas on how to improve that website. And it’s really easy to do that, it’s really easy to look at a website, and in even just half an hour come up with 100 different small things that you could do to that website. And that’s typically how people approach experimentation, there might be some research behind it. But the output is usually the same is that there are lots of different random ideas, which are focused on trying to improve a metric as we’ve talked about. And that’s where your prioritization frameworks like pie, and ice, and all these kinds of things come in, because what they’re trying to do is to just simply figure out which of those random ideas will be the will be the, the thing that delivers the most valuable against that metric. And again, none of this has got anything to do with strategy of innovating the business or doing anything different, you’re just coming up with a ton of random ideas that nobody ultimately cares about, beyond the fact that they’re going to deliver revenue. Of course, senior leaders care about revenue, and you can go and say we did this, it’s going to deliver this. But it’s ultimately not as interesting as other big things that move the business that pivot the business and change the way the business is running. And that’s what we’re trying to trying to do, we’re trying to get to the point where experimentation can support those bigger, more interesting things. In order for experimentation to play a bigger role in business innovation and business strategy. We need to think about what experimentation is, and how it works and how it operates within the business. If we go back to what CRO is, and the way CRO usually works, really, it’s an after the fact thing, you have a product development roadmap, or an engineering roadmap. And most of that is coming from opinion and guesswork. So most of what goes into the development roadmap for it for anything to do with engineering website or anything is coming from a set of ideas that people just conjure up from their heads. And then CRO is something that usually slots into the bottom of that you have a small team or an individual person coming up with small UX ideas, and trying to get that product or engineering team to deliver them in amongst all the other stuff that they’re doing. And that’s typically what CRO is. And it doesn’t really make sense. Because if you think what experimentation is, is simply testing your ideas and learning from them and trying to drive innovation. Why would you do anything that was not that why would you have some ideas that you just came up with? So we move on from CRO to what is experimentation where experimentation should be? And experimentation really should work by being something that drives all of those ideas. So rather than there being lots of stuff that people came up with based on opinion, then you’ve got experimentation driving that and experimentation should be doing that on multiple layers. You should be using experimentation to drive minimum viable product ideas. Then get bigger that then become fully fledged features.

Jonny Longden 10:04
However, even in this layer, experimentation is still something which is just focused on product engineering, it’s just focused on a website or a product or something like that. And of course, a business is a wider thing than just that. A business has a wider set of operations and ways of doing things. And that’s what we want to try and focus on. So the utopia, the third vision is that actually, you use experimentation to help a business become more scientific. And how that works is really the same sort of setup. But instead of just driving a product engineering roadmap, you’re actually driving the innovation of the business, the ideas that drive the business as a whole things that are going to impact a wider set of ways of doing things, products, services, the way people operate, than just engineering. And that’s what experimentation should be aimed at. That’s what that’s what we’re trying to deliver. Now one of the most important ways we can achieve that is by looking at science itself, ultimately, what we’re trying to do is replicate the scientific method and the scientific approach in a business context. And that’s not just about how you run controlled experiments. In fact, it’s a lot more than that. And experimentation itself in terms of a b testing, or whatever you want to call it, is actually just one part of a bigger method and approach. One of the most important things to look at is the way science deals with theory and law. A really great example of this is the law of gravity. And Apple always falls straight down from a tree. Why is that it’s because of the law of gravity. That is a scientific law that can be verified through experiment. But the theory of gravity is a bigger set of ideas, and a theory about why that happens. And importantly, what you might do with that information, what else you could do with that information. And there’s this interplay between law and theory, which allows scientific discovery to develop and to grow. And this is easily translatable into business language, because a theory is a business strategy, a business strategy is really is really a theory is really an understanding of how the business should be growing of how it should be moving of things that you should be doing in order to win against the competition. And a law, like a scientific law is really customer behavior. When you run experiments, you are learning things, real things about customer behavior, and how customers will interact with certain things in binary sense. So those two things can interplay in exactly the same way that they do in science, you can have a theory about the business that you can then validate through experiments that tell you whether or not that is how customers are going to behave against your theory, that in turn can then either change or develop or advanced the theory, and so on and so forth. So that’s what we’re trying to replicate, you’re trying to basically get to a place where experimentation is directly supporting theory and vice versa. And that is achievable with how we approach experimentation. So here’s a concrete example. Let’s imagine that we understand the business strategy, the big questions that our senior leadership team have in the business. And one of those The biggest one is how do we appeal to a younger audience? Now, this is actually a real problem that a client of mine has that I won’t mention. But this is a real problem that they have. They want their brand to resonate with a younger audience for lots of different reasons. But how to do that is the big strategic question, the big problem in the business, how do you do it? And that things they’ve tried have not necessarily worked? Because their brand is very well established and well known with a particular audience. So we want to try and help them understand that how to do that. And already you can see, if you’re able to answer that sort of a question, you suddenly put yourself in a much deeper place within the business than just fiddling around with UX. So how do we do that? Well, the first thing is to use critical thinking to break that down into what are called strategic hypotheses. So these are big, broad hypotheses of how you might achieve that. Number one, we could achieve that by changing the product portfolio. Perhaps younger customers don’t want to buy this brand, because the products they sell are not things that they’re interested in. And the answer to that would be to change the product portfolio to change what you sell. Number two, it could just be about the targeting. Perhaps it’s not the products themselves and anybody would be interested in those products. But the way marketing and branding has worked in the past has targeted an older audience, and you need to shift that direction. So from here, we can work out functional hypotheses. And you see here the difference between a strategic hypothesis and the functional hypotheses. These are starting to work out experiments that we can run that ladder back up to this big strategic question. We’re trying to answer a strategic question by discovering information and facts through experimentation. For example, if you want to change the product portfolio, we could come up with functional hypotheses, for example, I might just need to merchandise different products in marketing. So it might just be about on the homepage of my website, on emails that I send out on advertising content, whether that’s in site or outside, what are the products that I’m leading with. And all that’s needed is to change those products, a different hypothesis would be to actually cut out products altogether from the range. Because obviously, the products in your range are going to be filtered out into SEO, into other kinds of external content like Google Shopping, Google Shopping that attract customers, to perhaps you need to completely cut out the ranges of products that you know, aren’t attractive to the people you’re trying to attract. If we’re looking at targeting, which is the the other strategic hypothesis, there’s a couple of other ideas there. Perhaps we need to restrict advertising to only channels where we can laser target those people. So and by doing that we can we can target on on age range, and make sure that targeted advertising is being delivered to the people that we care about. And maybe that is what is required. Maybe that’s all that’s needed in order to attract a younger audience. Another idea there would be that it’s actually to modify the messaging, modify the creative of the messaging that we’re sending out. And so rather than talking about things in a certain way, we talk about them in a way that would appeal to a younger audience. And from there, you can come up with concrete experiments, actual individual experiments, which may be multiple manifold. And I’m not going to go through what those would be. But ultimately, you’re then coming up with experiments that are based on these themes, which ladder back to this big strategic question. And hopefully, you can see how different this is to coming up with a whole load of random ideas, and trying to prioritize them based on what will be what will deliver the greatest revenue. And I’m not saying that’s not important, you can still do that. And actually strategic ideas can come from those ideas. But if that’s all you’re doing, then you’re not answering these bigger questions. So hopefully, this shows a framework for how you can start to guide your experimentation and guide your ideas based on things that senior people are going to be more interested in. And this is a better way to try and increase people’s respect for experimentation than just saying, you know, they need to buy into it, or they need to believe in it. Because you’ve got to think about what these people actually want and what they value and what their role is, as well. And their role is to innovate the business, we all want experimentation to be taken more seriously. That’s what a lot of people are going to be talking about this conference. If you want to do that, then you need to help the business innovate, and you need to do the things that senior people care about. Hopefully, I’ve shown you that today. But if I leave you with two things, one, it’s about trying to replicate the scientific method. And the scientific method is not just a method is a method that delivers against a vision. Somebody didn’t accidentally discover how to get into space by testing a lot of random different things. They had a vision for getting into space. And that is what a business strategy is a business strategy is a vision for the business and experimentation and the method that goes with it can help achieve that. But you have to start out with the vision

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK