eCommerce CRO veteran Johann Van Tonder shares what usually goes wrong when coming up with test ideas and how to fix it. We also discuss the merits of quantity vs. quality of ideas, the problem with brainstorming, and what to do instead for more effective ideation sessions.

Find Johann on Twitter: @amaclickclick

Learn more about AWA Digital: awa-digital.com

https://open.spotify.com/episode/6CPiQZFej7XrffJ9sbYCUU?si=e16886975d7f4b62

AI-Generated Transcript

Tracy Laranjo 0:01
Before we get into today's episode, experiment nation encourages you to support Doctors Without Borders and their response toward the current humanitarian crisis in Ukraine. If you can spare $10 Go to donate dot doctors without borders.org. Thank you for listening onto today's episode.

Johann Van Tonder 0:17
People, whether they operate in a in a solo environment or whether they operate in a group environment in ideation tend to jump into solutions too soon. So we tend to start coming up with ideas and solutions to a problem before we properly understand the problem.

Tracy Laranjo 0:42
Hey, experiment nation, Tracy here. Our special guest today is E commerce optimization veteran of 15 years Johann van tonder Johan was forging a path as one of the web's earliest optimizers while I was a kid begging my mom to let me have an MSN account. You may recognize Johann today as the Chief Operating Officer of specialist CRO agency, Awa digital, or as the co author of e commerce website optimization, why 95% of your website visitors don't buy and what you can do about it. In this episode, Johan shares the four pitfalls you will probably encounter when coming up with test ideas and how to solve them. He also debates the merits of quantity versus quality for test ideas, what's wrong with brainstorming, and the techniques you should use instead to create a high quality ideation process? Here's the show. Hello, yo, Han, thank you so much for joining us today. And welcome to the show. Tracy, it's

Johann Van Tonder 1:37
good to talk to you again.

Tracy Laranjo 1:39
So in your in your book, ecommerce website optimization, you talk a lot about the pitfalls associated with the testing phase of an experimentation or optimization process or program. But I often find it's hard to find information on what could potentially go wrong throughout the ideation process, which is where a lot of things can really go wrong. So I'm really interested in finding out from you what you've found from working with clients and other businesses, what the most common pitfalls are that tend to occur throughout the ideation process in an optimization program.

Johann Van Tonder 2:21
Yeah, Tracy, and this is not something that I find that's really probably addressed in in business literature, where you'll find information on this is in academic literature. And I've been amazed at the amount of research that has been done decades and decades worth of research. And, you know, it's that that I draw on mainly, but to answer your question, what goes wrong? What are the pitfalls in an organizational setting? The first one is simply that there is no formal ideation process. And if you think about your own organization, or organizations that you've worked in, in the past, you know, very few places of work actually have a formal process for coming up with ideas. And the reason that so perhaps, is because there's never a shortage of ideas, right? There's always too many ideas to act on. And so why would you bother having an ideation process, but there is a lot of research that points to this being a really weak spot. And not the best ideas aren't the ones that that come to light if you don't have that formal structure. And I'm sure we'll talk a lot more about that, in this episode, just a few other things, three other pitfalls that are mentioned. The one is, and the second one, then, is that most ideation tends to be done in isolation, right? So you've got individual, on their own in solo coming up with ideas, and there are a number of problems without, but perhaps the biggest one for me is, while again, there's a lot of research that shows that the quality of ideas that are generated in that environment is inferior to the quality of ideas generated in a group environment. And one of the reasons for that is, when we come together as a group, you know, we we leverage the different pockets of knowledge that exists in the organization, and we all approach problem solving from a different angle, you know, given our own background, our own exposure to the problem around understanding and so on. And it's, if you, if you don't bring that into your process, then you've really got one point of view. And that's not the way to surface the best ideas. The third one is a horrible jargon, but I'll explain what it means. It's something called boundary spanning and it's a lie Due to the second one, this is Boundary Spanning is where ideas are cross pollinated among functions, departments and so on, you know, when you break down the silos, and people start collaborating. And the reason that's important is because again, there's research that this is one of the best predictors of innovativeness in organizations. So you, you want to, you definitely don't want to do it in isolation, you want to bring people together, but not just that you want to bring people together from different departments. And I'll come back to that later on as well. And then the fourth Pitfall, the last one that I mentioned, is that people, whether they operate in a in a solo environment, or whether they operate in a group environment, in ideation, tend to jump into solutions too soon. So we tend to start coming up with ideas and solutions to a problem before we properly understand the problem. And there's a, there's a quote that's attributed to Albert Einstein, it's not clear whether we actually said it, but it's a good quote, nevertheless, if I had an hour to solve a problem, I'd spent the first 55 minutes to define that problem, and the remaining five minutes to actually solve it. And it's that principle that we want to bring into the ideation process where we really clear about the task at hand really clear about the problem. If we don't understand the underlying causes, for example of a problem, then we might be coming up with ideas that solve one aspect of the problem and make other aspects worse. So you want to bring all of these together in your solution.

Tracy Laranjo 6:47
Wow, there is so much that that I want to address there. But as as someone who also does this full time for a living, optimizing ecommerce experiences, you really just hit the nail on the head and about so much of what I experienced during the ideation process, it's, it's almost funny to to work with, with peers who, in an ideation process, we just kind of throw ideas out there. And when I share their, you know, the research process behind coming up with ideas for tests, it's almost like a eureka moment across the team where it's like, oh, we don't have to copy this website that we like and just kind of take a shot in the dark. There's, there's so much more to ideation. And you also kind of made a great point that it really kind of is a team sport in many ways. It's very difficult just being kind of one person on your own, looking through mounds and mounds of data without that UX expertise. Without that statistical expertise, it really is much more likely to net out in higher quality test ideas and hypotheses when you have all of these talented minds in the room joining in on the conversation. Now, other than the biggest pitfall of not really having much of an ideation process, which of these do you see as kind of the most detrimental to any optimization program?

Johann Van Tonder 8:26
I think all of these really run together. And it's it the fundamental one is that first one that there is no formal ideation process, I think, if you want to, if you want to do a good job at ideation, and why wouldn't you want to because it's the it's the lifeblood of experimentation. But more than that, of really an organization's existence, innovation, and, you know, organizations need ideas, they need good ideas, in order to prosper. So it's a really important part of you of running a business. So if you want to do it, right, if you want to do a good job, you should have a formal ideation process. Apart from that, I'd say the, the the other one, if you ask me, what would be sort of on the same level, or maybe just level below that, what would be the second most detrimental one it would be jumping into into solutions, if not, not really understanding the problem before you move. And that's quite common. And I see this even where we have well thought through and well structured ideation, regimes, even there. I see that problem surfacing. Where the the opportunity that we're exploring that we that we're ideating around, there's not sufficient knowledge or data made available around that it's not sufficient he articulated exactly what it is how will we measure whether we, you know whether we've solved this. And one good way to do that, and this is stepping slightly into the practical stuff. And that one good way to overcome that is to Well, firstly, to, before you have any ideation session, whatever that ideation session looks like, whether it's on your own or in a group context, or whatever form it takes, is to really obsess over the data and the problem. understand exactly what is the problem, what is the opportunity to working with? What What data do you have to, to color it in your understanding, and then to, to reframe it in a way that you end up with an infinite number of solutions, as opposed to a finite number of solutions. So to give you a quick example, simple example, if you if you're looking for, let's say, a new vendor, it's typically that's going to be a finite number of of options that you choose from. But if you're trying to solve a particular underlying problem that this vendor is called to do address, that opens it up, and it gives you a blank canvas where there's a lot more potential solutions. And one of the things you want to do at the outset is to reframe the opportunity, the problem that you're addressing in a way that it it opens it up. It's an infinite number of solutions. And there are good reasons why that's the case. But that would be my advice.

Tracy Laranjo 11:49
That's pretty insightful. So what's more important than is it to have a lot of test ideas, or is it to have a few good test ideas? Yeah,

Johann Van Tonder 12:00
that that segues into one of the core themes, when you look at the literature around ideation, that you'll find this theme of around quantity breeding quality. And as you would imagine, there is a lot of debate in academic circles among researchers about this, there is a camp that firmly believes that quantity breeds quality, and you know, they can back it up. And then there's there's another camp that questions that. But but really, there's a there is there's a lot of truth in that. And there's a model actually, that's been constructed around this. And I won't bore you with the theoretical detail. But I'll mention the name of the model anyway, for people who want to look it up afterwards. And I'll explain that the practical implications. It's called the BI t, which is it stands for bounded ideation theory, the ID. And what that does is it gives you I think it's five or six different factors or variables that determine to what extent there is a correlation between quantity and quality of ideas. And if you understand that, then you're able to really make this quantity breeds quality mindset work for you. So so we actually find that a lot of a lot of the practical ideation methodologies that have been proposed and that are that are used in business are based off that bi T theoretical model. And so that's something that I that I'm not going to go into into the theory of, but I think the point that comes out of all of this is that you're really doing yourself a disservice by having just one or two or three, or even five ideas. From my experience, and from the research, I can tell you that and this is this is the point of brainstorming, right, is to generate initially as many ideas as possible, and then to kind of narrow down towards the final selection. And I want to say straight off the bat that I don't recommend brainstorming as a tool. I think it's better than doing it individually. There is a there is there are elements of brainstorming that I recommend that you incorporate in your process, but not brainstorming as it was proposed in the 50s. And as it's still practiced today. But but the the takeaway from brainstorming and the good thing about it is that a you get people working together which we've already covered, why that's important. And B it helps you to degenerate that a broad array that quantity of ideas. And then if you do things correctly, and if you follow all the advice and and, you know, put put it together in a good way or process, then you'll end up with with higher quality ideas. And of course, in a business environment, that's what we're after. We're not really after the quantity, we're after the quality ideas. But how we get there is by starting with, with a big number of ideas.

Tracy Laranjo 15:31
Yeah, yeah. Now, if brainstorming is not necessarily ideal, but neither is solo ideation, just kind of in your own vacuum, what really is the best way to sift through a high volume of ideas and really pick out those high quality ideas from the lower quality ones? Is that the process of prioritization for you? Is it bringing your ideas as a solo optimizer into a wider setting, to then sift through them? Tell me more about the best kind of approach there?

Johann Van Tonder 16:11
Well, this is really the heart of it, isn't it? Because that now we start talking about what should we do? What should that process look like? It's not prioritization, let's start there. Because what we ideation is way before the prioritization stage. With prioritization, what we try to do is we we try to ensure that the the ideas with the highest potential and the lowest effort, however, we defined that they they bubble up to the surface. Ideation is the process that gets the ideas on to the prioritization list. Right? So it's not prioritization, in fact, with ideation, you want to limit prioritization. As part of that process. As much as possible. You don't want to have too many filters. As part of the ideation process, you want to keep it as open as possible. In fact, I'll make this point quickly before I forget it. And it's it is a key point in in ideation sessions, one of the things that I always say to participants is the following. We want your wackiest ideas as well, if you because people often think, and this is this is one of the problems with brainstorming that there are a lot of issues with brainstorming. And this is one of them, is the reticence to contribute your own ideas and could be for a number of reasons. But one of those reasons could be you don't think it's good enough. You think it may be too wild. And actually, the wild ideas are often the ones that prompt other ideas that open new avenues of exploration that weren't there before. So even though in the end, it's not that crazy idea that gets taken forward that survives. It is the crazy idea. That was the genesis that that led to the birth of ultimately the winning idea. That's really the value of those crazy ideas. Okay, so what to do, then, if it's not brainstorming, and if it's not operating in, in a vacuum, as you said, the answer, for me is a combination of the two. Alright, so and this can be done in multiple ways. Indeed, there are, again, there's there's a lot of research about this. And again, I'm going to throw out a quick acronym and a quick theoretical model, but I won't, I won't bore you with a with a theory people going can go and read up about this, but it's, it's something called MGT nominal group technique. And this is never used in business. That label is never used in the business context. But I'm mentioning it because that's the foundation for what I'm about to say. And what the ng 20 ng t is all about is that each person does if you will brainstorming individually. So they go off and they generate in their own little bubble, they generate a number of ideas, but then they present it back in a group context. So that's the the thinking behind ng t. And that's the foundation for many methodologies that work really well. And one that I'll tell you about that, that I'm very fond of. And then I've had great success with that, but that that's for now, let's leave it there to say that whatever your your process looks like, you want to combine those two things. You want to find a structure where you get a number of different individuals together. We've discussed why that's important. Those individuals should be from different backgrounds. They should be from different departments, different funds. shins. And I found that works really well, if you've got a good mix of domain expertise, and people who are close to the problem. So in an E commerce context, I try to get people there who are close to the customer. So people who may be in the call center, maybe they work with customer services, you know, they deal with customer problems every day. So you want that small group of diverse group. And then you want to very clearly articulate the problem that you're solving in this session. And that's important, you're not solving every problem, you're not coming up with ideas to optimize the website, that's way too broad, you're not coming up with ideas to optimize the PD, optimize the PDP, the product detail page, that's way too broad. You're coming up with an app with ideas around a very particular problem, or opportunity you've identified on the PDP, you present the data, you present the customer insights, you explain, clearly, what it is, that makes this opportunity important. Then you ask each individual and you give them a time limit, you ask them to come up with many ideas. And you can, you can put numbers on that. I typically ask people do generate between five and seven ideas, sometimes it's more, but you'll find that people struggle even doing that. And then to make it a little bit more interesting, you give them very limited time. So typically, I would ask people to come up with five to seven ideas in five minutes. So that's less than a minute, per idea. There's good reason to do this, it's, as soon as we give people too much time those filters start coming in, are, this isn't good enough. Maybe somebody else will laugh at me when I say this. And if you if you timebox it, if you put people under pressure to come up with a number of ideas in a limited time, you get rid of those filters. So that's, that's the individual brainstorming part. And then the last part of the process is then to let people come back into a group context and replay their ideas to each other. And that's the start of it. That's not the entire process. But that is how you merge this individual ideation with, let's call it brainstorming.

Tracy Laranjo 22:45
It sounds like you have to really come to the table and be really specific about the problem you wish to solve, like you mentioned, saying, Let's optimize the site is a little too broad. Now, what's an example of a strong problem to bring to your nominal group technique? Session?

Johann Van Tonder 23:06
Well, I'll think of a recent example where we looked at product listings page, a PLP for a retailer, and they discovered that this site, it's a retailer that sells a broad range of products. And so this path to product is a really delicate problem on the site, you know, how do we get people to, to make sense of this vast array of items and, and help them to get to the product that they want, in the quickest way possible. And we found that the site isn't bad, it's, it's a big brand, it's been around for a long time, it's a well optimized site, but there are still opportunities to improve on that particular aspect. But even that itself, this improving the path to product on the PLP even that itself was too broad. To to attack as a as a as a brainstorming session. So what we did was we we broke that down into even smaller sub problems. And in one particular ideation session, and there were a few and there's still continues, but in one particular session that is fresh in my mind, because it happened recently, we looked at just the way that their products are presented in a row and and how, you know, how we, how we lay them out in this grid format. And this is before we get into the detail of what images to show what content to share with it how the different variables was like color and, and size and all these things, how that comes into it for now in that session. And this is a three hour session, looking at just that one part of the problem, which is how do we structure this in a way in the grid in a way that makes it easier for our customers to find the product, the product that they want, in the least amount of time. And to feel that session, we had a lot of customer data, we had anecdotes, we had usability testing session, insights from those sessions that are fed into this weird session recordings that were summarized. We had survey results we had results from, from customer interviews, we had analytics data, you know that there was a vast amount of intelligence that enabled us to really understand just that small part of the problem. That is a good session, it's a productive session, the moment you open it up too wide. There's not enough focus. So what what I would suggest. And also, I would say that not every opportunity, not every problem should be addressed in this way. You know, you'll spend your entire life in ideation sessions if you did it this way. But certainly the biggest opportunities, the big problems, the big, the big needle movers, you want to identify and you want to you want to put them through a proper formal ideation process.

Tracy Laranjo 26:38
Yeah, that was that was a really great explainer is really interesting to see how you handle your own ideation sessions with with this technique, other than MGT, are there any other methods and tools that you'd recommend to overcome or prevent these ideation pitfalls that you just shared with us?

Johann Van Tonder 27:02
Yeah, I'm gonna give you the tool that the methodology that I really like and drawn to, and I've, I've seen great success with it. In this environment, it's maybe not suitable for for every environment. So if you, for example, if you tried to come up with, with business strategy at a very high level, this is not what I would recommend. But for experimentation and for optimization, I'm sure you will have heard about the design studio. And that that's what I recommend. And you know, if you Google it, you'll find some information on it, but I'll summarize it quickly. Firstly, it before we dive into design studio, we spoken about brainstorming. So that's, that's one methodology. There is a another methodology that's fairly similar to it, which is called Speed storming. And that's almost like speed dating. And the way that works, and I've used this as well, and it works well, if you if you're trying to speed up the process, is you divide people into pairs, you get them, you give them three, three to five minutes. And you give them a very focused topic to discuss for three to five minutes, and then to come up with as many ideas as they can, or you give them a number of ideas you want. And then after the three to five minutes are up, they go to the next partner. And that's really good. Because it also promotes that social interaction. And you know, getting people together and breaking down silos and which, which is helpful in terms of socializing and evangelizing experimentation. Right. That's one of I think, we're not talking about that today. But it's, it's, it's one of the topics that come up so often, around experimentation is how do we promote this culture of experimentation in our organization that actually, part of the answer is ideation. It is the stuff we talking about today that I find is, is a great way to promote the culture of, of experimentation. This the second one, apart from speed storming is something called Brain writing, which is similar to brainstorming, but you write down your ideas and share them in silence. And then the third one is, is something called Sea sketch or collaborative sketching. And that's where I want to focus on a little bit. So we in that sea sketching and collaborative sketching and sketching in general, we tried to get people to sketch out their ideas rather than sort of verbally communicate their ideas. So this is what happens in a design studio, and I'm going to summer dries up very quickly, I'll go through some of the things we've mentioned already, but that there'll be some new answers. So firstly, we get all the people together, as we've discussed from different groups, different backgrounds, get them together for two or three hours, anything less than that really is not worth doing. In my experience, we give them a very, very focused problem to address, we give them lots of information about it lots of data. And this is usually pre work, right? We want to, we want them to arrive at the session understanding their problem. And then we get them to generate in a limited time to generate five to seven ideas. But the difference is the following. They are going to sketch their ideas. And usually when you announced this, there's a lot of why wide eyes and in the room, because everyone convinces themselves that they can't sketch. And what I say to people is, if you can put a pen to paper, then you can sketch because all you need to be able to do is draw rectangles, circles, triangles, it's very rough sketches, we don't want anything fancy, it's got to be there's no color involved, just sketching out your idea with some annotation. So draw an arrow and say what you have in mind, and five or seven of those in maybe 10 minutes for the first round. Then after that, you get each participant to put their ideas on a whiteboard, or on something like mural, or Miro that works quite well, obviously, in the in, in a pandemic, we we've all become accustomed to doing this online, anything works really well. You don't, you don't lose anything. In fact, some people would say it's a lot better. So whether you you together in a room, and you do it on the whiteboard, or we whether you do it online, on a virtual whiteboard, there's really no difference I found. So you get people to put their sketches up. And then you give them one minute to explain their idea. And each one does this. So you go through all the participants. And that's why you want to limit the number of participants I find a good number is if 677 is the absolute max that I will allow. And those are two disciplines who sketch so I won't sketch Oh, and the facilitator won't participate. So once everyone is explained their sketch, then there's a round of clarifying questions. So now each person can ask clarifying questions to the other participants around their sketches. That's then followed by a round of positive reflection. And, and positive reflection in this context isn't Oh, I really like that that appeals to me, it's more how you think that will solve the customer's problem, or how you think that will solve the problem that we tried to address, we always focus it on the problem. And then looking perhaps at elements that you elements of it, that you think will work well. And then the last round is a round of critical evaluation, critical reflection, and this is, you know, your opportunity to to be a little negative. But again, it's not, it's not an opportunity for you to say I really like that I really dislike that it's not going to work. It's it's more to, again, ask any questions or raise any challenges, concerns about how that will address the problem that we're trying to address? Are there perhaps any technical issues that we may not have taken into account? And what I find is that this discussion, those three rounds of discussion, that's really the value of a session like this, you've got to have a really sharp notetaker because you'll find that the nuggets, the real jewels, that that allow you to build on this happens in that discussion. Often it's not the sketches themselves, it's not the ideas themselves, it's the the banter, and, and, and the, you know, the the, the insights that came out as part of those discussions, then that's not the end of it yet. Then there's another round. And again, it's, it's around a sketching, but this time, you ask the disciplines to to come up with fewer ideas. And now I make it interesting. So and again, this is drawing on research, and I'm not going to bore you with the detail but but it's research back. What happens next. You now tell them to steal from somebody else So if there are ideas that somebody else presented that you really liked, or there are parts of the idea or elements of it that you really liked, then steal that and build on it. So that's the one instruction. The other instruction is either that, or take an idea. And, and we'll take two ideas and merge them. So two different ideas, at least two different ideas that could be more, but ideas you've just seen, being kicked around, and then put them together. And the third option is to come up with something entirely different that, you know, we've not seen, but typically, it's the first two. And that's then followed by another round of, of discussion similar to what I described. And at the end of that, we go away with a smaller group. And we now narrow down that, by now you should have between 30 and 50 ideas. So we take those, and we based on everything we've heard, we narrow them down and end up with maybe five to 10 or so. One thing I haven't mentioned, and I'll quickly squeeze it in there is it could also work well to to incorporate voting, so to ask people to vote for their favorite ideas. And that helps you to, you know, to narrow down that list. But that's that's the process. What did you call it? That's the the overview of the process that I find really powerful and highly recommend.

Tracy Laranjo 36:38
This was a great conversation. Thank you so much, Johan. For all the insight, there were a few things that stood out to me such as ideation really being a team sport, and of course ideating after you know what the problem is at hand, and then also how to really run an effective brainstorm. So thanks again, and great to have you on the show, Johan. Thanks, JC

Johann Van Tonder 37:03
was good to talk to you. Hi,

Rommil Santiago 37:05
this is Rommil, founder of experiment nation. If you liked this episode, then consider signing up to our bi weekly newsletter, where we share great interviews like this editorials, memes and other great content. You could subscribe at experiment nation.com forward slash newsletter

If you liked this interview, signup for Experiment Nation's bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25