Adventures in Experimentation: The importance of the voice of customer to conversion optimization - as well as to the entire business Experiment Nation

Home / Adventures in Experimentation / Adventures in Experimentation: The importance of the voice of customer to conversion optimization - as well as to the entire business

Rommil Santiago 0:06
From Experiment Nation, I’m Rommil. And this is Adventures in Experimentation. Our panel of CRO professionals leverage their years of experience in UX and conversion copywriting, to field common CRO questions. If you are new to the field, or even if you are a veteran, you’ll always learn something new.

Hello, everyone. Welcome to another episode of Adventures in Experimentation, where CRO experts answer questions that they get. On today’s episode, we have Eden and John, and I’ll let them introduce themselves.

Eden Bidani 0:40
Hi, everyone. Nice. Thanks to see you’re here. If you’re joining us for the first time, welcome. And if you’re joining us for the second time, welcome. Yeah, great copy there. I’m Eden a conversion copywriter, and I’m also a growth mentor. I’ve collaborated with Positive John on a few things and Rommil as well. And we’re just looking to answer some really burning CRO questions.

Positive John 1:01
How’s it going everyone? Positive John here, adding value to today’s experimentation discussion from a growth perspective and dropping some extra lines regarding market performance here and there.

Rommil Santiago 1:15
So, before we start, I need to ask Eden, what’s the temperature like where you’re at?

Eden Bidani 1:22
It’s actually it was 21 Celsius today. It was this is a really mild winter. It’s winter in Celsius. Yes. Yes.

Rommil Santiago 1:31
Celsius.

Eden Bidani 1:33
Celsius. I look, I came from Melbourne where our winters were like, zero minus one minus two. You know, at a period it’s like 21 degrees. And it’s winter. I’m like, Yeah, I’m fine. I’m fine. I’m fine with it.

Rommil Santiago 1:48
So John and I were talking just before this, and he’s saying it was really chilly. There was like, what was it John, seven minus seven?

Positive John 1:56
Exactly wrong. Oh, can you believe minus seven? I just want to say my Brazilian body’s not ready for this kind of temperature, man. It’s this is this hurts.

Eden Bidani 2:07
How cold is it in Canada?

Rommil Santiago 2:09
So I was born and raised in Canada. And it’s minus seven here. But she was like minus 14 and I can’t handle it. So yeah, I don’t know why I’m here. I can’t leave obviously, at this point. But it’s cold. And I don’t want to but I appreciate the cold because it essentially ensures that we don’t have venomous snakes and scorpions like they have in Australia. So I’ll take it for a few months of the year.

Eden Bidani 2:37
Okay, fair enough. Fair enough.

Rommil Santiago 2:41
Especially where you are. Everywhere in Australia. We have the worst the most deadliest snakes alive if I’m not mistaken.

Eden Bidani 2:48
Yeah, they’re like, the top 10 like deadliest animals like in the entire world, from snakes to spiders to jellyfish. You name it.

Rommil Santiago 2:57
The jellyfish? I forgot. All right. So let’s jump into it. Let’s pull up a question from Growth Mentor. Let’s see here. We are redesigning an important conversion funnel, we need a review from an expert and suggestions for analytics setup. So either one of you can start but be good to know. How do you guys usually go about this after having you know, your experience in the industry?

Eden Bidani 3:28
Hey, John, you want to kick this one off?

Positive John 3:30
Of course. So redesigning an important conversion funnel, and then I’m in the usually books that call for suggestions, fresh eyes, or questions regarding analytics set up to me, I want to share, of course, a little bit of how I’m doing this in growth mentor. But first, I need a disclaimer here. And that disclaimer is basically us taking it with a grain of salt, that undergrowth mentor setting, you know, the mentor me, Eden, we usually have a limited amount of business context, right? Because it’s a half an hour call. And even though in my sessions, I always send some strategic questions beforehand, so I can grab a little bit, a little more context about the problem that they’re trying to solve. It’s it’s never enough. And it’s very different from when I’m an experiment owner for a conversion program, and I’m doing that process I’m running that exercise, or when a business brings me as a business consultant. I still think it’s relevant to share a little bit of that experience because it might ring a bell, it might spark some different points of view from people listening to us who do similar exercises from a consulting point of view, for example. Starting with an overview, I developed this quick system that mentees seem to be happy with it and I split into three steps. Basically, I tried to get three sessions to make it more complete. The first one involves about, you know, me going through the page with the mentee doing a screen share, sharing my gut feelings, you know my reactions on spot about the page.

Rommil Santiago 5:09
So John just quickly, what are some of the things that you’re saying? Are you going through the site and pretty much saying things that you notice from an experimentation slash growth perspective, or you’re pretending to be a, you’re pretending to be the target audience.

Positive John 5:23
You know, with more mileage running analysis for multiple clients, I feel more confident on my gut feeling when it comes to heuristics and giving suggestions on design solutions, and messaging. And I like to clarify this idea of gut feelings with I have this rule of thumb, it’s a passage from this book, how to take smart notes that suggests that gut feeling is not a mysterious force, but an incorporated history of experience, learn through numerous feedback loops of success and failures. So this adds more context to what gut feeling actually means to me. And I tried to explain it this way, for mentees. I also suggest that, you know, this screen sharing moment works almost as an over the shoulder user test for them, because I try the most to not look much into their websites before the call, so I can get the best on spot reactions to them. Even though I’m most of the cases, I’m not the best target persona. But as a trained eyes, marketeer, if you will, I can provide relevant suggestions that they’re generally happy with.

Rommil Santiago 6:33
So now that you’ve gone through the site, and you’ve you’ve basically called out the things that you’ve seen using your you know, your experience, what do you do next?

Positive John 6:43
The second session usually involves Okay, remember all those good feelings that we took note? How could we do research or what types of research would help us adding evidence on those gut feelings so we can prioritize things better? And the third one, the third, and last step usually involves more commitment with the mentee. But we’re in a more frame worked exercise. And basically, I’ve been exercising a lot with the user psyche analysis from various contractor, which is basically a qualitative view of a commercial funnel.

Rommil Santiago 7:18
So just just I wanted to get a little bit of color there, where as you’re getting all this research, how do you make sure that you don’t miss any of the things that you noticed during your your first pass in the first round? Are you recording this session is who’s taking notes? How does that work?

Positive John 7:35
How me Trey don’t miss anything? This is an important question. I nowadays consider myself a professional note taker after reading this, how to take smart notes book. I loved that one. But I honestly start my sessions usually suggesting to the mentee, like what do you think about us recording this session? So we can overlook a little bit of notes. And we focus on what’s important here, which is understanding or your challenge your objective, you know, your expected outcome from this conversation? Half an hour conversation again. I’m curious to know, Eden, what exactly do you do for your calls? Are you always taking notes? Are you suggesting recording?

Eden Bidani 8:14
Yeah, absolutely. So the same, the same idea here, I usually record the calls. Either I record, I record them myself. And then I share with them after I lead. I let them in to record them as well, because it’s just, it just lets you immerse fully in the call. And it so you don’t have to worry about scribbling pages of notes. And then worrying if you miss something while you were taking the notes. And you know, because like you said, you only have half an hour to speak with him, which is not a great deal of time when you’re dealing with so many different elements at the at the same point. So usually follow up. And then we follow up in a chat if there’s anything else, if there’s any other major takeaways or things that we discussed that we need to follow up on afterwards.

Rommil Santiago 8:53
So just have you ever and this is you don’t have to tell me, but I’m really a very curious person. Have you ever gone through your process went through the site, like open it up? And you mentioned that you don’t really look things up before the session? So you’re looking at it for the first time? Have you ever seen a site that may go what the EFF Oh, this is gonna be a long day.

Eden Bidani 9:17
Yeah, so yes, it does happen. Often it’s because you know, one of the one of the things for business owners as well they like to do, they read though, they’re often very well versed in things like best practices from the CRO perspective, or from a marketing perspective, or even from a messaging perspective, like they know, they have an idea about the right thing that they should be doing for their site, but they tried to do all the right things at the same time. So the website looks it’s so there’s so many bells and whistles on the website that you can’t even tell. Like it’s really hard to figure out who they are and what they do or that it’s so pared back because they’re trying to make it so a minimalist that you have no idea who they are what they do for the second you get on the website. So actually with. So with regards to these, I do actually usually go through the website really quickly to get an idea of what questions I would like to ask ask the mentee on the call as well. But then I really like to walk through with the mentee and the website at the same time, simply because there’s more info, you can get a lot more information from there and I and building off your points, john as well. Things that I like to do, because usually they’re approaching me specifically about messaging with regards to conversion optimization. So we’re primarily about the copy, and a bit of the UX as well. And some of that did the design solutions in it. And as that all integrates together. So one of the things that we figured out is that I also like to ask them, things like where were the traffic is coming from? What did they see? But were they you know, is that are they sending paid traffic to this page? Is it organic? Primarily, how are users getting that what kind of messages of content are people seeing before they get to the get to the page or get to the site, what’s their likely stage of awareness and things if they have an idea to actually try and figure out more as to if this page is really serving their needs. And if if the traffic that is getting to this page is able to this if there is enough information for them to make a decision, or to take action take the action that we want them to take to make to help make a decision about the about the business or about the company?

Rommil Santiago 11:26
I actually want to touch on that where you’ve raised a really interesting point where you asked about the traffic that comes to the site? How would you go about this analysis to cover all the different types of users that go through the funnel? Are you pretending to be each one of these? I’m kind of curious how that would work. Let’s say you had new customers, old customers, old customers, existing customers. Those are like shoes, and those are like you know, different things? How does analysis like this, take into account all the different segments audience personas, however you want to phrase that?

Eden Bidani 12:05
Oh, that really that really depends. At any given time. One of the major principles that you know we have in copywriting is that you try to you try to create a page or you try to create messaging specifically for one reader. For one reader for one kind of audience segment, when you try to mix too many different messages and too many buyer personas on a single page, it just gets messy, it doesn’t matter. You know, it really doesn’t matter how well you can almost segment them out on the page, it just gets really messy in terms of trying to find a message that crosses all these different groups of people and meets them all at exactly the same level as they should be met. So we really try and focus the page for one type of user or one type of or one stage of awareness that we’re focusing on, or is it for example, just at least knowing what stage of awareness they’re most likely to be when they’re coming to this page? And then what’s the state of awareness supposed to be by the time they reach the end of the page? If they, they decided that they love the page so much. And they read all the way through? Or they absorb all the information? What’s their likely stage of awareness to be by the end of it?

Rommil Santiago 13:10
So this raises the question where one page one message, I think that makes a whole lot of sense. And a lot of folks try to push for that. But in terms of a homepage, let’s say where it is going to be the landing page for a lot of organic for different types of people. How would you take that into account where you might be having to address a lot of folks, uh, how do you figure out the messages in the order? And I know, we’re kind of deviating. But I’m really curious to see what you think.

Eden Bidani 13:44
Yeah, absolutely. Well, this is something that we worked on a while, I helped out with bush for the product led Institute when they were redoing the homepage. So because they have different people coming with different levels of knowledge with regards to product led growth, and they were all coming to the same page at the same time. So it really that it. It, it depends on the business, it depends on the business model depends on the niche, it depends on the type of audience as well. But really, kind of two core ways to structure your homepage, for example is one of them is treating it like a signpost. So you know, someone comes to a crossroads. And there’s a signpost standing there. And it’s like this way, you know, you can get to this information by taking rides, you can get to this information by clicking here or so. And that’s something that usually suits a lot of different audiences. Whereas if it’s a specifically if it’s a product, if it’s a SaaS product, or if it’s sorry, if it’s a SaaS or if it’s an eCommerce store, you can actually almost treat the homepage like a landing page. And all those little pieces of information you don’t need to give them the full rundown and treat it like a full heavy full form landing page. But you can treat it the fact of the landing page by that they should have enough information by the time they get to the bottom of the page to make a decision, and so then all the other pages of just are almost additional benefits.

Positive John 15:07
adding my five cents of value on this homepage discussion, I’m currently running two homepage experiments into two different products. One of them works exactly as this signpost that Eden mentioned before, the other one uses the concept of the next best action. Let me drill down a little bit into the specifics, the one that acts as a signpost. It’s one product that we’re currently developing the second most important persona for to business, right. So historically, we’ve been targeting this number one persona. But right now, because of market expansion, we started noticing that okay, for markets outside the US, that secondary persona might be more powerful by any factors. So the test basically involves adding, okay, you have your primary CTA, which still targets that primary persona. But above the fold, you now have your ghost button or your secondary CTA and retesting. It’s an ABCDEF, as far as I understand. So there’s a lot of traffic involved in that experiment. And all the statistical considerations to get meaningful results. We’re actually running that with CRE. And the test has been the test so far has been very interesting, giving us directions that okay, developing the second persona that are passing by the homepage is actually very relevant. For the second one that we’re using the second product that we’re using this next best action approach. It’s basically informed by beta analysis from users. So getting a segment of from users who land on the homepage, what is usually the next best action, considering our conversion touchpoint. So okay, they usually go to this page here, because it’s relevant to their journey. Are we able to optimize this for bringing more attention? Let’s, maybe a CTA above the fold on the homepage, to suggest the next best action for the user. So these are two of the fresh examples that I have regarding homepage experiments on my end.

“Quote” – Author
Tweet

Eden Bidani 17:16
Yeah, absolutely. And that’s something that’s something that’s fantastic. And when you can get all that when you can get that much information, that’s, that’s great. I mean, and that the fact that you were able to find that, that messaging that value in the product, that it was that it was the same, and then you were able to distill that into two different ways. That’s also something to do look there, there are also there are a lot of different ways that you can structure a homepage as well. One recent one where we did as well, it’s, it’s actually for there were two different audiences completely, because the people who were visiting the page, need the product and want the product, it was for a SaaS platform, they need and want the product, but they don’t have, but they’re not the decision maker that can actually make that decision. So we’re at the same time, the people who are the decision makers, were not likely to actually be visiting this website, because there were higher ups. So we had to balance those two messages, both those types of personas coming to potentially coming to the page, not necessarily for the audiences, but because we knew that the the, the people who are going to be viewing this page would take that information to the higher ups to help influence their purchasing decision. So that so we actually had kind of two columns on the page one addressing the target audience specifically, and then one addressing that, at that kind of secondary audience that what might not necessarily be visiting the page at all. But that would give enough information for the audience, the core audience to actually take back and empower their and empower them and push it kind of through the process. internally.

Rommil Santiago 18:54
I really like that example. It reminds me of a time, many moons ago, when I used to work in eCommerce for a software company and sales weren’t doing super well until we did a lot of user research and figured out that those who are purchasing our software and that it was really expensive, there was 1000s of dollars. It was never them that we had to convince, it was always the purchasing officer or their boss. So a lot of the messaging and copy and documents that were on our site, were focused towards giving them the materials to make the proper presentation and make the case that the software makes sense. So I love that point. And I think that’s it still rings true today. Oh, so you know, we’ve taken a little bit of a detour. I’m pretty sure you have a little bit more in your process. John, sorry to interrupt but you know, I let’s close that up. But what’s left in your process

Positive John 19:48
Rommil, that was a good detour. But yeah, there’s a third step in the process. I’ll be brief on this. The third step here basically consists in a more in depth analysis. So we started on the gut feelings on best practices covered potential research methodologies that help us to be more comfortable with what we found. And the third and last step, I’ve been exercising a lot with this idea of the user site framework from the Darius Contractor, he’s ex-head of growth in Airtable, in Dropbox, absolutely big brain, I got this framework from Reforge, I exercise a little bit more even leveraging Miro. So imagine like a Miro board with a bunch of screenshots that maps the user journey across pages. And we keep adding stickies and try to map this user psych, which is basically the level of motivation that the user have, or, you know, potentially have based on our analysis. But we tried to map this numerically, and see where are the bumps in the road or the friction points in the journey of the user. What I found it very interesting is that usually this qualitative way of mapping tends to correlate very nicely to the quantitative drop offs that we see, for example, in a Google Analytics, goal Funnel Report went well set. And one of the things that we’re sharing with the community Rommil, we were sharing one of the materials that I put together regarding this user psyche framework, using the LinkedIn premium product conversion funnel that I mapped under this system, this framework, and we’re going to make it available for everyone listening to us today.

Rommil Santiago 21:38
Cool, that’s awesome. Yeah, definitely, we’re gonna include that link in the description of the podcast as well as our webpage. You know, we’re here to share information and tools for all our listeners in. So let’s, let’s move on to the next topic. Let’s see here. Again, from growth mentor. So on the topic of getting feedback and collecting data from our users, how do you collect information around voice of customer? Let’s start with Eden this time?

Eden Bidani 22:08
Yeah, absolutely. So I know, we were just briefly discussing this before, before we hopped on the call. But so in terms of in terms of collective voice of customer data, it’s nice to run, I find it’s nice to run site polls and things like that. But not everyone likes to answer site polls. I know there’s a study, I can’t remember where it was. And if I find the link, definitely I’ll share it in the resources for this podcast.

Rommil Santiago 22:35
You’re telling me there’s people who don’t like to answer polls online.

Eden Bidani 22:38
Yes, I think it’s good question. When was the last time when’s the last time? You answered a survey or poll online? When was the last time you clicked? Yes. On a pop up. When was the last time you actually did things yourself? Without thinking, Oh, I’m going to help this marketer out that’s trying to collect data on me while I’m browsing the site.

Rommil Santiago 22:58
I can honestly say this morning because I set up one on my own site. I was QAing. Does that count? No?

Eden Bidani 23:07
John?

Unknown Speaker 23:08
Eden, I agree with you people don’t like answering polls at all. I remember reading something from Nielsen about that. Maybe it’s where you saw as well. But I’m taking the blame. I feel like I’m just too positive. Navigating online. I remember myself answering just, you know, earlier this week a pol from time you alar they prompt prompted me in the product is the software that I use for time tracking, and they’re pretty awesome. I feel like helping.

Rommil Santiago 23:37
But that actually speaks to the bias. Right?

Eden Bidani 23:39
Well, so that so that’s the thing. So the idea is, unfortunately, as much as it’s useful for us. And this is part of a huge other bigger discussion. In general, if you’re looking in terms of user experience, it’s like pop ups of any kind. Do people really like pop ups? Do people really like engaging with on site polls, I’m not here to challenge it, because I’m not going to we’re not going to unpack all of this, because that’s a really deep rabbit hole, we could go down. But if you can collect data, that way, you have enough traffic coming to the page, and people are willing to respond, that’s fine. One thing we also find, though, that people that tend to respond to these, they might be of a certain type of personality, or they might be like you Positive John, there might be like of like to help out. They don’t have a problem answering. But whereas it might only get us a small segment of actual user data or actual voice of customer data from it’s that type of customer personality or in terms of how they approach life, rather than a wider than a wider kind of approach. Again, this is this is with all kinds of user testing, you have to you are going to be meeting with certain types of people because they’re the only ones who respond to this kind of thing. Most people will not be you know will not be inclined to respond or participate simply because they’re busy. Not because they don’t want to, just because it’s part of their daily life, they’re rushing, they’re reading through this, they’re doing this they have other things that’s better to do. It’s not It’s not an insult, it doesn’t mean that it doesn’t work it just in terms of the overall user experience, you know, this page, what this this page poll is this one tiny, tiny, tiny section of their entire enormous, long, 24 hour long day.

Rommil Santiago 25:24
And there’s, there’s actually, I had a story relate to that, which is interesting that you brought it up, because it reminded me of the pain where we did do surveying, and we did on site, we’ve also did email and we tried different mechanisms. And we essentially tried to figure out, you know, what the customer what our target audience wanted. And we found a lot of responses in one channel, which was email. And we ended up designing features and even a whole subscription plan around that. And we had like, it had to be 10,000 responses, but it was it was it sucked to read, but you know, it’s all good. So when we launched this product, we found it fell flat. And of course, that’s not not that great. Only in post analysis, kind of like a post mortems. Like, why did this this not work? We figured out that the demo behind the email and the on site and the app and all that were different. So those who answered emails, skewed towards like, well, above 60, they had the time, they were tired. So they were like, this is what we want. And those that use the app and the website were much younger and didn’t have time for this jazz. And the stuff that we were talking about did not resonate with them. So not only is it like this bias in terms of the type of people who would answer these answers these these polls, it’s a lot about the channel as well. You have to think about, you know, what is the platform that they’re on? And is this going to skew in terms of a certain type of person? So yeah, yeah, that was great. Yeah. John, do you have any thing to add on that?

Positive John 27:04
Still on the more tactical side, and I’m curious to get some feedback from from you both. I had very good results, running polls in the thank you page. Of course, like, we’re talking ecommerce setting here. What almost held you back from purchasing or subscribing from us today? I really liked this one very insightful in most of the cases that I tested with it. And if not mistaken, I this first came to me first crossed with this question from Avinash Kaushik book, analytics 2.0. Anything that you guys can share about using this particular question

Rommil Santiago 22:35
You’re telling me there’s people who don’t like to answer polls online.

Rommil Santiago 22:58
I can honestly say this morning because I set up one on my own site. I was QAing. Does that count? No?

Eden Bidani 23:07
John?

Rommil Santiago 23:37
But that actually speaks to the bias. Right?

Eden Bidani 27:45
I usually end up doing is we actually conduct one on one customer interviews, as well. And that’s one of the questions that I asked that I asked them as well face to face and you get, you get so much rich, contextual, contextual, and voice of customer information data, it’s just it’s just incredible, the things that kind of things that come out of that question. But when I’m hopping on a customer interview, it’s usually going to be at least 20 to 30 minutes long. And that’s one of the questions that I like to that I like to ask because I’m trying to gauge what again, what was if anything that almost that almost stopped them from signing? Or was there anything that made them hesitate? And that can be that can be anything? Another rich question that we find this really useful to us is what was going what was happening in your business or in your company or in your in your life, for example, eCommerce what was happening during your day, when you decided to make this purchase, or when you decided that you needed help? So you’re not they’re not asking them specifically about, you don’t say about, you know, what their pain or problem they were struggling with? So they end up launching into this story, which shows all this contextual information that otherwise it’s that otherwise, it’s sometimes difficult for people to type out that if it’s going to take them a really long time to do it, or it’s going to take them? Or it’s just it’s this long, it’s this long, contextual answer. That is just really fascinating. You learn all the different things, what was the different problems and the pains that they were that they were struggling with? And what brought them to this realization? What kind of brought them themselves from this stage of unaware to pain problem aware to host aware with high intent,

the what was going on in your world is absolutely a great question. It’s a lot of value. When it comes to customer development. I usually have them either in jobs to be done, or in, you know, General surveys.

Rommil Santiago 29:39
So you’ve all you’ve all mentioned these great sources of ideas and feedback and information. And I’m hearing that some of these sessions take 30 minutes and they’re somewhat a lot of quality writing, but how often do you take this kind of effort on is this a continuous thing? Do you do it in one big shot before Like, when and how often do you do this?

Eden Bidani 30:04
Oh, ideally, you should do it on a regular basis, every cup every few months or as as new features get released or things like that you should try to, you should try to get this information regularly. Because because it changes, things change, people change, customers change, you know, the the world around us changes. So we need to understand things, what’s happening in that context. And if anything has shifted, even if it’s just to confirm what you learnt previously, or to see that you’re still going in the right direction, or if anything has shifted, in the meantime, socially, culturally, economically, whatever that might also might also be in the scope that might reflect that might impact on the data. But if it’s if it’s for a specific project, if it’s for a specific project, usually we do it prior to prior to a launch or prior to refurbishing a website or something like that, to get that information. So that that helps drive the direction of the new design of the new copy of the of the new layout, things like that to get started, recommend people do it as much as possible or if not at least, have call recordings that you can listen to have the you know, the customer service team recorded recall, speaking with them, listen to sales calls, listen to the demo calls, listen to you know all just try and collect and go through that feedback on a regular basis as well. Just to enrich just to be enriched with you want it because a lot of it’s from a messaging and copywriting perspective, a lot of it comes down to semantics. Sometimes it’s how people are expressing their pains and their problems and, you know, their experience their successes or their failures with a product or products, you know, whether it’s SaaS or eCommerce or anything else, go with the service. So it’s, it adds it adds just that qualitative layer that I just find that adds that extra depth, which is really useful in terms of in terms of just moving things forward.

Positive John 32:02
Yeah, talking about interviewing internal teams, talking to sales and customer support is always absolutely gold, isn’t it?

Eden Bidani 32:09
Yeah, absolutely customer support, if there’s a head of customer success for head of customer support, but really cuts who’s answering those tickets, like as they come in, really, that’s who you want to, that’s where you want to speak with because of part sales. Yeah, this is biting my time, while I’m saying this, and you know, sales and marketing and closed sales and marketing are close to the customer, but who’s really close to close the customer is customer support. That’s who’s really, really close to the customer on that kind of that troubleshooting that problem solving that almost day to day level. And that can be you can get some incredibly powerful insights just from that. But the benefit here is not just you’re not just collecting on, you’re not just collecting for the sake of collecting and seeing if there’s something that you can iterate on. Now, this is the kind of information that can get that might spark ideas for what you can do moving forward with the company can do moving forward, if someone raises an idea about a product feature, or an idea about something or that something in the world has changed. You know, I look like the comparison between between pre COVID and post COVID. Like the different things change, and how are people dealing with them. So maybe it was a social change, again, economical, cultural, something else was happening on in their in their slice of the world. And then that can and that can help shape which products you release next quarter, or which products you release next year, or any features you might decide to add and things like that. So you can actually just keep tapping into the customers to help even better the product and to better the company moving forward. It’s not just for the sake of conversion optimization, but it’s just an informed product as well. With regards to customer reviews, usually try to get between five to seven and even closer to 10. If possible. Usually a good mix, though, look for three to four that are customers who are happy with the product, maybe a couple of them that journey and a couple of them that were previous customers and have left for whatever reason.

Positive John 34:10
So five to 10 interviewees. That’s what you aim for in trying to diversify the people you talk to. That’s very nice.

Eden Bidani 34:18
Yes, you try again, you aim, you aim for it, you can’t always get it. But if you get some you can get some really, really deep insights as well as to why they left or why they why they abandoned the process. For example, for those who have left off.

Rommil Santiago 34:34
From from a tactical perspective, do you I was just gonna say bribe, but that’s the wrong word. Do you have incentive for folks to participate? Like, is it what what’s in it for them? Is it altruistic or do you motivate them in any way?

Eden Bidani 34:50
So that and that’s a good question. So we initially try to not incentivize the interview, see who responds. Again, that’s kind of that’s playing off the altruistic idea that we’re who answers on site polls? Are these altruistic people just want to help out these poor marketers who are trying to collect information. But at the same time you try to not incentivize it or you incentivize it with something that’s not related to the product, or service. Sorry, I just don’t you incentivize with something that’s not related to the product or service. The reason being, because you don’t want that you kind of don’t want that to, to try and influence their experience. It’s like, Oh, so they might. So if they know they’re getting a, you know, a special gift, or if they know they’re getting a month free of a Saas product or something like that, in exchange for their time, they might be more willing, more inclined to give a positive review or positive share positive only feedback when you just really want to hear everything you want to hear the good, the bad, and the ugly.

Positive John 35:51
One fresh example that I’m running, again, the go to market project that we’re running with Survey Monkey, we start testing the answer rates by first sending the survey to the best 5% of our email list base, based on open rates. Alright, so we targeted those users. And we saw that okay, answer rate compared to answer quality. Is this where we wanted to be? And how will that scale? Do we have any guesstimation for that? What I usually do next is a $10, Amazon gift card tends to increase answer rates quite a lot. You know, I had good experiences, using that tactic, if you want to call it. But more than a $10 amazon gift card, I start basically considering buying an amount of users from Survey Monkey panel to get a better answer rate with better quality of answers. So this is pretty much how like how I think through that process.

Rommil Santiago 37:01
Like I’m currently literally doing this for Experiment Nation. So I’m a very keen listener today. Like, yeah, yeah, need to talk to these customers. Yeah, I do. Yeah. Awesome.

And that’s it for another episode of Adventures and Experimentation. What did we learn today, we learned that recording your sessions with clients reduces the need for taking amazing notes. We also learned that qualitative research is the key to not just CRO, but for all things business. There’s just nothing like the voice of customer. And finally, we learned that no one likes surveys except Positive John. If you’ve enjoyed this episode, and you think we’ve earned it, please consider subscribing joining our Experimenter directory and telling a friend about Experiment Nation. Until next time.

Transcribed by https://otter.ai

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.