CR-No: Episode 2
The following is an auto-generated transcript of the podcast by https://otter.ai with very light manual editing. It’s mostly correct but listening to the actual podcast would be wildly more understandable.
Rommil Santiago 0:02
Okay, so this is awkward. This is who has the page open? Could I ask for someone to lead this part of this question as I take five minutes and deal with my crying son?
Siobhan Solberg 0:23
Yeah, I’ll do it.
Rommil Santiago 0:25
Sorry about that. I’ll be back.
From Experiment Nation, I’m Rommil Santiago, and this to CR-No.
CR-No is a series that pulls back the curtain on the conversion rate optimization industry. Listen in as a panel of experienced CRO veterans talk about some of the joy and a lot of the pains of our industry. On today’s panel, we have Siobhan Solberg, from Raze, Shiva Manjunath from Gartner, Kenya Davis from Evolytics, and Eddie Aguilar from Cicle Media Labs, let’s listen in.
Siobhan Solberg 1:17
Let’s talk a little bit about how we balance between our optimization projects or how we balance optimizing the funnel as a whole or just optimizing the website. Shiva, why don’t you go first.
Shiva Manjunath 1:32
Yeah, I think it’s it’s interesting to think about optimizing the website versus the whole funnel in that sometimes it could be a chicken and egg situation where at least from my experience, like what do you optimize the ad? And then have the landing page match the ad? Or do you know, the audience that’s coming to the website, and you create a perfect landing page for them and back into the ad? like to have the ad kind of match that landing page? Or is it both? And I mean, I, in my experience, I’ve run tests where we’ve had a landing page that matches the ad copy exactly. And it’s done terribly. And then we have an ad that is completely different than the landing page. And it like conversion rate goes through the roof. And it’s, it’s just super strange. I mean, I’ve seen mixed results. I don’t know if any of you guys have had any experience with any of that stuff, too.
Siobhan Solberg 2:28
Yeah, looks like Kenya might have.
Kenya Davis 2:30
Um, yes. I’m actually working with the client where they got a little overzealous with the design changes, and, and feature changes. And they changed the entire funnel, which makes it extremely hard to articulate what really drove the customer in any direction leaving pushing through. Right, and it’s good. I’m finding that that increases with the more exposure to testing. I feel like you know, they, they do the crawling phase of like I’m going to start start with my first test and cool, I did everything right, I got my primary KPI and my hypothesis, and then, you know, they get excited, and they’re like, boom, here’s 10 – 12 changes, tell me which funnel is better, like, you know, that there are times when you need to completely optimize the funnel. But there’s still so much that has to go into the research part of that on where do you start first and identifying the true problem? Or, or even if there is a problem, I just find it it’s funnel optimization can be a little stressful, and that is a huge investment in terms of time. When I hear that, yeah, I just think of like, Okay, are you ready to dedicate this full year on optimizing your funnel?
Eddie Aguilar 3:57
Yeah, to piggyback off Kenya, that’s something I definitely noticed is you have to basically research to see where the problem is, and if it’s the website or the funnel. And that’s where you end up starting, especially if the if the client comes to you and ask you, oh, I’m having issues converting. Can you see what’s going on? And you start researching and you notice, it’s probably the ads are just not converting then you’re you’re basically looking to start there and try to push those users out into the website. And then you can possibly start up the website. One thing I’ve, one thing I’ve noticed or I’ve done is I’ve had clients where they are spending so much money on all these different channels, when they can solely focus on their best converting channels and be optimizing inside of those rather than all these different types of channels where they’re really just spreading their time and efforts to infinity. And that’s usually the case that I run into, and that I always try to start wherever there’s like a leak in their, in their process.
Siobhan Solberg 5:20
Yeah, I feel like a lot of people come to you initially, right? And they want you to optimize their website. And then as you’re working through all the research and analyzing everything, you start realizing that there’s a bigger problem. And it’s just sometimes I struggle with this concept of where do I stop? And because, yes, I am supposed to be optimizing the website, but I feel like I can only do so much on the website, if let’s say, the journey of the user of the customer hasn’t been optimized, or just hasn’t been addressed properly. I think a lot of it just comes because I come from, like, I’ve done a lot of conversion, copywriting. And, you know, when you write any kind of a landing page or anything, you’re always looking at these kind of, you know, stages of awareness, and you really try to present a problem, you agitate it, all these kinds of things. And when I look at it that way, and then I apply this to the whole funnel, I started thinking that it’s really important to optimize the funnel as a whole, because, you know, if the ad is making it user aware, then how do I address the landing page to address that, like, you know, taking it past this concept of message matching, let’s say, but where in the journey is the user? And I feel that if I was just optimizing, let’s say, the website or the landing page, I would address it one way, whereas by looking at the whole funnel, and I really start looking at those stages of awareness, and where’s the user within that journey, or how many ads have they seen, and I can optimize the website itself better. So for me, they’re Blurred Lines. And for most of my clients, I end up doing, I, I optimize the website, and I advise on the rest, meaning I work with the other teams to make sure that we’re in line. And we have one kind of cohesive strategy for the whole funnel, but that each of us is in charge of optimizing on our own, but we’re very aware of what the others are doing. And this is kind of how we are trying to not get overwhelmed. like Kenya says like, I really don’t want to spend a whole year optimizing someone’s funnel.
Shiva Manjunath 7:22
Yeah, and I was gonna say like, on that point, when you think about the users coming to the site, it’s like, it can be difficult if things that are maybe out of this quote, unquote, CRO’s control, which is like traffic quality, and volume and other things like that, that might be difficult to optimize performance, when the users coming to your site, are inherently just not the the highest quality traffic, I mean, you can have the best website with the most streamlined conversion funnels. But if the traffic bluntly, if the traffic sucks, you know, it may not matter how good your website is, but kind of to that same point, when you have a large bucket of traffic coming to your site with kind of a single experience, I’ve had a lot of success with trying to like, segment out the behaviors that users are taking and putting them on unique pads as much as you can. And obviously, that’s a very big lift, but like, it’s kind of in the personalization vein, but like, if you know that a user or you can target users that you know, aren’t ready to convert, you try and give them a unique type of experience, which is maybe education focused, maybe it’s more in the like, learn more about the brand path, maybe that’s a little bit more there versus users who you know, are ready to convert, you kick them off to like a streamlined conversion path, like and that’s like something you could look at your email list and say, Alright, well, these are the users that have bought frequently from us. So you build an experience for those users, which is streamlined for what they’re looking for, versus maybe a new email subscriber. They’re not necessarily ready to convert, maybe you need to focus more on value prop or other things like that.
Siobhan Solberg 8:56
Yeah, I agree. And I have to say on this one, I actually make people self segment within an opt in email for this exact purpose, because then I start my segmentation quite early. I think segmentation is a huge thing. I’m really happy you brought it up, because it’s, to me really, it isn’t personalization, like in the creepy way, but it is personalized enough to make a user feel like you understand what they need. Looks like Kenya has something to say about this. So
Kenya Davis 9:25
yeah, no, I I definitely love that you brought that up, Shiva, because I think one of the biggest things that I’m pretty sure we all have to constantly bring up to our clients is that you have to have consistency. If you’re doing any funnel optimization or any segmentation, like if you can identify those people who are coming in and they’re just researchers or those that are ready to convert, making sure that the entire process to getting to conversion is is at least in a like the same messaging the same color, the same pathing or the same pages like that has to be a part of the whole process. And then you kind of fall out of that, typical AB testing and very strict experimentation methodology. You start running into, like, you know, there are a lot of variables changing. But if you are testing against, like an entire experience on the funnel, according to a segment, it does actually, you know, bring it back to a more narrowed focus. There’s the big thing is just making sure that it is consistent all throughout,
Siobhan Solberg 10:33
I was just gonna say like, how do we bring in segmentation, then into the funnel? You? So I’m saying like, how are we still just segmenting the users coming in and then giving them an optimal experience? Or do we still want to work through the whole funnel? Or does that allow us to just focus on the website now,
Eddie Aguilar 10:51
in my opinion, that depends on the sample size that you’re you’d have in those segments, at that point, because I’ve seen segments where they only have like 100 people. And I don’t think you can really optimize for just a specific 100 people unless you start getting to like personalization at this point. And that’s a whole different topic.
Shiva Manjunath 11:17
One of the underrated things about creating segments like that is, technically there is lower traffic volume. But oftentimes, what you’ll see is a lot of traffic performing in the same way will yield that statistical significance quicker. Like let’s say you’re trying to optimize desktop versus mobile. As you know, as we’re all well aware, mobile generally is more research focused and desktops, more conversion focus. So if you’re trying to say optimize the whole experience on desktop and mobile at the same time, you’re going to see a mishmash versus you’ll see consistent behavior generally on mobile users versus desktop. So that’s an example of like segments, where when they start to perform very, very similarly, then you start to see that statistical significance is actually a lot easier to accomplish. When you have a consistent source of traffic that performs are similar. To be fair, 100 100 Sessions is not going to be, you know, easy to optimize in any vein. But as you start thinking about like these, these segments, maybe a little bit bigger than that, as long as they start to get more and more consistency, and you start to target a unique set of users, it’s actually easier with statistical significance from my experience.
Rommil Santiago 12:29
So I guess it’s like, it’s a stronger signal that you’re trying to detect. And yeah I’m back. Sorry.
Kenya Davis 12:39
Like, that’s different.
Rommil Santiago 12:43
Oh, sorry, I interrupted someone.
Eddie Aguilar 12:46
Yeah, that’s, I was gonna say that’s something I’ve generally done. And especially when I’ve worked with like, predictive models, trying to optimize for specific segments, we just kind of we will mix a few segments, just because like you said, it’s the users are working are basically doing the same thing across those segments. So we just mix them in together in order to reach that set, see much quicker.
Siobhan Solberg 13:18
But I don’t know, I think segmentation, you know, now that we’ve kind of ended up on this, it’s essential nowadays. Even if it’s a sub segmentation mobile, that mobile desktop, you know, it’s a given, but even users behavior is coming from different ads, people are optimizing their landing pages based on where people are coming from. It’s essential, but obviously, you know, Shiva has pointed this out now that, yes, we have a lot less outliers, let’s say. So statistical, statistical significance can be reached quicker, with less traffic. But you still need the luxury to have that traffic. So you know, the biggest segments tend to be, you know, devices.
Shiva Manjunath 13:59
So I was just gonna say, I don’t know if you guys see this often, but like, kind of like the previous conversation about best practices, but I’m so sick of people writing articles telling us that desktop users are more likely to convert and mobile users are more likely to be in research and articles that are telling us that you have to have a mobile optimized website. I’m still seeing these articles, and I’m so sick of it.
Siobhan Solberg 14:28
Especially now I feel like suddenly COVID hit everyone’s at home. And guess what mobile has is not as important right now. It’s still being done for research and stuff. But I’m noticing, especially in eCom, where I used to get, you know, really 90% was mobile, and now it’s dropping to like 60. And it’s just because people are at home, they’re using their laptop, it’s more comfortable. And then these articles are so again, they’re best practices right they’re so general, and it’s just like this generality of it. It’s just Nothing is general you have clients at 90% mobile and you have clients. I got a shocker the other day when I looked at it, I looked at it and said 40% was mobile. I wasn’t shocked when you see an eCommerce site, we’re only 40% as a mobile device. You know, it’s, it’s really client dependent also, on articles, forget it.
Shiva Manjunath 15:23
I, I just think if you if someone’s writing an article, and they’re telling me I should have a mobile optimized website, I immediately discredit the person. Because if you’re if you’re not, if you don’t have a mobile optimized website in 2020, what are you doing with your program?
Rommil Santiago 15:43
By the way, I’m 100% writing that article this to cheese you off.
Shiva Manjunath 15:46
I expect no less from you.
Siobhan Solberg 15:56
There are people who don’t have mobile optimized websites, you know,
Shiva Manjunath 16:00
I agree. And I would never buy from that person. It’s it’s absurd. It just, I mean, it’s the same conversation we had where like, people still think CRO’s button color testing. Like, there’s, there’s still people who do not have mobile optimized sites. And it’s it’s sad to see.
Siobhan Solberg 16:17
And it’s really sad that when we hit those sites, we get so critical. I know that I get I get to a site and it’s not mobile optimized, like I’m not buying from here. But my partner’s like, buy it, we need it.
Rommil Santiago 16:29
I’d love to be a fly on the the wall in your houses as you’re shopping online? Why are they running this test and looking at inspecting all the tags using?
Eddie Aguilar 16:42
Have you tried using Google tag manager or Google Analytics through Safari on your mobile? Definitely not optimized for mobile.
Siobhan Solberg 16:54
I feel like Google products are so behind when it comes to mobile optimization.
Shiva Manjunath 17:03
But they have their own testing tool, right?
Siobhan Solberg 17:07
They don’t need it. Isn’t that it? Like when you’re big enough? You don’t need it.
Shiva Manjunath 17:12
Yeah, I was I think I wrote about this a couple days ago, like does Google Optimize use analytics to look at Google Optimize experiments for Google Analytics? Like there’s some weird Inception going on with that? there? It
Kenya Davis 17:23
Definitely is I just gonna test and I am not a fan. Um, I will pass. I think they have some work to do.
Rommil Santiago 17:37
So sorry, I missed that so you’re not a fan of what?
Kenya Davis 17:40
Google Optimize in the Google Analytics relationship.
Rommil Santiago 17:44
Oh, yeah. That’s a I Oh, yeah. I did that a couple of weeks ago, actually. And I’m going the same company. Why? Why is it this hard?
Siobhan Solberg 17:57
But unfortunately, I mean, fortunately, or unfortunately, right? It’s free. So it’s an entry level thing. So now it opens the whole testing and personalization world to a lot more people. And it also kind of backfires. Because people like us who would love to, you know, use a better tool sometimes kind of get stuck in Google Optimize because clients like that, that’s free. Um,
Shiva Manjunath 18:23
But that’s a blessing and a curse. Because it’s a blessing that people who have a basic and good foundation of CRO can run tests and be good. But then there’s the con of like, people who don’t understand CRO principles targeting all this stuff. They could be running tests that break their sites, it could be running tests, that they’re interpreting the results in the worst ways possible, because they don’t know what to setup or they’re breaking their site, or, I mean, I would talk to someone before and I was looking at an experiment, they were running, and they made no changes on the site, it was an AA test. But effectively, like they didn’t do it on purpose, but was effectively an AA test because the code broke. And they were like interpreting the results as Oh, no change. Like it can be a blessing and a curse if, if you don’t know what you’re doing. So it’s being free. I don’t know if that’s necessarily the absolute best thing for everyone.
Kenya Davis 19:12
Sometimes it could be like, if you happen to have a savvy, CRO person or optimization person and a dev of when it comes to Optimize, you know, you can use the tool to properly split your audiences and do all of the calculations on your own on the side. But with that, you know, you do have to have enough QA along the way to ensure that you’re not jeopardizing the experience of your site. Like in our setup, we went into that that very issue of we thought that the modal was working, and instead it wasn’t really working. It was it was popping up every single time someone made a click. And so you know, the Google Optimize, this experience is working. We’re like, what That’s shocking to this verdict this quickly and we go on the site, and it’s like the nuisance of a pop up. And it turns out it’s broken. And you know, there’s no alert or any sort of that. So yeah, it’s, you could use it, but you’d have to have a savvy person who could manipulate it.
Siobhan Solberg 20:18
Okay. Oh, yeah, we use it a lot in the agency. I have three devs who only work on Google Optimize. And we have two QA who are causing QA. It’s the only way I’ll do it.
Rommil Santiago 20:30
I see any has a comment.
Eddie Aguilar 20:31
My biggest pet peeve with Google Optimize is they’re fixed to flashing. Let’s just add a big white page. Oh, yeah. seconds before it flashes, like
Rommil Santiago 20:46
But you can change the delay. You can make it shorter or longer.
Eddie Aguilar 20:49
Yeah, I know. But like, why would you want to make a bigger flash than the original flash?
Siobhan Solberg 20:58
You know, I never put the anti flicker in. I feel like the anti flicker causes a worse user experience at any test that just, like clicks a second could. I guess.
Rommil Santiago 21:09
That’s interesting. What do you what do you folks think about the anti flicker tags? I know he’s way off the original topic. Like there’s a real obsession about putting it in ensuring the sight doesn’t flicker? Do you think it matters?
Siobhan Solberg 21:24
I think it matters, if you put it in. I think it’s it seriously, two seconds, sometimes just hiding and it can, it’ll get stuck. I’ve seen it, it gets stuck. And they can’t load something on the back end. And you’re making the experience so bad by just hiding the page from the user. Whereas most of the time, you could open the page and the user doesn’t even notice that you just switched a test. Like a you know, like, for example, I know is on board with this one. I was just reading an article of his few years, like a year ago about this. And back then I wasn’t sure and I start doing research. And I started like looking at the experience of test. It’s horrible. I’m talking about the Google optimized anti flicker, there’s I’m sure that our testing tools have a good kind of non flicker concept. But I don’t know, I feel like every time I QA a test to look at it, test it. Yeah, I think that delaying the page for the user is probably the worst scenario.
Shiva Manjunath 22:22
Yeah, I kind of agree with what Siobhan is saying. I generally think it just depends on the test. Ideally, if you can, with the tests that are above the fold, and you’re doing something above the fold, which could cause a flicker as much as you can try and run it server side or try and minimize the amount of code being executed. I that’s where I generally just try and prioritize the test with the devs to run it server side versus client side. But if you do have to go with the flicker, I i’ve similarly had some weird issues with optimized with the anti tag, anti flicker tag that I just, I just turned it off and optimize the heck out of the code for it to run in the fastest and most efficient way possible.
Rommil Santiago 23:07
Yeah, so we’re at time, thank you again for having that to carry on without me like FYI, background. My son has a little bit of an allergy to certain foods. So he thought he ate something. He came in all Oh, I gotta eat something. I’m gonna die. He’s still alive. That’s why I’m on this. Really awkward. Yeah, he’s at the hospital but let’s continue. I may have to edit all this out.
Shiva Manjunath 23:41
Rommil Santiago 23:46
And that’s it for another episode of CR-No. What did we learn today? We learned about the importance of segmentation and consistency when optimizing a funnel or website. We also learned that the panel is just not a big fan of Google Optimize and how it integrates with Google Analytics. If you’ve enjoyed this episode, and you think we deserve it, please consider subscribing. Until next time.
(Transcribed by https://otter.ai)
About Experiment Nation
Experiment Nation connects Experimenters from around the world to new ideas – and to each other.
We send a newsletter every week.
- Adventures in Experimentation (4)
- Analysis (1)
- Announcements (5)
- Community (1)
- CR-No (5)
- Experimentation from around the world (2)
- Frameworks (2)
- Growth (6)
- Interviews with Experimenters (107)
- Management (1)
- Marketing (2)
- Podcasts with Experimenters (13)
- Point/Counterpoint (1)
- Product Experimentation (5)
- Product Management (1)
- Profile (9)
- Question of the week (5)
- Sponsored by Experiment Nation (1)
- The Craft of Experimentation (1)
- The Cultures of Experimentation (1)
- The weekly buzz (12)
- The year in review (1)
- Weekly Newsletter (52)
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.