A podcast with Gartner’s Shiva Manjunath about Experimentation
The following is an auto-generated transcript of the podcast by https://otter.ai with very light manual editing. It’s mostly correct but listening to the actual podcast would be wildly more understandable.
Experiment Nation 0:00
You’re listening to Experiment Nation, the podcast with this guy.
Hi. This is Rommil, and Experiment Nation. We interview experimenters from around the world. We share their stories, their lessons and advice with you. our listeners, you’re listening
Experiment Nation 0:15
to Experiment Nation, listening to Experiment Nation,
Unknown Speaker 0:21
all over the place man in the news. It’s time for the lightning round.
Unknown Speaker 0:25
Am I recording this for you? What’s up for me?
Unknown Speaker 0:30
figured that out and
Unknown Speaker 0:32
you’re listening to Experiment Nation, the podcast.
Hello, everyone, and welcome to the show, Experiment Nation. The podcast is brought to you today with the support of our friends over at VWO. Today, I have shivah menginap. He’s from Gardner shivah. Welcome to Experiment Nation, the podcast.
Thanks for having me,
man. So Shiva, we’ve been chatting for a while. And you’re our second guest. Congratulations on that. I
Save the best for second, right.
No, you know, I guess that that sounds trite on my part. But honestly, I did reach out to you, because I follow you on LinkedIn. I love your stuff. We have great conversations, at least online. I think you’re really knowledgeable and a funny guy. So I had to have you on the show. So let’s kick this off with a little bit about yourself.
Sure, well, I could tell you, I didn’t really start in CRM, or digital analytics or any of that stuff. I actually started off as a doctor in the path of medicine so to speak. I was pretty interested in like science as a whole and experiments kind of at a young age, but not really like a B testing just like running scientific experiments doing you know, little things like that. But I think I wanted to do kind of medicine, my dad is a veterinarian, so I thought I wanted to go into medicine. But when I got to, like senior year of college into, like thinking about going to veterinary school, I realized I really didn’t want to do that I wasn’t really passionate about it. So ended up making sure that people
you didn’t care about people or living things anymore.
Uh, well, I guess that’s one way to put it. I really didn’t. I just wasn’t passionate about like, I love my dog. To be clear. I love pets, I love animals. But there’s just something about medicine that just, it didn’t like, I didn’t wake up in the morning like, man, I really want to, like open up a dog and do stuff inside its intestine to deal with blood and stuff like that.
It makes you wonder about folks who you know, you know, wake up in the morning and say,
yes, yes, this
is what I get to do.
I’d like to think a large part of it is to helping the animals piece. I’m sure that’s the reason they want to do it for them alone. Some of them don’t. But yeah, I just when I realized that veterinary medicine wasn’t going to be like my long term play, I ended up taking a job at a startup company that did digital marketing, and kind of, you know, consulting in that way. And, you know, I worked very closely with the CMO and the CEO over there, and just the put me on a great career trajectory to just being super fascinated about not just digital marketing, but specifically a B testing and auditing websites and saying, wow, this is really cool. Or man, I don’t like how they do this and continue to gain my zero knowledge kind of on the fly with a lot of hands on work. And, you know, that kind of led me to where I am today with CRM, just running websites running for running tests on websites and kind of just doing what we call CRL, which isn’t really the best descriptor as we all know, but basically just doing experimentation and, you know, just loving it every day.
So you did mention the word zero. For for those who aren’t as familiar with the term, I’d love to hear how you’d describe it to say, your parents or someone who isn’t familiar with the term.
Well, to be fair, every time I talk about my job to my mom, she always thinks I work in it fixing computers, so
my mom thinks I’m in sales.
I mean, technically, you are right. It’s definitely a branch of that.
It’s simple to explain. Yeah. But see what I was not. So yeah. How do you describe it? Yeah,
I guess. People in the CRL world feel weird about the term CRM, they don’t like the word. Personally, I think it’s, it’s fine. I think generally speaking, using the descriptor, like a lot of people’s job titles aren’t necessarily exactly what they do. So to that vein, CRM isn’t really conversion rate optimisation for a number of reasons. One, if you’re optimizing only for a conversion rate, you’re kind of missing the forest for the trees. If you hyper focus on conversion rate, you might be missing kind of the engagement piece, you might miss the branding piece. You might not be looking at the end result of lifetime value or you know quality of conversions, ao v etc. So Strictly speaking, CRL you shouldn’t be Optimizing only for conversion rate optimization.
And would you say the SEO quote unquote, should optimize for them?
I mean, it really depends on your business. But it’s more just the way that I try and define CRM is just experimentation at its core, but finding better ways to create a better experience for the website, using experimentation as a way to kind of get to that end result.
So we’ve kind of, so we’ve covered what it is, and kind of what it isn’t. So I’d love to hear about some of the misconceptions around zero that, you know, you’ve been in this field for quite some time now, and I’m sure to be hurt. I’m sure you’ve heard a few.
Yeah. So one of the big things that I feel like people always, they always infer CRL is a button color testing. And, man, that is my existence, isn’t it? Where people just say like, Oh, so have you tested green versus blue? Like, please stop talking?
Let’s, let’s make it
Have you tried a different word in the CTA. And like had, it’s not only about buttons, why’s whoever started this notion, I really hope just has bad things happen to them. Because it’s just, it’s just people just always assume it. So like, that’s the first thing whenever I talk to people actually ran like a CRM presentation for some folks internally at Gartner. So they could learn a little bit more about what we do. And that was, the big point that I tried to staple home is like, please, just let’s quell this myth. It’s not button testing that’s missing the forest from the trees, it’s just not what we do, we do so many more things. That’s not to say we don’t test buttons. It’s not like we are we ignore that facet. That’s not part of what we do. But to just boil it down to that. And this is so much of what we do, right we can, we could test so many different things that’s far beyond just a website, like one of the one of the things I love about what we do and experimentation is this, it’s we learn about our audiences in pretty unique ways that we could test things that you might say, this is an interesting learning, let’s take that to the brand team, let’s take that to, you know, let’s take that to the sales team. Let’s take that to marketing, let’s, let’s take these learnings and apply it to all these other functions. So like, it’s not just in a static bubble, where you’re just doing button color testing, UI testing, partnering, maybe with the UX team, UI team, whatever, it’s, you should be taking these learnings and being very collaborative across all these functions. And you have this tool to help you accomplish all these things. So you know, just doing button color testing, man, you’re missing the forest in the trees.
So if I get this, right, it’s it’s more of a partnership, rather than you being off on your own just pushing buttons around. It’s it’s more of a joint effort where you’re trying to improve the customer experience. And and you’re partnering with folks in marketing and sales and product and, and all those teams and, you know, working all together really?
Yeah, I mean, that was one of the big mistakes I had when I first started out at CRL was kind of just keeping it pretty simple with just, I would say, honestly, my first few tests were button testing, because that’s kind of what I assumed it was when you were doing it. But it’s so much more than that, and keeping that collaboration in place. So that you’re you’re letting the people know, what these tests are, what are the results. And you know, you can’t just the other thing that people I think often miss is the UX team, and the UI team and the dev team, the brand team, they all need to be involved in the process. It’s not just you saying, here’s my idea, I’m gonna run the test. All these other functions have these important ideas that can help influence what you’re doing, help you get better test ideas and run better, more efficient test. So if you’re doing it on your own, you’re you might be successful. But if you partner with everyone else, you’re not only influencing and making your own business better, but you’re helping the whole, you know, company, gain
So, okay, we’ve, we’ve talked about SEO, and I did bring you on, because I get a kick out of following you on LinkedIn, you know, you post great memes and stuff. And I was wondering, what’s your inspiration for these things? And do you think that CR O’s and experimenters have a good sense of humor? You know, they’re,
they’re kind of, you know, there’s this
thought there’s this thinking that, you know, there’s science or data or statistics geek, so they’re the kind of dry? Oh, yeah, I’d love to hear your take.
Yeah, honestly, the inspiration for my means comes from me just sitting at home in quarantine life watching the opposite I’ll watch something, I’ll laugh and I’ll be like, Huh, that’s reminds me of a CR o thing that I post that I’ve been thinking of and, and to be clear, I think a lot of my meetings are more of just like humorous ways to talk about a concept. A lot of my posts are what I try and do is use the meme as a way to talk about something that I want to talk about with the CRL world. And kind of to that point, I feel like sorrow. It’s not really unique to Sierra but zero can definitely benefit from this where it’s, you have to use a vehicle to kind of talk and communicate the findings and the things that you have. Because it’s, you know, as data people, right, we can definitely be a little bit more bone dry in terms of not it’s like, is it binary? Yes or no? Did it win? Yes or No, but like, there’s a lot of times where you have to have a little bit more of a finesse to get by in Sierra was a lot of politics, right, you have to get buy in from a lot of people, you have to give and take and push and pull in terms of getting resources getting brand, buy in, etc. And it’s there’s, there’s definitely a sales part of the job that you have to have in the politics part of the job that you have to be successful. Because if you’re just a, you know, a pretty straightforward person that says, here’s the test, here’s a day to hear the results. A lot of you have, a lot of people will want to listen to you. And they’re definitely a little bit of charisma to it. So is it humor? Maybe Maybe not, I like humor as a way to kind of sell my points and talk about the things I want to talk about. But you definitely have to have a little bit of like a sales mentality to sell what you’re doing and have people be interested in what you’re doing to be able to help get accomplish the things you want to accomplish.
So actually, I wanted to touch on something you mentioned, you said politics, that’s actually that’s something you see written, like anywhere, I’ve seen many job descriptions for zeros, optimization, experimentation. And I don’t think I’ve ever seen a line in there saying, able to navigate politics. How important is that in CRM,
very important. I guess it just depends on the job that you have and the function like where you’re live in terms of a function, where zero lives in terms of the function like if you’re on the marketing team, or you will or you your own CRM function, or whatever. But I think politics is really important when it comes to getting buy in from people to run tests getting buy in to prioritize your test. There’s obviously a huge data piece of it, where Sierra will help you get that data piece. But you know, when you think about things like Hippo, right, where a lot of times people will have a test idea and be like run this test, run it this way. Or they might have preconceived notions about the statistics and be like, Oh, look, it’s up by 20%. After three hours, let’s throw it all to 100%. Like, you have to have a certain finesse and a way of navigating that relationship, especially when it’s your boss’s boss’s boss telling you to do something, when you know it’s wrong. And we know that’s not how things are supposed to be done.
Okay, that’s, that’s interesting. So does humor impact politics?
I don’t think so. I’m not usually but I will tell you one time I did get in trouble at work. I although I don’t want to say it in trouble, per se, if it did get something but the potential for me to get in trouble was pretty, pretty significant. So one of my previous jobs, I won’t name which but one of my previous jobs when I first started with the company, their process was, let’s run a test for a couple of days. If that, look at the results and see how it’s doing. And if that’s winning, then push it out to 100%. On to the next test. Right? And as I say that, all the rowers on the board, remember, everyone’s like, Oh, that’s not good. But, you know, there’s, like I mentioned before, there’s people who don’t hundred percent know the statistics, they don’t really understand. And to that point, like, one of the things I wanted to plug was man, don’t, don’t always always trust your testing tool, 100%, you really need to stress test what goes on to those results. Because a lot of times you might see like a 20%, lift at 100% static or whatever that your tool might be telling you. But you shouldn’t trust that because especially statistical statistically wise, like, after two days, you shouldn’t trust those results. I wouldn’t trust those results. But that sometimes those tools do give you that and you have to have a layer of like, well, that’s what my tool is telling me. But I know I’m not going to trust that until I follow up my own internal, you know, statistics to vet that out entirely. But yeah, with this particular example, with them kind of pushing a test at 100%. It wasn’t even following statistics. They just were looking purely at the lift and saying, Wow, plus 40%. This is great. Let’s roll this out and move on to the next one. But obviously, we know that tests take a while to normalizes business cycles. There’s other reasons why we have to let the test run, you know, the duration we needed to get the traffic that it needs. So when I first was looking at this process, I was like, Man, this isn’t good. But you know, I wasn’t. It was what three or four days into the job seeing this process and I didn’t feel super comfortable with my poll to tell my boss’s boss’s boss that this is not good. You’re doing it wrong. So my superstar way of doing it was the next test that I was tasked to run and build. I ran it as an a test. And it was in a test. So when they peaked at the results, and they saw plus 20%, or whatever number they saw, so great was a lot of fun on her birthday. I said, Well, are you guys sure about that? And they’re like, Yeah, why not? Why wouldn’t we this is great. 20%. I said, Well, this isn’t a test. And they’re like, I don’t know what that means giving me some weird, confused blocks. And I told them, well, this is, this is the control versus the control, right? And making no changes, and you’re seeing a 20% lift. And you could see the gears turning, they’re like, Huh, okay, I think I get what you’re saying. And I said, this is really bad for us to peek at these results, because it takes a while for this test to normalize. If you make these decisions, early in the process, you’re pushing out false positives and false negatives. And you’re just, this is breaking every statistical model ever. And they’re like, Okay, I think we understand. So I said, Give me a chance to prove this out, let’s let this test run for two weeks, or whatever, I think was about two weeks. So reach the traffic, they needed to hit the statistical numbers, but it ran for the two weeks. And then we saw that there was like a minus, like point 5%, lift with some super low statistical threshold number there. And they’re like, Okay, it looks like it did normalize down to a normal number. And I said, that’s why we can’t just make these decisions. So super snarky way of doing it. But
I think eight tests are the reason I have trust issues. Don’t know if I have the guts to pull that off. But I should I should try to do that when they. Okay, let’s see. So zero. Where do you think you’ll be in 50? years?
That’s a great question. I actually did write about this maybe like a month ago, with another beam attached to it as well, but
turned over and see that or that didn’t inspire this question.
But yeah, I feel like CRM is gonna continue along the automation path, I know that there are a couple of CRM vendors that are testing tool vendors that are looking into automation and kind of creating more intelligent ways to test in an automated fashion. But I definitely think that there’s going to be a level of automation. But I definitely think there’s going to be a human hybrid, or human machine hybrid, where humans are going to have to oversee the tests, they’re going to have to oversee the execution. But the testing tool is going to be able to more intelligently help infer and take you on like tests and iterations of those tests. And also, I feel like the statistics between these and the testing tools are is going to get so advanced, you’re going to be able to run like maybe 60 or 70 variations of like a single experiment. And it’ll, it’ll tell you like the winning variation, much more quicker, versus like having to run like five variations Now, something like that. I think the statistics are really gonna, you know, I’m sure Georgie, Georgie is going to be the person who’s going to spearhead that, but there’s definitely going to be a layer of statistics behind that where you’re going to be able to run more variations quicker and get to learnings quicker using kind of an automation approach using the testing tools.
So one, I wonder if he’s listening, and too, it sounds like the future of Skynet for Surely, you know,
I wonder if that reference data may, by the way, but
you know, when you let the computers make all the decisions, yes, I do get interesting results. Anyway, so this brings me to the section that I call in the news.
Recently, I was digging around the interwebs. And I found a couple of links I wanted to talk to you about so I saw articles talking about Amazon’s product page optimization and its best practices. And I also found one in Forbes that claims it has the top the top, the top five e commerce convert conversion optimization tricks. Now, best practices, tricks, that are lovely words that get a lot of clicks. I want to get your thoughts on best practices and quote unquote, tips. Are they a good thing or a bad thing? do go with them, do you not?
What’s your take?
Yeah, as a sidebar. I’ve vented about this so many times. But I absolutely hate the word best practice when it comes to CR O. It really grinds my gears because I think the assumption around best practice is Oh, this is a thing that we should just do. It’s a best practice. It’s vetted. So let’s just do it. Let’s roll it out. Let’s not test it. And it’s man that makes me cringe because I’ve tested a number of quote unquote best practices that by its definition, I should have just done without testing it and they’ve lost in pretty significant fashion. Although to be fair, I’ve had a couple of quote unquote best practice. That one.
So those things don’t pay. When you’re like, this is this, this is not going to work. And it ends up getting like 5% 10% lift and you’re like,
well, that’s one of the things about being a CEO or right is you have to there’s a layer of hub built hability is that a word? Whatever. You have to be super humble in the way that you approach it. And I guess there’s a objectivity to because it’s not just man, I really like this test. I want it to win. Although I’d be lying if as we run tests, we don’t always secretly root for certain tests over
Oh, I’m, I’m always rooting. So like, when I work with folks, I’m like, very calm. And let’s learn from this experience. But deep down, I’m like, come on, you’re the son of a god.
Yeah, hundred percent. But obviously, as CRL was, when we look at the data, we take an objective view, because we want the best for the business. We’re not looking at something to just lose for the sake of losing or winning for the sake of winning, because we want to prove a co worker on however much we want. But
yeah, we look very common.
But yeah, I mean, we look at it objectively, right, we want to look at the data and and find the right answer the data. I forget what tangent I came on to get to this point. But always best practice. Yeah, okay. So So generally speaking, just testing these best practices outright, just test them Don’t, don’t just assume, because Amazon’s doing it, because Walmart’s doing it and they’re a big company, and they make a lot of money. Don’t just take what they’re doing and do it, test it. And just because I tell you something’s in best practice, which I will never do. But just because they tell you something as a best practice, don’t just assume it’s going to work test it. Yeah, it might be worthwhile for if you have a lot of data where other people are like, Man, this really works. That might give you an opportunity to prioritize it in your framework. Sure, but test it, don’t just do it. Yeah, I think the article you shared to me is another one of those examples coming to that point where a lot of the things on there definitely makes sense. But to that point, are these my question would be are these things that are Amazon best practices? Exactly? specific template? Or are there other? Or are these things that you could take and apply on your site? Because it makes sense,
right? It’s like the idea. Makes sense. But the exact execution in a different environment may look may have to look different to succeed.
Yeah, I mean, the other thing I wanted to call out was, as looking at these articles, man, there’s some of these things that they say it’s a best practice to, let’s say, Campbell case, your product titles, but there’s a lot of branded sites where they want to do weird, wonky things, and they don’t ever use like, a capital letter on their brand, because there’s some super fancy brand. And that drives me crazy,
by the way, it’s like, how do I start a sentence with this.
But like, if that’s part of your brand strategy, then I, I, there’s some things that aren’t worth testing. And like, if that’s part of your brand strategy to be all lowercase, because that’s what you’re trying to push out or you’re trying to push with that. But, you know, if you’re trying to push that, then that’s not something you would apply as a best practice on your site, because the brand team would slap you over the face by doing.
So it’s that time of the show again, I think you’ve done this in the, in our written interview before. So you’re very familiar with the lightning round. It’s gonna be very fast answers. You won’t try it. Try your best not to overthink these things is like gut reaction type thing. So I’m gonna throw these out at you. You may have seen this before. But it’ll be fun in person. So Bayesian or frequentist. Patient your favorite car? Oh, mean. Maybe that’s hard to explain, but do your best.
Honestly, I don’t have a favorite. And that’ll be my answer. Because I like everything that I post.
They’re all they’re all perfection. They’re all the best. They’re all my babies. Shifting gears randomly to those who don’t know, to the hockey the NHL. So that the season is actually I don’t know when the season starting. It’s supposed to start in January, but that’s unsure. exact date. So last I heard last I remember is your favorite teams Rangers. I
don’t that’s still true. It is always going to be true.
It is always to be true. Okay. Good to know. Good to know. How will they do the season?
A middle middle of the road? Okay. Okay. Yes, we have some good players. We have the first overall pick. We got the bread man. I’m feeling really good about the season, but also, we lost Henrik Lundqvist and yeah, I’m never gonna get used to him in a caps jersey.
Yeah, like, I don’t know how to the metro commands are going to do I never know how they’re gonna do, but it may look good doing it though.
Have you seen the retro jerseys?
Yeah. I think the Pretty sweet if you didn’t know they kind of remind me a little bit of the Rangers jersey. Same color scheme.
Yeah. Like, yeah, exactly.
It’s like, hey, these are cool women. I know that. Hey. It’s uh, I like that. Hold me grab one myself. What did you think of the last season’s best format?
That’s a good question. You know, it’s interesting because I want to give that team a ton of credit. The team is in the NHL, because they ran the whole freaking playoffs without any positive covid tests. Like that’s absurd to do that in a whole bubble strategy. That’s it. I anyway, from an infrastructure wise, I can appreciate everything that went into it. But Loki, I did not like the fake crowd noise that they pumped into the arena. I
know it felt so. I don’t know. I don’t want to get off on a huge tangent here. But I was watching. I was trying to get into it. And something felt just so wrong. And I don’t know what it is. But it was hard for me to watch.
It was strange. That was really strange.
And to close, how about you describe yourself in five words or less?
I think I’ve answered this before, but I’ll try it again. Hilarious, weird optimization wizard is probably I reuse a lot of it.
I love that. That’s a, that’s great.
I’m humbled to write
those six word.
Okay, so, you know, thank you for coming on the show. It’s always a pleasure to chat. And it’s been great to talk to you live. Is there anything you’d like to plug as we end today’s show?
Sure. If you guys are interested in seeing some more CRL names, I’d love for you guys to follow me on LinkedIn. I love the conversation. I Oh, sometimes we’ll post things that I’d like to encourage some more conversation around and people give their thoughts. So always feel free to jump in with your thoughts to mesirow means and Yeah.
Awesome. I encourage everyone to follow because I’ve been following Shiva for quite some time now. And I always get a hoot from all the conversations. He sparks so yeah. Thanks for coming on. And
yeah, that’s that’s it for the show.
Thank you very much.
And I guess that’s it.
Hope to see you soon.
(Transcribed by https://otter.ai)
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.