Product Experimentation: Aligning the need to deliver and the need to learn with OKRs Experiment Nation

Home / Product Experimentation / Product Experimentation: Aligning the need to deliver and the need to learn with OKRs

Podcast transcript

Rommil Santiago 0:01
From Experiment Nation, I am Rommil. And this is Product Experimentation. Product Managers everywhere are leveraging the power of experimentation to solve customer problems and build better products. Learn from seasoned PMs and find out how to up your product management game with the latest experimentation strategies.

Jaya Gupta 0:28
Hey, welcome, everybody. Welcome to another session of Product Experimentation with Experiment Nation. I’m your host, Jaya. And with me is

Siddharth Taneja 0:36
Hi everyone this is Sid.

Jaya Gupta 0:38
So today in this session, for those of you who haven’t listened to our first episode, do tune in, our last session was talking about the product development lifecycle. We talked a little bit about how to get started on an experiment within this cycle. And today, we’re gonna actually dive into once you’ve made the decision of kind of getting into experimentation. What might a good product experiment look like? What’s not a good experiment? And why aren’t more companies adopting the practice? We may not be able to answer but it’s definitely a hot topic. in all areas, I think you’ll agree right, Sid?

Siddharth Taneja 1:16
100% percent, I already feel, you know, a little tinge going back my spine, because this is gonna be a really tough topic to look at.

Jaya Gupta 1:26
So I thought maybe why don’t we start with examples of good experiments we’ve seen maybe not have conducted, but when we’ve seen Do you want to kick it off?

Siddharth Taneja 1:36
Sure. Yeah. This is a really tough one, you know, because once you start experimenting with things and features that you’re trying to build, is sort of like an addiction, you know, at least personally, for me. I constantly feel the need to keep experimenting, keep pushing the boundaries of what I’m building. And I think it’s something that you need to really control as well. For me, I think what makes a good product experiment would probably the first thing that comes into mind is this, this broad experiment should focus on your Northstar metric, right? If it’s not focusing on your Northstar metric, I feel that experiment is not really of any value. For example, assume that you’re trying to build a restaurant delivery service, right? So a Northstar metric is a metric, which is the one metric that you will focus on. And this will this will really evaluate the success and failure of your product. So you know, in this case of a restaurant delivery service that I’m trying to build, maybe the number of daily meals delivered is my Northstar metric. Right. And, you know, in this case, maybe our determine the number of pageviews affected by the quality of search results, right, and whatever hypotheses are built, if it’s not focused on trying to address this Northstar metric, then ultimately, this, you know, this, this, this metric won’t really matter. What do you think?

Jaya Gupta 3:13
Well, that’s, that’s really I know, you’re really testing me, I think you gave a really great example. It’s very to the tee. And it’s, it’s helpful to have techniques to exactly get to your measurement of where do you want to be and scale it back to what could contribute to that noise to I think that was amazing.

Siddharth Taneja 3:30
Thank you. And I just feel like, the second thing that comes into mind is probably, you know, identifying, and clearly, clearly focusing on the right segmentation. You know, your near future might be having a 30% penetration for a group of users, but also as a 5% 5% penetration in the target user group. This might be insignificant for the hypotheses and can nullify the experiment, right?

“…you don’t have to restrict experimentation to your professional lives, …you can apply it on buying an IPO, buying a house… experimentation is everywhere.” – Siddharth Taneja
Tweet

Jaya Gupta 3:57
Oh, that’s so true. Yeah, I think like, looking at the user groups is really important. When you’re looking at the experimentation design, it’s important to define even things like looking at are you looking at new users? How do you define new users? There’s always this talk about, you know, what’s a daily active user? Or a monthly active user? Are you? Are you able to kind of qualify what makes an active user? You know, how have you confirmed through your experiment that you’re targeting these people? Sometimes questions can be right off the bat, do you do this activity? Or have you done it? Do you do it regularly? It’s really important to get the basics confirmed. So when you’re looking through your experiment design, like you said, not only the metrics but look at what’s part of it, who are you targeting, what are you targeting with? Have you made that experiment, robust enough to actually contribute make a contribution towards your hypothesis, you know, supporting negation whatever the case is, I think there’s a lot of opportunity for starting with myself to be able to create that mindset that experimentation is something we can do and how do we do it? I had a funny story, actually, I thought we’d start me off on my experimentation journey myself. In my role, which I haven’t actually been in long, I think I started, it feels like so long, because the pandemic, I joined October 2019, I snuck away, able to meet my team have face to face conversations they are such an amazing group. And over this year, I thought, okay, it’s a great opportunity. Why don’t I try to invest in an IPO an initial public offering, and I’ve never done anything like that first time kind of investing. For those of you who don’t know me, I’m kind of a product manager in the wealth space. So it’s a learning for me. Right? Right. Yeah, so what I did was I thought I’d try creating an experiment about my first experience investing in an IPO. I wanted to test out whether I could get the basics of the the format, right. So I had Okay, purpose of the experient. I think I said something like to determine whether the number of shares and the timing to order the, to place the order impacts the fulfillment, because it’s, it’s really, there’s a lot of traffic that day, and like, whether you get in or not, is the is the game, right? Right. Yes, okay. hypothesis. I even I would like apparatus in like, you remember those science school they were in? Yeah, I had like, things like that, and then the method. And then I caught myself, I started making mistakes, because I was changing the conditions while I was going through. In the end, where I was like, you know, what, the design itself is failing. And that was my, I went through observation still pursued. My conclusion was that the design of the experiment was unsuccessful. Rather, we did get something out of this. And that was I learned, for those of you who don’t know, in the banking space, if there’s an IPO that’s going to release the next day, make sure you put your order the previous day at four o’clock.

Siddharth Taneja 7:25
You wanna cash on that premium, right.

Jaya Gupta 7:30
Yeah. And, and, to me, this was a great experiment, because I learned that that experiment design wasn’t great. But I learned something out of it. Yeah. So So to me, like, I’m in my very baby steps of getting into a good experiment format. I think not to say there’s, I don’t think there’s a I could be wrong. When I when I say like, what’s not a good experiment? I think they’re what I’m trying to get to is that what areas of improvement can you have? I don’t think that’s there actually can be a bad experiment as long as you’re learning. So you know, things like throwing in the practice of creating a problem statement, just doing that, right? gets you one step closer to learning, throwing in another step of Okay, let’s make a hypothesis. If we’re going to test it, then what do we think’s going to happen? moving in these components, I think, can be really helpful.

Siddharth Taneja 8:28
Yeah, Oh, 100%. And you know, what I think they should do, we just skipped over the first highlight of our session today, that you cannot, you don’t have to restrict experimentation to your professional lives, you can bring it in, you know, you can apply it on buying an IPO, buying a house, you know, for getting into getting a mortgage approved. So experimentation is everywhere.

Jaya Gupta 8:51
Yeah.

Siddharth Taneja 8:52
That’s amazing.

Jaya Gupta 8:53
And you don’t have to consider yourself as a nerd. Although you can call me a nerd if you want. Right, it’s finding passion in in what you do in life, work. And that’s what engages you to make a difference.

Siddharth Taneja 9:08
You really inspired me to be honest. So thank you for that. I think I totally agree with you, you know, in this context, that there’s never really a bad or a wrong experiment. I think there’s a technical aspect that I really like to look at, when I’m framing my experiments. One of the things that I keep in mind is, you know, when you’re building an experiment, we shouldn’t really impact the performance of the product, right? If you’re trying to create a new variant, or if you’re trying to create a new iteration that you want to experiment with, you’re not supposed to decrement or decrease the performance of your existing product. You have to keep that benchmark there or else, what’s the point? You know, like, it’s kind of it’s kind of obvious, like, when I just speak this out, I kind of feel like it’s common sense, but I thought I’ll put it out there because I made mistakes, I created UX designs where I went from a three step process, you know, three click process to a goal to an action, and from three, to five, and I’m like hmm. That doesn’t make sense, you know, like, and you know what, once you once you put this in production, and you involve different people who are trying to build this for you, you don’t want to you don’t want to waste those resources and that time, so it’s important to be careful about the performance. Um, another thing that comes into mind loose is the technical debt. And this is a little more, you know, this goes a little more towards a code base, you know, experimentation typically involves inserting conditionals, into your code base, right, this can result in a lot of technical debt to actually mitigate this team’s, you know, can really implement some kind of cleanup policy for the code. And this, this is, this is something that I kind of realized, when I was working with my developers. You know, we were, we were working on this new web framework for our product. And I want to try out different UI elements. And what eventually happened was, the code base got so inflated, because we were trying so many things that it was hard to keep a track. And we had to set some new policies in place where we were using a lot of comments, a lot of different functions to try and keep out the experimentation part aside, and have like, you know, a basic structure or the basic version of the product.

Jaya Gupta 11:42
You know, and this is what I love about experimentation, and you’re raising so many good points here, I don’t know where to start. Like, the dev aspect of it, needs to be continuously enhancing continuously, you know, refined, especially when you got tech debt, because that can be preventing enhancements, right? in parallel, though, you want to be learning and you don’t necessarily want to be bogging down your dev team to support experimentation, right? You you be blessed with a team who can prototype in dev cycle, but it’s not like full implementation. It’s like a concept, right? You also might be blessed with, you know, a design team. And when I say design team, it’s like, people who have a high level of intelligence and appreciating not only the aesthetics, but almost the experience engineering, is that even a concept like it’s, it’s the blueprints of how people think, where they anticipate they’ll be taken to if they click on a page, what the structure the informational hierarchy is, and, and I think the leaner, you, you think about experimentation, the better it is, because you don’t have to kind of introduce complex processes to incorporate that. But like you said, You need to be able to keep learning and keep developing in parallel. So here’s the here’s the thought, like, I think we’ve we’ve talked about good product experiments, and maybe not, like, not good product experiments, but like areas of opportunity. This is the thing, I think, very commonly in my experience. And you tell me, if you share this, too, you’ll have fight the battle, it’s almost like you’ve got two races in parallel going on. Let’s say, that’s a better analogy to hunters side by side, one that’s trying to progress with development, one that’s trying to progress with learning. But then the, you know, the person who’s running the race is saying, we need to get to the finish line so that we can get the prize. How do you actually incorporate the learning runner to have a fair game? That’s the that’s the question.

Siddharth Taneja 13:53
Oh, wow, that’s, that is such a great question. And I don’t know if there’s a straight answer for this, you know, because it’s varies so much. On a lot of the times, on teams, and senior management, they don’t really understand the need of experimentation. Right there. They feel like whatever has been decided, or whatever has been the norm that is set in stone. I had this challenge a few times where, you know, you couldn’t really put that put a new set of features across on the table, right. And personally, what really helps me is to have the OKRs right in front of me. And I use the OKRs and the different priorities set by different stakeholders, as a launchpad to building out new experiments, if that makes sense.

Jaya Gupta 14:46
All right, like getting in. So So sorry to interrupt. I just wanted to say with you objectives and key results, I think that’s the common language that can bring others the table right?

Siddharth Taneja 14:59
Absolutely. 100% like, if I, if I can imagine myself walking into, you know, a senior management room and just saying, Hey, I had this feature that I really want to build, and it’s not gonna happen. But like, I mean, I almost want to like, you know, put some kind of VR headset on my head and show them, listen, this is what I’m imagining, you know, and just project them on their screens, hey, listen, this is what I’m imagining, this is the vision, but I don’t think that’s how it works, you know, but the OKR tech terminologies really helps me like open doors with my stakeholders. And I also like, you know, when I kind of got used to the velocity of my dev team, I also tried to set aside some time post every sprint to just, you know, brainstorm ideas. I felt like, you know, just having like, a one hour two hour session in a month to just brainstorm ideas on the current state of the product, really help with the experimentation, you know, I feel like, like all the other artifacts you have in an agile process, which are living, I feel experimentation is also a living artifact of some sorts. You know, that’s, that’s the way I look at it. It’s a constant constantly moving and living process. You know, does that does that make sense?

Jaya Gupta 16:29
I think so. You know, it’s interesting, because I don’t think I’ve ever seen or witnessed like a repository of experiments, not to say that if you’ve done it before, then this is going to be the result thereafter. But kind of understanding the context around, you know, how did you run the experiment? What was the idea involving people, and then as you evolve as time evolves, situations evolve, not only do you get better at experiments, but you’re also learning at ways to incorporate more perspectives. I guess going back to that point around, essentially, how to balance experimentation with development efforts. I think you’ve you’ve kind of nailed it, it’s having starting the conversation with Why should I care? Right, the OKRs, the objectives and key results. I mean, if you use key performance indicators, they’re complimentary. So don’t think that they compete with each other. Objectives and key results, the way I’ve seen it are a little bit broader, actually much more broader key performance indicators, as I understand it, are ways that you can measure key results. And, and to bring that all together, we can really have real conversations with leaders on the value of experimentation. The more we get comfortable with sharing results, like let’s say we do something off the side of our desk, we run an experiment. And, you know, nobody, but the team knows, well, that’s kind of unfair, because our leaders and managers then won’t know that we actually got to value because of the experiment, right? And the more and more they see it, the better they’ll see the the role for experimentation. I think that kind of gets to the point of why aren’t more companies adopting the practice, because it’s just the awareness, just like the design, practice, product practice. These are all knowledge areas to learn more, right? If we don’t know the role, if we are not specialists in the role that how would we know? Right?

Siddharth Taneja 18:32
Yeah, yeah, that’s a really good point. I think we just makes me wonder now, when you just said that maybe it’s also about the leadership and the culture, right? Maybe the maybe the leadership of your team, or is, is not really open to experimentation, right. Sometimes I also feel maybe it’s, this is something that has to be top to bottom, right. There’s only so much you can do from when you go from bottom to top, I think it’s also about enabling your, your own team to, you know, not be averse to failures, and just be willing to think out of the box.

Jaya Gupta 19:13
Yeah, yeah, I totally agree. The more we we can align with each other and share our thoughts, the more we’ll think, at least for the purpose of why we’re doing things we’ll think similarly. And that’s when we’ll all feel like we can bring something to the table. I think you know what, this is a really great conversation. We’ve we’ve touched on each of the topics really nicely, with examples of what makes a good product experiment, even the components of it, some funny stories about how we’ve incorporated into our personal lives, how we see it playing out at work, and then areas of improvement if it’s not good product experiments, then what are the areas that we can actually start to incorporate experimentation with hopefully, we And start more conversations with our, our leaders and senior managers as product professionals because there’s a role there. But honestly, I want to share this responsibility with my peers. You know, their practices like design, experimentation, research, marketing, if we continue to create awareness, I think you know what will go a long way where we can all be learners and get better at delivering products that people love that works for the business. With that, I want to wish you guys a happy day. And yeah, hopfully we connect on lots of great topics in the future. Sid, any last comments?

Siddharth Taneja 20:41
Oh, this was this was great. I feel. This opens a lot of new conversations. And I love that personal touch of experimentation.

Jaya Gupta 20:50
Yeah, for sure. All right. Goodbye, guys.

Siddharth Taneja 20:53
Goodbye.

Transcribed by https://otter.ai

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.