Product Experimentation: Episode 1
By Jaya Gupta, Siddharth Taneja, and Rommil Santiago
Published on: January 23, 2021
Jaya and Siddharth explore how Experimentation fits into the Product Development Life Cycle, how to work with challenging stakeholders, and share stories about how they’ve leveraged Experiments to generate results.
The following is an auto-generated transcript of the podcast by https://otter.ai with very light manual editing. It’s mostly correct but listening to the actual podcast would be wildly more understandable.
Rommil Santiago 0:01
From Experiment Nation, I am Rommil. And this is Product Experimentation. Product Managers everywhere are leveraging the power of experimentation to solve customer problems and build better products. Learn from seasoned PMs and find out how to up your product management game with the latest experimentation strategies.
Siddharth Taneja 0:28
Alright, welcome everyone. I’m Siddharth and I am a data science professional, and Product Manager. I have a broad industry experience. And I’ve shipped products in legal services, and manufacturing industries. I like to leverage data to make informed decisions. And I leverage data to create value throughout the product development lifecycle. passing it over to you, Jaya.
Jaya Gupta 0:52
Thanks, Siddarth, along the same lines as you I’m a Product professional with over 15 years of experience in the FinTech industry, I guess you’d call me a specialist. I’ve been working towards delivering digital solutions that people love, and that works for the business. I kind of like to say that that sounds familiar. And if it’s not, try to familiarize yourself with Marty Cagan. I think that’s the one definition that’s resonated with me well.
Siddharth Taneja 1:19
Alright. Um, so for our listeners today, this first session, we we will, we will walk you through just what a product development lifecycle looks like. We know PDLC, you know, varies from a product manager to a product manager, and varies from company to company. So we’ll we’ll talk about the different flavors of the general PDLC. From there, we’ll we’ll walk into how can someone start with experimentation in a PDLC. And we’ll try and share our experience with the whole process. And for this episode, we’ll also close on with how can PMs, designers, engineers, and just your team can incorporate experimentation in your day to day work? So to start us off, Jaya, your experience, what do you think is a typical product development lifecycle like?
Jaya Gupta 2:13
Yeah, I think so depending on whether you’re kind of adopting a more iterative development process, or you’re kind of being handed down a process based on a concept. Generally, there’s some form of strategy and some form of delivery that complements each other, of course, when before the other, but depending on where you’re at in the cycle, things can keep repeating, until you get to product deliverables that you can measure, right? So essentially, what you’ve got is at the beginning, there’s either some form of research idea where you’re trying to discover or validate, you know, whether there’s any value to pursue this idea, this can be followed up with research, or support, whether it’s from white papers from customer research from competitive intel, and the product managers basically got that responsibility to synthesize all this information and make some sense out of it. Right? How often are you bombarded with so much information? And you have to tell a story? Yeah, like, why are you doing this? What’s the problem that you’re trying to solve? But it goes a long way to bring this all together in a cohesive way. To answer those important questions. Right, who are you targeting? What is it that you’re trying to address? What’s the opportunity? And some of the nitty gritty details like what are the benefits? Can you quantify that? Once you’ve got some substance into this idea that you want to pursue, you bring in experts of different skill sets, be it from the development side, from the design side, from the product side by side, and you collaborate to a point where you try to figure out, o kay, now that we’ve got something, how might we be able to get this done? Then you’ve kind of got if you’re in the softer space, then the implications could be around application solution architecture, you know, connections and integrations between systems. In other cases, it might actually be processing improvements, and they don’t necessarily depend on software. So it’s just the idea, the concept around developing idea how can we get to it? Ideally, then, when you get to a point where you can qualify whether it’s going to work or not, you’re testing something, whether it’s functionality, whether it’s the development progress, to make sure that you’re on the right track. And then you know, there’s some form of feedback that you want to get to, to see whether it’s working or whether it’s not. Here’s where we kind of test our conscious where we’re, you know, we’ve got, see enough to say, yes, it worked or no, it didn’t. You know, often in traditional systems, you might not be able to track back and say this didn’t work. Let’s start again. Yeah, I think nowadays, we’re much more open to this concept of learning, failing, learning the iterating, and then passing on, you know, an idea that could become something bigger. Does that, right? In a nutshell, from start to finish, the basic product development lifecycle, synthesizes I think, in two different industries, I’d probably love to hear what you’ve seen, the product development lifecycle turn into and maybe even in your world,
Siddharth Taneja 5:47
Right, yeah. I feel I totally resonate with you on the Discovery aspect, right. While working with legal services, as well, I felt that the discovery aspect is, is pretty consistent right across different industries, you either have a problem that you’re trying to solve, or you’re trying to improve a situation. This can be brought forth towards you, either by an internal stakeholder or an external stakeholder. That is it might be bring, it might be brought to you by someone in the management, or you figure out a problem that one of your users are facing. But what’s interesting is the ideation part, right, which follows after you’ve identified the problem that you’re trying to solve. I feel like from my experience, in different industries, the ideation or brainstorming phase is different. I worked with teams, which were really siloed. And for them, the ideation involved more of stakeholder management and less of brainstorming and whiteboarding. You know, in this situation, I really had to bring different people together and understand their limitations. But I’ve also worked with teams which are absolutely flat, you know, and that is where the actual creativity bumps in. And everyone is literally given one marker each and there’s a big whiteboard, where you can go out and lay out your ideas, right? And I feel that’s the kind of ideation I personally enjoy. Um, have you ever felt a scenario where, you know, the ideation aspect was really itchy?
Jaya Gupta 7:20
Yeah, it’s, it’s really interesting with the dynamics, sometimes it’s facilitated by external parties where you’re unbiased towards any party, the less structure I found, the better it actually turned out. When I say less structure, I don’t mean, how this session is being held, but rather, in terms of how you’re thinking, like, what, how can you kind of necessarily tell somebody how to think right, it’s, that’s your, that’s your creative juices that’s kind of flowing. But if you kind of focus on the constraints, or the context, why you’re here, the basics around the session, how it’s going to work, if you’re going to dot voting, or if you’re going to walk through a scenario, if you’re going to paint out a picture and literally envision what could the experience look like? You really put the participants around the table, you know, in their own free imaginary world where you see so many different perspectives, I think that’s where it ends up being successful versus on the flip side, if you’ve got, you know, more of a narrowed focus of, you know, this is the idea, now, how can we get it done? There’s so many imaginative side of that, but the creativity is kind of killed right at the onset, because you’re given the idea.
Siddharth Taneja 8:41
Yeah, I think you’re absolutely right about that. Um, my personal favorite is the dot, dot approach, where, you know, it just offers a really a lot of ambiguity to the people responding to different ideas. And I felt, you know, when people are asked to throw in their suggestions and their opinions, that is when the this practice really comes in. But I think the key thing here is, how can someone get started with the product experimentation throughout the whole product development lifecycle? In your experience so far, Jaya, how have you managed to experiment with the products that you build?
Jaya Gupta 9:22
I know, I’ll be really honest here. It’s I’ve been really influenced by my design partners constantly, where, you know, UX research has become such a huge space, you know, in recent years, but it’s it’s existed for quite some time. And just like product management somehow now it’s just boomed to be like a very interesting role, but it’s existed for quite some time. It’s just a matter of realizing really the value that you get out of all of the knowledge that we come together with, right so not only do we have the opportunities to run with experiments within user research, and by that I mean, any form of user testing, whether it’s interviews, whether it’s surveys, usability tests. But also, I almost want to ask a question, because I’m not certain. I was having this thought that even in the development phase when there’s like tech spikes, right, if you wanted to discover a way to do something, and you’re not sure how it might not follow a scientific method, but I wonder if there’s an opportunity to do do that, right.
Siddharth Taneja 10:31
Yeah, that’s a really good question. Um, I feel it depends a lot on the structure of the team. And the current alignment of resources, right? I feel there, there could be scenarios where you could create some variants for the control feature that you’re trying to map out, right? And you could try it out with a bunch of target users and see how they feel about it. But I feel in other scenarios, maybe you don’t have that bandwidth, other resources that can create this new, this new set of experiment for you. And what do you do then? Right? Do you like, just go in, you know, experiment and just try it out on a live product? Or do you hold back this creative idea until you get to a point where you can actually test it out?
Jaya Gupta 11:24
I almost want to think that there’s a mythbuster around here, where, you know, we almost want to be interested in what is part of an experiment? What do you really mean to run an experiment? And kind of from my really narrow and small perspectives on things, you need an experiment, design, problem statement or hypothesis, you need to be able to be ready to make observations. And then make that conclusion to say whether your hypothesis was supported or not, and take the next step to progress on that conclusion. Does that require a lot of resource or a lot of extra work and tools and stuff? I think depending on what you’re trying to achieve, you might require more advanced tools, tooling, but on the onset of it, just to even think about experiments as a way of working through a problem space. I don’t think you need a lot.
Siddharth Taneja 12:34
I want a fence here because, you know, I’m, I’ve been on the other side of you. And my last role, I did not have a UX researcher in my team or in the organization itself. Right. So everything was on me as a product manager. So for me, I had to put in a lot of extra effort to scope out the best practices for UX for a digital product that I was building, I had to enroll, believe it or not, on a part time course, for UX design. And I was like, Hey, I see a gap here. And that cannot be fulfilled in my organization, and I really need it. So I ended up, you know, investing some time on understanding those principles. And I really felt empowered, you know, as a, as a PM, I felt really connected with that product once I actually got to understand the UX aspect of building a product. And with that, I felt, that I kind of had a framework when I was building out my experiments, so I was trapped with the question that that was that that I was trying to answer. For example, how can we decrease the bounce rate on the different states of the product or different screens of the product? From there, I would go on to building a few, you know, set of hypotheses to answer this question, for instance, in this case, if we change the call to action button to red, more people will move to the next page. Right? And then I would go on to creating different variants, right, and identifying the target users to release these variants to. And finally, I would get some feedback and data to prove my hypothesis, either valid or invalid, you know?
Jaya Gupta 14:24
That’s an amazing experience.
Siddharth Taneja 14:26
Well, it’s a quite a lot of effort to be honest.
Jaya Gupta 14:30
I can see I can see how that could be very time consuming, especially if it’s just you, on top of kind of data science responsibilities, appreciating that as a product manager is a lot because here’s the thing as product professionals, we’re talking to so many different people in so many different areas. Not to say that other roles aren’t, but there’s a lot of hats switching defined like you know, when you need to think about working with your marketing partners, it’s, it’s kind of looking at, you know, through the sales funnel or the e-commerce platform strategy. What are you trying to support? In that angle? When you talk to your design partners? It’s it’s not only kind of the end goal, but what are you trying to achieve, achieve from a user experience on the digital platforms? There’s always like a slight, there’s either a drastic or a slight different kind of thought process, depending on who you’re talking to me because they’re different skill sets.
Siddharth Taneja 15:30
100%. Yeah. And in your experience you know, while we are still, while we’re still on this topic, what was the most difficult team or set of stakeholders that you had to work with?
Jaya Gupta 15:47
This one’s a tough one. When you say difficult, like, in terms of working through discussions, or hard-headed people?
Siddharth Taneja 15:57
You know, I would say with hard-headed people, right? Like some people that you that you had the hardest time connecting with and bringing onboard in your experimentation projects.
Jaya Gupta 16:07
And this one’s interesting, I think. So the first thing to do, I guess, is to appreciate why is this person appearing to be hard headed? Why am I thinking that this person is hard headed? It starts with me, right? I think, first going into these sessions, I can think of an example where I think we were trying to change a pricing strategy. This is when I was kind of assuming the role of a traditional Product Manager. And it was so complex, because it wasn’t just changing the pricing strategy or upgrading plans, it was also incorporating a cross-selling opportunity with another package of products, you know, pulling me into the digital space, I couldn’t actually leave it. I think at that time, it was it was me appreciating the fact that we started with different goals and objectives from from both sides. And I think working acknowledging that only came in, embarrassingly, later on, we started with, okay, here’s our package, here’s how, like, we want to incorporate it. But we didn’t really start with what is it that we’re all trying to achieve? By the time it was, you know, shambles, and we were just trying to clean things up, because it was, you know, we’re also tied down with risk managing communications, and legally, you’re bound to communicate each account per person. We had 10 accounts, you were sending 10 letters, and that’s pretty much not changing in the Canadian Space. It was it was streamlining on communications. What we had tried to do was iterate on different versions of letters, actually send these sample letters to not only employees, but also customers to see how they feel and customer support groups. Oh, well. It was really risky. We, you know, we were like, Oh my god, what if this blows up and stuff, but we tried, we trusted each other and said, you know, what, the best way to know is to find out how this could blow up and what questions we could get? Yeah. And so we ended up getting really, really good feedback, including feedback from risk partners, from customer experience folks, from customers. At the end of the day, you know, we were on the fence with changing certain things, because either way, change, especially in the pricing side is sensitive, right? That’s right. So you, you have to kind of appreciate and accept that. Some things you can’t change, but you can optimize. And I think that’s the that’s a benefit of experimentation. If you’re not looking to perfect it, you’re definitely looking to optimize to improve, you know, the next best thing. Yeah, I strongly think that we achieved that.
Siddharth Taneja 18:55
Yeah. Oh, I think that’s a great point. I really love the fact that you said, you know, experimentation, and the change really starts with oneself. Right? And until well is, you know, you’re open to change, and, you know, willing to try and see if any new thing works on what you’re building. You know, you can’t really expect anyone else to be on boarded on board with your idea, right? So really love that fact. And having worked with the pricing team, as well, I thoroughly know where you’re coming from, there, there’s only so many things you can move and change. But it’s really interesting to see that you could really look at experimentation as a way to optimize something, right? It’s not just trying to come up with something new, but you’re just trying to optimize an existing process.
Jaya Gupta 19:47
And I noticed like I want to add like one more piece at more, the more you actually inspire people with the same fail and see focus, the more we were able to align to each other. aligned with each other, the more People got actually motivated energetic to actually try different things. I think that’s the thing at the beginning, it’s everybody’s on a different page. If you spend time at the onset to set out, you know, what do we all agree to be here for, people will, you know, in generally will form a team and that will be your foundations to creating a really successful experiment design?
Siddharth Taneja 20:24
Yeah, I think it’s, the theme has to come together to actually heal to the result that we’re trying to aim for. Right? You can’t really move all the pieces by yourself. It’s a it’s a machine that’s, that’s working together. This kind of brings me to the to another interesting point, um, you know, how would you, how would you suggest your PMs, designers engineers, whom you’re working with to incorporate experimentation in your work?
Jaya Gupta 20:56
You know, it’s so interesting, I actually had a recent example of this, we’re partnering with another team with regards to a redesign concept. And those are really interesting, right? It’s not like a one and done. I’ve constantly thought about how to go through website redesigns. Is it like a full package of capabilities that you launch all at once? Can you even break it down and get into a Frankenstein? No, you can’t. So backing up, we actually got a lot of initial interviews done in I’d say, like a couple months ago, but they were still I’m trying to get to a practice of talking to users on a weekly basis. I love Teresa Torres for that. That’s my goal for the year trying to like, wow, once a week, it’s tough, I’m gonna do it. And so, you know, we wanted to brush up those interviews and start up again. Before my UX research partners dove into those interviews, I just asked a simple question. I said, you know, what, can we actually create some hypotheses before we get into these interviews? What is it that we think that could be true or not true? And she loved the idea. So she kind of took it away a couple of days, looked at some of our problem statements, what we created as concepts, and came up with a couple of statements. And I added to that from a product perspective. And it’s interesting, because the hypotheses that she wanted to confirm, from a design perspective, were very distinct from the ones that I wanted to support, which is great! It doesn’t have to be a certain discipline, right? Yeah, so just that change, I think, helped us feel comfortable getting into the interviews that after that, when I’m able to see what the synthesis looks like, we’re able to go back to those statements and say, you know, our research supports it negates it. And it shows very cohesively, what your next steps could be, when you’re trying to share that feedback to teams, they appreciate it, they see where you’re going with it, right?
Siddharth Taneja 23:04
Yeah, definitely. I’ve kind of felt the same thing while working with my teams. You know, I’ve always tried to urge my, at least my engineering team to be more creative and think outside the box. Um, one of the things that I felt that my team was having a challenge with was the, the, the apprehension, you know, and, and the averseness to fail, you know, people don’t like to fail. And, you know, failure is a part of experimentation, right? If you fail, you only get to know what does not work, right. So it was really hard to communicate, you know, this ideology, or this way of thinking to my team, and, you know, we’ve had so many sessions, you know, I actually had to take them out outside the office, you know, for a social to try and get them a little more comfortable with, you know, it’s alright, to fail. But let’s try out new things, right. There are tons of ideas available out in the market, right and out in the world to solve one problem. For instance, you know, if someone is looking for designs, design ideas, or design inspirations, they can refer to dribbble, or behance, some open platforms where people come out and share ideas for technical people for technical ideas. You could refer to stack overflows, GitHub, to actually look at how people are solving the same problem in different ways. And I feel, you know, it’s always about trying to just be a little more creative. And taking the chance.
Jaya Gupta 24:42
Oh, I love that. Take a chance. Right, start with me. Take a chance. Be ready for failure. These are really nice messages that that have kind of come across multiple times, ample times. You know, throughout my career. I love those.
Siddharth Taneja 24:57
Yeah. And just to wrap it up today, we spoke about, you know what a product development lifecycle really looks like. We spoke about the different flavors in product development lifecycle. From there, we moved on to discussing and sharing how someone could start experimentation in product development lifecycle. And we spoke about how the different industries vary in terms of experimentation. And we also spoke about a framework that that could be applied for starting experimentation right away. And lastly, we also spoke about how you as a product manager or product leader can enable your team to incorporate experimentation into their work. I’d like to thank all of our listeners for tuning in and we look forward to having you for the next session. This is Jaya and Sid
Jaya Gupta 25:56
Thank’s Sid! This is a really great first a session, I had lots of fun. Definitely tune in to us another time. And goodbye for now.
Siddharth Taneja 26:04
Signing off. Thank you.
(Transcribed by https://otter.ai)
About Experiment Nation
Experiment Nation connects Experimenters from around the world to new ideas – and to each other.
We send a newsletter every week.
- Adventures in Experimentation (9)
- Analysis (2)
- Announcements (48)
- Ask the Experiment Nation (11)
- Certified CROs (193)
- Community (2)
- Conversation Starters (2)
- CR-No (5)
- CRO Handbook (4)
- CRO Memes (18)
- Experiment Nation: The Conference RELOADED (41)
- Experimentation from around the world (2)
- For LinkedIn (121)
- Frameworks (2)
- Growth (14)
- ICYMI (2)
- Interviews with Experimenters (167)
- Management (1)
- Marketing (11)
- Opinion (8)
- Podcasts with Experimenters (15)
- Point/Counterpoint (1)
- Product Experimentation (5)
- Product Management (9)
- Profile (9)
- Question of the week (5)
- Sponsored by Experiment Nation (1)
- The Craft of Experimentation (1)
- The Cultures of Experimentation (1)
- The weekly buzz (13)
- The year in review (1)
- Uncategorized (352)
- Weekly Newsletter (183)
- Where I Started (2)
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.
- Making the case for data-backed design featuring Brian Massey - September 30, 2023
- Announcing the 2023 Conference Awards - September 24, 2023
- How to Maximize Hypotheses Testing ft. Eduardo Marconi Pinheiro Lima - September 9, 2023