Navigating Experimentation at Startups with Avishek Basu Mallick

AI-Generated Summary

Avishek Basu Mallick, a Senior PM at CheapOair talks about: – Experimentation at scale: Even with high traffic, rigorous experimentation is crucial. – Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas. – Communicating progress: Regular updates with leadership build trust and ensure alignment. – Personalization is nuanced: Distinguish between customization, segmentation, and true personalization. – Generative AI’s potential: Explore its use for simplifying the travel planning experience.

Audio

AI-Generated Transcript

(00:00) what I follow is I always so there’s a there’s a regular update on everything I work on on a weekly basis to higher up so they are aware of what I’m working on and where I’m blocked at any given point of time which happens fairly frequently because we are in an industry which is extremely highly regulated we are very

(00:15) high traffic so a little here and there could mean a lot for us so that’s the only way you know you can actually be very transparent and get things done is by communicating probably over communicating if required at [Music] times hi everybody my name is Madara I’m a senior optimization specialist with

(00:39) the upbound group and I’m your host for this episode of The Experiment Nation Podcast joining me today is abishek Basu Malik who is going to introduce himself here thanks M everyone my name is aishik um I’m a senior pm at chipo a prior to this I’ve been in the product space across multiple companies around in at

(01:00) Tech in retail currently in travel so yeah that that’s pretty much about me yeah absolutely I I want to touch on that a little bit some of your background and work experience you’ve been kind of all over the place in terms of industries that you’ve been in let’s let’s talk a little bit about what your

(01:16) current role is and how you work within the the travel industry and then kind of how all your previous industry experience from all the various different sectors you’ve been in how does that play into how you approach experiment mation in your current role sure So currently I manage user Journey at CheapOair so typically Cheo is the

(01:34) third largest OTA in North America and uh unlike xedia or Price Line you know our primary focus is the flight space so I manag the uh entire user Journey for the flight space which is from the search results to booking confirmation how does that sort of tie in what with what I’ve done in the past um so if you

(01:55) see all of my experience has been in the b2c space and U there’s a term I like to use you know it’s so for PMS I would like to say for b2c PMS something which is very important is something called Funnel Vision And You Know It essentially means you know where the user starts at the start of the funnel and how they go all

(02:13) the way down to you know actually converting which is pretty similar across Industries if you look at it that way so in a retail it will typically start off start with your again search and then funnel all the way down to cart and then you check out similarly edtech so I was in edtech for a while again

(02:29) started off pretty much with a let’s say cold calling prospecting that sort of a space and then funnels all the way down to signups and it’s for travel again it’s a booking right so it’s it’s really like customer Journey based your experience and what’s interesting is I just heard a statistic recently about

(02:46) how in a lot of e-commerce companies are likely to spend for every $92 they spend on traffic acquisition they’re only spending about $1 on traffic conversion so like stickiness all the way through to a transaction so it’s one of those things where does that reflect in your experience of like having to sort of you

(03:06) know there’s there’s a lot of spend on traffic acquisition because it’s very easy to attribute some sort of success to it but the actual experimentation of getting people to convert more it’s a little harder to attribute success there so how have you approached that and and what are your tactics there it’s a very

(03:23) good question you know especially and for me like you know I come from the other side of the world so I used to be in Market digital marketing so I’ve been you know a marketing guy who’s worked on sem campaigns social PPC all of that stuff uh over the years all of it has gone pretty I mean quite expensive and

(03:42) as you said I mean I’m quite surprised of I’m not very surprised to hear the $92 figure I mean I probably thought it would be a little higher than that we spend a lot of money on sem almost 50% of our marketing budgets go into sem but we are also very rigorous in terms of um you know the entire experimentation side

(04:02) of things and I would say we are blessed maybe in a way to have a lot of traffic for us to run experiments and that’s a challenge I see you know I’ve seen previously with you know the startups I worked is it’s a little difficult to run experiments you don’t have scale so and you don’t have time so two things and if

(04:22) you don’t have either of those two so you will probably not bother a lot about experiments you’ll just want to you know go out there you know throw money at whatever SES that is one of those things I was curious about with your sort of uh startup background and experience I I have always assumed startups move a

(04:40) little too fast to sit still long enough to let results mature and conversion rates sort of increase iteratively over time with you know efforts that stack into each other so so where in the startup space was there room for experimentation from your perspective even if you didn’t get to practice it necessarily honestly this is a difficult

(05:00) question because in a startup you run experiments I think experimentation as I said from a b2c products even let’s say for a B2B product manager as well I think experimentation is key now the question is you know the key question here is how long do you run an experiment for so just to give you an example so in my

(05:20) space you know if you’re probably at Expedia you’ll if you run an experiment for 7 days you’ll probably get you know significance to actually you know take your experiment live as a winner or whatever uh for cheaper probably it’s going to be anywhere between two to 3 weeks but let’s say you are somewhere

(05:35) you know you’re like an upstart and you know you you’ve entered the space you would likely have to probably run an experiments for let’s say six months to actually reach significance which would obviously not make sense as a startup so as a startup you’ll probably run an experiment just to get directional

(05:51) information as to where stuff is going and you know then take a call based on that right it’s very exploratory yeah absolutely it’s almost used as like a user research method as much as it is like a like a quantitative science so that’s a very interesting perspective So within your current role at Cheapo Air

(06:09) uh how are things structured what does your like experimentation program look like sure so I am sort of like the leor for anyone who wants to work in the flight space so like we’ve recently introduced hotels and you know we don’t really spend a lot of marketing dollars on promoting hotels so a lot of the

(06:28) hotel traffic has to be funneled from let’s say the flight space so I work with hotels I work with so and we have a you know multi-functional business approach in the sense we sell online we sell through our call center we sell through chat um so all of those teams work with me in terms of having their

(06:45) widgets their placements on the site so that’s how essentially my so I am sort of like you know the guardian the custodian of uh of the user Journey from search results onwards and then I have my own set of experiment which are done which are generally ux based in terms of improvements in terms of let’s say

(07:03) filters in terms of on-site banners in terms of let’s say payment methods and all of those things okay and so then within the actual structure of your org you’re the project manager and then do you have Specialists working on experimentation with you within that or is it sort of a little bit more distributed are you decentralized in

(07:20) that way some places like to centralize their experimentation and other places it’s like you know a product team will have someone who does experimentation mark marketing we’ll have somebody so on and so forth so kind of what reflects your experience more so here product runs its own experiment so as I said

(07:35) we’ve built our own Navy testing platform and at any given point of time You’ have anywhere close to around 10 experiments running again this is across the funnel so when I so this could be right up on the landing page or even on the confirmation page as well so we don’t let users go even after they’ve

(07:51) completed their booking uh we continue to experiment post that as well yeah so I would say we get a lot of our insights from let’s say ux I mean the user research team or marketing but uh the actual experiments are run by the product team themselves yeah that makes a lot of sense that’s one of those

(08:10) things you’re touching on there that I think is is really I think an important conversation that uh the cro space Maybe needs to have of like where do you get ideas right like how are you sourcing the the thought leadership that you need to run a successful experimentation program so it sounds like you take a lot

(08:26) of cues from ux research are there stakeholders within the organization that have ideas that are they know they can come to you specifically to try to enact those ideas absolutely so our leadership you know we’ve been around for what about 20 OD years so you know we’ve got some really you know established and

(08:44) experienced leaders hi this is Romo Santiago from experiment Nation if you’d like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the description now back to the episode who who’ve been in the space for you know like donkeys

(08:57) years and they have things which they know which have worked in their previous orgs and you know they would like you know us to test out but there’s never really a shortage of ideas so of course there there’s a lot of ideas within the org we do user research I I say I mean we probably want to do more than what we

(09:13) are doing right now but there’s still a fair amount of it going on and we are extensive in terms of you know collecting user feedback so a lot of our ideas actually come through our NPA surveys and I would say we never short of ideas that very interesting uh in my own organization we’re currently looking

(09:31) at utilizing some of our NPS VOC collection tools along with some of our user research tools I’ve sort of started this process where at the tale of an experiment after we know that there’s interesting results of some kind whether winning or losing or even inconclusive where we can’t for the life of us figure

(09:48) out why we got those results I’ll deploy per variation a survey to try to collect a little bit of feedback and see if there’s a qualitative answer to quantitative strangeness that I’m seeing so um it’s good to know that other people are are moving in a similar Direction that’s really really cool I am

(10:05) also curious when it comes to having a lot of ideas how do you build your backlog how do you turn that into a road map like what is your process for creating sort of a pro I guess a process how do you what’s your process for process is the question you’ve just asked for my secret sauce there but but generally you know

(10:28) prioritization is always you know it’s probably one thing a PM both loves to do and hates to do I would say um again so the way we structured is obviously you know we have the quarterly planning where we sort of plan out a road map for the next 3 months or so and out of that so we categorize items into so stuff

(10:47) which is going to go into the road map and stuff which is ba business as usual stuff so that would be you know regular changes on the side you know small us changes and stuff like that the road map we would pretty much identify three or four so we’ essentially look at numbers so as I mentioned you know I work with

(11:04) different teams so we have different metric such as you know growing traffic to call Ratio or growing newsletter signups or growing you know funnel movement from listing to payment and we track these metrics extensively you know on a daily basis on a weekly basis so at the start of a quarter we know which

(11:21) metrics we are good at and which metrics we are sort of you know lagging behind and those again serve as directional indicators on items we need to work on you know in that particular quarter that’s really awesome do you guys ever like have to align with some of those ba like okrs or like Strat level goals or

(11:35) anything like that like that come from maybe a higher level of leadership so as an or you have to then also Orient some of your testing in that direction we do so as I said a lot of so I would roughly say you know if you’ve got like 100% uh capacity You’ be spending anywhere close to almost close to let’s say 40 to 50%

(11:57) on ba items you know stuff which are recuring items and then within the rest 50% you’ll have probably 20% Which come you know flow down from the higher level and then 30% is what you manage on your own and from the product life cycle perspective MH within that road map of expected releases is it budgeted to have

(12:18) some amount of variability for whatever test results end up coming out like knowing that there’s a little bit of uncertainty in what may or may not get coded because a test may or may not have results absolutely and you know that’s why we also budget for multiple iterations of tests so and so that’s the

(12:34) approach I think we’ve we’ve learned it the hard way over the years you know that of course you know when we started out we used to be very optimistic in our budgeting used to say like you know maybe we’ll do like so the my entire team would let’s say do about 80 tests in a quarter but I mean we’ve scaled

(12:50) down significantly because we’ve realized that you need to run multiple iterations of tests because through user research qualitative user research you can only know so much you know when when you actually run something you get more insights yeah that’s definitely true and plus it’s like there’s qu like there’s

(13:04) you can have a quantity of of results or you can have quality results like you could generate a bunch of random little wins that don’t really mean anything in the larger picture or so yeah that’s all very very interesting I am also curious uh you’ve talked we talked a little offline about you have some thoughts on

(13:24) sort of testing Within like creating a product net new versus testing within established products is there a difference in process and and thought there absolutely I mean even here for example right now I’m working on a let’s say Green Field project and within the or itself the approach is drastically

(13:44) you know different I mean for an established product the process is always iterative you know I mean so let’s say let’s talk about the flights booking part of it you know it is pretty much a standard product I mean I’m sure there are opportunities for Innovation there but it’s going only going to happen incrementally but let’s say some

(14:01) of my projects I’ve done in the past with decathlon or with osmo and and the one I’m working on right now it’s sort of a green field so you know you are probably not going to see a lot of experimentation right away so the MVP you build out is going to be like a barebones version of the product which

(14:16) you can release as soon as possible get it out to users see what’s happening there good bad ugly whatever it is collect the feedback and then continue to iterate and once you to reach let’s say a certain amount of scale which could be let’s say an X number of users after let’s say 6 months that point of

(14:33) time you start adding the incremental things on top of it that makes a lot of sense yeah I am I’m curious like in your workflow kind of what percentage of new versus sort of iterative within an existing product tests are you running so this is what this is my second year with Cheapo and this is the first new you know project of product

(14:56) I’m working on so that should pretty much tell you in the last one and a half years I’ve significantly been on you know building up on what we already have and that’s one of those things I think anyone who’s spinning up an experimentation program or taking ownership of one that was already in place there is a sort of Runway that you

(15:12) have to walk or like to run to to get to where you can really move quickly and flexibly to to get results and and that’s one of those things where results are also with business and business leadership communicating results is its own process it’s a whole part of the job so within your role how are you finding

(15:35) it easiest to communicate out wins and losses and representing your program success I think the best way to do that and again this is something I’ve learned the hard way you know over the last 10 years or so is I think you should be constantly communicating I mean especially on you know Marquee projects

(15:53) or projects you know which take time to build because things do happen you know people will leave your team you know there could be an earthquake or or whatever so things do happen and you don’t want to be you know 3 months down the line into a project you don’t want to say you know Hang on we’re going to

(16:09) need you know another 5 weeks here so what I follow is I always so there’s a there’s a regular update on everything I work on on a weekly basis to higher up so they are aware of what I’m working on and where I’m blocked at any given point of time which happens fairly frequently when because we we in an industry which

(16:28) is extremely highly regulated we are very high traffic so a little here and there could mean a lot for us so that’s the only way you know you can actually be very transparent and get things done is by communicating probably over communicating if required at times I I’m in complete agreement with you there in

(16:47) my own organization we are lucky enough that our like marketing leadership up to the CMO has a lot of trust in our program we’ve built that trust but um even from the beginning we kind of had a little bit of autonomy and one of the ways we expanded that trust and created that that conversation was there’s like

(17:04) a weekly Wednesday morning meeting so you can be sure that my Tuesday afternoon every week is updating a deck of slides with current initiatives completed initiatives results handoffs it it is definitely a very necessary thing and then for the larger organization outside the direct line of leadership in my position we are

(17:22) spinning up a process of like quarterly reports uh and and also inviting other people to workshop with us within from other parts of the organization that maybe have visibility on operations or or merchandising or pieces that we don’t within our little corner of the company don’t have insight into and I I’m hoping

(17:42) that facilitating that kind of thing will sort of democratize the responsibility of innovation if that makes sense it does and we follow a similar process like this is probably more product specific so we are about 80 P.M across the org you know across multiple geographies so that’s India Europe North America uh once a month we

(18:03) have this of course you know not all 0 join but you know Friday we have this like a two-hour session which is more like a brainstorming session where you know people so there’s like couple of presentations people talk about things they’re working on there’s a general discussion so there are ideas being

(18:18) thrown about and and I think it’s it’s very important because you know especially if you’re working from home so you’re pretty much in a silo most of the time you know you’re doing your own thing you just have meetings in the morning and then in the afternoon you’re working on your backlog stories whatever

(18:32) so it’s important to have a larger perspective and I think as an or we’ve we’ve realized that so I mean I told you about the product meeting but they’re also trying to sort of expand it to other function so so like PMS like me now have a lot more exposure to teams such as marketing or even Finance for

(18:49) that matter because again at the end of the day we need to understand what works for the business I mean because we look at things sometimes we look at things with a lot of blinkers on and those need to be taken off oh yeah yeah it’s it’s very easy to get comfortable in the perspective you’ve developed and it’s

(19:07) not like it’s a bad perspective it’s very valuable for others to hear it but you do need to hear others as well I I think that’s really great let’s let’s pivot a little bit to the topic that is at every conference and every keyote this year and last year personalization we’re in the process in our org of

(19:26) spinning up a personalization and it’s of growing very naturally out of our optimization program so you say that’s something that’s happing with you and what’s that approach look like I would say we following a similar process I mean again we’ve we’ve been an or which has been very good at collecting

(19:43) data over the years but not in terms of using it but that’s that’s slowly changing slowly but surely changing I think the first approach we took was to essentially build out our own AB testing platform where you know we started testing things and experimenting stuff I think from there on we started so and

(19:59) that’s and I think roughly about I think what five or years back we sort of built out a extensive data science team who started working on you know these internal projects and I think our initial experiments around most of anything personalization was rule based so you know that would mean you know extensive sessions with analysts you

(20:17) know looking at multiple SE segments and all of those stuff running a lot of experiments again seeing what works what doesn’t we’ve actually mve moved on you know to machine learning recommenders and yeah I can’t tell you exactly where all we use them OB but we do deploy a lot of them across different aspects of

(20:37) the site so there’s probably at any at this point of time there’s probably three to four recommenders running across the entire flow I would say it’s something uh again all of it goes through a process it’s not like you know once we build a recommender we like oh it’s something shiny let’s just throw it

(20:53) out out there we experimented as well and and we’ve and we’ve had massive failures where something a recommender has actually led to a significant drop in a particular metric which it was supposed to improve but as I said you know multiple rounds of experimentation iteration tweaking things here here and

(21:10) there fixing it and constant communication between us and the data science team is I think what’s helped us help some of these experiments to succeed yeah I think you’re right on the money there uh the business intelligence aspect of personalization I don’t think is being talked about enough uh we we are approaching it as a very

(21:29) like sort of I I I also okay let me pivot here I think personalization is too broad of a term and I think it’s being thrown around like candy right now uh the same way AI was the same way any sort of Industry term gets thrown around like at the end of the day you’ve got customization you’ve got personalization

(21:46) you’ve got audience segmentation they’re not necessarily independent of each other and there’s crossover but they do slightly different things and I think you would know especially from that user Journey perspective customizing a user Journey based on say behavioral interaction and like what a user has

(22:02) done in session is different from identifying a user based on attributes they’re carrying in some data profile and saying well this is this kind of customer that’s shopping for this kind of thing to me that’s like true personalization of being able to then say okay here’s your personalized user experience that is based on those data

(22:20) points that are maybe independent of your behavioral data as well and so that’s one of those things where I think I don’t know that anybody has a lot of maturity yet especially because with this mve to first party data and everybody having to rely on you know customer data platforms and things like that I don’t know that the Integrations

(22:39) work well or reliably yet or have been attributed well and uh it’s it’s one of those challenges we’re facing as well in spinning up our personalization program and I made the controversial statement to a group of cro Specialists recently that uh I think anytime you run an experiment where you segment the

(22:58) audience based on like a user attribute it’s technically a personalization as much as it is a test and that sparked an argument so what are your thoughts on that you’ve hit the nail bang on the head m i mean here’s the thing right you know it’s it’s a buzz word so everyone who who’s doing anything close to

(23:17) segmentation says you know they’re personalizing I think the way we uh use these terms internally is a little different probably so for us personalization is if we know the user if it’s an identified user so and there are multiple ways we can identify the user right so you’ve gone you made a booking so yes we have details on you uh

(23:38) you’ve signed up for the newsletter so we have something on you as well and um again all of this is with your consent so this so I know that mtad has done something you know he has booked the last five times on my site and he’s used this payment method so that is pure personalization in my opinion I yeah on

(23:57) the other hand customization is again it’s it’s more about creating larger segment so anyone booking a flight from let’s say New York to Miami and booking seniors only that’s basically a custom segment and so there obviously there are opportunities to explore there but I wouldn’t call that as you know pure

(24:16) personalization that would be more around customization or segmentation yeah I I think I’m in total agreement there I guess I’ll open the floor here at this point you know uh what what’s kind of moving with you right now that you think is something very interesting within the space within your role within

(24:34) the industry that that you’re excited about right now of course again you know I’ll resort to one more buzz word which is generative AI course it it makes a lot of sense in the industry you know because um I love traveling and but planning for a trip is is hectic it’s crazy and um over the last couple of

(24:54) years they’ve been a couple I mean you know everyone in the space from xedia to kayak to booking.com have been have been doing a lot in this regard uh uh I definitely see a lot of opportunity there because this entire trip planning space can can essentially be simplified using generative AI it’s just that again

(25:15) who should be doing it should it be an otaa like us or Expedia should it be someone like a trip advisor or should it be someone else you know like a third party startup who comes in and does all of it um that’s that’s what uh we we need to figure out but there are definite opportunities here you know we

(25:35) at the end of the day you know as a company in the travel space you want to make life easier for people and let’s say booking a flight today is not easy there are multiple things you need to factor in cancellation charges are high you and then you need to look at a lot of stuff flight prices fluctuate you

(25:50) know indefinitely so it’s it’s not easy being a traveler if I may put it that way and we as a travel company I think or anyone in this space it’s their responsibility to make life easier you know for your customers and I think you can definitely do that using generative AI yeah absolutely and so that leads me

(26:08) to my next question which is are you using generative AI in your experimentation process right now and if so how um we are sort of Dipping our toes into the water you know as I said you know some of the others are you know competitors have already been there done that I think we always always we we’ve

(26:26) always are a little I wouldn’t say hesitant but I think we sort of take our time to see how the market responds to it again I mean we don’t want to go out there with a Half Baked product and I mean we’ve we’ve done so obviously I mean we use some part of it uh internally we’ve used generative AI internally for let’s say our own

(26:46) internal work but externally I mean we’ve we are there I mean we’ve done a couple of tests but I would say everything’s still pretty inent for us I think that’s where a lot of people are and I think you’re right on the money there of like needing to wait a little bit for some maturity in this space my

(27:01) uh VP that I roll up into uh his name is Damen Delgado and he always says you never want to be first to Market you want to be second to Market because the first person has already like borne the brunt of the backlash and you can then Sid step it so uh and I think there’s a lot of wisdom in that so yeah that’s all

(27:20) really very cool I think what I like to do sort of towards the last few minutes of any interview is a couple of uh let’s call them culture questions I I do them when we’re interviewing for roles on our team so I thought I’d bring them in here as well so we we’ll start with uh tea or coffee tea yeah I’m a chai guy myself uh

(27:40) it’s a podcast I want to start it’s like the try guys but it’s the try guys one day we’ll get there uh okay milk chocolate or dark chocolate dark chocolate any day also correct if the only wrong answer was white chocolate white chocolate’s not chocolate it’s not it’s fully not um okay cats or dogs that’s a tough one um

(28:04) gats perfect uh not everybody cares about this one but DC or Marvel can I say neither or but okay yeah absolutely not everybody cares about that yeah it’s lost a lot of the like when I when I started asking this question it was a lot more relevant culturally but the last few years there’s been a sharp

(28:24) decline in superhero interest uh almost as if the real world is scary enough I mean yeah um yeah final question is always uh what’s your favorite Cuisine oh I love eating so it would be hard to let’s say pick one thing particular but okay you’re putting me in a tough spot I would say indo-chinese yeah oh absolutely that’s a that’s a

(28:47) great choice yeah yeah yeah I am a I’m a big fan personally of like like DIY food Pakistani Indian food in general but uh if I had to choose something that I wouldn’t get at home it’s really tough to say uh I’m a big burger guy I don’t know if that counts as a Cuisine but I love Burgers so you know I tend to be

(29:06) very snobbish about my Burgers now so um can you be actually snobbish about Burgers I thought you can’t I think the snobby thing is to say you’re snobby about Burgers that’s well put yeah I I don’t know um I’d like to think there’s an elevated Burger but really I kind of like every Burger so I’m sure there they

(29:29) just need to do better sem maybe for you to find out there you go there you go yeah absolutely hi this is Romo Santiago from experiment Nation if you’d like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the description now back to the

(29:45) episode

If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

---