AI-Generated Summary

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight. Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration. Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically. Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction. Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

https://www.youtube.com/watch?v=Y2FhgrUR4WQ

AI-Generated Transcript

Time (Duration) Text 0:00 and it's one thing to nurture a culture 0:02 of experimentation in a company that 0:03 already has that inertia but it's a very 0:05 different Beast to start with a company 0:07 that does not have that and sort of 0:08 figure out what are the boundary 0:10 conditions that we need to put in place 0:11 in order to make experimentation more 0:13 possible and one of the things that I I 0:15 found is that um a lot of it revolves 0:20 around um the how teams decide what to 0:26 work on and how they collaborate between 0:29 different functions 0:30 [Music] 0:36 hey everybody it's Richard here from 0:38 discrimination podcast today I have 0:42 Lucas verir with us today uh it's been a 0:46 while since I've booked this 0:48 conversation with you uh for those who 0:50 don't know 0:53 Lucas Lucas was the experimentation um 0:57 director at booking.com so as you know 1:01 booking.com is one of the um yeah the 1:04 big 1:05 experimentation 1:07 um Role Models so to speak for how to 1:11 you know create a culture of 1:12 experimentation how to scale that and um 1:15 he was there for how long were you there 1:17 for 10 years eight 1:20 years yeah so you basically started off 1:23 it looks like from LinkedIn you started 1:24 off as a diet scientist and pretty much 1:27 mov into various roles of product 1:28 manager and um to finally director and 1:32 it looks like now you're pretty much um 1:34 doing several things you're obviously on 1:37 the public speaking circuit so you're 1:39 invited to pretty much every conference 1:41 known to man 1:43 uh every siero conference out there 1:48 um no you're an advisor on ab smartly 1:52 and you're currently acting as the 1:55 Director of 1:57 experimentation at Vista so welcome to 2:01 the podcast Lucas thank you great to be 2:04 here look a nice summary of my career or 2:08 most of it anyway I'm I'm really hying 2:10 that up it AR I aren't 2:11 [Laughter] 2:13 I um yeah I usually like to you know 2:17 have a quick chat about 2:20 um you know your education and um you 2:23 know just prob going over that and maybe 2:26 how you got into experimentation because 2:28 everyone's got some Pathways and um by 2:31 some people fall into by accident some 2:33 people scratch their own itch and um 2:36 yeah maybe just give us a brief of your 2:39 education and how you got into this uh 2:41 crazy 2:42 field huh it's not crazy what you call 2:46 crazy I'm not crazy are you 2:48 crazy I think we're all how did I get 2:50 into 2:54 um so how much education do you want so 2:57 my my parents were both academics I 2:59 think that I think that helps um so so 3:02 from from a young age but they were both 3:05 they were both linguists so very 3:07 different field um I think I got lucky 3:10 at a very young age that my my father 3:13 and his faculty there was one colleague 3:15 who was working on language in computers 3:18 or computers parsing language I think 3:20 it's is relevant because now with these 3:23 large language models like the world has 3:24 shifted right it's completely different 3:26 perspective on sort of how computers can 3:29 do language but at the time was still 3:30 very much structural uh still like 3:33 computers trying to parse grammar in a 3:35 very explicit way and so that got me 3:37 first interested in computers and and in 3:39 complex systems uh then uh in high 3:44 school I was very interested in in 3:45 biology in economics or as as as other 3:48 examples of complex systems but then 3:50 when it came to sort like going I wanted 3:52 to go to university I I wanted to study 3:55 uh a field um I couldn't really find 3:58 anything un like I liked until I ran 3:60 into to uh a a 4:03 specific really an experiment at the 4:06 University of utre uh in in the 4:08 Netherlands which they called the 4:10 technical artificial intelligence and it 4:13 was on the Confluence between psychology 4:16 and uh computer computer science so I 4:18 would have both psychology uh courses as 4:21 well as uh as computer science courses 4:23 and this was really early early days for 4:26 or not early days but it was a different 4:27 era for for machine learning right so so 4:30 I learned a lot of the basic 4:31 reinforcement learning algorithms so 4:33 like Bandits were obviously part of the 4:35 curriculum and all these things but I 4:37 learned them from a computer science 4:38 background yeah and then when I hit the 4:41 hit the job market uh I went into 4:43 Consulting at first uh did some business 4:46 intelligence work and mostly helped uh 4:50 companies who were trying to use 4:52 reinforcement learning for um marketing 4:54 automation or or next best action models 4:58 it's like figuring out which customers 4:59 want what on what and I think that was 5:02 really the first time that I ran into 5:04 the concept of a control group because 5:07 one of the one of the tools that I 5:09 worked with had a control group built in 5:11 so you could see how well the machine 5:13 learning model was was functioning 5:15 compared to some ground truth so for 5:18 most customers that would either be a 5:20 fixed calendar or a fixed offer or some 5:23 sort of popularity ranking but something 5:26 that was a simpler model than than the 5:27 complex machine learning or complex the 5:30 time it was complex now we would say 5:31 like it was a basic Bandit um and so 5:35 that I think that was the first time I 5:36 ran into a control group it's also the 5:38 first time that I ran into trouble with 5:40 sort of explaining why that was useful 5:42 to stakeholders because I I would have 5:45 clients who ask me to turn off the 5:47 control group or who would would ask me 5:50 like why are we showing customers an 5:51 inferior version and so like this these 5:54 were the first conversations I had 5:55 around sort of like the importance of 5:57 testing and sort of the the the value of 5:60 this 6:02 and very much as a consultant at the 6:04 time right so if if a client asked to 6:06 turn off the control group I would try 6:07 to convince them not to but I didn't 6:09 really have much much to say in the 6:12 matter then I uh got lucky really I mean 6:15 this the first I've gotten lucky many 6:17 times in my career but this first time I 6:19 got lucky I think was uh when I ran into 6:23 a man who worked for booking.com and 6:26 that that was very much in the scaleup 6:27 phase at the at the time yeah and they 6:29 were looking for someone to to work on 6:32 their recommendation system models uh 6:35 their 6:36 Bandits uh and so I joined uh booking in 6:39 2013 I think as a as their first data 6:42 scientist and because I was the first 6:45 data scientist the and there and they 6:46 were so much in scaleup mode the HR 6:48 department actually didn't have a job 6:50 description or a job role for uh data 6:53 scientists yet and so they they made me 6:55 a product manager because that in terms 6:58 of like role description and salary 6:60 bands that seemed to be the closest to 7:02 to what I was looking for and so I are 7:05 like my first interactions with product 7:07 were like having the product manager 7:09 title while doing a while doing a data 7:12 scientist job and so for the first I 7:14 think two years of booking I sort of 7:16 worked on their recommendation system 7:17 models and because this was very much in 7:19 scaleup mode uh you can imagine like 7:22 they were still operating very much as a 7:24 startup and so as a data scientist I 7:27 wasn't just building models in like a a 7:29 data or like many companies function now 7:32 but I was also doing some of the 7:33 engineering I was also doing some of the 7:36 product management I was also doing some 7:37 of the stakeholder management so so in 7:39 effect like the product manager role 7:41 wasn't all that strange like I was doing 7:43 some of those tasks and so it felt very 7:45 natural for me to then go from that into 7:49 a more formal product manager role when 7:50 we started hiring more people for the 7:52 team I became the product manager for 7:54 recommendation systems and later for for 7:56 ranking uh so the order that you saw if 7:60 you went to booking.com between say 2014 8:02 and 2016 the order in which you saw the 8:05 hotels on booking.com when you made a 8:07 search that was my responsibility at the 8:10 time um then uh I got increasingly 8:14 interested in the ab test that uh that 8:16 booking was running and so I spent more 8:18 and more time looking at the tests that 8:20 other teams were running uh and joyas 8:22 Alvis at the time was was responsible 8:25 for that platform uh but he decided to 8:27 leave the company and and so I was asked 8:29 to to step forward and become the 8:31 product manager for the experimentation 8:32 platform and that you mentioned AB 8:33 smartly earlier so one of the reasons 8:35 I'm working with with AB smartly is that 8:37 join us Office later founded AB smartly 8:40 because he he realized that as started 8:43 as a consultant and he was asked time 8:45 and time again to build the same product 8:48 or the same platform that that 8:50 booking.com had to build it for other 8:53 companies and they realized that there 8:54 might be a market for selling it and so 8:57 with AB smartly what they've essentially 8:59 done is rebuild booking.com AB testing 9:01 platform but now anyone can buy it so I 9:05 because I have Affinity with that 9:07 platform I was responsible for the 9:09 growth of of that platform between 2016 9:11 and 9:12 2021 um and Jonas and I sort of like 9:15 between us have like two decades of 9:17 experience in that uh I think that's 9:19 that was a sort of a natural fit to work 9:22 with them so that's a little bit of how 9:24 I rolled into into experimentation hi 9:27 this is Romo Santiago from experiment 9:28 Nation if you'd like to connect with 9:30 hundreds of experimenters from around 9:31 the world consider joining our slack 9:32 Channel you can find the link in the 9:34 description now back to the episode 9:36 thanks thanks it's a interesting story 9:38 and look um I'm particularly Keen in in 9:42 terms of learning 9:44 how 9:46 um you know you developed and scal that 9:50 culture of experimentation um at 9:54 booking.com you know I think our 9:56 audiences you know a lot of them would 9:58 know how to set up a 9:60 at least beginners would know how to set 10:01 of basic AB test using the the current 10:04 client side tools and sort of things 10:06 like that but um you know before 10:09 the podcast yesterday we were talking 10:11 about like you know how we 10:13 can you know just use a client side tool 10:16 but not credit culture and just have a 10:18 silo effect there and where it's just 10:20 you know usually done through marketing 10:21 um so I won't I won't steal your funder 10:24 you discussed um how you scaled it 10:26 yesterday so it be good to explain it to 10:27 audiences and really like how to really 10:30 truly embed experimentation um not just 10:32 in one area of the company but like you 10:35 know through throughout the whole 10:37 company um so I'll let you talk about 10:40 that yeah I mean that's so that's an 10:42 interesting topic that's been like the 10:45 the necklace of my focus in the last few 10:48 years there's many many things to talk 10:50 about I think one is that uh I don't 10:54 think we should take booking as a as a 10:56 model nor should we take my experience 10:58 there as like a as vouching for my sort 11:00 of expertise in this area uh booking was 11:03 already very much an experimentation 11:05 culture when I joined like I didn't 11:07 create that it was already there and I 11:09 think it was very much driven by the 11:11 founders and the people they hired and 11:13 the way they organized their teams it's 11:15 not something that I created it's 11:16 something that I nurtured um what I did 11:20 help was sort of taking at the time when 11:23 I joined that was very focused to the 11:26 web side of things so the the the 11:28 website was running a lot of experiment 11:30 but there's many other aspects of of 11:32 booking account that were not and so you 11:34 can think of booking sort of divided by 11:37 different uh product areas uh one of 11:40 them being the sort of the website 11:42 that's consumer facing but there's also 11:44 a website is that is Hotel facing that 11:46 they use to sort of add rooms or change 11:48 availability and then there's obviously 11:51 the customer care the call center 11:53 there's many other aspects of that of 11:54 that business that were're not really 11:55 running experiments when I when I 11:57 started um and so one of the things that 11:59 I I tried to do in in a time was there 12:01 it was like on the one hand sort of like 12:03 increase the quality of the experiments 12:05 that were being run in the website and 12:07 at the same times or like scale out that 12:10 approach to other parts of the 12:11 organization which mostly entailed on 12:14 the uh um figuring out technically how 12:17 to sort of uh move into those areas 12:21 because for example if you want to run 12:23 experiments in a call center you have to 12:24 figure out how to integrate into a a an 12:28 ivr like a phone system them you need to 12:30 figure out how to do a coin flip based 12:31 on a phone number you need to figure out 12:33 how to do metrics based on that so 12:34 there's some technical challenges 12:36 related to that and then there's some 12:37 organizational challenge related to how 12:39 do we educate those people how do we get 12:40 them on board like how do we how do we 12:42 help them understand that this is 12:43 important I think this is much easier 12:45 when you have a large share of the 12:47 organization that is already running 12:48 experiments and so one of the ways that 12:50 we scaled into partners for example was 12:53 to take people developers designers 12:55 product managers who were already 12:57 running experiments on the website on 12:59 the customer side of things and say hey 13:01 can you can you join one of the teams on 13:03 the partner side and help them 13:05 understand how to use experimentation as 13:07 part of product development and so like 13:09 we we could cross-pollinate between 13:11 these departments and sort of help them 13:12 help them scale out I think that that 13:14 made the job a lot easier one of the 13:16 reasons that I I left booking actually 13:18 was I I wanted to figure out like if I 13:19 go to a different company that does not 13:22 have this scale like how what other 13:25 challenges do I run into what is 13:27 difficult about doing it there and it's 13:28 been actually very very very interesting 13:30 learning experience because I I found to 13:32 to your point like there's a lot of 13:34 people who know including me like know 13:36 how to set up an experiment and it's one 13:39 thing to nurture a culture of 13:40 experimentation in a company that 13:41 already has that inertia but it's a very 13:43 different Beast to start with a company 13:45 that not have that and sort of figure 13:47 out what are the boundary conditions 13:48 that we need to put in place in order to 13:50 make experimentation more possible and 13:52 one of the things that i' I found is 13:54 that um a lot of it revolves 13:58 around um 13:60 the how teams decide what to work on 14:05 yeah and how they collaborate between 14:07 different functions and so um when I 14:10 joined 14:12 Vista they were they were organized in 14:14 such a way that there was a data 14:16 organization there was an engineering 14:17 organization there was a product 14:18 organization and a marketing 14:19 organization that included design and to 14:22 me this was very surprising because for 14:24 for eight years of booking I had worked 14:26 in a model where there were cross 14:28 functional teams and so if you think 14:30 about ranking my my responsibility when 14:33 when as was a product manager I had a 14:35 team that included data scientists 14:37 developers and designers as part of one 14:40 single team and our job was related to 14:43 make to making it easier for people to 14:45 find a hotel that they could stay at 14:47 yeah and so we we were not organized 14:50 around craft but we were organized 14:52 around a customer objective and we and 14:55 in I could argue to leadership and say 14:58 well in order to accomplish this 15:00 objective here are the skills that I 15:02 need right I need a designer because I 15:05 need to explain to people why this is 15:06 the right hotel for them I need a data 15:09 scientist because I need to build 15:10 machine learning models that help pick 15:11 the right the right hotel right and so 15:14 that given the objective there can be 15:16 conversations about which skills are are 15:18 needed on the team and and these teams 15:20 were long lived and they are considered 15:22 to be primary teams so if you had 15:24 approached my team members at the time 15:26 and asked them like which team are you 15:27 part of they would say the ranking team 15:29 or they would say the search team now 15:31 when I joined Vista I noticed that when 15:32 you ask an engineer which team are you 15:34 part of they would name their 15:35 engineering team and you ask them what 15:37 do you own they would name the service 15:39 that they owned and the same for the 15:41 data data analyst and the same for 15:42 product managers like they weren't 15:44 thinking in terms of problem customer 15:47 problem focused teams they were thinking 15:50 in terms of craft teams and so this 15:53 makes experimentation much more 15:55 difficult because there's many more 15:56 communication Hops and there's much more 15:58 room for misalignment between the 15:60 different crafts even though we're all 16:01 working together to solve the same 16:02 customer problem and an experiment 16:05 serves to figure out whether the 16:06 customer problem is solved yeah but but 16:08 if we're not all aligned on what 16:10 customer problem we're solving together 16:11 then the experiment doesn't really have 16:13 value and and and no one really sees a 16:15 need to experiment in the first place 16:17 and so I started to think that we needed 16:19 to change the organization of of Vista 16:21 towards a more uh cross functional 16:24 product team organization and and I will 16:28 not in anyway claimed that I was the one 16:31 who drove this change like there were 16:33 many people with me that had the same 16:35 opinion I have tried to push for this 16:37 for the last three years and I'm happy 16:39 to report that the things are going 16:40 really well and that we're slowly moving 16:42 towards this product operating model um 16:45 and but I do think it's one of the 16:46 boundary conditions that needs to be in 16:48 place for experimentation to be really 16:51 scalable and so to your earlier point 16:53 like I think a lot of organizations and 16:55 and especially the older so like the the 16:58 first generation of experimentation 16:59 platforms they are very marketing 17:01 Centric and they're very marketing 17:03 focused and the pitch is essentially uh 17:06 put this script on your website and you 17:08 will never have to talk to it again to 17:10 make to to run an experiment that's 17:12 that's the pitch and I think that is 17:14 marvelous 17:15 technology but it sort of completely 17:19 circumvents the core problem that we 17:21 need to solve which is that marketing 17:23 and it should not be separate in the 17:25 first place right and they usually kind 17:28 of into taking this well they can't be 17:30 antagonistic toward towards each other 17:32 because marketing will be like okay 17:34 We've ran these experiments um here's a 17:36 few winning experiments that we want it 17:40 produ to 17:41 productionize and then for it it's like 17:44 they're rolling their eyes and they 17:45 thinking oh well I'll just got to I'll 17:47 just add to my backlog of 100 other 17:49 items on my to-do 17:51 list you know what I mean yeah yeah yes 17:54 and and and they're PRI prioritizing in 17:56 different ways right so so one of the 17:57 reasons that marketing struggles to put 17:59 stuff on the it backlog is it just have 18:01 different has different priorities and 18:02 that's why why I think that these people 18:04 should be together in one team and have 18:06 the same set of objectives ideally 18:08 around a customer because then the the 18:10 conversation changes I I mean you tell 18:13 you an anecdote we had a uh we had a re 18:16 a Booking.com reunion a few months ago 18:18 where like people who worked for 18:19 booking.com over the years sort of got 18:21 together and sort of chatted about their 18:22 life after booking it's very interesting 18:24 because it's very much a almost like a 18:26 cult 18:27 following if I might say so uh and uh I 18:31 I was talking to a group of Engineers 18:34 and and they were all surprised about 18:36 sort of the engineering cultures that 18:38 they found outside uh of bookan aom and 18:42 I said oh yeah I I've noticed the same 18:44 when when I say to people like at 18:48 booking it is very it was very normal 18:51 that you go to an engineer and you say I 18:53 would like you to put like on this on 18:56 these landing pages I want you to change 18:58 this thing 18:59 yeah it is very common for an engineer 19:02 to then say well that page gets about 19:05 10,000 visitors a day which translates 19:07 into about 19:09 0.001% of our total revenue so if I were 19:12 to spend two days of work on this like 19:14 that is already more expensive than a 19:17 100% uplift on conversion that we would 19:20 that we would need to to cover like the 19:22 base cost and there's no way we would 19:23 get that because the change that you're 19:25 proposing like psychologically I don't 19:27 see why that would cause that much 19:29 changing conversion so I'm not going to 19:30 prioritize 19:32 it and and I'm sort of look around the 19:35 look around the circle of Engineers and 19:36 they all like yes that is that is 19:39 exactly what is missing from the 19:41 engineering cultures that we see in 19:42 other organizations where Engineers 19:44 would not they are not expected to and 19:47 so they do not talk in business terms 19:51 and I don't think this is a lack of 19:52 skill they just really just thinking how 19:54 do we technically set up this test but 19:56 then not thinking like correct um like a 19:60 would where it's like okay we get 10,000 20:02 uniques per day or month or whatever and 20:05 um you know these are the business 20:07 implications if I set up this test if 20:09 it's high then I'll just put it high up 20:11 my back in my my Q in my road map if 20:14 it's pretty low then I'll just push back 20:17 against you and say no this is not good 20:18 or put it lower down the the the um the 20:21 road map 20:24 um yeah like that's um that's that's not 20:27 something I've experienced with 20:28 Engineers cuz usually they just like 20:29 okay I'll just I'll build it for you 20:30 because you you told me to yes and I 20:33 think that's if that's the working model 20:35 then getting into an experimentation 20:36 space is is difficult because you're 20:38 just speaking different languages and 20:40 that's why I I contrast this with a sort 20:42 of the newer generation of tools AB 20:44 smartly is one of them there there's 20:46 others of course and they they're all 20:48 founded by people who came from these um 20:52 these these organizations that are 20:54 already running experimentation at 20:55 Scales so so Facebook Airbnb booking.com 20:59 like these are all companies that are 21:00 running experimentation scale people 21:02 coming from those companies they design 21:04 experimentation Platforms in a 21:05 completely different way because they 21:07 aren't thinking about it in terms of put 21:09 this pixel on your website so you can 21:10 avoid it they're thinking of it in terms 21:13 of it is at the table when you're 21:16 designing this this experiment like they 21:18 are involved in the design and the 21:20 execution of the experiment the vast 21:22 majority of experiments that are run 21:23 at.com are run by Engineers like they're 21:26 they're most of the tests are set up by 21:28 by an engineer in person rather than bu 21:30 a product person so I think this is like 21:33 this is the biggest differ when we talk 21:34 about the experimentation Gap I this 21:37 article that came out I think a year 21:39 ago like this is a large part of the Gap 21:42 like it's not just scale it's scale that 21:45 comes from a different way of working 21:47 and a different way of working that 21:49 encourages especially engineering to 21:51 think in a different way yeah so um 21:55 maybe for our audiences um who are 21:57 listening to this like say they are are 21:59 in that sort of 22:00 um they product manager or they're C 22:03 Specialist or whatnot and they're in 22:06 that sort of um early to midstage of 22:10 experimentation maturity and you know um 22:14 they still relying on just the client 22:15 side tool and they're thinking okay um 22:18 you know if I really want to 22:20 embed a real culture of experimentation 22:24 I've got to go engineering first and um 22:28 have a tool that's going to 22:30 be servici side cuz it's going to be 22:33 faster and Engineers you know they care 22:36 about speed and those sort of things um 22:40 what kind of advice would you give to 22:42 those people um from your experience 22:44 looking at Vista where um they want 22:47 they're in that sort of Silo 22:50 structured um to go into the cross 22:53 pollination structure like you you 22:56 mentioned at booking.com I mean 22:59 you have to restructure the the whole 23:01 damn organization or how the hell do you 23:03 do it I mean that's a scary proposition 23:06 that's a scary proposition um I 23:10 think please take my advice here with a 23:12 huge grain of salt yeah I think I think 23:15 there if you you have to adjust your 23:17 prior this this is a company that did 23:19 not have a scaled organization of 23:21 culture uh of experimentation at the 23:23 time that I joined um but they did hire 23:27 me as I and so like that already tells 23:30 you something about like the leadership 23:32 Buy in that there was for for 23:34 experimentation yeah at the time right 23:37 because I think I talked to a lot of our 23:38 other organization and just merely 23:40 suggesting that you should hire a 23:41 director of experimentation is already 23:43 like a bridge too far for many many 23:45 organizations in this case we had a a c 23:48 Vista has a CEO that is very much bought 23:51 into uh the the idea that 23:53 experimentation is an important part of 23:55 product development and the same for CTO 23:57 and CPO there very strong uh leadership 24:00 Buy in there's also a long history of 24:03 experimentation at Vista it's just very 24:05 decentralized so so I didn't know this 24:07 when I joined but Vista actually started 24:09 running experiments before booking.com 24:10 did so booking started in 2005 February 24:14 I believe yeah and Vista ran their first 24:16 experiment somewhere in at the end tail 24:17 end of 20 2004 so they were just barely 24:21 sooner um but some somehow it never h it 24:26 did scale actually but somehow it it 24:29 like uh slumped again and I was hired 24:32 sort of as the slump was sort of sort of 24:35 paning out and now now we're going going 24:36 back up again um and so I don't think 24:40 Vista is like a a a good template for 24:44 like how other organizations can do this 24:46 because there I had a lot of wind in my 24:48 sales 24:50 um given that like when I joined there 24:54 they did not have a product operating 24:55 model um but me coming from that uh that 24:60 is the only way that I know how to 25:01 operate like I don't really know how you 25:05 would make things work if you don't have 25:06 a cross functional team that has all the 25:08 skills that they need to sort of solve 25:10 the problems that their T to solve so 25:12 one of the first things I did was sort 25:13 of explain to people like how I saw 25:17 teams should function uh even if that 25:21 was not the reality of organization at 25:22 the time uh sort of advocating for that 25:25 model and then applying it within my own 25:28 team like I I had a very very small team 25:31 of I believe three analysts or two 25:33 analysts when I joined and one of the 25:35 first things I did was I I went to an 25:37 engineering leader and said well uh I 25:39 want to try this new model where we take 25:41 those two analysts and we take the two 25:43 Engineers that you have assigned to this 25:45 topic and we put them together and we 25:46 tell them that they are one team and we 25:48 give them one name and we give them one 25:50 identity and together we give them 25:52 objectives and I was lucky to find an 25:54 engineering leader that that bought into 25:56 that Paradigm and said yes let's do it 25:58 and so together we set the objectives 25:59 for this one team and started treating 26:01 them as if they were 26:03 one and very quickly then I hired Uh 26:06 Kevin Anderson as a product manager I 26:08 found sort of like an organizational 26:10 loophole where I could hire a a product 26:12 manager as part of my organization even 26:14 though I wasn't in product by calling 26:16 him a data product manager that was the 26:18 that was the loophole like Kevin very 26:22 much operates as a product manager he's 26:24 excellent um but he was called a data 26:26 product manager because that allowed me 26:28 to then hire in into the team and so we 26:30 could we could sort of treat them as a 26:32 one like as an island of a an an 26:35 empowered product team within within a 26:37 larger organization that was still 26:38 grappling with the with the concept and 26:40 use them as a as a role model and go to 26:43 leadership and say like hey here's how 26:45 this works see the artifacts that that 26:47 they are creating and again Kevin is 26:49 excellent for this because he's a he's 26:50 very much a documentation driven person 26:52 he will write things down and he does so 26:54 very publicly and so all of the 26:56 artifacts that this team was creating 26:57 that we could also show them into 26:58 leadership and say hey here's what the 26:60 proit team is doing and here's how and 27:01 here's the results they're having and 27:02 here's how they're tracking their kpis 27:04 and so that becomes a role model and 27:06 example that you can use for rest 27:07 organization at the same time I I have 27:09 to say like I very much had to wind in 27:11 my sales because we have a a chief 27:13 product officer who's very much bought 27:14 into the same Paradigm we worked at 27:16 Amazon before uh we have a CTO that was 27:19 very much bought into this CEO that also 27:21 came to from Amazon so we had all these 27:23 people who understood this model um and 27:27 then we got help from uh from outside 27:29 consultancy uh including Marty Kagan 27:32 who's uh who's known for what what is 27:34 called the product operating model which 27:36 I was not familiar with but when I read 27:39 all the material it's like oh yes that's 27:40 how booking aom operated so so it feels 27:44 like there's some sort of convergent 27:45 evolution where like a lot of these 27:47 companies are finding out that if you 27:48 put people of different crafts together 27:50 and you give them a customer problem to 27:52 solve and you give them flexibility on 27:54 how to how to sort of explore solutions 27:56 that is a very effective model 27:59 and Marty has done some great work in 28:00 sort of documenting how the differences 28:03 between the different companies and sort 28:04 of the commonalities between different 28:05 companies approaching this I was lucky 28:07 to be hired into a company that did this 28:09 now I'm I'm lucky to be in a company 28:11 that is slow that is not slowly that's 28:14 rapidly adjusting to the style of 28:16 working and so like three years ago we 28:19 were completely siloed now like we are 28:21 completely uh product operating mod 28:24 there's no more separation between uh 28:27 engineering uh product and and design 28:29 they're all they're all one so terms of 28:33 um cultural 28:36 change 28:38 um would you say that there a lot of um 28:42 growing growing pangs and in in in going 28:46 from The Silo model to 28:48 the the sort of cross Silo model like I 28:51 mean oh it's yeah do you get people 28:53 leaving for these reasons you get people 28:55 that are like oh that's company's 28:57 changing I mean do 28:59 is is a common so there so so two things 29:01 there are definitely growing pants so so 29:03 so it's not easy I'm not at all sugges 29:05 and it it does take time like this this 29:07 is a process is going to take years yeah 29:10 it it does and I think it's difficult 29:12 for two reasons like people have to find 29:14 a new way of working they have to new 29:16 find new rhythms and so we have a we 29:18 have a separate part of the organization 29:20 that is specifically tasked with 29:22 figuring out those organizational 29:23 rhythms and they also operate as product 29:26 teams and so they set their own 29:27 objectives they make meure their 29:28 outcomes in terms of how teams are 29:30 operating and this is not treated as a 29:32 as a like a one-off program this is an 29:35 ongoing Improvement thing with in the 29:36 organization like how do we work that's 29:38 one I think the other is that you said 29:40 are people leaving 29:42 um I I don't think we I don't think we 29:45 we see like people leaving in dros like 29:48 every company has attrition right and no 29:50 different I do see like people have to 29:53 stretch themselves because this model 29:54 does ask different things so to go back 29:57 to the earlier example about engineers 29:58 right booking called these t-shapes you 30:01 want you want people who are not just 30:03 very good in one craft you want people 30:06 who are good in one craft but then know 30:09 a little bit or know enough about each 30:11 other craft to have conversations with 30:13 their peers right a an engineer should 30:16 know enough about the business and about 30:18 the metrics of the product they're 30:20 responsible for to have conversations 30:22 with a designer and a product manager 30:23 about the product they're working on and 30:25 a designer should know enough about the 30:27 engineering problems to be able to talk 30:29 to the engineer about solutions to 30:31 designs that they have come up with 30:33 right and so this requires people to 30:35 think a little bit outside of their 30:36 craft and there are people who want this 30:38 and there are people who don't want this 30:39 I think that is that is a shift we're 30:41 slowly seeing and I think it's one of 30:43 the reasons that you see that not only 30:44 are there Growing Pains there's also uh 30:48 I think it was CRZ was it this famous um 30:51 quote like the future is already here 30:53 it's just not evenly 30:55 distributed of that yeah right it's we 30:58 have teams are operating really well in 30:60 this product operating model and we have 31:01 teams are really struggling in this 31:03 product operating model and so we're 31:05 trying to find ways again 31:07 cross-pollinate like we did a booking 31:08 like how do we take people who are very 31:10 effective at this like put them in teams 31:12 that are struggling it's like how do we 31:13 how do we make a rising tide Floats or 31:15 boats and that is just a process that we 31:17 need to go through I in no way 31:19 suggesting that this is easy um and your 31:22 again even if you have leadership eying 31:25 it is still not easy you've mentioned 31:27 this a few times that you know you had 31:29 leaders like the seite execs um having 31:33 huge buying and I think that's sounds 31:36 like that's a 31:38 common it's a a huge factor for for 31:42 driving this don't you think like if 31:43 it's not coming from the Top If It's s a 31:46 sort of bolted on the side like 31:47 marketing it's always going to be in 31:49 that sort of 31:50 Silo um sort of Paradigm where it's like 31:54 marketing versus technology or whatever 31:56 but if it's coming from the top down and 31:58 it's like you know really embedded from 32:01 the exacts and it's flowing throughout 32:04 the whole organization not saying it's 32:06 easy but that sounds like that's a key 32:09 element that you need would you agree 32:13 like I I would say if you don't have it 32:15 you think you have it yes and that's not 32:17 sufficient it's not sufficient because 32:19 because I would say yes leadership needs 32:21 to be byy in and Leadership needs to 32:23 understand that it has implications for 32:25 how they organize their their business 32:28 yeah and how they operate 32:31 themselves and and this is the part that 32:34 is not 32:35 easy you might have a leader that says 32:37 yes I want you to run experiments 32:39 marketing go run experiments but what 32:41 we'll keep the engineering organization 32:44 aside we're not touching it then they're 32:46 not bought in enough yeah then you kind 32:49 of end up being siloed um yeah so look 32:54 it's um unfortunately we can't go on for 32:56 too long but um any any closing thoughts 32:59 for our 33:04 audiences uh I I hope I hope what I 33:07 share 33:08 helps I hope I've also given you enough 33:11 caveats to say like this is my 33:12 experience at two companies so so like 33:16 and I I I again I've been very lucky 33:18 I've had the wind in my sales for a long 33:19 time I like if you struggle with this i' 33:22 I'd be happy to chat like to hear hear 33:23 the things you're struggling with I 33:25 maybe I maybe I can help so please do 33:26 reach out let's talk 33:29 we are all learning right this like we 33:31 are experimentation people let's 33:33 experiment with how we do the 33:34 organization around experiment um how 33:37 can people contact you Lucas sure uh so 33:40 just go to my website my name. NL NL for 33:43 Netherlands uh and then there's a link 33:45 at the top that allows you to book some 33:46 time in my calendar or send me an email 33:48 whatever you want that's the quickest 33:50 way awesome thanks for joining the 33:52 podcast thank you for having me it was 33:54 great bye see you hi this is Romo 33:57 Santiago from experiment Nation if you'd 33:59 like to connect with hundreds of 33:60 experimenters from around the world 34:01 consider joining our slack Channel you 34:03 can find the link in the description now 34:04 back to the episode

If you liked this post, sign up for Experiment Nation's newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK