AI-Generated Summary
Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight. Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration. Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically. Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction. Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.
AI-Generated Transcript
Time (Duration) | Text |
0:00 | and it’s one thing to nurture a culture |
0:02 | of experimentation in a company that |
0:03 | already has that inertia but it’s a very |
0:05 | different Beast to start with a company |
0:07 | that does not have that and sort of |
0:08 | figure out what are the boundary |
0:10 | conditions that we need to put in place |
0:11 | in order to make experimentation more |
0:13 | possible and one of the things that I I |
0:15 | found is that um a lot of it revolves |
0:20 | around um the how teams decide what to |
0:26 | work on and how they collaborate between |
0:29 | different functions |
0:30 | [Music] |
0:36 | hey everybody it’s Richard here from |
0:38 | discrimination podcast today I have |
0:42 | Lucas verir with us today uh it’s been a |
0:46 | while since I’ve booked this |
0:48 | conversation with you uh for those who |
0:50 | don’t know |
0:53 | Lucas Lucas was the experimentation um |
0:57 | director at booking.com so as you know |
1:01 | booking.com is one of the um yeah the |
1:04 | big |
1:05 | experimentation |
1:07 | um Role Models so to speak for how to |
1:11 | you know create a culture of |
1:12 | experimentation how to scale that and um |
1:15 | he was there for how long were you there |
1:17 | for 10 years eight |
1:20 | years yeah so you basically started off |
1:23 | it looks like from LinkedIn you started |
1:24 | off as a diet scientist and pretty much |
1:27 | mov into various roles of product |
1:28 | manager and um to finally director and |
1:32 | it looks like now you’re pretty much um |
1:34 | doing several things you’re obviously on |
1:37 | the public speaking circuit so you’re |
1:39 | invited to pretty much every conference |
1:41 | known to man |
1:43 | uh every siero conference out there |
1:48 | um no you’re an advisor on ab smartly |
1:52 | and you’re currently acting as the |
1:55 | Director of |
1:57 | experimentation at Vista so welcome to |
2:01 | the podcast Lucas thank you great to be |
2:04 | here look a nice summary of my career or |
2:08 | most of it anyway I’m I’m really hying |
2:10 | that up it AR I aren’t |
2:11 | [Laughter] |
2:13 | I um yeah I usually like to you know |
2:17 | have a quick chat about |
2:20 | um you know your education and um you |
2:23 | know just prob going over that and maybe |
2:26 | how you got into experimentation because |
2:28 | everyone’s got some Pathways and um by |
2:31 | some people fall into by accident some |
2:33 | people scratch their own itch and um |
2:36 | yeah maybe just give us a brief of your |
2:39 | education and how you got into this uh |
2:41 | crazy |
2:42 | field huh it’s not crazy what you call |
2:46 | crazy I’m not crazy are you |
2:48 | crazy I think we’re all how did I get |
2:50 | into |
2:54 | um so how much education do you want so |
2:57 | my my parents were both academics I |
2:59 | think that I think that helps um so so |
3:02 | from from a young age but they were both |
3:05 | they were both linguists so very |
3:07 | different field um I think I got lucky |
3:10 | at a very young age that my my father |
3:13 | and his faculty there was one colleague |
3:15 | who was working on language in computers |
3:18 | or computers parsing language I think |
3:20 | it’s is relevant because now with these |
3:23 | large language models like the world has |
3:24 | shifted right it’s completely different |
3:26 | perspective on sort of how computers can |
3:29 | do language but at the time was still |
3:30 | very much structural uh still like |
3:33 | computers trying to parse grammar in a |
3:35 | very explicit way and so that got me |
3:37 | first interested in computers and and in |
3:39 | complex systems uh then uh in high |
3:44 | school I was very interested in in |
3:45 | biology in economics or as as as other |
3:48 | examples of complex systems but then |
3:50 | when it came to sort like going I wanted |
3:52 | to go to university I I wanted to study |
3:55 | uh a field um I couldn’t really find |
3:58 | anything un like I liked until I ran |
3:60 | into to uh a a |
4:03 | specific really an experiment at the |
4:06 | University of utre uh in in the |
4:08 | Netherlands which they called the |
4:10 | technical artificial intelligence and it |
4:13 | was on the Confluence between psychology |
4:16 | and uh computer computer science so I |
4:18 | would have both psychology uh courses as |
4:21 | well as uh as computer science courses |
4:23 | and this was really early early days for |
4:26 | or not early days but it was a different |
4:27 | era for for machine learning right so so |
4:30 | I learned a lot of the basic |
4:31 | reinforcement learning algorithms so |
4:33 | like Bandits were obviously part of the |
4:35 | curriculum and all these things but I |
4:37 | learned them from a computer science |
4:38 | background yeah and then when I hit the |
4:41 | hit the job market uh I went into |
4:43 | Consulting at first uh did some business |
4:46 | intelligence work and mostly helped uh |
4:50 | companies who were trying to use |
4:52 | reinforcement learning for um marketing |
4:54 | automation or or next best action models |
4:58 | it’s like figuring out which customers |
4:59 | want what on what and I think that was |
5:02 | really the first time that I ran into |
5:04 | the concept of a control group because |
5:07 | one of the one of the tools that I |
5:09 | worked with had a control group built in |
5:11 | so you could see how well the machine |
5:13 | learning model was was functioning |
5:15 | compared to some ground truth so for |
5:18 | most customers that would either be a |
5:20 | fixed calendar or a fixed offer or some |
5:23 | sort of popularity ranking but something |
5:26 | that was a simpler model than than the |
5:27 | complex machine learning or complex the |
5:30 | time it was complex now we would say |
5:31 | like it was a basic Bandit um and so |
5:35 | that I think that was the first time I |
5:36 | ran into a control group it’s also the |
5:38 | first time that I ran into trouble with |
5:40 | sort of explaining why that was useful |
5:42 | to stakeholders because I I would have |
5:45 | clients who ask me to turn off the |
5:47 | control group or who would would ask me |
5:50 | like why are we showing customers an |
5:51 | inferior version and so like this these |
5:54 | were the first conversations I had |
5:55 | around sort of like the importance of |
5:57 | testing and sort of the the the value of |
5:60 | this |
6:02 | and very much as a consultant at the |
6:04 | time right so if if a client asked to |
6:06 | turn off the control group I would try |
6:07 | to convince them not to but I didn’t |
6:09 | really have much much to say in the |
6:12 | matter then I uh got lucky really I mean |
6:15 | this the first I’ve gotten lucky many |
6:17 | times in my career but this first time I |
6:19 | got lucky I think was uh when I ran into |
6:23 | a man who worked for booking.com and |
6:26 | that that was very much in the scaleup |
6:27 | phase at the at the time yeah and they |
6:29 | were looking for someone to to work on |
6:32 | their recommendation system models uh |
6:35 | their |
6:36 | Bandits uh and so I joined uh booking in |
6:39 | 2013 I think as a as their first data |
6:42 | scientist and because I was the first |
6:45 | data scientist the and there and they |
6:46 | were so much in scaleup mode the HR |
6:48 | department actually didn’t have a job |
6:50 | description or a job role for uh data |
6:53 | scientists yet and so they they made me |
6:55 | a product manager because that in terms |
6:58 | of like role description and salary |
6:60 | bands that seemed to be the closest to |
7:02 | to what I was looking for and so I are |
7:05 | like my first interactions with product |
7:07 | were like having the product manager |
7:09 | title while doing a while doing a data |
7:12 | scientist job and so for the first I |
7:14 | think two years of booking I sort of |
7:16 | worked on their recommendation system |
7:17 | models and because this was very much in |
7:19 | scaleup mode uh you can imagine like |
7:22 | they were still operating very much as a |
7:24 | startup and so as a data scientist I |
7:27 | wasn’t just building models in like a a |
7:29 | data or like many companies function now |
7:32 | but I was also doing some of the |
7:33 | engineering I was also doing some of the |
7:36 | product management I was also doing some |
7:37 | of the stakeholder management so so in |
7:39 | effect like the product manager role |
7:41 | wasn’t all that strange like I was doing |
7:43 | some of those tasks and so it felt very |
7:45 | natural for me to then go from that into |
7:49 | a more formal product manager role when |
7:50 | we started hiring more people for the |
7:52 | team I became the product manager for |
7:54 | recommendation systems and later for for |
7:56 | ranking uh so the order that you saw if |
7:60 | you went to booking.com between say 2014 |
8:02 | and 2016 the order in which you saw the |
8:05 | hotels on booking.com when you made a |
8:07 | search that was my responsibility at the |
8:10 | time um then uh I got increasingly |
8:14 | interested in the ab test that uh that |
8:16 | booking was running and so I spent more |
8:18 | and more time looking at the tests that |
8:20 | other teams were running uh and joyas |
8:22 | Alvis at the time was was responsible |
8:25 | for that platform uh but he decided to |
8:27 | leave the company and and so I was asked |
8:29 | to to step forward and become the |
8:31 | product manager for the experimentation |
8:32 | platform and that you mentioned AB |
8:33 | smartly earlier so one of the reasons |
8:35 | I’m working with with AB smartly is that |
8:37 | join us Office later founded AB smartly |
8:40 | because he he realized that as started |
8:43 | as a consultant and he was asked time |
8:45 | and time again to build the same product |
8:48 | or the same platform that that |
8:50 | booking.com had to build it for other |
8:53 | companies and they realized that there |
8:54 | might be a market for selling it and so |
8:57 | with AB smartly what they’ve essentially |
8:59 | done is rebuild booking.com AB testing |
9:01 | platform but now anyone can buy it so I |
9:05 | because I have Affinity with that |
9:07 | platform I was responsible for the |
9:09 | growth of of that platform between 2016 |
9:11 | and |
9:12 | 2021 um and Jonas and I sort of like |
9:15 | between us have like two decades of |
9:17 | experience in that uh I think that’s |
9:19 | that was a sort of a natural fit to work |
9:22 | with them so that’s a little bit of how |
9:24 | I rolled into into experimentation hi |
9:27 | this is Romo Santiago from experiment |
9:28 | Nation if you’d like to connect with |
9:30 | hundreds of experimenters from around |
9:31 | the world consider joining our slack |
9:32 | Channel you can find the link in the |
9:34 | description now back to the episode |
9:36 | thanks thanks it’s a interesting story |
9:38 | and look um I’m particularly Keen in in |
9:42 | terms of learning |
9:44 | how |
9:46 | um you know you developed and scal that |
9:50 | culture of experimentation um at |
9:54 | booking.com you know I think our |
9:56 | audiences you know a lot of them would |
9:58 | know how to set up a |
9:60 | at least beginners would know how to set |
10:01 | of basic AB test using the the current |
10:04 | client side tools and sort of things |
10:06 | like that but um you know before |
10:09 | the podcast yesterday we were talking |
10:11 | about like you know how we |
10:13 | can you know just use a client side tool |
10:16 | but not credit culture and just have a |
10:18 | silo effect there and where it’s just |
10:20 | you know usually done through marketing |
10:21 | um so I won’t I won’t steal your funder |
10:24 | you discussed um how you scaled it |
10:26 | yesterday so it be good to explain it to |
10:27 | audiences and really like how to really |
10:30 | truly embed experimentation um not just |
10:32 | in one area of the company but like you |
10:35 | know through throughout the whole |
10:37 | company um so I’ll let you talk about |
10:40 | that yeah I mean that’s so that’s an |
10:42 | interesting topic that’s been like the |
10:45 | the necklace of my focus in the last few |
10:48 | years there’s many many things to talk |
10:50 | about I think one is that uh I don’t |
10:54 | think we should take booking as a as a |
10:56 | model nor should we take my experience |
10:58 | there as like a as vouching for my sort |
11:00 | of expertise in this area uh booking was |
11:03 | already very much an experimentation |
11:05 | culture when I joined like I didn’t |
11:07 | create that it was already there and I |
11:09 | think it was very much driven by the |
11:11 | founders and the people they hired and |
11:13 | the way they organized their teams it’s |
11:15 | not something that I created it’s |
11:16 | something that I nurtured um what I did |
11:20 | help was sort of taking at the time when |
11:23 | I joined that was very focused to the |
11:26 | web side of things so the the the |
11:28 | website was running a lot of experiment |
11:30 | but there’s many other aspects of of |
11:32 | booking account that were not and so you |
11:34 | can think of booking sort of divided by |
11:37 | different uh product areas uh one of |
11:40 | them being the sort of the website |
11:42 | that’s consumer facing but there’s also |
11:44 | a website is that is Hotel facing that |
11:46 | they use to sort of add rooms or change |
11:48 | availability and then there’s obviously |
11:51 | the customer care the call center |
11:53 | there’s many other aspects of that of |
11:54 | that business that were’re not really |
11:55 | running experiments when I when I |
11:57 | started um and so one of the things that |
11:59 | I I tried to do in in a time was there |
12:01 | it was like on the one hand sort of like |
12:03 | increase the quality of the experiments |
12:05 | that were being run in the website and |
12:07 | at the same times or like scale out that |
12:10 | approach to other parts of the |
12:11 | organization which mostly entailed on |
12:14 | the uh um figuring out technically how |
12:17 | to sort of uh move into those areas |
12:21 | because for example if you want to run |
12:23 | experiments in a call center you have to |
12:24 | figure out how to integrate into a a an |
12:28 | ivr like a phone system them you need to |
12:30 | figure out how to do a coin flip based |
12:31 | on a phone number you need to figure out |
12:33 | how to do metrics based on that so |
12:34 | there’s some technical challenges |
12:36 | related to that and then there’s some |
12:37 | organizational challenge related to how |
12:39 | do we educate those people how do we get |
12:40 | them on board like how do we how do we |
12:42 | help them understand that this is |
12:43 | important I think this is much easier |
12:45 | when you have a large share of the |
12:47 | organization that is already running |
12:48 | experiments and so one of the ways that |
12:50 | we scaled into partners for example was |
12:53 | to take people developers designers |
12:55 | product managers who were already |
12:57 | running experiments on the website on |
12:59 | the customer side of things and say hey |
13:01 | can you can you join one of the teams on |
13:03 | the partner side and help them |
13:05 | understand how to use experimentation as |
13:07 | part of product development and so like |
13:09 | we we could cross-pollinate between |
13:11 | these departments and sort of help them |
13:12 | help them scale out I think that that |
13:14 | made the job a lot easier one of the |
13:16 | reasons that I I left booking actually |
13:18 | was I I wanted to figure out like if I |
13:19 | go to a different company that does not |
13:22 | have this scale like how what other |
13:25 | challenges do I run into what is |
13:27 | difficult about doing it there and it’s |
13:28 | been actually very very very interesting |
13:30 | learning experience because I I found to |
13:32 | to your point like there’s a lot of |
13:34 | people who know including me like know |
13:36 | how to set up an experiment and it’s one |
13:39 | thing to nurture a culture of |
13:40 | experimentation in a company that |
13:41 | already has that inertia but it’s a very |
13:43 | different Beast to start with a company |
13:45 | that not have that and sort of figure |
13:47 | out what are the boundary conditions |
13:48 | that we need to put in place in order to |
13:50 | make experimentation more possible and |
13:52 | one of the things that i’ I found is |
13:54 | that um a lot of it revolves |
13:58 | around um |
13:60 | the how teams decide what to work on |
14:05 | yeah and how they collaborate between |
14:07 | different functions and so um when I |
14:10 | joined |
14:12 | Vista they were they were organized in |
14:14 | such a way that there was a data |
14:16 | organization there was an engineering |
14:17 | organization there was a product |
14:18 | organization and a marketing |
14:19 | organization that included design and to |
14:22 | me this was very surprising because for |
14:24 | for eight years of booking I had worked |
14:26 | in a model where there were cross |
14:28 | functional teams and so if you think |
14:30 | about ranking my my responsibility when |
14:33 | when as was a product manager I had a |
14:35 | team that included data scientists |
14:37 | developers and designers as part of one |
14:40 | single team and our job was related to |
14:43 | make to making it easier for people to |
14:45 | find a hotel that they could stay at |
14:47 | yeah and so we we were not organized |
14:50 | around craft but we were organized |
14:52 | around a customer objective and we and |
14:55 | in I could argue to leadership and say |
14:58 | well in order to accomplish this |
15:00 | objective here are the skills that I |
15:02 | need right I need a designer because I |
15:05 | need to explain to people why this is |
15:06 | the right hotel for them I need a data |
15:09 | scientist because I need to build |
15:10 | machine learning models that help pick |
15:11 | the right the right hotel right and so |
15:14 | that given the objective there can be |
15:16 | conversations about which skills are are |
15:18 | needed on the team and and these teams |
15:20 | were long lived and they are considered |
15:22 | to be primary teams so if you had |
15:24 | approached my team members at the time |
15:26 | and asked them like which team are you |
15:27 | part of they would say the ranking team |
15:29 | or they would say the search team now |
15:31 | when I joined Vista I noticed that when |
15:32 | you ask an engineer which team are you |
15:34 | part of they would name their |
15:35 | engineering team and you ask them what |
15:37 | do you own they would name the service |
15:39 | that they owned and the same for the |
15:41 | data data analyst and the same for |
15:42 | product managers like they weren’t |
15:44 | thinking in terms of problem customer |
15:47 | problem focused teams they were thinking |
15:50 | in terms of craft teams and so this |
15:53 | makes experimentation much more |
15:55 | difficult because there’s many more |
15:56 | communication Hops and there’s much more |
15:58 | room for misalignment between the |
15:60 | different crafts even though we’re all |
16:01 | working together to solve the same |
16:02 | customer problem and an experiment |
16:05 | serves to figure out whether the |
16:06 | customer problem is solved yeah but but |
16:08 | if we’re not all aligned on what |
16:10 | customer problem we’re solving together |
16:11 | then the experiment doesn’t really have |
16:13 | value and and and no one really sees a |
16:15 | need to experiment in the first place |
16:17 | and so I started to think that we needed |
16:19 | to change the organization of of Vista |
16:21 | towards a more uh cross functional |
16:24 | product team organization and and I will |
16:28 | not in anyway claimed that I was the one |
16:31 | who drove this change like there were |
16:33 | many people with me that had the same |
16:35 | opinion I have tried to push for this |
16:37 | for the last three years and I’m happy |
16:39 | to report that the things are going |
16:40 | really well and that we’re slowly moving |
16:42 | towards this product operating model um |
16:45 | and but I do think it’s one of the |
16:46 | boundary conditions that needs to be in |
16:48 | place for experimentation to be really |
16:51 | scalable and so to your earlier point |
16:53 | like I think a lot of organizations and |
16:55 | and especially the older so like the the |
16:58 | first generation of experimentation |
16:59 | platforms they are very marketing |
17:01 | Centric and they’re very marketing |
17:03 | focused and the pitch is essentially uh |
17:06 | put this script on your website and you |
17:08 | will never have to talk to it again to |
17:10 | make to to run an experiment that’s |
17:12 | that’s the pitch and I think that is |
17:14 | marvelous |
17:15 | technology but it sort of completely |
17:19 | circumvents the core problem that we |
17:21 | need to solve which is that marketing |
17:23 | and it should not be separate in the |
17:25 | first place right and they usually kind |
17:28 | of into taking this well they can’t be |
17:30 | antagonistic toward towards each other |
17:32 | because marketing will be like okay |
17:34 | We’ve ran these experiments um here’s a |
17:36 | few winning experiments that we want it |
17:40 | produ to |
17:41 | productionize and then for it it’s like |
17:44 | they’re rolling their eyes and they |
17:45 | thinking oh well I’ll just got to I’ll |
17:47 | just add to my backlog of 100 other |
17:49 | items on my to-do |
17:51 | list you know what I mean yeah yeah yes |
17:54 | and and and they’re PRI prioritizing in |
17:56 | different ways right so so one of the |
17:57 | reasons that marketing struggles to put |
17:59 | stuff on the it backlog is it just have |
18:01 | different has different priorities and |
18:02 | that’s why why I think that these people |
18:04 | should be together in one team and have |
18:06 | the same set of objectives ideally |
18:08 | around a customer because then the the |
18:10 | conversation changes I I mean you tell |
18:13 | you an anecdote we had a uh we had a re |
18:16 | a Booking.com reunion a few months ago |
18:18 | where like people who worked for |
18:19 | booking.com over the years sort of got |
18:21 | together and sort of chatted about their |
18:22 | life after booking it’s very interesting |
18:24 | because it’s very much a almost like a |
18:26 | cult |
18:27 | following if I might say so uh and uh I |
18:31 | I was talking to a group of Engineers |
18:34 | and and they were all surprised about |
18:36 | sort of the engineering cultures that |
18:38 | they found outside uh of bookan aom and |
18:42 | I said oh yeah I I’ve noticed the same |
18:44 | when when I say to people like at |
18:48 | booking it is very it was very normal |
18:51 | that you go to an engineer and you say I |
18:53 | would like you to put like on this on |
18:56 | these landing pages I want you to change |
18:58 | this thing |
18:59 | yeah it is very common for an engineer |
19:02 | to then say well that page gets about |
19:05 | 10,000 visitors a day which translates |
19:07 | into about |
19:09 | 0.001% of our total revenue so if I were |
19:12 | to spend two days of work on this like |
19:14 | that is already more expensive than a |
19:17 | 100% uplift on conversion that we would |
19:20 | that we would need to to cover like the |
19:22 | base cost and there’s no way we would |
19:23 | get that because the change that you’re |
19:25 | proposing like psychologically I don’t |
19:27 | see why that would cause that much |
19:29 | changing conversion so I’m not going to |
19:30 | prioritize |
19:32 | it and and I’m sort of look around the |
19:35 | look around the circle of Engineers and |
19:36 | they all like yes that is that is |
19:39 | exactly what is missing from the |
19:41 | engineering cultures that we see in |
19:42 | other organizations where Engineers |
19:44 | would not they are not expected to and |
19:47 | so they do not talk in business terms |
19:51 | and I don’t think this is a lack of |
19:52 | skill they just really just thinking how |
19:54 | do we technically set up this test but |
19:56 | then not thinking like correct um like a |
19:60 | would where it’s like okay we get 10,000 |
20:02 | uniques per day or month or whatever and |
20:05 | um you know these are the business |
20:07 | implications if I set up this test if |
20:09 | it’s high then I’ll just put it high up |
20:11 | my back in my my Q in my road map if |
20:14 | it’s pretty low then I’ll just push back |
20:17 | against you and say no this is not good |
20:18 | or put it lower down the the the um the |
20:21 | road map |
20:24 | um yeah like that’s um that’s that’s not |
20:27 | something I’ve experienced with |
20:28 | Engineers cuz usually they just like |
20:29 | okay I’ll just I’ll build it for you |
20:30 | because you you told me to yes and I |
20:33 | think that’s if that’s the working model |
20:35 | then getting into an experimentation |
20:36 | space is is difficult because you’re |
20:38 | just speaking different languages and |
20:40 | that’s why I I contrast this with a sort |
20:42 | of the newer generation of tools AB |
20:44 | smartly is one of them there there’s |
20:46 | others of course and they they’re all |
20:48 | founded by people who came from these um |
20:52 | these these organizations that are |
20:54 | already running experimentation at |
20:55 | Scales so so Facebook Airbnb booking.com |
20:59 | like these are all companies that are |
21:00 | running experimentation scale people |
21:02 | coming from those companies they design |
21:04 | experimentation Platforms in a |
21:05 | completely different way because they |
21:07 | aren’t thinking about it in terms of put |
21:09 | this pixel on your website so you can |
21:10 | avoid it they’re thinking of it in terms |
21:13 | of it is at the table when you’re |
21:16 | designing this this experiment like they |
21:18 | are involved in the design and the |
21:20 | execution of the experiment the vast |
21:22 | majority of experiments that are run |
21:23 | at.com are run by Engineers like they’re |
21:26 | they’re most of the tests are set up by |
21:28 | by an engineer in person rather than bu |
21:30 | a product person so I think this is like |
21:33 | this is the biggest differ when we talk |
21:34 | about the experimentation Gap I this |
21:37 | article that came out I think a year |
21:39 | ago like this is a large part of the Gap |
21:42 | like it’s not just scale it’s scale that |
21:45 | comes from a different way of working |
21:47 | and a different way of working that |
21:49 | encourages especially engineering to |
21:51 | think in a different way yeah so um |
21:55 | maybe for our audiences um who are |
21:57 | listening to this like say they are are |
21:59 | in that sort of |
22:00 | um they product manager or they’re C |
22:03 | Specialist or whatnot and they’re in |
22:06 | that sort of um early to midstage of |
22:10 | experimentation maturity and you know um |
22:14 | they still relying on just the client |
22:15 | side tool and they’re thinking okay um |
22:18 | you know if I really want to |
22:20 | embed a real culture of experimentation |
22:24 | I’ve got to go engineering first and um |
22:28 | have a tool that’s going to |
22:30 | be servici side cuz it’s going to be |
22:33 | faster and Engineers you know they care |
22:36 | about speed and those sort of things um |
22:40 | what kind of advice would you give to |
22:42 | those people um from your experience |
22:44 | looking at Vista where um they want |
22:47 | they’re in that sort of Silo |
22:50 | structured um to go into the cross |
22:53 | pollination structure like you you |
22:56 | mentioned at booking.com I mean |
22:59 | you have to restructure the the whole |
23:01 | damn organization or how the hell do you |
23:03 | do it I mean that’s a scary proposition |
23:06 | that’s a scary proposition um I |
23:10 | think please take my advice here with a |
23:12 | huge grain of salt yeah I think I think |
23:15 | there if you you have to adjust your |
23:17 | prior this this is a company that did |
23:19 | not have a scaled organization of |
23:21 | culture uh of experimentation at the |
23:23 | time that I joined um but they did hire |
23:27 | me as I and so like that already tells |
23:30 | you something about like the leadership |
23:32 | Buy in that there was for for |
23:34 | experimentation yeah at the time right |
23:37 | because I think I talked to a lot of our |
23:38 | other organization and just merely |
23:40 | suggesting that you should hire a |
23:41 | director of experimentation is already |
23:43 | like a bridge too far for many many |
23:45 | organizations in this case we had a a c |
23:48 | Vista has a CEO that is very much bought |
23:51 | into uh the the idea that |
23:53 | experimentation is an important part of |
23:55 | product development and the same for CTO |
23:57 | and CPO there very strong uh leadership |
24:00 | Buy in there’s also a long history of |
24:03 | experimentation at Vista it’s just very |
24:05 | decentralized so so I didn’t know this |
24:07 | when I joined but Vista actually started |
24:09 | running experiments before booking.com |
24:10 | did so booking started in 2005 February |
24:14 | I believe yeah and Vista ran their first |
24:16 | experiment somewhere in at the end tail |
24:17 | end of 20 2004 so they were just barely |
24:21 | sooner um but some somehow it never h it |
24:26 | did scale actually but somehow it it |
24:29 | like uh slumped again and I was hired |
24:32 | sort of as the slump was sort of sort of |
24:35 | paning out and now now we’re going going |
24:36 | back up again um and so I don’t think |
24:40 | Vista is like a a a good template for |
24:44 | like how other organizations can do this |
24:46 | because there I had a lot of wind in my |
24:48 | sales |
24:50 | um given that like when I joined there |
24:54 | they did not have a product operating |
24:55 | model um but me coming from that uh that |
24:60 | is the only way that I know how to |
25:01 | operate like I don’t really know how you |
25:05 | would make things work if you don’t have |
25:06 | a cross functional team that has all the |
25:08 | skills that they need to sort of solve |
25:10 | the problems that their T to solve so |
25:12 | one of the first things I did was sort |
25:13 | of explain to people like how I saw |
25:17 | teams should function uh even if that |
25:21 | was not the reality of organization at |
25:22 | the time uh sort of advocating for that |
25:25 | model and then applying it within my own |
25:28 | team like I I had a very very small team |
25:31 | of I believe three analysts or two |
25:33 | analysts when I joined and one of the |
25:35 | first things I did was I I went to an |
25:37 | engineering leader and said well uh I |
25:39 | want to try this new model where we take |
25:41 | those two analysts and we take the two |
25:43 | Engineers that you have assigned to this |
25:45 | topic and we put them together and we |
25:46 | tell them that they are one team and we |
25:48 | give them one name and we give them one |
25:50 | identity and together we give them |
25:52 | objectives and I was lucky to find an |
25:54 | engineering leader that that bought into |
25:56 | that Paradigm and said yes let’s do it |
25:58 | and so together we set the objectives |
25:59 | for this one team and started treating |
26:01 | them as if they were |
26:03 | one and very quickly then I hired Uh |
26:06 | Kevin Anderson as a product manager I |
26:08 | found sort of like an organizational |
26:10 | loophole where I could hire a a product |
26:12 | manager as part of my organization even |
26:14 | though I wasn’t in product by calling |
26:16 | him a data product manager that was the |
26:18 | that was the loophole like Kevin very |
26:22 | much operates as a product manager he’s |
26:24 | excellent um but he was called a data |
26:26 | product manager because that allowed me |
26:28 | to then hire in into the team and so we |
26:30 | could we could sort of treat them as a |
26:32 | one like as an island of a an an |
26:35 | empowered product team within within a |
26:37 | larger organization that was still |
26:38 | grappling with the with the concept and |
26:40 | use them as a as a role model and go to |
26:43 | leadership and say like hey here’s how |
26:45 | this works see the artifacts that that |
26:47 | they are creating and again Kevin is |
26:49 | excellent for this because he’s a he’s |
26:50 | very much a documentation driven person |
26:52 | he will write things down and he does so |
26:54 | very publicly and so all of the |
26:56 | artifacts that this team was creating |
26:57 | that we could also show them into |
26:58 | leadership and say hey here’s what the |
26:60 | proit team is doing and here’s how and |
27:01 | here’s the results they’re having and |
27:02 | here’s how they’re tracking their kpis |
27:04 | and so that becomes a role model and |
27:06 | example that you can use for rest |
27:07 | organization at the same time I I have |
27:09 | to say like I very much had to wind in |
27:11 | my sales because we have a a chief |
27:13 | product officer who’s very much bought |
27:14 | into the same Paradigm we worked at |
27:16 | Amazon before uh we have a CTO that was |
27:19 | very much bought into this CEO that also |
27:21 | came to from Amazon so we had all these |
27:23 | people who understood this model um and |
27:27 | then we got help from uh from outside |
27:29 | consultancy uh including Marty Kagan |
27:32 | who’s uh who’s known for what what is |
27:34 | called the product operating model which |
27:36 | I was not familiar with but when I read |
27:39 | all the material it’s like oh yes that’s |
27:40 | how booking aom operated so so it feels |
27:44 | like there’s some sort of convergent |
27:45 | evolution where like a lot of these |
27:47 | companies are finding out that if you |
27:48 | put people of different crafts together |
27:50 | and you give them a customer problem to |
27:52 | solve and you give them flexibility on |
27:54 | how to how to sort of explore solutions |
27:56 | that is a very effective model |
27:59 | and Marty has done some great work in |
28:00 | sort of documenting how the differences |
28:03 | between the different companies and sort |
28:04 | of the commonalities between different |
28:05 | companies approaching this I was lucky |
28:07 | to be hired into a company that did this |
28:09 | now I’m I’m lucky to be in a company |
28:11 | that is slow that is not slowly that’s |
28:14 | rapidly adjusting to the style of |
28:16 | working and so like three years ago we |
28:19 | were completely siloed now like we are |
28:21 | completely uh product operating mod |
28:24 | there’s no more separation between uh |
28:27 | engineering uh product and and design |
28:29 | they’re all they’re all one so terms of |
28:33 | um cultural |
28:36 | change |
28:38 | um would you say that there a lot of um |
28:42 | growing growing pangs and in in in going |
28:46 | from The Silo model to |
28:48 | the the sort of cross Silo model like I |
28:51 | mean oh it’s yeah do you get people |
28:53 | leaving for these reasons you get people |
28:55 | that are like oh that’s company’s |
28:57 | changing I mean do |
28:59 | is is a common so there so so two things |
29:01 | there are definitely growing pants so so |
29:03 | so it’s not easy I’m not at all sugges |
29:05 | and it it does take time like this this |
29:07 | is a process is going to take years yeah |
29:10 | it it does and I think it’s difficult |
29:12 | for two reasons like people have to find |
29:14 | a new way of working they have to new |
29:16 | find new rhythms and so we have a we |
29:18 | have a separate part of the organization |
29:20 | that is specifically tasked with |
29:22 | figuring out those organizational |
29:23 | rhythms and they also operate as product |
29:26 | teams and so they set their own |
29:27 | objectives they make meure their |
29:28 | outcomes in terms of how teams are |
29:30 | operating and this is not treated as a |
29:32 | as a like a one-off program this is an |
29:35 | ongoing Improvement thing with in the |
29:36 | organization like how do we work that’s |
29:38 | one I think the other is that you said |
29:40 | are people leaving |
29:42 | um I I don’t think we I don’t think we |
29:45 | we see like people leaving in dros like |
29:48 | every company has attrition right and no |
29:50 | different I do see like people have to |
29:53 | stretch themselves because this model |
29:54 | does ask different things so to go back |
29:57 | to the earlier example about engineers |
29:58 | right booking called these t-shapes you |
30:01 | want you want people who are not just |
30:03 | very good in one craft you want people |
30:06 | who are good in one craft but then know |
30:09 | a little bit or know enough about each |
30:11 | other craft to have conversations with |
30:13 | their peers right a an engineer should |
30:16 | know enough about the business and about |
30:18 | the metrics of the product they’re |
30:20 | responsible for to have conversations |
30:22 | with a designer and a product manager |
30:23 | about the product they’re working on and |
30:25 | a designer should know enough about the |
30:27 | engineering problems to be able to talk |
30:29 | to the engineer about solutions to |
30:31 | designs that they have come up with |
30:33 | right and so this requires people to |
30:35 | think a little bit outside of their |
30:36 | craft and there are people who want this |
30:38 | and there are people who don’t want this |
30:39 | I think that is that is a shift we’re |
30:41 | slowly seeing and I think it’s one of |
30:43 | the reasons that you see that not only |
30:44 | are there Growing Pains there’s also uh |
30:48 | I think it was CRZ was it this famous um |
30:51 | quote like the future is already here |
30:53 | it’s just not evenly |
30:55 | distributed of that yeah right it’s we |
30:58 | have teams are operating really well in |
30:60 | this product operating model and we have |
31:01 | teams are really struggling in this |
31:03 | product operating model and so we’re |
31:05 | trying to find ways again |
31:07 | cross-pollinate like we did a booking |
31:08 | like how do we take people who are very |
31:10 | effective at this like put them in teams |
31:12 | that are struggling it’s like how do we |
31:13 | how do we make a rising tide Floats or |
31:15 | boats and that is just a process that we |
31:17 | need to go through I in no way |
31:19 | suggesting that this is easy um and your |
31:22 | again even if you have leadership eying |
31:25 | it is still not easy you’ve mentioned |
31:27 | this a few times that you know you had |
31:29 | leaders like the seite execs um having |
31:33 | huge buying and I think that’s sounds |
31:36 | like that’s a |
31:38 | common it’s a a huge factor for for |
31:42 | driving this don’t you think like if |
31:43 | it’s not coming from the Top If It’s s a |
31:46 | sort of bolted on the side like |
31:47 | marketing it’s always going to be in |
31:49 | that sort of |
31:50 | Silo um sort of Paradigm where it’s like |
31:54 | marketing versus technology or whatever |
31:56 | but if it’s coming from the top down and |
31:58 | it’s like you know really embedded from |
32:01 | the exacts and it’s flowing throughout |
32:04 | the whole organization not saying it’s |
32:06 | easy but that sounds like that’s a key |
32:09 | element that you need would you agree |
32:13 | like I I would say if you don’t have it |
32:15 | you think you have it yes and that’s not |
32:17 | sufficient it’s not sufficient because |
32:19 | because I would say yes leadership needs |
32:21 | to be byy in and Leadership needs to |
32:23 | understand that it has implications for |
32:25 | how they organize their their business |
32:28 | yeah and how they operate |
32:31 | themselves and and this is the part that |
32:34 | is not |
32:35 | easy you might have a leader that says |
32:37 | yes I want you to run experiments |
32:39 | marketing go run experiments but what |
32:41 | we’ll keep the engineering organization |
32:44 | aside we’re not touching it then they’re |
32:46 | not bought in enough yeah then you kind |
32:49 | of end up being siloed um yeah so look |
32:54 | it’s um unfortunately we can’t go on for |
32:56 | too long but um any any closing thoughts |
32:59 | for our |
33:04 | audiences uh I I hope I hope what I |
33:07 | share |
33:08 | helps I hope I’ve also given you enough |
33:11 | caveats to say like this is my |
33:12 | experience at two companies so so like |
33:16 | and I I I again I’ve been very lucky |
33:18 | I’ve had the wind in my sales for a long |
33:19 | time I like if you struggle with this i’ |
33:22 | I’d be happy to chat like to hear hear |
33:23 | the things you’re struggling with I |
33:25 | maybe I maybe I can help so please do |
33:26 | reach out let’s talk |
33:29 | we are all learning right this like we |
33:31 | are experimentation people let’s |
33:33 | experiment with how we do the |
33:34 | organization around experiment um how |
33:37 | can people contact you Lucas sure uh so |
33:40 | just go to my website my name. NL NL for |
33:43 | Netherlands uh and then there’s a link |
33:45 | at the top that allows you to book some |
33:46 | time in my calendar or send me an email |
33:48 | whatever you want that’s the quickest |
33:50 | way awesome thanks for joining the |
33:52 | podcast thank you for having me it was |
33:54 | great bye see you hi this is Romo |
33:57 | Santiago from experiment Nation if you’d |
33:59 | like to connect with hundreds of |
33:60 | experimenters from around the world |
34:01 | consider joining our slack Channel you |
34:03 | can find the link in the description now |
34:04 | back to the episode |
If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.
Advertisment
Categories
- Adventures in Experimentation (9)
- Analysis (2)
- Announcements (91)
- Ask the Experiment Nation (11)
- Certified CROs (193)
- Community (2)
- Conversation Starters (2)
- CR-No (5)
- CRO Handbook (4)
- CRO Memes (18)
- Experiment Nation: The Conference RELOADED (41)
- Experimentation from around the world (2)
- For LinkedIn (164)
- Frameworks (2)
- Growth (14)
- ICYMI (2)
- Interviews with Experimenters (209)
- Management (1)
- Marketing (11)
- Opinion (8)
- Podcasts with Experimenters (15)
- Point/Counterpoint (1)
- Product Experimentation (5)
- Product Management (9)
- Profile (9)
- Question of the week (5)
- Sponsored by Experiment Nation (1)
- The Craft of Experimentation (1)
- The Cultures of Experimentation (1)
- The weekly buzz (13)
- The year in review (1)
- Uncategorized (352)
- Weekly Newsletter (184)
- Where I Started (2)
- Navigating App Optimization with Ekaterina (Shpadareva) Gamsriegler - October 18, 2024
- Building Your CRO Brand with Tracy Laranjo - October 11, 2024
- Have a narrow and well-defined ICP with Dr. Else van der Berg - October 4, 2024