Leadership buy-in is essential with Lukas Vermeer Experiment Nation

AI-Generated Summary

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight. Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration. Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically. Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction. Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

AI-Generated Transcript

Time (Duration)	Text
0:00	and it’s one thing to nurture a culture
0:02	of experimentation in a company that
0:03	already has that inertia but it’s a very
0:05	different Beast to start with a company
0:07	that does not have that and sort of
0:08	figure out what are the boundary
0:10	conditions that we need to put in place
0:11	in order to make experimentation more
0:13	possible and one of the things that I I
0:15	found is that um a lot of it revolves
0:20	around um the how teams decide what to
0:26	work on and how they collaborate between
0:29	different functions
0:30	[Music]
0:36	hey everybody it’s Richard here from
0:38	discrimination podcast today I have
0:42	Lucas verir with us today uh it’s been a
0:46	while since I’ve booked this
0:48	conversation with you uh for those who
0:50	don’t know
0:53	Lucas Lucas was the experimentation um
0:57	director at booking.com so as you know
1:01	booking.com is one of the um yeah the
1:04	big
1:05	experimentation
1:07	um Role Models so to speak for how to
1:11	you know create a culture of
1:12	experimentation how to scale that and um
1:15	he was there for how long were you there
1:17	for 10 years eight
1:20	years yeah so you basically started off
1:23	it looks like from LinkedIn you started
1:24	off as a diet scientist and pretty much
1:27	mov into various roles of product
1:28	manager and um to finally director and
1:32	it looks like now you’re pretty much um
1:34	doing several things you’re obviously on
1:37	the public speaking circuit so you’re
1:39	invited to pretty much every conference
1:41	known to man
1:43	uh every siero conference out there
1:48	um no you’re an advisor on ab smartly
1:52	and you’re currently acting as the
1:55	Director of
1:57	experimentation at Vista so welcome to
2:01	the podcast Lucas thank you great to be
2:04	here look a nice summary of my career or
2:08	most of it anyway I’m I’m really hying
2:10	that up it AR I aren’t
2:11	[Laughter]
2:13	I um yeah I usually like to you know
2:17	have a quick chat about
2:20	um you know your education and um you
2:23	know just prob going over that and maybe
2:26	how you got into experimentation because
2:28	everyone’s got some Pathways and um by
2:31	some people fall into by accident some
2:33	people scratch their own itch and um
2:36	yeah maybe just give us a brief of your
2:39	education and how you got into this uh
2:41	crazy
2:42	field huh it’s not crazy what you call
2:46	crazy I’m not crazy are you
2:48	crazy I think we’re all how did I get
2:50	into
2:54	um so how much education do you want so
2:57	my my parents were both academics I
2:59	think that I think that helps um so so
3:02	from from a young age but they were both
3:05	they were both linguists so very
3:07	different field um I think I got lucky
3:10	at a very young age that my my father
3:13	and his faculty there was one colleague
3:15	who was working on language in computers
3:18	or computers parsing language I think
3:20	it’s is relevant because now with these
3:23	large language models like the world has
3:24	shifted right it’s completely different
3:26	perspective on sort of how computers can
3:29	do language but at the time was still
3:30	very much structural uh still like
3:33	computers trying to parse grammar in a
3:35	very explicit way and so that got me
3:37	first interested in computers and and in
3:39	complex systems uh then uh in high
3:44	school I was very interested in in
3:45	biology in economics or as as as other
3:48	examples of complex systems but then
3:50	when it came to sort like going I wanted
3:52	to go to university I I wanted to study
3:55	uh a field um I couldn’t really find
3:58	anything un like I liked until I ran
3:60	into to uh a a
4:03	specific really an experiment at the
4:06	University of utre uh in in the
4:08	Netherlands which they called the
4:10	technical artificial intelligence and it
4:13	was on the Confluence between psychology
4:16	and uh computer computer science so I
4:18	would have both psychology uh courses as
4:21	well as uh as computer science courses
4:23	and this was really early early days for
4:26	or not early days but it was a different
4:27	era for for machine learning right so so
4:30	I learned a lot of the basic
4:31	reinforcement learning algorithms so
4:33	like Bandits were obviously part of the
4:35	curriculum and all these things but I
4:37	learned them from a computer science
4:38	background yeah and then when I hit the
4:41	hit the job market uh I went into
4:43	Consulting at first uh did some business
4:46	intelligence work and mostly helped uh
4:50	companies who were trying to use
4:52	reinforcement learning for um marketing
4:54	automation or or next best action models
4:58	it’s like figuring out which customers
4:59	want what on what and I think that was
5:02	really the first time that I ran into
5:04	the concept of a control group because
5:07	one of the one of the tools that I
5:09	worked with had a control group built in
5:11	so you could see how well the machine
5:13	learning model was was functioning
5:15	compared to some ground truth so for
5:18	most customers that would either be a
5:20	fixed calendar or a fixed offer or some
5:23	sort of popularity ranking but something
5:26	that was a simpler model than than the
5:27	complex machine learning or complex the
5:30	time it was complex now we would say
5:31	like it was a basic Bandit um and so
5:35	that I think that was the first time I
5:36	ran into a control group it’s also the
5:38	first time that I ran into trouble with
5:40	sort of explaining why that was useful
5:42	to stakeholders because I I would have
5:45	clients who ask me to turn off the
5:47	control group or who would would ask me
5:50	like why are we showing customers an
5:51	inferior version and so like this these
5:54	were the first conversations I had
5:55	around sort of like the importance of
5:57	testing and sort of the the the value of
5:60	this
6:02	and very much as a consultant at the
6:04	time right so if if a client asked to
6:06	turn off the control group I would try
6:07	to convince them not to but I didn’t
6:09	really have much much to say in the
6:12	matter then I uh got lucky really I mean
6:15	this the first I’ve gotten lucky many
6:17	times in my career but this first time I
6:19	got lucky I think was uh when I ran into
6:23	a man who worked for booking.com and
6:26	that that was very much in the scaleup
6:27	phase at the at the time yeah and they
6:29	were looking for someone to to work on
6:32	their recommendation system models uh
6:35	their
6:36	Bandits uh and so I joined uh booking in
6:39	2013 I think as a as their first data
6:42	scientist and because I was the first
6:45	data scientist the and there and they
6:46	were so much in scaleup mode the HR
6:48	department actually didn’t have a job
6:50	description or a job role for uh data
6:53	scientists yet and so they they made me
6:55	a product manager because that in terms
6:58	of like role description and salary
6:60	bands that seemed to be the closest to
7:02	to what I was looking for and so I are
7:05	like my first interactions with product
7:07	were like having the product manager
7:09	title while doing a while doing a data
7:12	scientist job and so for the first I
7:14	think two years of booking I sort of
7:16	worked on their recommendation system
7:17	models and because this was very much in
7:19	scaleup mode uh you can imagine like
7:22	they were still operating very much as a
7:24	startup and so as a data scientist I
7:27	wasn’t just building models in like a a
7:29	data or like many companies function now
7:32	but I was also doing some of the
7:33	engineering I was also doing some of the
7:36	product management I was also doing some
7:37	of the stakeholder management so so in
7:39	effect like the product manager role
7:41	wasn’t all that strange like I was doing
7:43	some of those tasks and so it felt very
7:45	natural for me to then go from that into
7:49	a more formal product manager role when
7:50	we started hiring more people for the
7:52	team I became the product manager for
7:54	recommendation systems and later for for
7:56	ranking uh so the order that you saw if
7:60	you went to booking.com between say 2014
8:02	and 2016 the order in which you saw the
8:05	hotels on booking.com when you made a
8:07	search that was my responsibility at the
8:10	time um then uh I got increasingly
8:14	interested in the ab test that uh that
8:16	booking was running and so I spent more
8:18	and more time looking at the tests that
8:20	other teams were running uh and joyas
8:22	Alvis at the time was was responsible
8:25	for that platform uh but he decided to
8:27	leave the company and and so I was asked
8:29	to to step forward and become the
8:31	product manager for the experimentation
8:32	platform and that you mentioned AB
8:33	smartly earlier so one of the reasons
8:35	I’m working with with AB smartly is that
8:37	join us Office later founded AB smartly
8:40	because he he realized that as started
8:43	as a consultant and he was asked time
8:45	and time again to build the same product
8:48	or the same platform that that
8:50	booking.com had to build it for other
8:53	companies and they realized that there
8:54	might be a market for selling it and so
8:57	with AB smartly what they’ve essentially
8:59	done is rebuild booking.com AB testing
9:01	platform but now anyone can buy it so I
9:05	because I have Affinity with that
9:07	platform I was responsible for the
9:09	growth of of that platform between 2016
9:11	and
9:12	2021 um and Jonas and I sort of like
9:15	between us have like two decades of
9:17	experience in that uh I think that’s
9:19	that was a sort of a natural fit to work
9:22	with them so that’s a little bit of how
9:24	I rolled into into experimentation hi
9:27	this is Romo Santiago from experiment
9:28	Nation if you’d like to connect with
9:30	hundreds of experimenters from around
9:31	the world consider joining our slack
9:32	Channel you can find the link in the
9:34	description now back to the episode
9:36	thanks thanks it’s a interesting story
9:38	and look um I’m particularly Keen in in
9:42	terms of learning
9:44	how
9:46	um you know you developed and scal that
9:50	culture of experimentation um at
9:54	booking.com you know I think our
9:56	audiences you know a lot of them would
9:58	know how to set up a
9:60	at least beginners would know how to set
10:01	of basic AB test using the the current
10:04	client side tools and sort of things
10:06	like that but um you know before
10:09	the podcast yesterday we were talking
10:11	about like you know how we
10:13	can you know just use a client side tool
10:16	but not credit culture and just have a
10:18	silo effect there and where it’s just
10:20	you know usually done through marketing
10:21	um so I won’t I won’t steal your funder
10:24	you discussed um how you scaled it
10:26	yesterday so it be good to explain it to
10:27	audiences and really like how to really
10:30	truly embed experimentation um not just
10:32	in one area of the company but like you
10:35	know through throughout the whole
10:37	company um so I’ll let you talk about
10:40	that yeah I mean that’s so that’s an
10:42	interesting topic that’s been like the
10:45	the necklace of my focus in the last few
10:48	years there’s many many things to talk
10:50	about I think one is that uh I don’t
10:54	think we should take booking as a as a
10:56	model nor should we take my experience
10:58	there as like a as vouching for my sort
11:00	of expertise in this area uh booking was
11:03	already very much an experimentation
11:05	culture when I joined like I didn’t
11:07	create that it was already there and I
11:09	think it was very much driven by the
11:11	founders and the people they hired and
11:13	the way they organized their teams it’s
11:15	not something that I created it’s
11:16	something that I nurtured um what I did
11:20	help was sort of taking at the time when
11:23	I joined that was very focused to the
11:26	web side of things so the the the
11:28	website was running a lot of experiment
11:30	but there’s many other aspects of of
11:32	booking account that were not and so you
11:34	can think of booking sort of divided by
11:37	different uh product areas uh one of
11:40	them being the sort of the website
11:42	that’s consumer facing but there’s also
11:44	a website is that is Hotel facing that
11:46	they use to sort of add rooms or change
11:48	availability and then there’s obviously
11:51	the customer care the call center
11:53	there’s many other aspects of that of
11:54	that business that were’re not really
11:55	running experiments when I when I
11:57	started um and so one of the things that
11:59	I I tried to do in in a time was there
12:01	it was like on the one hand sort of like
12:03	increase the quality of the experiments
12:05	that were being run in the website and
12:07	at the same times or like scale out that
12:10	approach to other parts of the
12:11	organization which mostly entailed on
12:14	the uh um figuring out technically how
12:17	to sort of uh move into those areas
12:21	because for example if you want to run
12:23	experiments in a call center you have to
12:24	figure out how to integrate into a a an
12:28	ivr like a phone system them you need to
12:30	figure out how to do a coin flip based
12:31	on a phone number you need to figure out
12:33	how to do metrics based on that so
12:34	there’s some technical challenges
12:36	related to that and then there’s some
12:37	organizational challenge related to how
12:39	do we educate those people how do we get
12:40	them on board like how do we how do we
12:42	help them understand that this is
12:43	important I think this is much easier
12:45	when you have a large share of the
12:47	organization that is already running
12:48	experiments and so one of the ways that
12:50	we scaled into partners for example was
12:53	to take people developers designers
12:55	product managers who were already
12:57	running experiments on the website on
12:59	the customer side of things and say hey
13:01	can you can you join one of the teams on
13:03	the partner side and help them
13:05	understand how to use experimentation as
13:07	part of product development and so like
13:09	we we could cross-pollinate between
13:11	these departments and sort of help them
13:12	help them scale out I think that that
13:14	made the job a lot easier one of the
13:16	reasons that I I left booking actually
13:18	was I I wanted to figure out like if I
13:19	go to a different company that does not
13:22	have this scale like how what other
13:25	challenges do I run into what is
13:27	difficult about doing it there and it’s
13:28	been actually very very very interesting
13:30	learning experience because I I found to
13:32	to your point like there’s a lot of
13:34	people who know including me like know
13:36	how to set up an experiment and it’s one
13:39	thing to nurture a culture of
13:40	experimentation in a company that
13:41	already has that inertia but it’s a very
13:43	different Beast to start with a company
13:45	that not have that and sort of figure
13:47	out what are the boundary conditions
13:48	that we need to put in place in order to
13:50	make experimentation more possible and
13:52	one of the things that i’ I found is
13:54	that um a lot of it revolves
13:58	around um
13:60	the how teams decide what to work on
14:05	yeah and how they collaborate between
14:07	different functions and so um when I
14:10	joined
14:12	Vista they were they were organized in
14:14	such a way that there was a data
14:16	organization there was an engineering
14:17	organization there was a product
14:18	organization and a marketing
14:19	organization that included design and to
14:22	me this was very surprising because for
14:24	for eight years of booking I had worked
14:26	in a model where there were cross
14:28	functional teams and so if you think
14:30	about ranking my my responsibility when
14:33	when as was a product manager I had a
14:35	team that included data scientists
14:37	developers and designers as part of one
14:40	single team and our job was related to
14:43	make to making it easier for people to
14:45	find a hotel that they could stay at
14:47	yeah and so we we were not organized
14:50	around craft but we were organized
14:52	around a customer objective and we and
14:55	in I could argue to leadership and say
14:58	well in order to accomplish this
15:00	objective here are the skills that I
15:02	need right I need a designer because I
15:05	need to explain to people why this is
15:06	the right hotel for them I need a data
15:09	scientist because I need to build
15:10	machine learning models that help pick
15:11	the right the right hotel right and so
15:14	that given the objective there can be
15:16	conversations about which skills are are
15:18	needed on the team and and these teams
15:20	were long lived and they are considered
15:22	to be primary teams so if you had
15:24	approached my team members at the time
15:26	and asked them like which team are you
15:27	part of they would say the ranking team
15:29	or they would say the search team now
15:31	when I joined Vista I noticed that when
15:32	you ask an engineer which team are you
15:34	part of they would name their
15:35	engineering team and you ask them what
15:37	do you own they would name the service
15:39	that they owned and the same for the
15:41	data data analyst and the same for
15:42	product managers like they weren’t
15:44	thinking in terms of problem customer
15:47	problem focused teams they were thinking
15:50	in terms of craft teams and so this
15:53	makes experimentation much more
15:55	difficult because there’s many more
15:56	communication Hops and there’s much more
15:58	room for misalignment between the
15:60	different crafts even though we’re all
16:01	working together to solve the same
16:02	customer problem and an experiment
16:05	serves to figure out whether the
16:06	customer problem is solved yeah but but
16:08	if we’re not all aligned on what
16:10	customer problem we’re solving together
16:11	then the experiment doesn’t really have
16:13	value and and and no one really sees a
16:15	need to experiment in the first place
16:17	and so I started to think that we needed
16:19	to change the organization of of Vista
16:21	towards a more uh cross functional
16:24	product team organization and and I will
16:28	not in anyway claimed that I was the one
16:31	who drove this change like there were
16:33	many people with me that had the same
16:35	opinion I have tried to push for this
16:37	for the last three years and I’m happy
16:39	to report that the things are going
16:40	really well and that we’re slowly moving
16:42	towards this product operating model um
16:45	and but I do think it’s one of the
16:46	boundary conditions that needs to be in
16:48	place for experimentation to be really
16:51	scalable and so to your earlier point
16:53	like I think a lot of organizations and
16:55	and especially the older so like the the
16:58	first generation of experimentation
16:59	platforms they are very marketing
17:01	Centric and they’re very marketing
17:03	focused and the pitch is essentially uh
17:06	put this script on your website and you
17:08	will never have to talk to it again to
17:10	make to to run an experiment that’s
17:12	that’s the pitch and I think that is
17:14	marvelous
17:15	technology but it sort of completely
17:19	circumvents the core problem that we
17:21	need to solve which is that marketing
17:23	and it should not be separate in the
17:25	first place right and they usually kind
17:28	of into taking this well they can’t be
17:30	antagonistic toward towards each other
17:32	because marketing will be like okay
17:34	We’ve ran these experiments um here’s a
17:36	few winning experiments that we want it
17:40	produ to
17:41	productionize and then for it it’s like
17:44	they’re rolling their eyes and they
17:45	thinking oh well I’ll just got to I’ll
17:47	just add to my backlog of 100 other
17:49	items on my to-do
17:51	list you know what I mean yeah yeah yes
17:54	and and and they’re PRI prioritizing in
17:56	different ways right so so one of the
17:57	reasons that marketing struggles to put
17:59	stuff on the it backlog is it just have
18:01	different has different priorities and
18:02	that’s why why I think that these people
18:04	should be together in one team and have
18:06	the same set of objectives ideally
18:08	around a customer because then the the
18:10	conversation changes I I mean you tell
18:13	you an anecdote we had a uh we had a re
18:16	a Booking.com reunion a few months ago
18:18	where like people who worked for
18:19	booking.com over the years sort of got
18:21	together and sort of chatted about their
18:22	life after booking it’s very interesting
18:24	because it’s very much a almost like a
18:26	cult
18:27	following if I might say so uh and uh I
18:31	I was talking to a group of Engineers
18:34	and and they were all surprised about
18:36	sort of the engineering cultures that
18:38	they found outside uh of bookan aom and
18:42	I said oh yeah I I’ve noticed the same
18:44	when when I say to people like at
18:48	booking it is very it was very normal
18:51	that you go to an engineer and you say I
18:53	would like you to put like on this on
18:56	these landing pages I want you to change
18:58	this thing
18:59	yeah it is very common for an engineer
19:02	to then say well that page gets about
19:05	10,000 visitors a day which translates
19:07	into about
19:09	0.001% of our total revenue so if I were
19:12	to spend two days of work on this like
19:14	that is already more expensive than a
19:17	100% uplift on conversion that we would
19:20	that we would need to to cover like the
19:22	base cost and there’s no way we would
19:23	get that because the change that you’re
19:25	proposing like psychologically I don’t
19:27	see why that would cause that much
19:29	changing conversion so I’m not going to
19:30	prioritize
19:32	it and and I’m sort of look around the
19:35	look around the circle of Engineers and
19:36	they all like yes that is that is
19:39	exactly what is missing from the
19:41	engineering cultures that we see in
19:42	other organizations where Engineers
19:44	would not they are not expected to and
19:47	so they do not talk in business terms
19:51	and I don’t think this is a lack of
19:52	skill they just really just thinking how
19:54	do we technically set up this test but
19:56	then not thinking like correct um like a
19:60	would where it’s like okay we get 10,000
20:02	uniques per day or month or whatever and
20:05	um you know these are the business
20:07	implications if I set up this test if
20:09	it’s high then I’ll just put it high up
20:11	my back in my my Q in my road map if
20:14	it’s pretty low then I’ll just push back
20:17	against you and say no this is not good
20:18	or put it lower down the the the um the
20:21	road map
20:24	um yeah like that’s um that’s that’s not
20:27	something I’ve experienced with
20:28	Engineers cuz usually they just like
20:29	okay I’ll just I’ll build it for you
20:30	because you you told me to yes and I
20:33	think that’s if that’s the working model
20:35	then getting into an experimentation
20:36	space is is difficult because you’re
20:38	just speaking different languages and
20:40	that’s why I I contrast this with a sort
20:42	of the newer generation of tools AB
20:44	smartly is one of them there there’s
20:46	others of course and they they’re all
20:48	founded by people who came from these um
20:52	these these organizations that are
20:54	already running experimentation at
20:55	Scales so so Facebook Airbnb booking.com
20:59	like these are all companies that are
21:00	running experimentation scale people
21:02	coming from those companies they design
21:04	experimentation Platforms in a
21:05	completely different way because they
21:07	aren’t thinking about it in terms of put
21:09	this pixel on your website so you can
21:10	avoid it they’re thinking of it in terms
21:13	of it is at the table when you’re
21:16	designing this this experiment like they
21:18	are involved in the design and the
21:20	execution of the experiment the vast
21:22	majority of experiments that are run
21:23	at.com are run by Engineers like they’re
21:26	they’re most of the tests are set up by
21:28	by an engineer in person rather than bu
21:30	a product person so I think this is like
21:33	this is the biggest differ when we talk
21:34	about the experimentation Gap I this
21:37	article that came out I think a year
21:39	ago like this is a large part of the Gap
21:42	like it’s not just scale it’s scale that
21:45	comes from a different way of working
21:47	and a different way of working that
21:49	encourages especially engineering to
21:51	think in a different way yeah so um
21:55	maybe for our audiences um who are
21:57	listening to this like say they are are
21:59	in that sort of
22:00	um they product manager or they’re C
22:03	Specialist or whatnot and they’re in
22:06	that sort of um early to midstage of
22:10	experimentation maturity and you know um
22:14	they still relying on just the client
22:15	side tool and they’re thinking okay um
22:18	you know if I really want to
22:20	embed a real culture of experimentation
22:24	I’ve got to go engineering first and um
22:28	have a tool that’s going to
22:30	be servici side cuz it’s going to be
22:33	faster and Engineers you know they care
22:36	about speed and those sort of things um
22:40	what kind of advice would you give to
22:42	those people um from your experience
22:44	looking at Vista where um they want
22:47	they’re in that sort of Silo
22:50	structured um to go into the cross
22:53	pollination structure like you you
22:56	mentioned at booking.com I mean
22:59	you have to restructure the the whole
23:01	damn organization or how the hell do you
23:03	do it I mean that’s a scary proposition
23:06	that’s a scary proposition um I
23:10	think please take my advice here with a
23:12	huge grain of salt yeah I think I think
23:15	there if you you have to adjust your
23:17	prior this this is a company that did
23:19	not have a scaled organization of
23:21	culture uh of experimentation at the
23:23	time that I joined um but they did hire
23:27	me as I and so like that already tells
23:30	you something about like the leadership
23:32	Buy in that there was for for
23:34	experimentation yeah at the time right
23:37	because I think I talked to a lot of our
23:38	other organization and just merely
23:40	suggesting that you should hire a
23:41	director of experimentation is already
23:43	like a bridge too far for many many
23:45	organizations in this case we had a a c
23:48	Vista has a CEO that is very much bought
23:51	into uh the the idea that
23:53	experimentation is an important part of
23:55	product development and the same for CTO
23:57	and CPO there very strong uh leadership
24:00	Buy in there’s also a long history of
24:03	experimentation at Vista it’s just very
24:05	decentralized so so I didn’t know this
24:07	when I joined but Vista actually started
24:09	running experiments before booking.com
24:10	did so booking started in 2005 February
24:14	I believe yeah and Vista ran their first
24:16	experiment somewhere in at the end tail
24:17	end of 20 2004 so they were just barely
24:21	sooner um but some somehow it never h it
24:26	did scale actually but somehow it it
24:29	like uh slumped again and I was hired
24:32	sort of as the slump was sort of sort of
24:35	paning out and now now we’re going going
24:36	back up again um and so I don’t think
24:40	Vista is like a a a good template for
24:44	like how other organizations can do this
24:46	because there I had a lot of wind in my
24:48	sales
24:50	um given that like when I joined there
24:54	they did not have a product operating
24:55	model um but me coming from that uh that
24:60	is the only way that I know how to
25:01	operate like I don’t really know how you
25:05	would make things work if you don’t have
25:06	a cross functional team that has all the
25:08	skills that they need to sort of solve
25:10	the problems that their T to solve so
25:12	one of the first things I did was sort
25:13	of explain to people like how I saw
25:17	teams should function uh even if that
25:21	was not the reality of organization at
25:22	the time uh sort of advocating for that
25:25	model and then applying it within my own
25:28	team like I I had a very very small team
25:31	of I believe three analysts or two
25:33	analysts when I joined and one of the
25:35	first things I did was I I went to an
25:37	engineering leader and said well uh I
25:39	want to try this new model where we take
25:41	those two analysts and we take the two
25:43	Engineers that you have assigned to this
25:45	topic and we put them together and we
25:46	tell them that they are one team and we
25:48	give them one name and we give them one
25:50	identity and together we give them
25:52	objectives and I was lucky to find an
25:54	engineering leader that that bought into
25:56	that Paradigm and said yes let’s do it
25:58	and so together we set the objectives
25:59	for this one team and started treating
26:01	them as if they were
26:03	one and very quickly then I hired Uh
26:06	Kevin Anderson as a product manager I
26:08	found sort of like an organizational
26:10	loophole where I could hire a a product
26:12	manager as part of my organization even
26:14	though I wasn’t in product by calling
26:16	him a data product manager that was the
26:18	that was the loophole like Kevin very
26:22	much operates as a product manager he’s
26:24	excellent um but he was called a data
26:26	product manager because that allowed me
26:28	to then hire in into the team and so we
26:30	could we could sort of treat them as a
26:32	one like as an island of a an an
26:35	empowered product team within within a
26:37	larger organization that was still
26:38	grappling with the with the concept and
26:40	use them as a as a role model and go to
26:43	leadership and say like hey here’s how
26:45	this works see the artifacts that that
26:47	they are creating and again Kevin is
26:49	excellent for this because he’s a he’s
26:50	very much a documentation driven person
26:52	he will write things down and he does so
26:54	very publicly and so all of the
26:56	artifacts that this team was creating
26:57	that we could also show them into
26:58	leadership and say hey here’s what the
26:60	proit team is doing and here’s how and
27:01	here’s the results they’re having and
27:02	here’s how they’re tracking their kpis
27:04	and so that becomes a role model and
27:06	example that you can use for rest
27:07	organization at the same time I I have
27:09	to say like I very much had to wind in
27:11	my sales because we have a a chief
27:13	product officer who’s very much bought
27:14	into the same Paradigm we worked at
27:16	Amazon before uh we have a CTO that was
27:19	very much bought into this CEO that also
27:21	came to from Amazon so we had all these
27:23	people who understood this model um and
27:27	then we got help from uh from outside
27:29	consultancy uh including Marty Kagan
27:32	who’s uh who’s known for what what is
27:34	called the product operating model which
27:36	I was not familiar with but when I read
27:39	all the material it’s like oh yes that’s
27:40	how booking aom operated so so it feels
27:44	like there’s some sort of convergent
27:45	evolution where like a lot of these
27:47	companies are finding out that if you
27:48	put people of different crafts together
27:50	and you give them a customer problem to
27:52	solve and you give them flexibility on
27:54	how to how to sort of explore solutions
27:56	that is a very effective model
27:59	and Marty has done some great work in
28:00	sort of documenting how the differences
28:03	between the different companies and sort
28:04	of the commonalities between different
28:05	companies approaching this I was lucky
28:07	to be hired into a company that did this
28:09	now I’m I’m lucky to be in a company
28:11	that is slow that is not slowly that’s
28:14	rapidly adjusting to the style of
28:16	working and so like three years ago we
28:19	were completely siloed now like we are
28:21	completely uh product operating mod
28:24	there’s no more separation between uh
28:27	engineering uh product and and design
28:29	they’re all they’re all one so terms of
28:33	um cultural
28:36	change
28:38	um would you say that there a lot of um
28:42	growing growing pangs and in in in going
28:46	from The Silo model to
28:48	the the sort of cross Silo model like I
28:51	mean oh it’s yeah do you get people
28:53	leaving for these reasons you get people
28:55	that are like oh that’s company’s
28:57	changing I mean do
28:59	is is a common so there so so two things
29:01	there are definitely growing pants so so
29:03	so it’s not easy I’m not at all sugges
29:05	and it it does take time like this this
29:07	is a process is going to take years yeah
29:10	it it does and I think it’s difficult
29:12	for two reasons like people have to find
29:14	a new way of working they have to new
29:16	find new rhythms and so we have a we
29:18	have a separate part of the organization
29:20	that is specifically tasked with
29:22	figuring out those organizational
29:23	rhythms and they also operate as product
29:26	teams and so they set their own
29:27	objectives they make meure their
29:28	outcomes in terms of how teams are
29:30	operating and this is not treated as a
29:32	as a like a one-off program this is an
29:35	ongoing Improvement thing with in the
29:36	organization like how do we work that’s
29:38	one I think the other is that you said
29:40	are people leaving
29:42	um I I don’t think we I don’t think we
29:45	we see like people leaving in dros like
29:48	every company has attrition right and no
29:50	different I do see like people have to
29:53	stretch themselves because this model
29:54	does ask different things so to go back
29:57	to the earlier example about engineers
29:58	right booking called these t-shapes you
30:01	want you want people who are not just
30:03	very good in one craft you want people
30:06	who are good in one craft but then know
30:09	a little bit or know enough about each
30:11	other craft to have conversations with
30:13	their peers right a an engineer should
30:16	know enough about the business and about
30:18	the metrics of the product they’re
30:20	responsible for to have conversations
30:22	with a designer and a product manager
30:23	about the product they’re working on and
30:25	a designer should know enough about the
30:27	engineering problems to be able to talk
30:29	to the engineer about solutions to
30:31	designs that they have come up with
30:33	right and so this requires people to
30:35	think a little bit outside of their
30:36	craft and there are people who want this
30:38	and there are people who don’t want this
30:39	I think that is that is a shift we’re
30:41	slowly seeing and I think it’s one of
30:43	the reasons that you see that not only
30:44	are there Growing Pains there’s also uh
30:48	I think it was CRZ was it this famous um
30:51	quote like the future is already here
30:53	it’s just not evenly
30:55	distributed of that yeah right it’s we
30:58	have teams are operating really well in
30:60	this product operating model and we have
31:01	teams are really struggling in this
31:03	product operating model and so we’re
31:05	trying to find ways again
31:07	cross-pollinate like we did a booking
31:08	like how do we take people who are very
31:10	effective at this like put them in teams
31:12	that are struggling it’s like how do we
31:13	how do we make a rising tide Floats or
31:15	boats and that is just a process that we
31:17	need to go through I in no way
31:19	suggesting that this is easy um and your
31:22	again even if you have leadership eying
31:25	it is still not easy you’ve mentioned
31:27	this a few times that you know you had
31:29	leaders like the seite execs um having
31:33	huge buying and I think that’s sounds
31:36	like that’s a
31:38	common it’s a a huge factor for for
31:42	driving this don’t you think like if
31:43	it’s not coming from the Top If It’s s a
31:46	sort of bolted on the side like
31:47	marketing it’s always going to be in
31:49	that sort of
31:50	Silo um sort of Paradigm where it’s like
31:54	marketing versus technology or whatever
31:56	but if it’s coming from the top down and
31:58	it’s like you know really embedded from
32:01	the exacts and it’s flowing throughout
32:04	the whole organization not saying it’s
32:06	easy but that sounds like that’s a key
32:09	element that you need would you agree
32:13	like I I would say if you don’t have it
32:15	you think you have it yes and that’s not
32:17	sufficient it’s not sufficient because
32:19	because I would say yes leadership needs
32:21	to be byy in and Leadership needs to
32:23	understand that it has implications for
32:25	how they organize their their business
32:28	yeah and how they operate
32:31	themselves and and this is the part that
32:34	is not
32:35	easy you might have a leader that says
32:37	yes I want you to run experiments
32:39	marketing go run experiments but what
32:41	we’ll keep the engineering organization
32:44	aside we’re not touching it then they’re
32:46	not bought in enough yeah then you kind
32:49	of end up being siloed um yeah so look
32:54	it’s um unfortunately we can’t go on for
32:56	too long but um any any closing thoughts
32:59	for our
33:04	audiences uh I I hope I hope what I
33:07	share
33:08	helps I hope I’ve also given you enough
33:11	caveats to say like this is my
33:12	experience at two companies so so like
33:16	and I I I again I’ve been very lucky
33:18	I’ve had the wind in my sales for a long
33:19	time I like if you struggle with this i’
33:22	I’d be happy to chat like to hear hear
33:23	the things you’re struggling with I
33:25	maybe I maybe I can help so please do
33:26	reach out let’s talk
33:29	we are all learning right this like we
33:31	are experimentation people let’s
33:33	experiment with how we do the
33:34	organization around experiment um how
33:37	can people contact you Lucas sure uh so
33:40	just go to my website my name. NL NL for
33:43	Netherlands uh and then there’s a link
33:45	at the top that allows you to book some
33:46	time in my calendar or send me an email
33:48	whatever you want that’s the quickest
33:50	way awesome thanks for joining the
33:52	podcast thank you for having me it was
33:54	great bye see you hi this is Romo
33:57	Santiago from experiment Nation if you’d
33:59	like to connect with hundreds of
33:60	experimenters from around the world
34:01	consider joining our slack Channel you
34:03	can find the link in the description now
34:04	back to the episode

If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

Join the Directory

Advertisment

Leadership buy-in is essential with Lukas Vermeer

AI-Generated Summary

AI-Generated Transcript

Connect with Experimenters from around the world

Categories

Related posts: