AI-Generated Summary

Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues' goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results.

https://youtu.be/qfPUkCmzQr0

AI-Generated Transcript

(00:00) sometimes I read papers or Artic articles I mean or or videos or presentations like the six things to do to get to an experimentation culture and this person is absolutely right I'm really happy this person shares her or his ideas um about this situation and I'm sure it worked for them but every situation is different

(00:19) every company is different every Market is different every country is different every culture is different welcome to another episode of experiment Nations podcast I'm your host Charlotte bonford and today I going to speak to I'm a big fan of this guy his name is Ruben and I have learned a lot of things

(00:42) when I was going through um his HUD me course and I'm just stoked to be able to have this conversation and you know that opportunity to um finally speak to that person who is showing every presenting everything in that um in the small screen so welcome Reuben thank you thank you for having me wonderful to be here

(01:06) awesome why don't you introduce yourself and um uh probably your your background and how you led into experimentation your career oh that's that's a big story uh let me start with introducing myself yeah so uh yeah my name is Ruben uh I now work as a lead consultant at the agency online dietet dialogue and I have

(01:29) my own company conversion ideas with the goal to help people learn and Excel in Converse rate optimization and experimentation for affordable prices and one of the main products are my online courses which sell through yamy I've been in this business for 15 years it basically started with well first of

(01:46) all I was always better at data than than for instance languages at school I've always been result driven and I have a high interest in Psychology and human behavior ni and that kind of led me into Converse rate optimization and from a career in Converse rate optimization I still that that passion to to make an impact to to drive results

(02:06) to make measurable results and and human behavior but it also shifted now from not just digital products but also to through organizations how can we involve colleagues how can you motivate colleagues to work more data customer Centric and experiment driven that's amazing that's a very short version of um of uh you know your

(02:25) career um yeah so just wanted to know like the types of organ organizations that you work with like do they range from different Industries yeah absolutely mainly based in uh the Netherlands but also some International companies as well I now work for a very big uh publishing company I work for the Dutch raway

(02:46) stations work for a very big e-commerce website in the uh in the Netherlands um so it's very different uh which makes it fun and it's not just the client who's different also of course the the questions we get if I see like five years ago I was mainly working with e-commerce managers marketing managers

(03:04) um but now I'm also very active in the on the product side so now some marketing and a lot of product teams that's amazing because there's a lot of things that the question Ruben is like why do you think experimentation is crucial you how would you define experimentation why is it crucial uh for developing strategies with like

(03:28) organizations that you work with oh we suck at making decisions right we think we are very good at making decisions and and being able to estimate the impact of those decisions uh but if you if you look all in our in our work we see it of course are very often that that you think an experiment is going to win and

(03:47) you'll even want to place a bet on it but when you check the results it's not a winner um and it's not just in in our line of work right if you look at lots of scientific papers you will see that in many professions people love to guess to estimate something to make a decision but the outcome is very different than

(04:05) we uh we expected so with experimentation with AP testing um we make better decisions hopefully for to make better products for for happier and healthier lives yeah I agree with that because sometimes that happens to myself too and I agree that sometimes like it's a bit subjective and when you see the numbers

(04:29) you're like oh I was proving wrong you know and um yeah that's uh that's the fun of experimentation you just don't know what you're going to get I guess um absolutely stay curious and be willing to experiment yeah absolutely exactly exactly so uh what we're going to talk about today is more of like how we're

(04:48) going to integrate um those insights that were we've learned from experimentation to different aspects of the company right and um I think like I would want to know how that looks like um like how the processes is step by step and how you influence like different people within you know different you know you have

(05:14) your product you have your marketing you have you know your clients like actual decision makers now how does that look like I think I think the most important and and short advice is to use the same pH ophy the same mindset as we do with our digital users so when you look at our work and and standard zero process

(05:34) we start with thorough research right we do all kinds of user research all kinds of data research uh heat maps and some I can highly recommend do also scientific research we get all these insights from our digital users and then we create from this insights we create test ideas we create hypotheses we AB test them

(05:53) given you have enough traffic otherwise other experiment ments would we test our test ittis and hypothesis see if they're correct learn from it and reiterate and do it over again and do this as much as possible we try to test as much as possible to to fully align our digital product with the needs and motivations

(06:13) of our digital users hi this is Romo Santiago from experiment Nation if you'd like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the description now back to the episode then when we look in the organization um we just start

(06:26) sending a beat test email after a beat test email or we present result after result and we never Wonder like is this useful for our colleagues um does this match their needs does this match their motivations does it help them in their work let alone do they even open the email do they read the email or do they

(06:45) read all our select messages with test results well on our websites we we are very curious and eager to learn more and more about our users we forget to do that uh often within our AR organizations and I think we have to use the same mindset the same principles as we do with our digital users start talking to your colleagues to your

(07:06) stakeholders and fully understand what drives them what are their needs what are their goals what are their struggles what are their obstacles and really try to understand how you can help them with your experiments and ask them how they want to be involved and and how they want to see results or stay up to dat um

(07:24) so start with like colleague research whereas you do user research on your website start with colleague research colle from there adjust your approach and and also learn what works and what doesn't work within your organization oh so that's interesting so you try to understand their psychology as well to be able to explain better how

(07:42) to integrate your the the results of your experiments to that specific colleague is that kind of like what you're trying to say yeah both ways so um how can you use experiments to to help them achieve their goals and overcome their obstacles and struggles and when you did those experiments to help them how can you best share those

(08:04) results and how will they start using your insights in their work and how will you help them achieve their goals indeed yeah yeah so I I wonder like how that looks like on a real life example do you have like a certain real life example so you know our you our listeners would be able to like picture it um on a real

(08:23) life scenario yet so when I start with a client and of course I I I'm the lead I lead multidisiplinary teams I mainly let them do the the research for digital users and while they are doing all the research to understand digital users build a test their backlog and hypothesis I do the research within the

(08:42) company I start talking to product owners stakeholders managers but also of course those zro Specialists or or experimentation Champions themselves um I start doing a big scan of the whole organization to to understand um well the obstacles we can face with experimentation what the needs are um I basically do a big research lots of

(09:06) interviews I can do surveys within the organization to learn the maturity of experimentation and needs for experimentation in different departments and in different teams and they can be very very different from there I create a strategic document and a plan how to best help what team what their needs are

(09:25) and how to involve them and and then of course uh it's still experimentation of course we don't have the big numbers like AB tests um but generally my in my standup I do reflect stand-ups at the end I do reflect a few minutes like hey how are we involving colleagues and this is going well so for instance it's very easy to create some

(09:46) sort of form Google form an air table form to collect test ideas um but when no one used that form you can Wonder like okay should we continue with it or should we stop it and when we continue with it like how are you going to do this what worked for some clients is as we had like a huge big idea competition

(10:04) where we had like big prices and um we could have people use the form get used to using the form and they could win individual prizes or team prizes and of course when they uh hand in their test ideas and it turns out to be a good winner a nice winner we bring them a cake not to teach them that it's all

(10:23) about Winners but it's to show them like Hey we're doing something with your idea and here's a big cake you can share with the apartment so it stays top of mind for another client and this is an idea I got from anamarie from photo zigo uh we placed a big feedback button on the website targeted to the IP address of

(10:40) that office and then people uh really started using it because they saw the big feedback button and when they click on it they went to the form this case an air table form and we leave we received lots of uh ideas lots of bug fixes from the people at the office so again it's not just showing a a link to a form it's

(10:59) continuously experiment and see how yeah see how it can work within the organization same for for presenting a test results after you present a test results little bit bate later do ask people was it useful did you use it how should I change it yeah keep trying and and keep finding feedback on how you

(11:19) involve others how you share with others I actually like your ideas because it's not just you drop it and then you go like you guys go with whatever idea like that's the result you go with it or this is like I kind of like that you have practical and I think like what I've like when I was watching areu Demi you would say here's a

(11:39) principle and here's like a practical solution and I think that's one of those reasons that I kind of like oh you know this Ruben guy is really good but anyway um in terms of like so you're in your an organization do you have like because you've me mentioned there are people who it's kind of like creating a certain

(12:01) habit so that they get used to that habit and get awarded for it it's kind of like a 30-day free trial then it forms a habit and then they you know just give me my give you my money I can't kind of thing um um but how has this um influenced like what are the changes that you've seen after you've implemented those practical Solutions

(12:25) within the organ organizations like how was the organization were they more like receiving or or you know they're like oh you know I'm glad that you're giving me this feedback instead of just like okay thanks for the feedback and leave it alone uh what are those changes that you've seen um that you know your

(12:43) organization has gone through yeah good question uh of yeah with the for field it's easier because you get more and more test ideas but of course uh what I what I generally do is a a maturity scan like once a year for instance once every six months to see if it progresses and of course it's it's still very difficult

(13:02) it's not easy and you do need higher management to to grow in experimentation for instance when you could try really hard with with the team but as long as a team is output driven um it's going to be really really challenging because what happens there is the the team the manager or hire manager demands the team to release

(13:21) stuff uh on the websites new features or or use all Sprint points to build this new product and the product owner yeah she or he listens to uh the boss and does so course and the product owner does not have any Sprint points for a test for instance or if he does and and you start testing and you find out that

(13:43) 30% of the AP test is a winner so he can now only release 30% of the features um while his goal is to release 100% of the features he's going to miss his Applause his his bonus at the end of the year and his boss is not going to be happy with him because he only launched 30 features instead of 100 that year and so in that

(14:01) case the goal set up from higher management to the to the product team and experimentation really Collide because product owner just wants to launch features and with experimentation you're going to say like no 70% of those features make no sense because they're inconclusive or or losing experiments

(14:18) and you shouldn't release them so it's still you're still very dependent upon higher management this is just one example and the culture and the strategy and the tooling and but Etc to to grow experimentation but at least what you can do yeah try to really align and and yeah get that feedback that's right

(14:36) because I yeah I agree that there are still challenges but what's the common like what's the most difficult most challenging you know like push back that you've gotten so far in terms of like product owners like and how are you able to overcome them yeah good question I think I think the biggest pushback is a

(14:58) lot that output driven goals um mainly caused by culture and and Legacy uh for instance companies have been around for many many years have done the work in kind of the same way that is really really difficult and how I've overcome them that's again very different per client and very different per organization and it doesn't always work

(15:18) then then switch to another team and help that team first before spending more energy on that other team and I think that's crucial to understand because just like I state like we should use the same philosopy same mindset uh in our organization which we doing our website it's also the same thing with uh

(15:32) what works and what doesn't work sometimes I read papers or or Artic articles I mean or or videos or presentations like the six things to do to get to an experimentation culture and this person is absolutely right I'm really happy this person shares her or his ideas um about this situation and I'm sure it worked for

(15:54) them but every situation is every compan is different every Market is different every country is different every culture is different so for me like the six things to do to get to an experimentation culture is for me the same as as the six magical uh conversion optimization tricks to Triple your conversion rate overnight that's true

(16:13) yes um again it's the same here of course learn from Best Practices see them get inspired by them but it doesn't necessarily have to work for your organization that's true that's a good point it is kind of like like um customize all the time because it depends on the person depends on everyone that's working in what type of

(16:34) culture like you've said but um even really depends for each team like for one client now I have nine I'm in a center of excellence helping nine product teams experiment so I'm part of the center of excellence there coaching them them part the center of excellence as well as the teams helping them send

(16:50) strategy creating creating plans and two teams that are very close together work very close together one has a product owner that came from an experiment driven company so that team is very easy to help and very easy to grow very quickly in maturity and experimentation whereas the other team that works really

(17:09) close with them they have a product owner of 65 years old and he's counting down to his retirement and he has been a product owner for 30 years um there it's it's really are you allowed to say that he might hear this one no I mean I mean it's it's a wonderful person I I I think very person I can I can learn a lot from

(17:29) him because he has so much experience and so yeah it's so fascinating to hear his whole story as as being a product owner for so long so I learned a lot from him um but it's kind of hard to move him towards experimentation because he's never done that and his his managers also give him output driven

(17:47) goals so he's open for the conversation he's interested but it's yeah it's hard to change to change much harder to change such a team compared to a team with a product owner that just came from experimentation driven company yeah that's true though how do you start cultivating that so for example you get

(18:03) get a new client right and they're kind of like okay let's this person is kind of like that 65 year old guy who doesn't have any experience in in experimentation like how would you like probably this is again like a best practice you know question but how would you start putting like oh you know we need

(18:24) to experiment that before we do the output what like where would you start again ask ask a lot of questions uh really I think again um where does a Great Hero Specialist or consultant start it's with simply with asking questions it's not teaching yet because yeah if ione's going to teach such a person he's not motivated to learn a lot

(18:46) maybe he might be interested um but I will barely move the needle in that team towards experimentation maturity so it is really with asking questions understanding that person why does he do what he do and then you can also hear that maybe it's the goals he gets from his boss and what are his interests um

(19:03) how can I help him um but it's very difficult and challenging and it might be better roads to work from top down get to know his manager align with that person or work with that team that's closely aligned with them get really good results there so the other team sees how it works and how it can help

(19:23) them with their uh goals so as always it's like a nice puzzle it's a fun puzzle it's a challenge and sometimes it's have you seen it like you know like with your experience like have you seen a certain person handling a team and then turn around to not like knowing anything about experimentation to like

(19:42) okay let's experiment things all the time yeah yeah I I've had clients where some some were like someone said like um Z Specialists uh make their own work which I find a very funny comment and um after two years um he was also uh giving test ideas and helping with uh with experimentation yeah that's amazing

(20:06) that's amazing that's already a win I think absolutely that's wonderful that's so fun to see that that yeah you give people a new way of thinking new way of working and they actually like it and I think that's also uh I think what we can do with experimentation as well is to to involve people I think one of the

(20:25) wonderful things from experimentation is that every idea should be heard and every idea can be tested whether you're like a junior on your first day or or a CEO every idea can be tested and I think that's a wonderful thing for experimentation as well to to involve people have the ideas being heard test

(20:41) them learn together it makes for much funner and and nicer organizations that's true and and I agree and I agree it's not just like subjective Solutions it's more like let's look at this and objectively um databased and you know you have more of um ground when you're making decisions like I said we suck at making decisions and we do

(21:04) maybe also when you start with with a company um um all find those people that are enthusiastic create a community because that also again brings people together I agree I agree there you can help each other to grow and and support other teams now the question is like okay now that you've like um mentioned this AB

(21:27) testing results right how how do you help um different departments to make this you know ideas that are wins and not wins into reality how do you help them okay so on your department I think this um AB testing insights whether it's a win or a loss would benefit you how do you help them integrate those insights

(21:51) into their department and how they strategize things and how they prioritize things in their Department yeah so one of the questions you of course always ask is what are you working on what are the biggest customer problems you're working on what are the biggest opportunities you're working on and let me help you with experimentation

(22:10) to so we understand how to best work on these problems or opportunities and that that's when you start aligning your exp your experiments with with what they are working on you start Alig your experiments with with the business goals at that moment and with their okrs if they have them and that's when you can

(22:28) share those insights and share those results and that makes it uh much more interesting for the for the team to implement your winners but also much more interesting for the team to listen to your insights and and learnings because they can now learn what works and what doesn't work um where in the customer Journey related to that

(22:45) opportunity they are now working on as a team do people get excited on heat Maps first or the numbers I feel like get excited now people get excited about the insights that that they find interesting yeah exactly that's yeah okay awesome um anyway any final thoughts on uh what we've talked about today Ruben

(23:08) and um it like I've said like you always give out practical Solutions like on most of your talks and um teachings and yeah any yeah last minute I I think this sum up is what we started with start using the philosophy and mindset uh in your organization start doing colleague research start asking questions and start aligning with

(23:32) their needs and motivations just like you align with the needs and motivations of your digital users that's amazing thank you so much Ruben we're so delighted and I'm so excited to have finally like this chat with you again I'm a big fan and um thank you so much for your time and we appreciate it thank you this was uh fun

(23:53) this was very fun thank you thank you hi this is Romo Santiago from experiment Nation if you'd like to connect with hundreds of experimenters from around the world consider joining our slack Channel you can find the link in the description now back to the episode

If you liked this post, sign up for Experiment Nation's newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK