How to be a good Experimentation LEADER with Cristina Pujol

Video

Audio

Transcript

Cristina Pujol 0:00
Post a demo session. This is an a giant ceremony. Those who are familiar with the Dine know more about it. But basically, it’s just a session where all the team would meet. To show off work that has been done in a certain period. Usually, we do it monthly. And so this can be specific to experimentation or brother within your team. So you can have different presenters.

Cristina Pujol 0:26
And one or like a couple of slides can be around experimentation, what we have learned and what not. And I always recommend to include visuals in here as they are more engaging and also some key metrics that you have been able to improve or you got the learnings.

Cristina Pujol 0:55
Hello, I’m Christina undecide, merchandising manager at Vistaprint, overseeing the site merchandising strategy for non European markets. Today I’m going to talk about how I created an experimentation process for my team. In order to increase the speed, volume and value of experimentation, I will share some real examples and tips. For a bit of context. VISTA is part of the SIM press Group.

Cristina Pujol 1:19
We are the leading marketing and design partner for millions of small business owners around the world. And this year, I’m proud to say that we start one the experimentation, culture awards for building a culture of experimentation. I couldn’t be happier to be part of this team as experimentation ambassador, a key role on this accomplishment that I took over a year ago, and that I will explain more in detail in a second. I hope you enjoyed the presentation and find it useful. So without further ado, let’s get started.

Cristina Pujol 1:49
In this view, here, I tried to visualize how we are structured at VISTA and what the different roles and skill sets there are to effectively run experiments. Scaling experimentation is often the most difficult challenge for an organization. And this exercise becomes essential to make scaling a reality, especially when teams are in various geographies and time zones. So in our organization, we have the experimenters on the left side here in blue, that belong to a decentralized domain, for example, the email team or a country group or a specific marketing channel, they have their domain specific skills and knowledge regarding how experiments are prioritized, design, coordinated, executed and evaluated.

Cristina Pujol 2:33
Then we’ve got on the right hand side, in purple, the centralized experimentation hub, they create decentralized freedom to experiment with a shared framework. So they are responsible for collecting, documenting, evangelizing, and teaching best practices, as well as building necessary data products and overall technology and tools for experimenters to succeed.

Cristina Pujol 2:57
I won’t go into too much detail here, if you want to know more about this structure, you will find this link on the bottom left, where there’s a full article about it. So what I want to call out here is the ambassador role, which is the role I’ve been in. And it’s really the link between the experimenters in its domain and the experimentation hub. The reason why ambassador is in the middle is because if we were to have only a centralized two bar function for all the domains, it will require deep knowledge of all domains in one place, which is often not feasible and doesn’t really scale well.

Cristina Pujol 3:34
So having a decentralized experimentation Ambassador role, at least one per domain, in addition to a centralized support function is key to provide better support for each domain. So what are the things that we do? We are the first line of support for any experiment across the domain, we get trainings from the experimentation hub, who provide second line support. So they are the experimentation consultants. And here are some specifics about this role.

Cristina Pujol 5:06
Okay, so that was a bit of the theory. Let’s see, in practice how it looks like. This is the experimentation process that I put together for my domain. My domain is a country group. So it’s nine markets. And within that domain, I have people from different marketing channels pricing and site that would follow this process, they would be the experiment owners. And this is the process they would follow in order to mediate and then ultimately launch an experiment. So the first point is just to brainstorm and have all the ideas in a backlog fine. All ideas are good. But sometimes there are some ideas that are more impactful than others.

Cristina Pujol 5:48
So it’s important to have a list of everything that we want to do. And then the next step for this experiment owners is to review their ideas with analytics, that could be a dedicated analytics person or open hours, we used to have open hours every Friday, where people would jump in if they had any questions around how to how to size that the opportunity. There’s also some other tools as synchronously that the team could use, like a sample size calculator. And in general, there are other ways that we we supported them a synchronously

Cristina Pujol 6:52
measure. Again, here, there’s this strong partnership with analytics. So they would support on what are the metrics that they want to monitor? What makes sense? What not, what’s the success metrics? Or what’s the minimum detectable effect to make sure that overall, the testing plan made sense? And here, they would have all my support all the time. So where to create the testing plan?

Cristina Pujol 7:20
Or where where to ask a question, or they would reach out to me as well to help them on this process. And so after that, they would partner with the teams to develop the test variations on time, it’s, the email team needs support from the site team, because they need to create a landing page or something. So I will share an example in a minute, but there are there are collaboration between different teams. So again, here, it’s important to coordinate on the timings to make sure that the test will be executed on time.

Cristina Pujol 8:01
And just get the results. So obviously, the test owner is responsible for monitoring the results. And afterwards, identifying when, when it’s significant, or when to stop the test. So they will get the results. And the most important part for me here is to update the testing plan. So like the testing plan is really end to end source of truth of everything that’s related to that test, or even tests that are consequent of that.

Cristina Pujol 8:33
So in the testing plan, it’s good to link okay, what’s the next step? What happens after this after we got this learning. And so, once the testing plan is updated, we want to share broadly. So there are different ways to celebrate success that I will explain later in the presentation. But this part, I consider it very important to really increase the volume and enthusiasm within the team to run more experiments. So let’s take a real example. Let’s imagine that the email team wants to understand what’s the value of building a specific landing page for their campaigns, as opposed to driving traffic directly to the homepage.

Cristina Pujol 9:16
So someone within the email team is the experiment owner of this test. So their forgot their email. What they want to experiment with is what happens after customers click on this hero on top. Can we drive traffic to can we split the traffic into two pages one is a landing page that the site team owns. And the other is just the homepage that the site team also owns. So once they have this idea, and they have size it with analytics, what’s the next step? So the first thing they would do is they would create a testing plan in a public space again.

Cristina Pujol 9:58
In this example here it

Cristina Pujol 10:00
We’ll be in Jireh. That’s what we use. And they will validate with analytics, all the metrics to monitor to test the success metric, the minimum detectable effect how many customers they need, in order to read significance, and so on. So everything is documented. Next, they would request the variations to the site team, which is the team that owns the those pages in this particular example. And they would agree on the timing. So whether it’s one week or two weeks that the site team needs in order to build those pages. So they would agree on that. And here, again, the ambassador is supporting all the time, it as I said, is the first line of support for the experiment owner, in this case, someone within the email team.

Cristina Pujol 10:52
And after that, we would just QA and launch the test, once the test has been launched, the experiment owner, in this case, someone within the email team with monitor the metrics. And so in this particular example, because it’s just an email drop, then we would just wait a couple of days to get most of the traffic. And so we will get already the results in very few days. But sometimes, a test can take up to four weeks or more, to get the results. So it’s important to monitor the results on a regular basis to make sure nothing is off.

Cristina Pujol 12:22
And we’re coming to the end of the presentation. Here I listed some ideas to share learnings and celebrate success. Again, I would like to highlight the importance of having everything documented in a public space, meaning anyone in the company can access that experimentation plan, and can see your experiment end to end. And if it’s well explained, literally anyone that is not expert in your domain would be able to understand, that’s how I think it’s important to write everything well. And making sure there are attachments of how the different experiences look like, and so on. And so some ideas, some of them I’ve tried within my team, others I’m hoping I can try soon. The first one is to host a demo session.

Cristina Pujol 13:14
This is an agile ceremony, those who are familiar with the Dine no more about it. But basically, it’s just a session where all the team would meet to show off work that has been done in a certain period. Usually, we do it monthly. And so this can be specific to experimentation or brother within your team. So you can have different presenters. And one or like a couple of slides can be around experimentation, what we have learned and what not. And I always recommend to include visuals in here as they are more engaging and also some key metrics that you have been able to improve, or you got the learnings.

Cristina Pujol 13:59
The second one, I didn’t have a chance yet to do it. But I found it in a different network I’m in and is to run again, similar meeting or this can even be part of the demo session. It’s about guessing the winner. So you would present your experiment and the two variations, ideally with some visuals. And then people would have to guess which one was the winner. You can use Kahoot. It’s a free gaming online tool where people can use it with their phones. And I think it’s a fun way to get people engaged.

Cristina Pujol 14:36
And the third one is an experimentation newsletter. I’ve done this similar it’s not it was not an email, but it was like having an experimentation section in the business review that we do every week, month and quarter. But basically this can be just an email sent to stakeholders and you would celebrate a synchronously what the team has accomplished. What are the learning

Cristina Pujol 15:00
and success and what the next steps are. I think overall, the more you share, it really increases the trust within the team in experiments in the results, and overall encourages more testing. So I really recommend having some sort of sharing, ceremony or meeting or really a way to celebrate what the team has accomplished through experimentation.

Cristina Pujol 15:28
Well, that was all I hope you enjoyed the presentation and you found it useful. Here are my contact details. If you want to reach out, feel free to drop me an email or you can add me on LinkedIn. And I will be more than happy to answer any questions. Thank you

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK