Bolt's Gerda Vogt-Thomas shares how to approach standing up a CRO program Experiment Nation

A conversation with Bolt’s Gerda Vogt-Thomas about Experimentation

Standing up a CRO program is no easy feat. I recently spoke to Gerda about how she approaches getting people involved, measures success, and how Gerda tells whether the program is running well.

Rommil: Hi Gerda, how have you been? Thank you for taking the time to chat with me today!

Gerda: Hi Rommil, thanks for having me! I’ve been surprisingly busy given that the world has slowed down quite a bit over the past year.

Could you share what you do with our audience and a bit about your career journey?

I like to romanticise it and think that I make the world a better place, one website at a time. But in terms of how I got here- after finishing university I quickly realised that there was a lot to learn about digital marketing if I wanted to be anyone in business. I sort of fell into CRO when I managed to get an entry level opportunity at CXL. I hadn’t even heard about CRO at that point and I started as a project assistant/customer support and worked my way up from there. I spent three great years at CXL managing projects and learning more than I could’ve ever imagined.

After leaving CXL I started a consulting business with a fellow Optimizer (and now husband). We have some cool product development ideas as well that we’d eventually want to realise, but that’s more in an incubation stage at the moment. Quite recently an opportunity came along to join Bolt as their Website Product Owner, so my co-founder is leading the charge with the consultancy at the moment.

There’s a lot of misconceptions about the life of someone in CRO. I’d love to hear your thoughts about what people think the day-to-day is in CRO and what it’s actually like.

People in marketing are of course more familiar with optimization in general, but I think even then it sometimes surprises them how technical this role can get with understanding tracking, source codes and statistics for example.

Sometimes when I’m talking to people from different industries and say that I try to make websites better, I feel they imagine me sitting in a dark cubicle, pushing a single button that magically fixes everything.

So many folks I’ve spoken to recently have struggled with standing up new CRO programs. In your experience, how have you gotten people involved, and even more importantly, how do you keep them involved?

There will always be challenges with setting up a new program. I think the trick is to think about it the same way you do CRO. You want to make the experience better for the user right, so as with any product, you need to communicate how will this benefit everyone else, in this case all your stakeholders. I’ve seen many projects starting with not enough internal buy in and it is extremely hard to do your best CRO work and simultaneously sell it every day to the people you’re trying to work with.

If at first they don’t seem to be interested, do your research. Literally ask them questions, find out what they care about and what will make their life easier. Chances are you can help their cause with your CRO knowledge.

This applies to presenting your findings as well. I’ve had cases where I oh so proudly presented my winning tests focusing on great conversion rates and statistical significance only to realise that no one really cared. I did my best to explain all of the definitions and why they were important but still nothing. Then I pivoted into literally just reporting how much more dollars each test made and everything changed. These were business people without a lot of experience with testing so I needed to speak their language first. Of course this will not work with everyone and is highly dependent on your metrics, but the point is – know your customer!

How do you measure the success of the program?

That again depends on the business goals and the client or company you’re working for. Overall increasing ROI and conversions, having an inclusive culture where different parties are genuinely interested in what you’re trying to do and decisions backed by real customer insights is a good starting point.

When should you consider your program fully launched? Meaning, at what point would you feel that it’s running well.

This for me ties in with the last question but:

The analytics are in near mint condition, goals are triggered correctly, you know where the data comes from and you’re confident you can use it.
You’ve done bandwidth calculations on how many tests on which pages you can run, for how long and which areas bring the biggest ROI.
You’ve done proper qualitative research – onsite polls, customer surveys, heatmaps, session recordings, funnel analysis, form analysis, user testing, interviews with customer support and end customers. Basically analysed any sort of customer data you might get your hands on.
You trust your testing tool and its implementation, meaning there’s no flicker effect, you’ve run AA tests for calibration, you send your test data to an analytics platform other than just using the one inside the tool.
Based on your research you’ve built a solid backlog of test hypotheses.
You have a testing roadmap for at least a month ahead. When will what go live?
You have a good workflow set up with your designers and developers.
There is a proper QA process before the test goes live.
There’s a process for documenting your results and sharing them with relevant stakeholders.
You know what the next step after the test was concluded is. Should you implement or iterate?
There’s pretty much always some sort of experiment live somewhere.

I’m sure I forgot something, but basically rinse and repeat.

Do you have other tips for those looking to launch a CRO program?

I cannot stress enough how important it is to have your analytics sorted for CRO. Do you understand how and where events are triggered? Do you perhaps need to add more events to do proper analysis? Do the numbers in your analytics match with backend numbers? Do you have sampling issues that need solving? Basically be suspicious about everything in your analytics at first and ask if this is real customer behaviour or a tracking issue. There is honestly nothing more painful than running a bunch of tests and making recommendations only to realise that none of it is based on trustworthy data.

There are a million analytics health check articles and guides on how to find these things out and how to fix them, but if you don’t know where to start I recommend the CXL and Simo Ahava’s Blogs.

How is the CRO scene in Estonia? Are there any challenges that you face in your part of the world that other countries don’t?

CRO to me still seems like kind of a niche term in general, but Estonia is extremely tech driven as a nation. To be successful here you need to think globally as we only have a bit over a million people, so not a huge market. Companies that think globally generally understand that optimization is important either way, so I don’t really see radically different challenges compared to the rest of the world.

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

The stats nerd I’m married to won’t forgive me if I publicly endorse Bayesian…

I hear you loud and clear. What are your thoughts on “wife-carrying”? I hear Estonia is known for it.

Ha, I think I might be out of the loop on this one. I’ve definitely heard that there are world championships held somewhere and Estonia takes part, but that’s about all I know of this.

A way more enticing national sport is kiiking. https://en.wikipedia.org/wiki/Kiiking Basically they strap a single person in a swing and then they try to do a full circle with it. It’s an adrenaline rush just watching it. This sport originates from old village swings that were already a thing back in the 18th century and can still be found all over Estonia in the countryside.

https://en.wikipedia.org/wiki/Village_swing#/media/File:Keemu_sadam_2.JPG

On these you would sometimes get on with like 10 people, hold on for dear life and hope that everything will be ok. We have fun here in Estonia.

That looks like fun and looks incredibly risk at the same time. I can see why people like it haha!

If you couldn’t work in Experimentation, what would you do?

I would like to pick the music for cool movies and tv shows.

Describe Gerda in 5 words or less.

(Sometimes too) honest.

Thank you, Gerda, for taking the time to chat today.

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.