How Conversion’s Stephen Pavlovich convinces clients on Experimentation Experiment Nation

A Conversion Conversation with Conversion’s Stephen Pavlovich

Getting clients to embrace Experimentation is like baking bread. It takes great ingredients, warmth, and a good amount of repeated kneading to get things going. I recently spoke with Stephen from Conversion about how they leverage personas to drive Experimentation, the importance of an Experimentation Culture, and his favourite sourdough recipe.

Rommil: Hi Stephen, thanks for taking the time out of your busy schedule to chat with me today!

For those who don’t know, could you tell us what Conversion is and what your role is there?

Stephen: Conversion is an experimentation agency. Most of the time, we’re helping companies understand their customers, then optimizing their website to improve performance. But we use experimentation to answer any kind of question: what’s the best design for a landing page, which payment methods should we offer, which product should we launch next, and what should we charge for it?

I’m the founder and CEO, so I support the team on experimentation strategy and occasionally ideation. I also help to set the agency’s direction (and often get in the way).

How did you come up with the idea to start your company? And how is it different from all the other agencies out there?

In 2005, I took a job for a photo printing company. My remit was to increase sales through the website. So after getting up-to-speed on SEO (and doing undeservedly well at it), I looked at ways to get more sales from the same traffic. I started reading up on copywriting, usability, design, and psychology and began testing with Google Website Optimizer. After six months, I’d increased sales 6x. I set up the agency soon after (with the photo printing company as my first client).

“…we use experimentation to answer any kind of question…”

Not all companies are set up with Experimentation platforms?—?how do you handle those situations?

There are plenty of good platforms out there, and we’re always happy to advise what would be the best fit for a company. There’s a core level of functionality that’s pretty similar, but there’s a layer of detail that’s hard to assess?—?unless you’re an agency that’s using most of the platforms out there. More and more, we’re also being asked to support companies moving from one platform to another, rather than buying one for the first time.

“…run the experiment well, and an increase should materialize (it just may not be the exact uplift you saw in the experiment).”

Obviously, clients care that they are getting positive ROI from their investments in Experimentation. However, after running tests for a period of time and rolling the winning changes out, the projected increase in revenue doesn’t materialize. How do you account for this when you manage expectations?

Most of the time, this happens because the experiment wasn’t run properly in the first place. We’ve run A/A tests that have been reported by the platform as having a statistically significant difference. Funnily enough, if you rolled out the “A” variation, the uplift wouldn’t materialize… 🙂

That accounts for most instances of this. Of course, it’s possible that there are external factors too (i.e. the period when the experiment was run wasn’t representative longer-term).

And it’s worth noting too that we’re looking for confidence to roll out a change?—?we’re not looking for confidence in the size of the change (otherwise we’d have to run experiments for an unnecessarily long time).

So in short… run the experiment well, and an increase should materialize (it just may not be the exact uplift you saw in the experiment). It can be hard to educate and inform on this?—?especially when experiment results get shared more widely, and the statistical disclaimer gets lost 🙂

I’ve read that Conversion leverages personas. Could you briefly explain what a persona is and why you feel leveraging them in beneficial?

There are two ways we leverage personas.

One is to better understand the audience. So we look at key personas and how they differ. E.g. they may differ by goal or by motivation or by behaviour. When most people think of personas, they start giving them names, and saying what TV programmes they watch. That’s not what we’re trying to do?—?we want to understand how their goal, motivation and behaviour varies.

Then, we look for anything that overlaps (e.g. two distinct personas are motivated by the same thing) and anything that contradicts (e.g. two distinct personas are diametrically opposed in what motivates them). This forms the basis of our experimentation.

The second way to use personas is as the basis for personalization. This means we’re just not analyzing personas to help us better understand the audience?—?we’re actually looking to identify and target specific segments of users, and deliver an experience that better reflects their goals and motivation.

Both approaches allow us to create a more relevant and meaningful experience, which generates better performance?—?the second just requires us to be able to target the visitor when they’re on the website (which isn’t always possible).

“…any decision you make (including making no decision) comes with risk. Experimentation by its nature allows you to better understand that risk.”

I imagine you’ve faced clients who are very risk-averse?—?how do handle those situations?

Two ways?—?(1) we start slow and scale-up, and (2) we show the risk in what they’re already doing.

On the first point, we’ve been lucky to work with many of our clients for multiple years. Some started out pretty risk-averse. We’d run relatively simple UX and messaging experiments. Fast forward four years and we’re testing on product and pricing. It’s a gradual process that needs trust built up over time.

On the second point, any decision you make (including making no decision) comes with risk. Experimentation by its nature allows you to better understand that risk. So if a client is risk-averse and doesn’t want to test something?—?that’s still taking a risk. If anything, it’s more risky than testing it in the first place.

The first and second points go together. As you develop the relationship, you have the opportunity to show how experimentation could help them better understand the most important choices they have to make. Just by saying, “This is how we could test that…” helps to educate and inspire people to be more open to experimentation.

“…you need a culture that’s open to risk?—?where people aren’t afraid to experiment with multiple options and where ‘lore’ can be challenged.”

Obviously, a culture of Experimentation is key for your company. Can you describe how you define an Experimentation culture and how you ensure that your company embraces it?

You have to commit to valuing evidence and data above opinion and belief. And that’s really challenging to do. Most companies are built on a hierarchy where experience > data. Instead, you need a culture that’s open to risk?—?where people aren’t afraid to experiment with multiple options and where “lore” can be challenged. Otherwise, you’ll just do the same as everyone else (and get similar results).

Fareed Mosavat (ex-Slack) presents it well:

Good experiments are used as a tool of humility. They are used to test something we believe in, but we are humble enough to know that we aren’t sure if it will work. Bad experiments are used to defer decision making, settle disagreements, or let the data tell us what the right thing to build is.

I love that quote. I’m going to re-use that. That said, finally, it’s time for the Lightning round!

Bayesian or Frequentist?

No comment 🙂

What excites you most about the CRO industry?

Pushing the boundaries on what people test. I don’t want to run boring experiments (the stuff you’d do anyway if it weren’t possible to test)?—?I want us to test the ideas that make people uncomfortable but could just work…

What keeps you up at night?

Climate change.

If you couldn’t work in CRO?—?what would you do?

Bake sourdough bread.

I’ve never made sourdough. Do you happen to have a sourdough recipe you can share?

Ha, I don’t have an original but I tend to do the Tartine country loaf and mix in cheddar and jalapenos 🙂

Check out the recipe here: https://www.theperfectloaf.com/tartine-sourdough-country-loaf-bread-recipe/

Finally, describe Stephan in 3 words.

Who is Stephan? 🙂

LOL Stephen, if that is indeed your name, thank you for joining the Conversation!

My pleasure!

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.