How Conversion’s Stephen Pavlovich convinces clients on Experimentation

Stephen Pavlovich

A Conversion Conversation with Conversion’s Stephen Pavlovich

Getting clients to embrace Experimentation is like baking bread. It takes great ingredients, warmth, and a good amount of repeated kneading to get things going. I recently spoke with Stephen from Conversion about how they leverage personas to drive Experimentation, the importance of an Experimentation Culture, and his favourite sourdough recipe.


Rommil: Hi Stephen, thanks for taking the time out of your busy schedule to chat with me today!

For those who don’t know, could you tell us what Conversion is and what your role is there?

Stephen: Conversion is an experimentation agency. Most of the time, we’re helping companies understand their customers, then optimizing their website to improve performance. But we use experimentation to answer any kind of question: what’s the best design for a landing page, which payment methods should we offer, which product should we launch next, and what should we charge for it?

I’m the founder and CEO, so I support the team on experimentation strategy and occasionally ideation. I also help to set the agency’s direction (and often get in the way).

https://conversion.com/

How did you come up with the idea to start your company? And how is it different from all the other agencies out there?

In 2005, I took a job for a photo printing company. My remit was to increase sales through the website. So after getting up-to-speed on SEO (and doing undeservedly well at it), I looked at ways to get more sales from the same traffic. I started reading up on copywriting, usability, design, and psychology and began testing with Google Website Optimizer. After six months, I’d increased sales 6x. I set up the agency soon after (with the photo printing company as my first client).

“…we use experimentation to answer any kind of question…”

Not all companies are set up with Experimentation platforms — how do you handle those situations?

There are plenty of good platforms out there, and we’re always happy to advise what would be the best fit for a company. There’s a core level of functionality that’s pretty similar, but there’s a layer of detail that’s hard to assess — unless you’re an agency that’s using most of the platforms out there. More and more, we’re also being asked to support companies moving from one platform to another, rather than buying one for the first time.

“…run the experiment well, and an increase should materialize (it just may not be the exact uplift you saw in the experiment).”

Obviously, clients care that they are getting positive ROI from their investments in Experimentation. However, after running tests for a period of time and rolling the winning changes out, the projected increase in revenue doesn’t materialize. How do you account for this when you manage expectations?

READ  How Josh Sookman drives product development through Experimentation

Most of the time, this happens because the experiment wasn’t run properly in the first place. We’ve run A/A tests that have been reported by the platform as having a statistically significant difference. Funnily enough, if you rolled out the “A” variation, the uplift wouldn’t materialize… 🙂

That accounts for most instances of this. Of course, it’s possible that there are external factors too (i.e. the period when the experiment was run wasn’t representative longer-term).

And it’s worth noting too that we’re looking for confidence to roll out a change — we’re not looking for confidence in the size of the change (otherwise we’d have to run experiments for an unnecessarily long time).

So in short… run the experiment well, and an increase should materialize (it just may not be the exact uplift you saw in the experiment). It can be hard to educate and inform on this — especially when experiment results get shared more widely, and the statistical disclaimer gets lost 🙂

Advertisement

Connect with members of the Experiment Nation Directory
PhotoNameLocationShort Bio / SpecialitiesLinkedIn URL
Jonathan Deleon Orlando Experimentation https://www.linkedin.com/in/jonathantdeleon/
Gurudev Karanth San Francisco Bay Area One of the founding members of the experimentation team at eBay in 2005, helping build the digital experimentation platform ground up and then deployed at Paypal. Built the first omni-channel experimentation platform ground up at Target https://www.linkedin.com/in/gurudev/
Usman Janvekar Toronto, Canada Digital Data & Analytics Manager working on complex transformational solutions across digital that deliver on effectiveness and efficiencies. Specialities: — Consumer online behaviour and effectiveness targeting — Media Measurement and Attribution — Google Martech capability building https://www.linkedin.com/in/usmana

I’ve read that Conversion leverages personas. Could you briefly explain what a persona is and why you feel leveraging them in beneficial?

There are two ways we leverage personas.

One is to better understand the audience. So we look at key personas and how they differ. E.g. they may differ by goal or by motivation or by behaviour. When most people think of personas, they start giving them names, and saying what TV programmes they watch. That’s not what we’re trying to do — we want to understand how their goal, motivation and behaviour varies.

Then, we look for anything that overlaps (e.g. two distinct personas are motivated by the same thing) and anything that contradicts (e.g. two distinct personas are diametrically opposed in what motivates them). This forms the basis of our experimentation.

READ  TechTarget’s Renee Thompson on how to effectively test on B2B customer properties

The second way to use personas is as the basis for personalization. This means we’re just not analyzing personas to help us better understand the audience — we’re actually looking to identify and target specific segments of users, and deliver an experience that better reflects their goals and motivation.

Both approaches allow us to create a more relevant and meaningful experience, which generates better performance — the second just requires us to be able to target the visitor when they’re on the website (which isn’t always possible).

“…any decision you make (including making no decision) comes with risk. Experimentation by its nature allows you to better understand that risk.”

I imagine you’ve faced clients who are very risk-averse — how do handle those situations?

Two ways — (1) we start slow and scale-up, and (2) we show the risk in what they’re already doing.

On the first point, we’ve been lucky to work with many of our clients for multiple years. Some started out pretty risk-averse. We’d run relatively simple UX and messaging experiments. Fast forward four years and we’re testing on product and pricing. It’s a gradual process that needs trust built up over time.

On the second point, any decision you make (including making no decision) comes with risk. Experimentation by its nature allows you to better understand that risk. So if a client is risk-averse and doesn’t want to test something — that’s still taking a risk. If anything, it’s more risky than testing it in the first place.

The first and second points go together. As you develop the relationship, you have the opportunity to show how experimentation could help them better understand the most important choices they have to make. Just by saying, “This is how we could test that…” helps to educate and inspire people to be more open to experimentation.

“…you need a culture that’s open to risk — where people aren’t afraid to experiment with multiple options and where ‘lore’ can be challenged.”

Obviously, a culture of Experimentation is key for your company. Can you describe how you define an Experimentation culture and how you ensure that your company embraces it?

You have to commit to valuing evidence and data above opinion and belief. And that’s really challenging to do. Most companies are built on a hierarchy where experience > data. Instead, you need a culture that’s open to risk — where people aren’t afraid to experiment with multiple options and where “lore” can be challenged. Otherwise, you’ll just do the same as everyone else (and get similar results).

READ  The how, what, and why of copy testing with Green Light Copy’s Eden Bidani

Fareed Mosavat (ex-Slack) presents it well:

Good experiments are used as a tool of humility. They are used to test something we believe in, but we are humble enough to know that we aren’t sure if it will work. Bad experiments are used to defer decision making, settle disagreements, or let the data tell us what the right thing to build is.

I love that quote. I’m going to re-use that. That said, finally, it’s time for the Lightning round!

Bayesian or Frequentist?

No comment 🙂

What excites you most about the CRO industry?

Pushing the boundaries on what people test. I don’t want to run boring experiments (the stuff you’d do anyway if it weren’t possible to test) — I want us to test the ideas that make people uncomfortable but could just work…

What keeps you up at night?

Climate change.

If you couldn’t work in CRO — what would you do?

Bake sourdough bread.

I’ve never made sourdough. Do you happen to have a sourdough recipe you can share?

Ha, I don’t have an original but I tend to do the Tartine country loaf and mix in cheddar and jalapenos 🙂

Check out the recipe here: https://www.theperfectloaf.com/tartine-sourdough-country-loaf-bread-recipe/

Finally, describe Stephan in 3 words.

Who is Stephan? 🙂

LOL Stephen, if that is indeed your name, thank you for joining the Conversation!

My pleasure!



Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

You may also like

To build or buy? Solving the classic Experimenter’s dilemma with Convoy’s Chad Sanderson

A conversation with Convoy’s Chad Sanderson about Experimentation Those in this space eventually face the question of whether to use Read more

BMO’s Claire Yeh: Experimentation is key to Personalization success
Photo of Claire Yeh

A conversation with BMO’s Claire Yeh about Experimentation Personalization and Experimentation may seem like very unrelated practices — but this can’t be Read more

The how, what, and why of copy testing with Green Light Copy’s Eden Bidani

A conversation with Green Light Copy’s Eden Bidani about Experimentation I recently spoke to Eden about what conversion copywriting is, Read more

Bithika Mehra on partnering with HiPPOs and leveraging neuromarketing to drive conversion
Bithika Mehra

A conversation with Bithika Mehra about Experimentation Despite our best efforts, HiPPOs are key stakeholders that can make or break Read more