Launching a world-class Experimentation program and how to spot a healthy Experimentation culture

Jon Crowder

A Conversion Conversation with Journey Further’s Jon Crowder

Clarity and context are the foundation of any strong decision framework. Without getting to clarity quickly, you’ll end up going in circles on virtually everything — wasting time and potentially resulting in sub-optimum outcomes. Read on to understand how “Clarity at Speed” is a key tenet for Jon and his agency, Journey Further, as well as the motivation of building Sky’s Experimentation network.


Rommil: Hi Jon, thanks for taking the time to speak today! How are you?

Jon: I’m ok, thanks. The world is a strange place at the moment and it’s going to be strange for some time to come. For the benefit of anyone reading this from the future, we’re right in the middle of a global pandemic (SARS-CoV-2) at the moment, and it’s one of the first since Spanish Flu that’s managed to spread pretty much everywhere across the world. For most people in Europe, like me, it’s our first experience of a lot of things, and everyone has had to get familiar with the basics of epidemiology within a few weeks, to understand how we prevent the virus from hurting a lot of people. Wash your hands please everyone.

Here’s a great video on how to wash your hands effectively: https://twitter.com/SinghLions/status/1240686550939136003

Could you share with us what you do and the career path that led you to Journey Further?

Sure. I’m a Conversion Specialist. That’s not really what I’d like to call it, but it best explains to clients the service that we provide. It’s pretty different from a lot of traditional ‘Conversion Rate Optimization’ roles though. At Journey Further, we’re experimentation and proof focused and we’re very scientific in our approach. This lets us move really fast and to be really clear on the value we provide. The agency’s proposition is ‘Clarity at Speed’ so it’s no surprise that our model focuses on agility and proof of value.

Before working at Journey Further, I worked for Sky for 9 years and was a key part in building their experimentation programme, which is now considered world-class. We started at Sky with nothing but a half-broken analytics tool, and from it, we built a programme that delivered millions of pounds of business value.

Working for Journey Further (alongside Sky’s former Head of Digital Optimisation, Jonny Longden) means that we can offer that experience, skill and value to other businesses.

https://www.journeyfurther.com/

So, you mentioned you helped build a world-class Experimentation Programme at Sky. Could you share with us what the motivation was and how you ensured that the Programme was evolving properly?

Sky had multiple large aspects to its business. They are primarily a subscription model, so they rely on signing customers up to that subscription and then keeping customers subscribed. They’re also an entertainment provider and creator so there’s costs associated with that too, but when focusing on key revenue drivers there’s really two approaches. Having people purchase the service and keeping people happy with the service so they don’t want to cancel. A lot of people perceive sales channels as the only way to grow and make money, and that’s frequently reflected in sales budgets and incentives. In the same vein, service channels are considered a cost of doing business, but a cost to be limited.

Limiting cost can be done in multiple ways. Some are effective, some are less so. For example, we’ve all had experiences of using a company where the website sucks and you can never speak to anyone if something goes wrong or can’t find help. If you do that you really annoy your customers, and eventually, they get so annoyed they leave because they feel like they’ve got no other option. So building a digital-first approach to service channels isn’t something you want to do with just lines on a ledger. You need to be smart, and you need to try and achieve two goals which frequently can compete with each other. You need to improve your service and you need to make it more efficient.

Sky needed a methodology which wasn’t incredibly expensive to run, but that could tangibly improve customer experience whilst improving efficiency. We built a programme that could influence, measure and then prove both of those things, and that was flexible enough to serve multiple metrics. For example, reducing call volumes saves money, but must be balanced against goal achievement because just making the phone number harder to find will reduce call volumes, but won’t improve experience metrics like goal achievement or satisfaction, so ultimately that’s a short-term cost saving that you lose later on with damage to your customer retention.

There’s other ways to save money of course. You can make the phone calls that do happen shorter. That’s a nice healthy way to do things because most of Sky’s customers told them they actually didn’t want to phone at all. They just wanted to watch TV and use their broadband. So we could develop and optimize experiences that focus on that concept, allowing customers to solve an issue, contact us when they need to, but keep queue times down, make staff more accessible and ultimately get that customer back to enjoying the service they pay for.

That was the motivation behind building a world-class optimization team. Something that was cost-effective, skilled and could make a provable change in the organization.

As for how we knew it was progressing and evolving, we had our financial targets adjusted multiple times per year because the programme was so successful. The ROI on our small team with the right methodology and focus was huge.


Advertisement


Moving on to your new role at Journey Further, what’s the importance and your approach to achieving clarity when working with clients?

Clarity is crucial. Clients need to know you’re worth what they’re paying you and what we do is quite technical, especially when you really dive into the statistical analysis. Each explanation on how it works could be a presentation in its own right, so we need a way that clients can understand what we’re doing, why it works and how it works, without the decades of combined experience we have within the team.

What are some of the biggest challenges you face when working with clients and how have you tackled them?

Our clients are really cool, but they have challenges that any client could have. Different org structures, sometimes some internal politics to navigate, sometimes functions are under-resourced and that’s difficult, but it’s another challenge to be worked around. We’re lucky because we’ve got a model that does it all, so the specific challenge you have just determines the level of support you need from us. Nothing is really a show-stopper. Plus, every member of our team has worked client-side too, so we can really empathize and understand some of the challenges our clients have.

How do you know when a company has adopted an Experimentation culture?

They never release a bad customer experience again.

Even the most advanced experimentation organization, one that has it baked into their DNA won’t have fully transitioned to an experimental culture. You’re still going to have ego battles and well-paid important people who make decisions based on their own biases. Yes, even Google. Even Booking.com.

However, the goal of having a perfect experimentation culture keeps an organization aspirational of behaviours that recognize and value evidence and data.

Key things to look for include:

  • If your boss comes to you and asks ‘can we test this’
  • If your developers build a product with analytics in mind
  • If your CEO is asking for experiment results and not just launch dates
  • If your people get excited and not depressed when an idea fails
  • If no idea is thrown out without being researched and, if possible, tested

Experimentation, in a business context, is a generally new field. In your opinion, what are the most important skills one should have to excel?

  • Analytical competence and thinking are important. You’re going to spend a lot of time with data. Learn to know what it’s telling you.
  • Business knowledge is important. Everything you learn needs to be evaluated against with a ‘so what does that mean for us?’ question.
  • Communication skills are important. You’re going to have to explain difficult and complex concepts and outcomes and your audience won’t have the same background as you. You’re going to need to be able to speak a common language.
  • Technical skills help. HTML and CSS knowledge at a minimum. Get learning JavaScript if you can. You’re never going to be disadvantaged by knowing those languages.
  • Creativity is frequently overlooked and sometimes it’s hard to teach. Finding a problem is an analytical challenge, but solving a problem is a creative challenge. Have both skills and you can do both.

It’s time for the Lightning round!

Bayesian or Frequentist?

Frequentist. But either is better than closing your eyes and gambling.

Who do you consider as leaders in the business Experimentation space?

Spotify do some really good work. Sky too.

If you couldn’t work in Experimentation, what would you do?

I’m a musician in my spare time, so I’d do that.

Describe Jon in 5 words or less

“Have you considered testing it?”

LOL best answer I’ve heard yet!

Thank you, Jon, for joining the conversation!



You may also like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s