Conversion Studio’s Tracey Reed on making websites work harder Experiment Nation

A Conversion Conversation with Conversion Studio’s Tracey Reed

I feel that most Experimenters, at some time during their career, think about starting their own agency. But for whatever reason, many of us bail on that dream. However, the rare few with the right grit, enough energy, and ample talent go all the way and succeed. I had the pleasure of speaking to Tracey recently about her journey towards starting her own agency, how she describes what she does to her mom, and the first thing she’ll do after the pandemic.

Romml: Hi Tracey, how are you? I hope you’ve been staying healthy and safe. Thanks for chatting with me today!

Tracey: Hi Rommil, I’m well, thanks. I’ve been staying safe in my bubble, like many, I’ve spent a lot of time on conference calls over the past few weeks.

Same here. I dream of a day where we can start conference calls without any technical issues.

So, let’s start off with what you do today and a bit about your career journey so far.

Today I’m running my own CRO consultancy, Conversion Studio. This has naturally grown from an accumulation of experiences throughout my career which has opened opportunities to work with industry leaders and to share my experience in front of influential audiences.

My career highlight to date has been establishing Air New Zealand’s experimentation program where I worked with an amazing team of analysts, product managers and global marketing teams. As a CRO practitioner, I’m required to have a diverse skill set across UX, marketing, data and people management. Working alongside so many highly skilled individuals at Air New Zealand meant I learnt skills that would take years to master.

I think about these people today and how they have been impacted by the pandemic. Those who remain with the airline have a big job ahead of them. It also means there are a lot of highly skilled workers looking for new roles. An opportunity for smart companies to hire the best.

How does Conversion Studio stand apart from all the other CROs out there?

At Conversion Studio we wholeheartedly believe the best optimization programs are embedded. We’re also the only agency in New Zealand that offers full-stack experimentation. Our passion lies with working with businesses to develop a strategy, implement the program and train their teams to success. Of course, there are some businesses that need the pros to look after CRO while they get on with running business.

How about you? How has COVID-19 impacted your world?

Everything has changed. There’s been a fundamental shift in the way consumers shop online. Their physical (social distancing) and emotional needs have changed and my job is to help my clients understand how to adapt to this new audience. This means understanding and applying behavioural science practices are now critical.

I’ve also seen two types of new clients emerge as a result of the pandemic. Those in a panic state, whose businesses have been dramatically impacted by the pandemic, with sales simply dropping away, and the others who understand this is the ideal time to invest in their website and leverage the demand growth for eCommerce and omnichannel retail experiences.

As you work with these new clients, some undoubtedly aren’t very experienced with Experimentation. How do you explain its value?

How I explain it to my mum is, I make websites work harder.

Digital continues to play an increasing role in our lives, with the pandemic expediting online as a preferred channel. There are two ways to increase website sales; send more traffic to your website, which can be expensive and not always quality traffic, or you can invest in CRO, which increases the number of website visitors that turn into paying customers.

There are obviously other benefits to CRO, such as a better user experience resulting in brand preference and return purchases, and equally as important, it reduces risk and development costs by only implementing changes that have proven to increase sales.

I like that way of framing things. I should try that with my own mom! To be honest, I just tell her I help companies sell stuff. So I’m pretty certain she thinks I’m a salesman.

Over your career, I’ve read you’ve driven over $100M in incremental revenue for companies. There will be many skeptics out there who will question that. How do you go about proving them wrong and attributing that revenue directly to Experimentation?

It is a big number. I’ve had the privilege of working with large global companies that have equally large revenue, where even a 1% conversion increase is a substantial revenue lift. Through A/B testing, external factors can be eliminated to prove the revenue increase is attributed to the experiment. In my experience, the larger the revenue impact, the more robust the attribution model.

As you mentioned, an added benefit of CRO is better user experiences. How do you attribute improvements to customer satisfaction to Experimentation?

Like many businesses, I leverage the CSAT framework to measure customer satisfaction. CSAT being short for customer satisfaction, which is a commonly-used key performance indicator used to track how satisfied customers are with a company or brand, and is most commonly measured through website or email questionnaires.

Some A/B testing tools facilitate integration with the more popular CSAT tools, and if it’s not available out of the box, integration can be manually achieved by passing experiment IDs to the CSAT session. This allows you to measure how the experiment has influenced customer satisfaction. The caveat being that you require high amounts of data to reach significance on this measure.

Speaking of A/B testing, it is, arguably, the bread and butter of Experimenters. Why should Experimenters opt for A/B testing over, say, pre-post testing?

Anyone with access to Google Analytics will know conversion rates fluctuate month on month, even day to day. This is due to both internal and external factors such as traffic source and quality, environmental impacts (say… a global pandemic?), or competitor activity. By running pre/post analysis you cannot always tell if the conversion rate change is the result of your change/enhancement, or simply due to other factors. A/B testing removes this noise and tells us the true impact of our enhancement, as all other internal and external factors are true for both the A and B audience.

Getting from 0 to 1 is always a challenge. But once your clients have run their first experiment?—?how do you convince them to iterate?

The first experiment is always the most important. Focus on low hanging fruit, preferably at the lower end of the funnel (as this is more likely to result in visible revenue lift), and is something of interest to senior managers. Like any interaction, you have limited time to make an impression, and when the impression is large enough (again, visible revenue lift) the attention and trust will follow.

In your opinion, what are the key ingredients to a well-run Experimentation program? And how do you measure its success?

The most successful programs I’ve run have been embedded, meaning that experimentation is part of the product development cycle and every team member is equally invested. This is where the harmony of experimentation is focused on the customer outcome and harnesses the power to transform businesses.

“The most successful programs I’ve run have been embedded, meaning that experimentation is part of the product development cycle and every team member is equally invested.”

Changing gears a bit. Over your years of doing this?—?describe to us your favourite Experiment?

This would be A/B testing Air New Zealand’s redesigned mobile responsive booking website into production. The usual approach for a website launch in 2018 was to run a beta site, which always performs poorly due to audience bias (I could really go deep on why beta sites don’t work, but for all our sanity I’ll save that for another day), and then simply launch the site and pray.

However, through A/B testing we were able to quantify all the hard work of the product team with increased booking conversion and CSAT. Additionally, we identified a few areas for improvement which went straight to the top of the product prioritization roadmap. It wasn’t an easy experiment, and we learnt a lot about the customer through the process.

Since this is a new industry, who would you suggest newcomers follow online and what resources should they check out?

CXL Institute is by far the leader in this space. Peep Laja, CXL founder, has brought together the industries best to share their know-how in short, self-paced online courses.

Finally, it’s time for the Lightning Round!

Frequentist or Bayesian? I know I should take a side, but more importantly, is what platform is most suited to your business, and how they robust their statistics to deal with anomalies such as false positives.

If you couldn’t be Experimenting, what would you do?

Travelling. But I don’t think I’m the only one dreaming of this right now.

I’d have to agree. Travelling to the grocery store, while eventful, doesn’t make for much of an adventure.

Describe Tracey in 5 words or less.

Passionately curious, often hungry.

Ha! I’m with you.

Tracey, thank you for your time and for joining the conversation!

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.