AWA digital’s Johann Van Tonder on the experimentation industry’s maturity, approaches, and attitudes

Disclosure: This post contains affiliate links. If you use these links to buy something we may earn a commission. 

By Rommil Santiago
Published on: December 20, 2020

Hi Johann, thanks for taking the time to talk today, how are you?

Very well. Tomorrow I go on leave! Well, sort of, I have calls booked tomorrow and the day after and…

Well I hope you enjoy the time off, at least sort of lol. So, Johann, how about we start with you introducing yourself to our audience?

I grew up in South Africa, which is still where I’m mainly based. 200m from the beach in Cape Town, voted the best city in the world seven times in a row. I have a 2nd dan black belt in karate and train regularly, as well as teach kids. 

I live in Toronto, so I’m very jealous of that beach right now for sure! I myself have a 1st dan black belt in shovelling snow – or at least that’s what it feels like sometimes. On a more serious note, we’d love to hear a bit about your career journey. How did you become the COO of AWA Digital?

I cut my teeth in the corporate world, working for a global internet investment giant. Post-MBA I was given a portfolio of online businesses and told to ‘fix them, kill them or sell them’. This is where I discovered the power of an experimentation mindset. Two books had a great influence on me. The Four Steps to the Epiphany by Steve Blank shaped my approach to business. Soon after reading Lean Startup by Eric Ries, I took over operations at a struggling online retailer, where I led the turnaround team. 

Recently, AWA announced that it is launching a State of Online Experimentation Report. Could you tell us more about this project and what motivated AWA to launch it?

It’s to produce a snapshot of current levels of experimentation maturity, approaches and attitudes towards experimentation as well as assessing obstacles in the journey. The data will help organisations and teams to benchmark themselves, and hopefully serve as inspiration for some. 

How has the reception been thus far?

We’ve had responses from a range of organisations around the world, from owner-run stores to big corporations. Leading experimentation platforms encouraged their clients to participate in the survey. If you are involved in an experimentation programme, please complete the survey.

Very cool. We’ll share that with our readers, for sure. Changing gears a bit. As you work with clients, what are some of the biggest misconceptions you encounter?

A common misconception is that you run out of “low-hanging fruit” at some point and then experimentation comes to a natural stop. Experimentation is a growth strategy, not a bolt-on project that you get done with in a few months.

In your opinion, what makes a good Experiment? Or Experimentation program?

A good experiment gives you real-world data which lets you make a smart business decision. Not what people say, but what people do – because there is a difference. By isolating one variable, while holding everything else constant, you are able to measure the impact of that variable on a key business metric. You’ll make fewer bad decisions, stop wasting resources on bad ideas and ultimately take bigger risks, where the upside is bigger.

How do you measure the ROI of Experimentation?

It’s a topic I can keep you busy with for a long time. At AWA digital we have one of the most advanced methodologies for calculating ROI on Experimentation. It’s evolved over years and based on millions of data points. 

However, the best answer to this question comes from Prof Stefan Thomke at Harvard Business School: “What is the ROI of not doing experimentation?!”

I like that.

As we chat, it’s Black Friday and Cyber Monday season in North America. What are your thoughts around running Experiments during big holiday seasons? 

In many organisations, a significant portion of annual turnover is produced in this period. It would be insane to suspend experimentation. That said, there can be significant shifts in consumer motivation and user behaviour. So you might explore season-specific hypotheses, and you should always deal with questions of generalisation and transferability. 

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

Haha. Bayesian, but really it depends. If I’m comparing GA segments in pre-test analysis, a good ol’ T-test or Chi-Square analysis does the trick.

What is your favourite Experimentation tool?

I have a few favourites, depending on the context. But I have to say this: a tool is just an enabler, it’s not even close to being the most important factor. 

How would you suggest picking the right Experimentation tool?

If you’re starting out, Google Optimize is a good low-friction option because a) it’s Google and b) it’s free. As the programme matures, you’ll outgrow that and want to invest in a premium tool. By now you should know enough about your needs to inform your decision or approach a reputable independent agency for advice. My advice is to obsess over something else. Get the systems, process and culture in place. The tool will take care of itself if the bedrock is solid.

100%. There is far too much focus on tooling. Here’s an interesting question. Do you ever miss snake handling?

Oh I still do it. I’ve always been fascinated by snakes. As a young boy I went looking for them in nearby fields. In the area where I live we’re surrounded by nature, home to some rather venomous snakes. Occasionally I get called by people in the neighbourhood to remove a cape cobra or boomslang.

We’ll have to include a photo of you doing that. It’s a sight to see.

Describe Johann in 5 words or less.

Get to the point. Quickly.

On that note, haha, thank you for joining the conversation. I hope you enjoy your leave!