AWA digital’s Johann Van Tonder on the experimentation industry’s maturity, approaches, and attitudes

A conversation with AWA digital’s Johann Van Tonder

Disclosure: This post contains affiliate links. If you use these links to buy something we may earn a commission. 


By Rommil Santiago
Published on: December 20, 2020


Hi Johann, thanks for taking the time to talk today, how are you?

Very well. Tomorrow I go on leave! Well, sort of, I have calls booked tomorrow and the day after and…

Well I hope you enjoy the time off, at least sort of lol. So, Johann, how about we start with you introducing yourself to our audience?

I grew up in South Africa, which is still where I’m mainly based. 200m from the beach in Cape Town, voted the best city in the world seven times in a row. I have a 2nd dan black belt in karate and train regularly, as well as teach kids. 

I live in Toronto, so I’m very jealous of that beach right now for sure! I myself have a 1st dan black belt in shovelling snow – or at least that’s what it feels like sometimes. On a more serious note, we’d love to hear a bit about your career journey. How did you become the COO of AWA Digital?

I cut my teeth in the corporate world, working for a global internet investment giant. Post-MBA I was given a portfolio of online businesses and told to ‘fix them, kill them or sell them’. This is where I discovered the power of an experimentation mindset. Two books had a great influence on me. The Four Steps to the Epiphany by Steve Blank shaped my approach to business. Soon after reading Lean Startup by Eric Ries, I took over operations at a struggling online retailer, where I led the turnaround team. 

Advertisement

Connect with members of the Experiment Nation Directory
PhotoNameLocationShort Bio / SpecialitiesLinkedIn URL
Silvia Serrano Mateos Madrid CRO & UX Specialist https://www.linkedin.com/in/serranosilvia/
Sandeep Shah London AB Testing, Vendors, SaaS, Development, Product https://www.linkedin.com/in/sandeepshah89/
Zharina Pelea Vancouver, Canada Self-motivated Digital Strategist and Marketer with an MBA & 5 years’ experience leading digital Conversion Rate Optimization (CRO) strategies, driving business growth for brands across various verticals (e.g. travel, ecommerce, financial services, B2B, transport, entertainment). Digitally savvy with a passion for the digital landscape and making data-informed decisions. Always up to date with market trends and tools, possessing a working knowledge of platforms like Google Analytics, Optimizely, VWO, Maxymiser, etc. Effective communicator known for building strong relationships with clients & collaborating with internal tech teams to develop and implement solid CRO & digital marketing strategies. Skilled at anticipating and addressing clients’ needs with more than 12 years’ experience in account management. Proven experience of leading successful CRO programs, leading to clients increasing their investment in the ongoing programs. https://www.linkedin.com/in/zharina-pelea
READ  CR-No: Experiments should be backed by hypotheses - leave your best practices for your processes

Recently, AWA announced that it is launching a State of Online Experimentation Report. Could you tell us more about this project and what motivated AWA to launch it?

It’s to produce a snapshot of current levels of experimentation maturity, approaches and attitudes towards experimentation as well as assessing obstacles in the journey. The data will help organisations and teams to benchmark themselves, and hopefully serve as inspiration for some. 

How has the reception been thus far?

We’ve had responses from a range of organisations around the world, from owner-run stores to big corporations. Leading experimentation platforms encouraged their clients to participate in the survey. If you are involved in an experimentation programme, please complete the survey.

Very cool. We’ll share that with our readers, for sure. Changing gears a bit. As you work with clients, what are some of the biggest misconceptions you encounter?

A common misconception is that you run out of “low-hanging fruit” at some point and then experimentation comes to a natural stop. Experimentation is a growth strategy, not a bolt-on project that you get done with in a few months.

'A good experiment gives you real-world data which lets you make a smart business decision.' - Johann Van Tonder Click To Tweet

In your opinion, what makes a good Experiment? Or Experimentation program?

A good experiment gives you real-world data which lets you make a smart business decision. Not what people say, but what people do – because there is a difference. By isolating one variable, while holding everything else constant, you are able to measure the impact of that variable on a key business metric. You’ll make fewer bad decisions, stop wasting resources on bad ideas and ultimately take bigger risks, where the upside is bigger.

READ  Exploring the suitability of t-tests for A/B tests involving RPV with integrate.ai’s Summer Hu

How do you measure the ROI of Experimentation?

It’s a topic I can keep you busy with for a long time. At AWA digital we have one of the most advanced methodologies for calculating ROI on Experimentation. It’s evolved over years and based on millions of data points. 

However, the best answer to this question comes from Prof Stefan Thomke at Harvard Business School: “What is the ROI of not doing experimentation?!”

I like that.

As we chat, it’s Black Friday and Cyber Monday season in North America. What are your thoughts around running Experiments during big holiday seasons? 

In many organisations, a significant portion of annual turnover is produced in this period. It would be insane to suspend experimentation. That said, there can be significant shifts in consumer motivation and user behaviour. So you might explore season-specific hypotheses, and you should always deal with questions of generalisation and transferability. 

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

Haha. Bayesian, but really it depends. If I’m comparing GA segments in pre-test analysis, a good ol’ T-test or Chi-Square analysis does the trick.

What is your favourite Experimentation tool?

I have a few favourites, depending on the context. But I have to say this: a tool is just an enabler, it’s not even close to being the most important factor. 

How would you suggest picking the right Experimentation tool?

If you’re starting out, Google Optimize is a good low-friction option because a) it’s Google and b) it’s free. As the programme matures, you’ll outgrow that and want to invest in a premium tool. By now you should know enough about your needs to inform your decision or approach a reputable independent agency for advice. My advice is to obsess over something else. Get the systems, process and culture in place. The tool will take care of itself if the bedrock is solid.

100%. There is far too much focus on tooling. Here’s an interesting question. Do you ever miss snake handling?

Oh I still do it. I’ve always been fascinated by snakes. As a young boy I went looking for them in nearby fields. In the area where I live we’re surrounded by nature, home to some rather venomous snakes. Occasionally I get called by people in the neighbourhood to remove a cape cobra or boomslang.

READ  WeTeachCRO's Sam Irvine partners with clients and helps them not get bogged down with statistics

We’ll have to include a photo of you doing that. It’s a sight to see.

Describe Johann in 5 words or less.

Get to the point. Quickly.

On that note, haha, thank you for joining the conversation. I hope you enjoy your leave!

About Experiment Nation

Experiment Nation connects Experimenters from around the world to new ideas – and to each other.

We send a newsletter every week.


Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

You may also like

Ana Catarina Cizilio tells us about CRO in Brazil and the mindset needed for CRO

Ana tells us about the differences between the US and Brazil, as well as about the mindset you need to Read more

Product Experimentation: A chat with Moshe Mikanovsky about building competency and fluency with data (Part 2)

Jaya and Sid chat with Moshe Mikanovsky, a Product Manager with over 30 years of experience, about the importance of Read more

The 2021 Experiment Nation CRO Jargon Tournament

Can you guess 2021's most important buzzword? Check our leaderboard to see our best guessers so far!

Product Experimentation: Building a competency and fluency with data (Part 1)

Jaya and Sid chat about building competency with and a fluency with data to get a better idea of what's Read more