Test before you build anything or you’ll waste time course-correcting with Rommil Santiago

There’s a lot of talk about how to find things to test, how to analyze tests, and the importance of iterating on tests. This week, I was thinking about how I could add something a little different to the conversation and decided to share my thoughts about when to run a test – particularly when building products. 

Not every insight is derived from an A/B Test with Daniela Marquez

Curiosity needs to be a driver in every experimenter’s toolkit. What is going on in the head of my customer? What is really their job-to-be-done? When buying flowers, we typically think of the standard occasions: Mother’s Day, Valentine’s Day, new baby & sympathy. But if we were to structure our ecommerce website on those occasions alone, we may be addressing the majority, but we’d be missing out on the real motivation behind the purchase as well as other reasons people buy flowers.

What Hawaiian Pizza can teach us about creating a more inclusive CRO industry

Pineapple on pizza. If you ever want to have a heated debate with a group of people, ask them if pineapple belongs on pizza. You’ll hear arguments about how people should be allowed to like what they like and that should be OK, as well as, pizza is XYZ and that anything different cannot beContinue reading “What Hawaiian Pizza can teach us about creating a more inclusive CRO industry”

How Wordle proves that Experimentation works

If you haven’t tried it yet, Wordle is a popular online game where you have to guess a 5-letter word within 6 tries – and it’s quite fun and addictive. Every day there is a new word to guess, and everyone gets the same word. It cleverly lets you post how you solved (or didn’t) the daily puzzle – which often sparks some lovely conversation online. Interestingly, there are a lot of parallels between Wordle and the Craft of Experimentation. Find out how.

Can you run tests on sites with little traffic?

From a pure experimentation perspective – one absolutely can forego using statistics and just see what happens. We can believe the dictionary, the dictionary definition of an experiment is: “a scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact.” A scientific procedure (aka the scientific method) states nothing about having to use a statistical approach for analysis. However…

How to verify your data truly reflects user behaviour with Florent Buisson

Florent Buisson is a decade-long behavioral economist who published a practical guide to behavioral data analysis in business. It’s the book he wished he had when he started in analytics.

In this episode, he shares:

Why intuition can actually be a good thing in data analytics. If you need to know R or Python to understand your users. Why and when causal analytics are important.

How do I change people’s attitudes towards Experimentation?

Of course, there are a million ways to answer this question that range from demonstrating wins, being inclusive, and evangelism. This week, I thought of sharing a couple of the little things you can do that, if done consistently over time, can start to realize gains in the long run. Words matter Experimenters and CROsContinue reading “How do I change people’s attitudes towards Experimentation?”

How to make a better sandwich: An introduction to multivariate testing with Danielle Gunraj

A/B Testing can be intimidating to the uninitiated, but it doesn’t have to be! I want to introduce testing concepts via a real-world scenario that we can all relate to: Dessert. I’ll cover what an MVT is, hypotheses, independent variables, dependent variables, results (of course), and design flaws. And if you need any more convincing, I have four words for you: Monte Cristo Bread Pudding.

Choosing the right A/B testing tool with Product for Product 

Matt Green and Moshe Mikanovsky of Product for Product podcast help you navigate the plethora of A/B testing tools out there.

Whether you’re in product or marketing, listen for a beginner-friendly comparison of tools like Google Optimize, Optimizely, and more.

**This episode, nor those of Product for Product, are endorsements of any of the discussed tools.**

Experiment Cartography :: Building the Perfect Roadmap with Kenya Davis and Krissy Tripp

You can do better than testing button colors, but how do you decide what to test when and what type of test to run? Roadmapping. This session teaches you to be the GPS navigator, helping your team optimize experimentation routes. On this journey you will identify how each path can drive desired impacts by turning a formidable trek into a series of attainable milestones.

How to rally your team around Voice of Customer research with Tracy Laranjo

Tracy is a Toronto-based growth marketer with a knack for conversion optimization. In a past life she’s QA tested “adult” eBooks and helped make it easier for Canadians to write their wills online. Tracy is also one of the co-hosts of the Experiment Nation podcast where she chats with optimizers at the top of their game.

Beyond the 9-5, you can catch her knitting, crushing empires in Civ6, or napping with her cats.

Developing a Culture of Testing/Experimentation with Anil Batra

Anil is a Digital Marketing and Analytics professional with 18+ years of experience. He has worked with companies ranging from startups to Fortune 500 and has helped them improve marketing results as well as develop data driven culture. He has vast experience in Web, Email, Social, Display, Search, Paid Search, and CRM. He holds a B. Tech in Electronics and Communication Engineering from India and an MBA from the University of Washington, Seattle. He is the founder of https://academy.optizent.com – A Digital Analytics training and education company.

Where are the users? with Moshe Mikanovsky

When experimenting with our products, we need to find our users but it’s not always that simple! Where do we find the users? In this session Moshe will go over different types of products and different ways to find users for each of them. Coming out of this session, attendees will have some ideas on where to start looking for their users.

Sina Fak on why CRO is not a person but a multidisciplinary Experimentation team

Sina Fak is the Head of Optimization & Business Intelligence at ConversionAdvocates, the #3 top CRO agency in the world in 2020. Sina shares what businesses get wrong when they hire a shiny new in-house CRO strategist, what to look out for before taking an in-house CRO role, and when a business should consider going agency vs. In-house for CRO.

We Need More ART in CRO with Jimmy Barber

This session is about the conundrum that “data is everything” – a mentality circulating at large the past couple years. There’s so much data, and we often find ourselves simply staring at metrics looking for “+” or “-” to make decisions. The objective of this session is to draw on thinking strategies from physics, of how to approach CRO differently, without any real “data”, to then gleam positive impacts to CRO.

Moving Beyond Spaghetti Testing with Haley Carpenter

The success of your testing program depends on the quality of your hypotheses. But too many experimentation programs lack the processes and frameworks to move from spaghetti testing to strategic experimentation. Haley shares two practical frameworks to help you form better hypotheses and objectively rank test ideas.

Speero’s Experimentation Maturity Audit with Edgar Spongolts

Speero’s Director of CX Optimization, Edgar Spongolts, dives deep into the influential agency’s Experimentation Maturity Audit and why maturity matters. Learn about the four pillars of a successful experimentation program and what it takes to reach the highest levels of maturity. Take the audit yourself here: https://speero.link/ExperimentNation

Creating an experimentation program that’s more than A/B testing with Annika Thompson and Paul Randall

Too often experiments are squandered. But if you put customer problems first and design the right type of test you can ensure every experiment is valuable to your business. The following talk will give your experimentation programs guardrails to make sure you’re not testing for testing sake.

Best practices might be killing your experimentation culture – 3 reasons why with Ana Catarina Cizilio

Everybody is doing it, so you have to do it too, right? As my mom used to say: you’re not everyone! Best practices are not experimentation and you should know that. I want to prove to you this is the easy way, but it’s not the right way for a healthy, profitable and experiment-oriented business. These tactics don’t last long.  In this session, we will discuss why, statically, best practices don’t work. It’s fun to know about the competitors, their results and ideas. It may generate new ideas and grab your attention when you’re starting to experiment, but that’s the only good thing. To have a great experiment culture, you need to learn from your tests. Let’s talk more about this?

Ripple Benefits: CRO as the foundation of other digital marketing efforts with Spencer Gray

CRO should be the foundation that all of your other digital marketing efforts rests upon. Through what we call our ‘Ripple Benefits’, I’d love to showcase just how CRO not only improves your conversion rates, but your SEO, Content Marketing, Paid Ads, and more. Testing is key and should be the foundation that drives where your efforts are focused.

CRO não é um profissional e sim um processo (participação especial André Gillich)

Nesse episódio nosso convidado André Gillich, Head of CRO, compartilha sua experiência na Jussi crescendo a cultura de experimentação, começando pela nova abordagem em que “CRO não é um profissional e sim um processo.”Exploramos o impacto da cultura “Agil” e o desafio do foco em outputs não outcomes.Fechamos o podcast de forma prática compartilhando iniciativas tomadas em nossas empresas para acelerar a flywheel de experimentação.

The Science of Business with Jonny Longden

Our industry is focused on the individual elements of what we do and how we do it, but not why we do it and the overall philosophy for it. This is part of the reason that businesses don’t give it the traction it deserves; senior business leaders are not interested in the execution, they’re interested in the overall benefit.

Specsavers’ Steph Le Prevost on growing a top UK CRO program

Steph walks us through the challenges of growing Specsavers’ CRO team and how she overcame them, her experimentation red flags, and her take on if CRO certifications are really worth it. Plus: A comparison of the CRO job market between Canada vs. the UK, including how well you can expect to be paid as an optimizer.

Microsoft Digital Stores’ Tim Mehta on strategies for low and high maturity experimentation programs

Tim shares his tips for running low and high maturity experimentation programs, what it’s like to optimize in-house vs. agency, when it is (and is not) ok to make compromises, and to how make your program’s results actually drive action. Plus: Tim’s take on those controversial UI polls on LinkedIn.

Migrando para a área de experimentação vindo do Jornalismo (participação especial Fabricio Migues)

Hoje começamos um novo papo em nosso podcast: como migrar para a área de experimentação? Nosso convidado especial é Fabricio Migues, CRO Manager na Small World Financial Services, conta pra gente como foi sair da área de jornalismo e relações públicas e adentrar na experimentação, como aproveitar seus aprendizados e conhecimentos da área de comunicação dentro de CRO e como isso o tem ajudado.

Shopify’s Shanelle Mullin on tough buy-in conversations and how we can empower future optimizers

Tracy Laranjo chats with Shopify’s Experimentation & Analysis Lead, Shanelle Mullin, about tough buy-in conversations, her take on the “I don’t dream of labour” trend, making experimentation more representative of diverse voices, and how we can empower the next generation of optimizers.

Série: Estruturando uma área de CRO na sua empresa – episódio 2

Neste terceiro episódio contamos com a presença de André Vieira, fundador da Looptimize. Ele nos conta um pouco mais da sua visão sobre:

  • Qual o ponto de partida para começar a investir em CRO?
  • Popularização da experimentação: heurísticas e boas práticas — pontos positivos e negativos
  • A qual área CRO melhor pertence dentro de uma empresa?  

Série – Estruturando uma área de CRO na sua empresa – episódio 3 (Participação especial André Vieira)

Neste terceiro episódio contamos com a presença de André Vieira, fundador da Looptimize. Ele nos conta um pouco mais da sua visão sobre:

  • Qual o ponto de partida para começar a investir em CRO?
  • Popularização da experimentação: heurísticas e boas práticas — pontos positivos e negativos
  • A qual área CRO melhor pertence dentro de uma empresa?  

Frequentist & Bayesian: Comparison and Practical Application (R script attached)

By Andrea Corvi Canon vs Nikon and iOS vs Android are just two modern examples of “dualisms” that divide crowds. The long-lasting Frequentist vs Bayesian fight is not any different and like the other two examples above, it’s still far from finding a winner. This doesn’t affect just the CRO world but the same typeContinue reading “Frequentist & Bayesian: Comparison and Practical Application (R script attached)”

The weekly buzz: May 26, 2021

👉 Upsides to Unshipping: The Art of Removing Features and Products by Reforge

👉 Browser Privacy Policy Updates – Optimization Effects by Kelly Wortham and Cory Underwood

👉 Visual Hierarchy: A 2021 Optimization Guide For Marketers by Unbounce

⏰ Two days to go.. If you haven’t already, submit your case in individual, team or organization wide categories for Experimentation Culture Awards by May 28th. 📢

👉 Upsides to Unshipping: The Art of Removing Features and Products by Reforge

👉 Browser Privacy Policy Updates – Optimization Effects by Kelly Wortham and Cory Underwood

👉 Visual Hierarchy: A 2021 Optimization Guide For Marketers by Unbounce

⏰ Two days to go.. If you haven’t already, submit your case in individual, team or organization wide categories for Experimentation Culture Awards by May 28th. 📢

The weekly buzz: May 21, 2021

👉 Multiplication instead of division: Molly Stevens and Lukas Vermeer team up to tell us why research and experimentation are the “peanut butter and jelly” of customer insights.

👉 What you can learn from one extra experiment: Kevin Dunn explains how you may miss out if you are only doing A/B testing, or following the Change One Single Thing (COST) approach to experimentation.

👉 How does Failure help you in Experimentation? Craig Sullivan shares Tesla’s experience with experimentation and failure.

📢 If you haven’t already, submit your case in individual, team or organization wide categories for Experimentation Culture Awards by May 28th. ⏰

The weekly buzz: May 14, 2021

Here’s the latest Weekly Buzz. 🐝 Links in the doc!

👉 Why we use #experimentation quality as the main #KPI for our experimentation platform from Booking.com

👉 A/B Testing and Experimentation: Tactics From Matt Gershoff

📢 If you haven’t already, submit your case in individual, team or organization wide categories for Experimentation Culture Awards by May 21st. ⏰ And remember…

“It’s not about that one experiment, it’s about improving the process, structure, trustworthiness, democratization and motivation for experimentation and data-driven decision making.”

Adventures in Experimentation: The importance of the voice of customer to conversion optimization – as well as to the entire business

Eden and Positive John speak about how recording your sessions with clients helps reduce the need for taking notes, how qualitative research is key to not just CRO but for all things business, and how no one likes pop-up surveys except for Positive John.

SplitBase’s Raphael Paulin-Daigle talks about convincing clients to invest resources towards optimization

Hi Raphael, thanks so much for taking the time to chat with us! How have you been? I’ve been good! While the COVID situation here in Montreal is making things a bit boring, business has been stronger than ever, and we work with super fun clients, so can’t complain about that side of things –Continue reading “SplitBase’s Raphael Paulin-Daigle talks about convincing clients to invest resources towards optimization”

Point/Counterpoint: Exploring different experiment prioritization frameworks with Marion Ravel, Patrick Buffum and Trisha Utomi

On today’s episode of Point/Counterpoint, Trish leads Patrick and Marion through a discussion about: Their CRO New Year’s resolutions, and the pros and cons of several experiment prioritization frameworks.

Twilio’s Justin Coons shares what you need to know about SEO Experimentation

Rommil interviews Twilio’s Justin Coons and they talk about: Not only can you run experiments for SEO, but you also should, Which is better, short-form content or long-form content, and What metrics to optimize for when running SEO Experiments

Product Experimentation: Jaya and Siddharth explore how to Experiment during the Product Development Life Cycle

Jaya and Siddharth explore how Experimentation fits into the Product Development Life Cycle, how to work with challenging stakeholders, share stories about how they’ve leveraged Experiments to generate results.