Things I’ve learned about Experimentation in 2019
Highlights from 2019’s Conversion Conversations
For those who are new to my blog (practically speaking, that should be all of you), I started this thinking I’d write a few blog posts, share some mental frameworks, and pepper in some interesting interviews just to mix things up. But as soon as I started having “Conversion Conversations” with people?—?it took on a life of its own.
Since I started, I’ve had the pleasure of interviewing over a dozen people (though I couldn’t get all of the interviews posted this year) involved in experimentation from around the world. I never seek out anyone in any particular occupation or field, or whether they work on digital products or physical ones. I didn’t care if they had 20 years of experience or just 1. The only thing that matters is whether they love experimentation?—?and that formula has worked out thus far.
So with the new year around the corner, I figured it was time to not only say thank you to all the awesome folks who took time out of their busy schedules to chat with me but to also re-share some of the best nuggets of knowledge I’ve heard in 2019. It is the season of giving after all. With that said, let’s get this started!
On the topic of helping people embrace experimentation
“Understand different value currencies: People always argue…what is the value of doing this and that. Having a better understanding of the different currencies of value aside from monetary such as operational efficiency, relationship management, culture and confidence booster, etc. Data itself is a very rational thing yet users are emotional non-rational beings.”?—?TELUS Digital’s Jason Yang
“I think there is a fundamental disconnect in what experimentation can do and how it’s used. Experiments alone have very little predictive capabilities when it comes to the lasting impact of a feature. A feature that generates $50K during the course of a month-long experiment may generate $1M over the course of a year, or it could generate half that, less than half that, double that, or zero. Using confidence intervals as declarations about the future is the most surefire method to have PnL teams scratching their heads at your multi-million dollar optimization claims.”?—?Convoy’s Chad Sanderson
“I think a true culture of experimentation has to be grounded in humility, curiosity, a drive for growth, and a level of comfort with uncertainty. Experimentation can’t just be a means to an end, it needs to be seen as a cost of doing business and integrated in the foundations of the organization.”?—?Widerfunnel’s James Flory
On the topic of physical products and Experimentation
“Building out a prototype for a hardware product takes a long time but you still need to get feedback from your customers along the way. We found that verbal descriptions, drawings and renders can be great substitutes until a prototype is complete.”?—?Stabilo Medical’s Josh Lobo
On the topic of Growth and Experimentation
“In my opinion, a ‘growth marketing’ role without rigorous experimentation is not growth at all.”?—?Growth Marketing Today’s Ramli John
“I thought experimentation was all about driving traffic online, optimizing websites, copy, etc., and seeing the results. I would remind people that while this may work in some cases?—?consumer, e-commerce, SMB SaaS, etc., it may not work at all.”?—?Coohom’s Kareem Azees
On the topic of the importance of statistics in Experimentation
“Statistics should act as the foundation for the design of experiments. It is needed to establish valid and replicable results. That said, I believe statistics is not sufficient at providing a complete picture. Substantial domain expertise is also essential in designing effective experiments. It should assist in guiding the experimental design process: identifying potential treatments, evaluating the feasibility of designs, and synthesizing the results obtained.”?—?Angela Kang
On the topic of personalization and Experimentation
“Starting with a basic business rules is always a good idea. You may not come up with the best rules, but you will discover important features from the set of rules. Those features can be further used in machine learning models.”?—?Ritual’s Grace Du
“If you’re not careful, personalization can exhaust your design and content team’s resources quickly.”?—?Cloudflare’s Scott Olivares
On the topics of UI and User Experience Research
“I think it’s so critical to eliminate any biases and assumptions as early as possible. A lot of times, our clients come in with very strong opinions about what a landing page should look like, so a lot of times, we try to do research to either prove or disprove our clients’ hypotheses. We usually end up creating two landing page options- one that corresponds to exactly what the client asks for (usually where all key components are “above the fold” *cringe*) and another version that is based on the audience research and measurement framework that we’ve established.”?—?Edelman’s Nicola Ambrogio
“…I disagree with any forms of taboo. I think the right approach is to ask questions and continue to measure effects without any popularized stigma. Taking this approach we have discovered flat patterns, as well as ones with low and high effect potentials?—?that’s a good thing I think.”?—?GoodUI’s Jakub Linowski
So with that, I’ll end this by saying that I look forward to the new year, all the new learnings ahead, and having even more Conversion Conversations with you!
If you’d like to be featured on a future post reach out to me at rommil@experimentnation.com!
Also, I’m aiming to hold an Experimentation and Personalization event this year in Toronto, if you’d like to be involved?—?reach out to me!