Best practices might be killing your experimentation culture – 3 reasons why with Ana Catarina Cizilio Experiment Nation

Home / Weekly Newsletter / Best practices might be killing your experimentation culture – 3 reasons why with Ana Catarina Cizilio

AI-Generated Transcript

Ana Catarina Cizilio 0:24
Hello have experimenters from all over the world. My name is Anna Katharina, I’m from Brazil. I’m a compression analyst, and UX specialist, and I’m so happy to be part of this conference. And I’m here today to talk about best practices. Yeah, I know, you already followed one, I already followed one. It’s a weird thing to say out loud. Yes, maybe. But we definitely now need to talk about this.

Ana Catarina Cizilio 0:56
Best practices are everywhere, are every common in every marketing area. And we want to know what everybody is doing. So we can just copy and paste the same strategy and cross our fingers that it’s gonna work for us, too. When I first started, as a CRO analysts, I loved to test best practices. Because it’s very easy, when you simply don’t know where to begin, or you don’t know what to study when you don’t have much references. And I learned that actually, it’s very easy to win a test. It’s very easy. But it’s very hard to dig in, actually learn why it has won. And when you don’t know why youth has one, you can have the best storytelling in the whole world. It’s not gonna work. And the results won’t last long.

Ana Catarina Cizilio 1:57
And I will tell you why. And I will tell you why. In my experience, I had worked with zero agencies, ecommerce companies. And now I’m currently working in a b2b business. And I’d like to show you three examples of this. Why? Why best practices don’t work. The first one, it’s very common in best practices for some CRO groups, saying that you should decrease the number of fields to increase your conversions. But how about lead qualification? Do they know that my sales team is gonna kill me if I don’t correctly qualify my leads? I don’t know.

Ana Catarina Cizilio 2:46
But it’s something to think about. Also, when I work in E commerce, a designer asked me to test fix it add to cart button in the bottom of the page in the mobile version of the website, we were so excited to test this because the new layout was awesome. And in the first two weeks, we increase our sales by 15%. With this test, but the control and the variation was suddenly performing more and more similar. And I discovered that novelty bias. I will talk more about this later, but keep going. This third example, it’s very important to it’s not about best practices, but it’s having to learn from your resume. It’s one of my successful tests was a small change of copy and creating a new CTA in the construction companies website. It increase 100% The number of fleets awesome, right?

Ana Catarina Cizilio 3:58
Awesome. I know. But this change was also not good for SEO, nor for branding. And we had to fight a battle inside the company to justify our changes. Why we were proposing that new CTA that new copy even weighed all the salespeople extremely busy with all the qualified contacts. We made some changes to the winning variation, so we could share our strategy among the company and share our goals with other areas. And this happened only because I had a deep understanding of what of why the test was so important, and most important why users were behaving that way.

Ana Catarina Cizilio 4:51
Otherwise, the client would have throw all those results in the trash and Finally, let me tell you once and for all, why best practices don’t work. Correlation is not causation. Those A B testing results you see in these gurus website can know for sure, if the results are due to the new changes they made, or due to something random, like a sale holiday, some pandemic virus new ads, maybe I mentioned on the medium things that we can’t, and we cannot control. And I’d like this quote so much. Einstein said, no amount of experimentation can ever prove me, right? A single experiment can prove me wrong.

Ana Catarina Cizilio 5:49
The second one novelty bias. In the example I gave you, before, we added a new cool thing in the website, people like it in the first days, but then it became just one more feature, just like every other ecommerce store has a Add to Cart button, fix it. We want the tests because it’s, it was a novelty, but it didn’t impact the revenue. And what’s more important for your business. And if we were talking about b2b, for example, every test we do only has an effect, like two months or more from now. So we could have won this test with this best practice. But would it have made a difference to the business?

Ana Catarina Cizilio 6:44
I don’t think so. And this leads us to the last motive test in to learn, test to win tests to learn. If the test doesn’t have any direct impact on the revenue, it’s very difficult to prove the value. If you did learn from it. Let’s say you’re changing the copy and the image of a button color for and the reason the tests increase your conversion rate. How would you explain this to your clients or to your boss?

Ana Catarina Cizilio 7:23
Ah, we change it because we like better dissuade, oh, this is not simply part of the experimentation culture. This will lead you to in your business to nowhere nowhere. And I would like to, to quote the Stephen Tunc. In his book experimentation works. He wrote all experimentation, whether conducted in Edison’s laboratory a century ago, or in an online retail channels to date should generate knowledge.

Ana Catarina Cizilio 8:01
That knowledge comes as much from failure as it does from success. Finally, the output of this presentation is test to learn not to win. Because nowadays, when winning a test only takes a day or two until you discover that actually, you were completely wrong about your business, about your users, about your clients about your customer. Thank you so much for writing this. Let’s connect on LinkedIn and talk more about experiments. Share your thoughts. Thank you

—–

Also available as a podcast below:

If you liked this interview, signup for Experiment Nation’s bi-weekly newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://lnkd.in/g6rM8w25

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.