The top 3 things CROs forget to do with Alexandra Matheny

Also available in audio format:


Alexandra Matheny 0:10
Hi, everyone, thank you for tuning into my session optimizers thing they know data, but few do these three things. My name is Alexandra Matheny, and I am the Director of Product Marketing. I’ve been working in consumer data and intelligence for most of my career. So this is a topic that is very near and dear to my heart. I worked at Optimizely. And because of these two roles, I’ve had this extreme privilege to see hundreds, if not 1000s, of experiments planned and executed. And something that I have seen both at Optimizely.

Not enough anyway. So I wanted to take this opportunity to talk about this because there’s so much opportunity left on the table to leverage customer data and customer insights. What I’m proposing is that good experimenters are data driven, but great experimenters, they build data driven segments, they analyzed by business goals, and more importantly, they take action on their experiment results by connecting them to analytics and activation tools. And by doing so, they create a virtuous cycle of breaking down data silos, and leveraging insights to create better customer experiences. Now, what makes me say this, let me do a little bit of table setting. Around 2017, we saw that the market was getting more mature and more ready for server side testing. We had more and more requests coming in from growth and marketing teams who were willing to move to servers that experimentation, because they wanted to put in place more advanced and sophisticated experiments, but also because an increasing number of developers were involved in configuring those servers that experiments. And as we were talking to our customers, what we learned is that teams wanted to use the same analytics and data integrations with the tools that they use every day. Having the same UI, the same data model. For server side and client side experiments came secondary to having access to the tools that they are used to using. But at that time, this was a really difficult task for server side experiments. It needed extensive developer support. And that was exactly the challenge that we wanted to solve. And our solution was chameleon hybrid, a capability that allows users to take action on their customer data very easily, because it lets users create server side experiments with client side capabilities, which includes plug and play integrations, to other martec tools. And this is what makes me excited about chameleons hybrid capability that this is an easy means for turning a good experimenter into a great one, because it makes the syncing and leveraging of customer data so much easier. So that is the background or the context for my thesis. Now, let me jump into the details. The number one characteristic of a great experimenter is that they consolidate their customer data for precise and persistent segmentation. And they use the same segments across all their tools in their tech stack. In other words, in in the context of experimentation. What this means is that they sink their third party audiences from other martech tools into their experimentation platform, so that they could use those audiences for precise targeting. At scale, of course, they don’t waste their time recreating their existing audiences in every single tool that they use, or worse yet create poorly defined proxy audiences for them. And these great experimenters rely on customer data to determine who to target with what message what offer or feature. Now, because of my background, I do have a soft spot for integrating with customer data platforms where all the information that we have about our customer, their purchase history, their onside behaviors, loyalty points, support ticket information, even platform usage is unified under one persistent profile. But the good news is is that you do Want to have to have a CDP to have a great experimentation strategy that you can then apply to your experiment targeting. Because a chameleon, we have seen plenty of clients who grew their experimentation programs and expanded their optimization opportunities, leveraging audiences that they created an analytics platforms, including Google Analytics, or Mixpanel. But personally, I have met too many experimenters who felt like third party audience targeting was just an extra step that they couldn’t afford to take. So I always like to challenge them. Listen, you have invested so much time and effort into segmenting your customers in your messaging, or analytics or customer data platform. But what is really the point of having that segmentation strategy if you are not putting it into action, if you are not making good use of it. So take that extra step, I implore you use the segments that you have worked so hard to create, whether they are in Google Analytics, Mixpanel, or, and bring them into your experimentation platforms, so that you could target your visitors more precisely, go beyond what’s available out of the box in your A B testing tool. The other day, I saw someone from the CRO community share a post that began like this. A summary of your AB test results is not the same as its learnings. And this rang really true to me. Great experimenters analyzed by meaningful business goals, and they draw actionable conclusions from their experiments. Too many programs today are set up to have you believe that the results of an experiment that the winner is this and the percentage of lift that we have seen is this. We are made to believe that that is the end of the road, when in reality is just the beginning. I have seen this at Optimizely. And I have seen this at chameleon as well. An experiment concludes our experimenter identifies the winner. And then they kind of just leave it at that. Great experimenters don’t stop at acknowledging the metrics that they see in their experimentation platforms reporting dashboards, they think about the implication of those results and make recommendations for next steps. My litmus test for telling who is a good versus agreed experimenter, is this when you are presenting your experiments result, what is your last slide? Do you finish with results? Or do you finish with recommendations? If it’s the latter, you are a great experimenter. And this brings me to my third and final point, great experimenters take action on experimentation results. In order to complete the virtuous cycle that I was talking about earlier, experimenters need to get their results available in other third party tools in their tech stack, because they need to further analyze their customers behaviors and activate against the members of their experiments based on test results. And this matters regardless of the type of experiment or you are a marketer, a product manager or a dedicated optimization expert or pro expert. Taking action on experiment results will help you develop a deeper understanding of your customers. And if you’re a marketer, this is also an excellent opportunity to continue interacting with your customers uninterrupted. When we sync our experiment results to an ESB email service provider for instance, we can follow up with our customers and re engage them with personalized and relevant messages that are aligned with the experience that they have seen on our website earlier. Because imagine how poor the customer experience would be without this alignment. On the other hand, when you are experimenting and sending experimentation data to your analytics platform, you can gain better understanding of the context of your tests. If your analytics platform, for instance, offers a customer journey analysis tool like content square does, you can look and see how your visitor arrived on your site. Where did they go after seeing your experiment? And this really gives you the context of the customer journey that includes your optimized experience in the middle.

And not only can you analyze how changes to your messages or page layout have impacted the customer behavior or conversion rates, you can also identify new opportunities for testing. And you can implement more impactful experiences the next time you’re optimizing something. And that brings us to full circle. Once again, we are looking at our third party data. And we are possibly creating third party audiences that we can then sync over to our experimentation platform to target in our next tests. And that is our complete virtuous cycle for breaking down data silos and leveraging customer insights to create better experiences. But Alexandria, all of this makes sense. But it’s so hard to achieve for server side tests with my current tool. And I don’t disagree, you have to have sufficient resources in order to do these things. That means that you either have developers helping you code your integrations and analytics tracking, or that you have an experimentation platform that makes it easier for you to work with server side experiments. And chameleon is one such solution. It gives you the power to create server side experiments by leveraging client side capabilities. And it plugs into your existing tax doc for quick and easy integrations. So by these definitions, are you a great experimenter? Do you leverage existing customer data for experiment targeting? Do you analyze by meaningful business goals and make recommendations? And do you take action on your experimentation? If yes, I’d love to hear and learn from you. If you’re not there yet, I would love to talk to you as well about how we can help you get there. Thank you for listening. Again, my name is Alexandra thenI. And I’m looking forward to connecting with you in real time, or even in person if you’re in the Washington DC area. You can reach me anytime on LinkedIn or drop me an email at a martini at chameleon calm. And of course, visit us on Thank you. Bye

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox:

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

Rommil Santiago