Category Archives: Interviews with Experimenters

Data-backed web design refers to the approach of utilizing empirical evidence and user insights to inform the design and optimization of websites. By analyzing data such as user behavior, click-through rates, conversion rates, and heatmaps, designers can make informed decisions about layout, navigation, content placement, and visual elements. A data-backed approach helps identify user preferences, pain points, and areas for improvement, resulting in enhanced user experiences. A/B testing, user surveys, and analytics tools are commonly employed to gather data and validate design choices. By leveraging data, web designers can create user-centric interfaces that drive engagement, conversions, and overall business success.

In the world of Conversion Rate Optimization (CRO), Eduardo, a seasoned practitioner, shares his journey and the evolving landscape of CRO in Brazil. He underscores the importance of maintaining a problem-solution mindset and learning from mistakes to drive meaningful impact. One core principle he highlights is the need to focus on hypotheses that yield significant results, comparing them to squeezing fruit for juice. This strategy involves exploring promising hypotheses further and discarding those with neutral or unimpressive outcomes, a practice essential for effective experimentation, whether in-house or at an agency. The key takeaway is to persevere with ideas, akin to squeezing every drop of insight from hypotheses, and leveraging comprehensive data for a deeper understanding, a competitive edge held by in-house roles in the experimentation field.

Integrating landing pages is crucial for tailored user experiences. Without it, visitors are directed to generic pages lacking personalized messaging. Landing pages are designed to enhance conversion rates and ROI. E-commerce companies benefit by using various types of landing pages, like the classic hero for focused products, listicle hero for storytelling, collection hero for multiple items, and content advertorials for top-funnel content. Mistakes include using one-size-fits-all pages and lacking customer-oriented content. Improve by collaborating with ads teams, using customer voices, and continuous experimentation. Claire More from Splitbase shared insights on optimizing landing pages for effective campaigns.

Don’t test everything. Experimentation isn’t about proving how clever you are. Mistakes in testing ideas from your own head are common; personal bias clouds results. High-performing experimentation needs executive buy-in and integrated roles. Successful teams engage diverse players with vested interest, ensuring hypotheses formation and metric selection are comprehensive. Valuable experimentation saves time by testing what not to do. Rather than just chasing conversion lifts, recognize when improvements won’t make a difference. Cultural shifts, humility, and prioritizing efficient use of time are key components of effective experimentation programs. Test what makes sense.

In this podcast episode, Mark Eltsefon, a senior data scientist and experimentation evangelist, discusses building an experimentation culture in different companies. He shares his journey from starting as a software developer to becoming passionate about data science and experimentation. He contrasts the experimentation culture at TikTok, where data-driven decisions were central and democratization was encouraged, with his current role at an e-commerce company specializing in print-on-demand. He emphasizes the importance of educating stakeholders about data-driven decision-making, and he highlights the need for strong experimentation infrastructure and high-quality hypothesis testing. He believes that while democratization is valuable, it’s important to balance it with maintaining the quality of ideas and statistical rigor.

Juliana Amorim emphasizes that personalization is not a silver bullet and recommends its use when aligned with strong hypotheses and business goals. She co-founded Croct, a personalization experimentation platform, focusing on adapting user journeys to context. Examples include tailoring websites for diverse buyer personas within B2B companies and customizing content for users at different stages, like newcomers or existing customers. AI-based personalization and rule-based personalization can coexist. Brands should prioritize personalizing their websites and consider first-party data support.

Building a personal brand as a conversion rate optimizer is crucial in today’s competitive digital landscape. Your personal brand serves as a powerful tool to establish credibility, showcase expertise, and attract new opportunities. By cultivating a strong brand identity, you position yourself as an authority in the field, gaining trust from potential clients or employers. A well-developed personal brand helps differentiate you from others, highlighting your unique approach and methodologies. It opens doors to collaborations, speaking engagements, and thought leadership opportunities. Additionally, a strong personal brand creates a lasting impression, leading to increased visibility, a larger professional network, and ultimately, a steady stream of clients or job offers.

By implementing ideation strategies and fostering a culture of experimentation, companies can drive innovation and stay ahead of the curve. Building an experimentation culture is crucial for companies that want to innovate and stay ahead of the competition. It starts with creating an environment where experimentation is encouraged and failure is seen as an opportunity to learn and improve. Teams should be given the freedom to try new ideas and test them quickly using small-scale experiments. It’s important to collect and analyze data from these experiments to make informed decisions about what works and what doesn’t. Lucas Vos interviews Lucia van den Brink in this great AMA session.

Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.

David Mannheim, author and expert on personalization, discusses the importance of personalization in marketing. He emphasizes the need for brands to focus on building relationships with customers based on trust and care. Mannheim distinguishes personalization from segmentation, highlighting personalization as a communication principle rather than just a marketing tactic. He also mentions the challenges faced in implementing personalization, such as the pressure for immediate returns and the balance of metrics.

We sat down with Jason Chappel, founder of Defiant, a specialist Customer Value Optimization agency, to discuss what sets CVO apart from CRO. Jason shares his insight on the importance of understanding and optimizing the entire customer journey, from acquisition to retention, and how CVO can help businesses increase customer lifetime value and drive sustainable growth in your business strategy.

We all know that humans are very complex and are hard to predict how they will behave. This is why we want to do use multiple methodologies in tandem to understand our customers.

To have any real influence or buy-in, you need to be able to tell a good story. Khalil recently interviewed Priya Bhatia to get their advice on how to tell a good story.

Optimizing forms during checkout is crucial for a seamless user experience. By simplifying and streamlining the process, businesses reduce cart abandonment rates and increase conversions. Shortening form length, utilizing autofill options, and providing clear instructions enhance efficiency, minimize user frustration, and ultimately boost customer satisfaction, loyalty, and revenue.

Nicoleta walks us through a case-study where she explores a 1 column form vs a 3 column form in checkout.

Learn A/B Testing Statistics, how to use Overall Evaluation Criterion, Twyman’s law, how much traffic do you need to run A/B tests, and other statistical concepts all Experimenters must know.

Are you looking to replace Google Optimize? On Experiment Nation’s recent podcast, Charlotte April Bomford chats with Kenneth Kutyn about replacing Google Optimize, how to select a new vendor, and whether it’s worth building your own Experimentation Platform (spoiler alert: the answer is usually no).

Tim Thijsse, a senior CRO (Conversion Rate Optimization) consultant at Orange Valley, discusses AI and the future of CRO, and the importance of incorporating qualitative research into company-wide processes. He emphasizes the challenge of linking user behavior and mindset to knowledge and information. Tim shares his background, starting as an interaction designer and later transitioning to UX design and CRO. He highlights the evolution of CRO, the need for a culture of experimentation, and the role of qualitative insights in optimization strategies. Tim also discusses the challenges companies face in adopting an experimentation mindset, such as aligning strategy, operations, and technical aspects, and the importance of having a common metric and collaboration among teams.

Ayat Shukairy emphasizes the importance of addressing the functional, social, and emotional aspects of customer needs when designing websites. She shares seven lessons learned from her experience in conversion rate optimization.

Georgi Georgiev performed a meta-analysis of over 1000 A/B tests and has a great deal to share about:
– What you should expect your win rate to be
– The benefits of sequential testing
– Why you should report on confidence intervals

Let’s face it, experimentation is completely misunderstood. The power and the benefit and potential of it is to innovate and develop a business. And when you think about it, that’s what experimentation has always been in relation to the scientific method. If you think about spaceflight, medicine, anything great that’s happened in the history of modern civilization, this has all happened as a result of the careful application of the scientific method.

Experimenters obsess about many A/B Testing Statistics concepts – but not all of them are worth worrying about. Statsig’s Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels – some of his thoughts may surprise you.