Category Archives: Announcements

Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways: Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses. Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate. QA is critical but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly. Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden. Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results.

Avishek Basu Mallick, a Senior PM at CheapOair talks about: – Experimentation at scale: Even with high traffic, rigorous experimentation is crucial. – Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas. – Communicating progress: Regular updates with leadership build trust and ensure alignment. – Personalization is nuanced: Distinguish between customization, segmentation, and true personalization. – Generative AI’s potential: Explore its use for simplifying the travel planning experience.

Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers: Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals. AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance. Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results. People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing. Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences.

Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game:

– Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning!
– Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks.
– Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success.
– Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving.
– Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!

Join us for a deep dive into conversion rate optimization on low-traffic sites with these actionable insights Chris Mercer

Measure before optimizing: Understand the current user journey and behaviors to identify areas for improvement.
Test sequentially: Iterate faster on low-traffic websites by testing changes sequentially instead of using AB tests.
Focus on the right metrics: Prioritize actions that drive meaningful outcomes, such as lead generation or revenue.
Listen to the market: Gather user feedback to inform your optimization strategy and meet their needs.
Automate measurement: Leverage Google Analytics 4 to collect and analyze data effectively to guide decision-making.

Discover the power of experimenting to learn and win! ?

Nick Murphy shares his expertise on building learning libraries. Here are 5 key takeaways:

– Learning libraries capture the “why” behind successful and unsuccessful tests.
– Organize your learning library with consistent and digestible fields.
– Use “broad” and “specific” learning columns for a layered understanding.
– Encourage cross-team collaboration by sharing insights company-wide.
– Learning libraries foster a culture of experimentation and growth.

In this episode, senior customer experience leader, Courtney LeBlanc, reveals the ideal partnership between CX and experimenters and shares invaluable insights on:

Forging powerful partnerships between CX and experimenters
Navigating the daily demands of a CX leader
Simplifying CX workflows with customer insights
… and so much more!

Don’t miss out on this enlightening conversation from an expert in the CX game!

How do you get Product Managers to adopt Experimentation? Shagun Aulakh, the Director of Product Management – Experimentation at American Express shares with Rommil here tips.

In this episode, we have Joey Herzberg who shares his thoughts on how AI fits in the experimentation life cycle, the utility of keeping easily referenced records of test results, and the importance of experimentation strategy with regard to sample ratios.

Learn about human-first engineering in this in-depth interview where Shawn David discusses strategies to organize work and to break down tasks into smaller digestible sessions. He explained the most effective ways for teams to discover the work that should be automated the most by focusing on the desires and strengths of individuals, to remove work that is menial especially to them. We explored the impacts on company culture that menial tasks can cause and discussed some of the pitfalls with new technologies as well as lessons to keep in mind while reviewing new tools and technologies.

Join Nima Yassini as he dives into experimentation culture, the journey of building a brand UX and digital marketing agency focused on conversion rate optimization, the integration of digital marketing and experimentation culture, the struggles and benefits of adopting a culture of experimentation, and the need for leadership and reinvestment in the experimentation industry.

CEO and founder of Delivery Layer, Solomon Kahn shares why he thinks that most companies should NOT be A/B testing and what we should do instead. He also shares details about his recent exchange with Ron Kohavi on LinkedIn on this same topic. What side are YOU on?

It’s that time of year again! It’s time for Experiment Nation’s 2024 CRO Salary Report. In this episode, we explore some of the report’s highlights with Tracy Laranjo as well as ask her for her thoughts on how CROs and Experimenters can get paid fairly.

Zach Lebovics shares insights on experimentation and product management. He discusses the importance of documenting assumptions, testing the best-case scenario first, and understanding that experiment failure isn’t solely tied to the outcome. Zach also shares personal experiences and learnings from working in companies like Zynga, Ritual, and Tonal, providing valuable tips for product managers.

Media.Monk’s Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.

Media.Monk’s Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.

Fractional Head of CRO, Tracy Laranjo talks about the power of Mentorship for Conversion Rate Optimizers (CROs). We cover:

Why would CROs need mentorship?
How do you find one?
When are you ready to be a mentor and how do you get that word out?
Would Tracy be a mentor for our listeners?

In an insightful interview at Experiment Nation’s Conference, Bryce York, Director of Product Management at Tatari, delves into several key insights. He explores the parallels and distinctions between CRO and product management, especially in smaller companies where overlapping skills are crucial. Bryce shares his personal journey from entrepreneurship to product management, underscoring the significance of both qualitative and quantitative data in experiment design. Additionally, he emphasizes the importance of documentation and knowledge sharing in cultivating an experiment-driven culture.

Documentation. The unsung, yet incredibly important part of any experimentation program. Great documentation allows you to look back at past experiments and find patterns of what were the most effective levers to inform your testing roadmap and levers you probably can deprioritize. In today’s episode of Experiment Nation, our own Charlotte April Bomford recently spoke to Edmund Beggs about the “Levers Framework” which allows Convert.com to deliver results for their clients in shorter amounts of time. If you’d like to learn more about this interesting framework, watch the full interview.

Simon Girardin emphasizes the importance of learning from every test, capturing value by reviewing data, segmenting results, and analyzing user behavior through session recordings and heatmaps.

Koalatative’s Gerda Vogt-Thomas asks Rommil Santiago questions about how he started Experiment Nation and how he manages it while working a full-time job at Optimizely.

Research is important to Conversion Rate Optimization, but are you leveraging the power of Diary Studies? If not, you’re missing out. Learn more from Mike Green.

Learn from André Morys, CEO of KonversionsKRAFT, on gaining crucial stakeholder buy-in for experiments. André highlights the challenges faced by optimizers and advocates for understanding team dynamics through stakeholder mapping. He emphasizes approaching stakeholders, using empathy to build relationships, and overcoming objections. André shares practical insights on conflict management, including a stakeholder map with color-coded power dynamics. Discover the significance of regular check-ins, fostering trust, and achieving tangible results in experimentation. Master the art of securing support for your experiments effectively.

Will Laurenson shares his journey from startup to becoming a CRO (Conversion Rate Optimization) expert. Will discusses the significance of usability, anxiety, and motivation in CRO, drawing from his extensive experience in running tests and optimizing various businesses. Will’s approach to CRO involves deep insights into user behavior and a focus on aligning products with customer needs, challenging the common pitfalls of assuming product-market fit based solely on initial traction or advertising success.

In this insightful video, Ishan Goel, transitioning from a software engineer to a researcher, delves into crucial CRO statistical concepts. He sheds light on statistical significance in experimentation, explores A/B testing intricacies, emphasizes the efficiency of sequential testing, and discusses challenges in stopping tests prematurely. The talk concludes by highlighting the intersection of experimentation and statistics in the realm of Conversion Rate Optimization.

So much is spent on ads that it’s a no-brainer to invest in ad optimization. Embracing experimentation and learning from failures is essential for long-term success in performance marketing. Experiment Nation’s Tracy Laranjo spoke with Joe Fitzpatrick about reasons why your ads aren’t working and what you can do about it.

Maximizing results in ad network optimization requires a systematic approach. Start with A/B testing on Google Ads and gradually introduce more complex campaigns, balancing technical and creative aspects. Analyze your channel’s performance, ensuring sufficient impressions and conversions for statistically significant results. Shape your product to align with the ad network’s requirements and consider user experience. Prioritize velocity in testing initially, gradually enhancing quality, and encourage collaboration across different platforms for valuable insights in the quest for optimization.

Lucas, a senior conversion specialist, shares his journey in the media industry, emphasizing the transformative impact of A/B testing. He discusses myths about experimentation – debunking them along the way as well as highlights the importance of validation, integration into essential projects, and showcasing negative results. Lucas encourages testing the obvious and unexpected, questioning assumptions, and showing the value of experimentation to reduce risk and aid decision-making, making it an asset for senior managers and specialists.

Eric Melchor, an evangelist at Optimonk, emphasizes the importance of focusing on the product page for effective conversion rate optimization (CRO). He suggests treating website visitors as individuals and creating a personalized and enjoyable customer experience in real-time. Melchor also discusses the significance of capturing visitor interest early on and leveraging smart CRO tactics and personalization timing throughout the customer journey. Additionally, he mentions the value of optimizing the product page, as it can yield significant returns on investment.

We explore various angles of CRO Organizational Structure. We discussed primarily the differences between centralized CRO teams vs a company-wide experimentation culture, and how this is best looked at as a continuum. Companies starting out might benefit from a centralized CRO team whereas more mature optimization programs can benefit more from working experimentation into all facets of the company. How can a company who does 0 experimentation introduce and adopt an experimentation process/culture? We also chatted about experimentation at the customer level, what this means, and how it could be the future of optimization.

Heatmaps to the rescue! Creating an impactful website hero section above the fold is vital for user engagement. Clear value propositions and compelling calls to action are essential, as users often don’t scroll past this section. Heat mapping, combined with other techniques like exit polls and A/B testing, provides a comprehensive understanding of user behavior. While urgency and scarcity tactics can be effective, users have become more skeptical of overused strategies like countdown timers. Understand your audience through analytics data and tailor your optimization strategies accordingly, whether it’s for B2C, B2B, or other sectors. Mobile interactions may have imprecise heat map data, so use it alongside other sources for a well-rounded view.

To ensure the success of CRO projects, it’s vital to involve all stakeholders, including executives and directors, right from the start. This inclusive approach not only helps them understand the intricacies of the CRO process but also demonstrates the depth of strategic thinking behind it. Even skeptical executives can be won over by involving them in CRO calls, showcasing the complexity that goes beyond mere marketing tactics. Such transparency and involvement lead to a more informed and supportive team, fostering successful CRO initiatives.

In the world of Conversion Rate Optimization (CRO), Eduardo, a seasoned practitioner, shares his journey and the evolving landscape of CRO in Brazil. He underscores the importance of maintaining a problem-solution mindset and learning from mistakes to drive meaningful impact. One core principle he highlights is the need to focus on hypotheses that yield significant results, comparing them to squeezing fruit for juice. This strategy involves exploring promising hypotheses further and discarding those with neutral or unimpressive outcomes, a practice essential for effective experimentation, whether in-house or at an agency. The key takeaway is to persevere with ideas, akin to squeezing every drop of insight from hypotheses, and leveraging comprehensive data for a deeper understanding, a competitive edge held by in-house roles in the experimentation field.

Integrating landing pages is crucial for tailored user experiences. Without it, visitors are directed to generic pages lacking personalized messaging. Landing pages are designed to enhance conversion rates and ROI. E-commerce companies benefit by using various types of landing pages, like the classic hero for focused products, listicle hero for storytelling, collection hero for multiple items, and content advertorials for top-funnel content. Mistakes include using one-size-fits-all pages and lacking customer-oriented content. Improve by collaborating with ads teams, using customer voices, and continuous experimentation. Claire More from Splitbase shared insights on optimizing landing pages for effective campaigns.

Don’t test everything. Experimentation isn’t about proving how clever you are. Mistakes in testing ideas from your own head are common; personal bias clouds results. High-performing experimentation needs executive buy-in and integrated roles. Successful teams engage diverse players with vested interest, ensuring hypotheses formation and metric selection are comprehensive. Valuable experimentation saves time by testing what not to do. Rather than just chasing conversion lifts, recognize when improvements won’t make a difference. Cultural shifts, humility, and prioritizing efficient use of time are key components of effective experimentation programs. Test what makes sense.

In this podcast episode, Mark Eltsefon, a senior data scientist and experimentation evangelist, discusses building an experimentation culture in different companies. He shares his journey from starting as a software developer to becoming passionate about data science and experimentation. He contrasts the experimentation culture at TikTok, where data-driven decisions were central and democratization was encouraged, with his current role at an e-commerce company specializing in print-on-demand. He emphasizes the importance of educating stakeholders about data-driven decision-making, and he highlights the need for strong experimentation infrastructure and high-quality hypothesis testing. He believes that while democratization is valuable, it’s important to balance it with maintaining the quality of ideas and statistical rigor.

Juliana Amorim emphasizes that personalization is not a silver bullet and recommends its use when aligned with strong hypotheses and business goals. She co-founded Croct, a personalization experimentation platform, focusing on adapting user journeys to context. Examples include tailoring websites for diverse buyer personas within B2B companies and customizing content for users at different stages, like newcomers or existing customers. AI-based personalization and rule-based personalization can coexist. Brands should prioritize personalizing their websites and consider first-party data support.

Building a personal brand as a conversion rate optimizer is crucial in today’s competitive digital landscape. Your personal brand serves as a powerful tool to establish credibility, showcase expertise, and attract new opportunities. By cultivating a strong brand identity, you position yourself as an authority in the field, gaining trust from potential clients or employers. A well-developed personal brand helps differentiate you from others, highlighting your unique approach and methodologies. It opens doors to collaborations, speaking engagements, and thought leadership opportunities. Additionally, a strong personal brand creates a lasting impression, leading to increased visibility, a larger professional network, and ultimately, a steady stream of clients or job offers.

By implementing ideation strategies and fostering a culture of experimentation, companies can drive innovation and stay ahead of the curve. Building an experimentation culture is crucial for companies that want to innovate and stay ahead of the competition. It starts with creating an environment where experimentation is encouraged and failure is seen as an opportunity to learn and improve. Teams should be given the freedom to try new ideas and test them quickly using small-scale experiments. It’s important to collect and analyze data from these experiments to make informed decisions about what works and what doesn’t. Lucas Vos interviews Lucia van den Brink in this great AMA session.

Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.

David Mannheim, author and expert on personalization, discusses the importance of personalization in marketing. He emphasizes the need for brands to focus on building relationships with customers based on trust and care. Mannheim distinguishes personalization from segmentation, highlighting personalization as a communication principle rather than just a marketing tactic. He also mentions the challenges faced in implementing personalization, such as the pressure for immediate returns and the balance of metrics.

We sat down with Jason Chappel, founder of Defiant, a specialist Customer Value Optimization agency, to discuss what sets CVO apart from CRO. Jason shares his insight on the importance of understanding and optimizing the entire customer journey, from acquisition to retention, and how CVO can help businesses increase customer lifetime value and drive sustainable growth in your business strategy.

We all know that humans are very complex and are hard to predict how they will behave. This is why we want to do use multiple methodologies in tandem to understand our customers.

To have any real influence or buy-in, you need to be able to tell a good story. Khalil recently interviewed Priya Bhatia to get their advice on how to tell a good story.

Optimizing forms during checkout is crucial for a seamless user experience. By simplifying and streamlining the process, businesses reduce cart abandonment rates and increase conversions. Shortening form length, utilizing autofill options, and providing clear instructions enhance efficiency, minimize user frustration, and ultimately boost customer satisfaction, loyalty, and revenue.

Nicoleta walks us through a case-study where she explores a 1 column form vs a 3 column form in checkout.

Learn A/B Testing Statistics, how to use Overall Evaluation Criterion, Twyman’s law, how much traffic do you need to run A/B tests, and other statistical concepts all Experimenters must know.

Are you looking to replace Google Optimize? On Experiment Nation’s recent podcast, Charlotte April Bomford chats with Kenneth Kutyn about replacing Google Optimize, how to select a new vendor, and whether it’s worth building your own Experimentation Platform (spoiler alert: the answer is usually no).

Tim Thijsse, a senior CRO (Conversion Rate Optimization) consultant at Orange Valley, discusses AI and the future of CRO, and the importance of incorporating qualitative research into company-wide processes. He emphasizes the challenge of linking user behavior and mindset to knowledge and information. Tim shares his background, starting as an interaction designer and later transitioning to UX design and CRO. He highlights the evolution of CRO, the need for a culture of experimentation, and the role of qualitative insights in optimization strategies. Tim also discusses the challenges companies face in adopting an experimentation mindset, such as aligning strategy, operations, and technical aspects, and the importance of having a common metric and collaboration among teams.

Ayat Shukairy emphasizes the importance of addressing the functional, social, and emotional aspects of customer needs when designing websites. She shares seven lessons learned from her experience in conversion rate optimization.

Georgi Georgiev performed a meta-analysis of over 1000 A/B tests and has a great deal to share about:
– What you should expect your win rate to be
– The benefits of sequential testing
– Why you should report on confidence intervals

Let’s face it, experimentation is completely misunderstood. The power and the benefit and potential of it is to innovate and develop a business. And when you think about it, that’s what experimentation has always been in relation to the scientific method. If you think about spaceflight, medicine, anything great that’s happened in the history of modern civilization, this has all happened as a result of the careful application of the scientific method.

Experimenters obsess about many A/B Testing Statistics concepts – but not all of them are worth worrying about. Statsig’s Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels – some of his thoughts may surprise you.