Category Archives: Interviews with Experimenters

Veteran experimenters, Rommil Santiago and Tracy Laranjo, decided to approach CRO interviews in a new direction with their new series entitled CRO TMI. In this short, rapid fire format, Rommil and Tracy answer questions that folks are often too shy to ask and share more detail than most people care to share – giving listeners insight into what it’s REALLY like to be an experimenter.

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways: Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning. Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself. Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations. Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial. Document everything: Detailed documentation helps track results and avoid surprises.

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways: 1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas. 2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges. 3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona. 4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future. 5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes.

Dr. Elsa van der Berg, a B2B SaaS product management expert, shares some key advice: Don’t try to build for everyone. ICP discrepancies are a common problem. ICP descriptions need to be evidence-based. Embrace focused testing. Understand the difference between signal and noise: Not all feedback is actionable. Focus on pain points that users have actively tried to solve.

We recently had the pleasure of chatting with Rishi, a Shopify product expert. Here are 5 key takeaways for optimizing your product pages: 1) Product pages are the Grand Central Station of conversions. They’re where buyers make the decision to buy or leave. 2) Pop-ups aren’t as effective as you think. Focus on valuable content, not annoying interruptions. 3) Content is king (but don’t be afraid to go long!). Detailed product descriptions are more effective than short ones. 4) Build trust above the fold. Establish your company’s credibility before highlighting product features. 5) Understand the filters buyers use. Address common concerns like “too good to be true” to build confidence.

Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos. Here are five key takeaways from his chat with Tracy Laranjo: 1) Start now! Don’t wait until the last minute to get buy-in and start testing. 2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits. 3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing. 4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement. 5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales. Get ready to conquer Black Friday!

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.
Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.
Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.
Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.
Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way.

In this podcast episode of Experiment Nation, hosted by Charlotte Bomford, guest Iqbal Ali, an experienced web designer, researcher, and comic book writer, discusses the concept of solution bias and the importance of identifying underlying problems before diving into solutions. Ali emphasizes using AI to help understand problem landscapes, avoiding hastily jumping to solutions, and improving experimentation processes. The conversation covers techniques like premortems and frameworks to minimize potential issues and ensure experiments address real user problems. Ali highlights the collaborative potential of AI tools in maintaining the connection between problem statements and solutions, advocating for a more thoughtful approach to A/B testing and problem-solving in product development.

Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues’ goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results.

It’s a wrap – Experiment Nation’s Product Experimentation event may be over but you can still watch it all. A huge thanks to all our amazing speakers: Ellie Hughes ?Aarushi Raizada ?Stefanie Grimmling ?Anastasia Shvedova ?Olga Kosyreva ?Marilyn Koutsonikolas ?Jyoti Malec ?Dennis Nederlof ?Marylisa Santos With over 9K views, it has been one of our most successful events ever! Thanks to everyone who took the time to watch.

Tymur Donets, a seasoned Growth PM and experimentation consultant, shared some powerful insights during his Experiment Nation talk. Here are five key takeaways:

Growth PMs are holistic: Don’t be siloed! Growth PMs work across the entire customer journey, ensuring marketing and product teams are aligned.
Commercial awareness is key: Both Growth PMs and CROs must understand the business impact of their work and how it contributes to the bottom line.
Empathy is vital: CROs need to understand the pressures their PM colleagues face and proactively offer support.
Leverage your expertise: If you’re a CRO, consider specializing in Growth PM to expand your impact and skillset.
Transparency is key: Clear communication between teams helps everyone understand their roles and collaborate effectively.

Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out: – CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability. – Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow! – Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal. – UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests.

Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways: Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses. Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate. QA is critical but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly. Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden. Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results.

Avishek Basu Mallick, a Senior PM at CheapOair talks about: – Experimentation at scale: Even with high traffic, rigorous experimentation is crucial. – Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas. – Communicating progress: Regular updates with leadership build trust and ensure alignment. – Personalization is nuanced: Distinguish between customization, segmentation, and true personalization. – Generative AI’s potential: Explore its use for simplifying the travel planning experience.

Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers: Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals. AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance. Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results. People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing. Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences.

Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game:

– Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning!
– Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks.
– Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success.
– Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving.
– Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!

Join us for a deep dive into conversion rate optimization on low-traffic sites with these actionable insights Chris Mercer

Measure before optimizing: Understand the current user journey and behaviors to identify areas for improvement.
Test sequentially: Iterate faster on low-traffic websites by testing changes sequentially instead of using AB tests.
Focus on the right metrics: Prioritize actions that drive meaningful outcomes, such as lead generation or revenue.
Listen to the market: Gather user feedback to inform your optimization strategy and meet their needs.
Automate measurement: Leverage Google Analytics 4 to collect and analyze data effectively to guide decision-making.

Discover the power of experimenting to learn and win! ?

Nick Murphy shares his expertise on building learning libraries. Here are 5 key takeaways:

– Learning libraries capture the “why” behind successful and unsuccessful tests.
– Organize your learning library with consistent and digestible fields.
– Use “broad” and “specific” learning columns for a layered understanding.
– Encourage cross-team collaboration by sharing insights company-wide.
– Learning libraries foster a culture of experimentation and growth.

In this episode, senior customer experience leader, Courtney LeBlanc, reveals the ideal partnership between CX and experimenters and shares invaluable insights on:

Forging powerful partnerships between CX and experimenters
Navigating the daily demands of a CX leader
Simplifying CX workflows with customer insights
… and so much more!

Don’t miss out on this enlightening conversation from an expert in the CX game!

How do you get Product Managers to adopt Experimentation? Shagun Aulakh, the Director of Product Management – Experimentation at American Express shares with Rommil here tips.

In this episode, we have Joey Herzberg who shares his thoughts on how AI fits in the experimentation life cycle, the utility of keeping easily referenced records of test results, and the importance of experimentation strategy with regard to sample ratios.

Learn about human-first engineering in this in-depth interview where Shawn David discusses strategies to organize work and to break down tasks into smaller digestible sessions. He explained the most effective ways for teams to discover the work that should be automated the most by focusing on the desires and strengths of individuals, to remove work that is menial especially to them. We explored the impacts on company culture that menial tasks can cause and discussed some of the pitfalls with new technologies as well as lessons to keep in mind while reviewing new tools and technologies.

Join Nima Yassini as he dives into experimentation culture, the journey of building a brand UX and digital marketing agency focused on conversion rate optimization, the integration of digital marketing and experimentation culture, the struggles and benefits of adopting a culture of experimentation, and the need for leadership and reinvestment in the experimentation industry.

CEO and founder of Delivery Layer, Solomon Kahn shares why he thinks that most companies should NOT be A/B testing and what we should do instead. He also shares details about his recent exchange with Ron Kohavi on LinkedIn on this same topic. What side are YOU on?

It’s that time of year again! It’s time for Experiment Nation’s 2024 CRO Salary Report. In this episode, we explore some of the report’s highlights with Tracy Laranjo as well as ask her for her thoughts on how CROs and Experimenters can get paid fairly.

Zach Lebovics shares insights on experimentation and product management. He discusses the importance of documenting assumptions, testing the best-case scenario first, and understanding that experiment failure isn’t solely tied to the outcome. Zach also shares personal experiences and learnings from working in companies like Zynga, Ritual, and Tonal, providing valuable tips for product managers.

Media.Monk’s Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.

Media.Monk’s Juliana Jackson shares the truth about app development, optimization, and how web and app should work together.

Fractional Head of CRO, Tracy Laranjo talks about the power of Mentorship for Conversion Rate Optimizers (CROs). We cover:

Why would CROs need mentorship?
How do you find one?
When are you ready to be a mentor and how do you get that word out?
Would Tracy be a mentor for our listeners?

In an insightful interview at Experiment Nation’s Conference, Bryce York, Director of Product Management at Tatari, delves into several key insights. He explores the parallels and distinctions between CRO and product management, especially in smaller companies where overlapping skills are crucial. Bryce shares his personal journey from entrepreneurship to product management, underscoring the significance of both qualitative and quantitative data in experiment design. Additionally, he emphasizes the importance of documentation and knowledge sharing in cultivating an experiment-driven culture.

Documentation. The unsung, yet incredibly important part of any experimentation program. Great documentation allows you to look back at past experiments and find patterns of what were the most effective levers to inform your testing roadmap and levers you probably can deprioritize. In today’s episode of Experiment Nation, our own Charlotte April Bomford recently spoke to Edmund Beggs about the “Levers Framework” which allows Convert.com to deliver results for their clients in shorter amounts of time. If you’d like to learn more about this interesting framework, watch the full interview.

Simon Girardin emphasizes the importance of learning from every test, capturing value by reviewing data, segmenting results, and analyzing user behavior through session recordings and heatmaps.

Koalatative’s Gerda Vogt-Thomas asks Rommil Santiago questions about how he started Experiment Nation and how he manages it while working a full-time job at Optimizely.

Research is important to Conversion Rate Optimization, but are you leveraging the power of Diary Studies? If not, you’re missing out. Learn more from Mike Green.

Learn from André Morys, CEO of KonversionsKRAFT, on gaining crucial stakeholder buy-in for experiments. André highlights the challenges faced by optimizers and advocates for understanding team dynamics through stakeholder mapping. He emphasizes approaching stakeholders, using empathy to build relationships, and overcoming objections. André shares practical insights on conflict management, including a stakeholder map with color-coded power dynamics. Discover the significance of regular check-ins, fostering trust, and achieving tangible results in experimentation. Master the art of securing support for your experiments effectively.

Will Laurenson shares his journey from startup to becoming a CRO (Conversion Rate Optimization) expert. Will discusses the significance of usability, anxiety, and motivation in CRO, drawing from his extensive experience in running tests and optimizing various businesses. Will’s approach to CRO involves deep insights into user behavior and a focus on aligning products with customer needs, challenging the common pitfalls of assuming product-market fit based solely on initial traction or advertising success.

In this insightful video, Ishan Goel, transitioning from a software engineer to a researcher, delves into crucial CRO statistical concepts. He sheds light on statistical significance in experimentation, explores A/B testing intricacies, emphasizes the efficiency of sequential testing, and discusses challenges in stopping tests prematurely. The talk concludes by highlighting the intersection of experimentation and statistics in the realm of Conversion Rate Optimization.

So much is spent on ads that it’s a no-brainer to invest in ad optimization. Embracing experimentation and learning from failures is essential for long-term success in performance marketing. Experiment Nation’s Tracy Laranjo spoke with Joe Fitzpatrick about reasons why your ads aren’t working and what you can do about it.

Maximizing results in ad network optimization requires a systematic approach. Start with A/B testing on Google Ads and gradually introduce more complex campaigns, balancing technical and creative aspects. Analyze your channel’s performance, ensuring sufficient impressions and conversions for statistically significant results. Shape your product to align with the ad network’s requirements and consider user experience. Prioritize velocity in testing initially, gradually enhancing quality, and encourage collaboration across different platforms for valuable insights in the quest for optimization.

Lucas, a senior conversion specialist, shares his journey in the media industry, emphasizing the transformative impact of A/B testing. He discusses myths about experimentation – debunking them along the way as well as highlights the importance of validation, integration into essential projects, and showcasing negative results. Lucas encourages testing the obvious and unexpected, questioning assumptions, and showing the value of experimentation to reduce risk and aid decision-making, making it an asset for senior managers and specialists.

Eric Melchor, an evangelist at Optimonk, emphasizes the importance of focusing on the product page for effective conversion rate optimization (CRO). He suggests treating website visitors as individuals and creating a personalized and enjoyable customer experience in real-time. Melchor also discusses the significance of capturing visitor interest early on and leveraging smart CRO tactics and personalization timing throughout the customer journey. Additionally, he mentions the value of optimizing the product page, as it can yield significant returns on investment.

We explore various angles of CRO Organizational Structure. We discussed primarily the differences between centralized CRO teams vs a company-wide experimentation culture, and how this is best looked at as a continuum. Companies starting out might benefit from a centralized CRO team whereas more mature optimization programs can benefit more from working experimentation into all facets of the company. How can a company who does 0 experimentation introduce and adopt an experimentation process/culture? We also chatted about experimentation at the customer level, what this means, and how it could be the future of optimization.

Heatmaps to the rescue! Creating an impactful website hero section above the fold is vital for user engagement. Clear value propositions and compelling calls to action are essential, as users often don’t scroll past this section. Heat mapping, combined with other techniques like exit polls and A/B testing, provides a comprehensive understanding of user behavior. While urgency and scarcity tactics can be effective, users have become more skeptical of overused strategies like countdown timers. Understand your audience through analytics data and tailor your optimization strategies accordingly, whether it’s for B2C, B2B, or other sectors. Mobile interactions may have imprecise heat map data, so use it alongside other sources for a well-rounded view.

To ensure the success of CRO projects, it’s vital to involve all stakeholders, including executives and directors, right from the start. This inclusive approach not only helps them understand the intricacies of the CRO process but also demonstrates the depth of strategic thinking behind it. Even skeptical executives can be won over by involving them in CRO calls, showcasing the complexity that goes beyond mere marketing tactics. Such transparency and involvement lead to a more informed and supportive team, fostering successful CRO initiatives.

In the world of Conversion Rate Optimization (CRO), Eduardo, a seasoned practitioner, shares his journey and the evolving landscape of CRO in Brazil. He underscores the importance of maintaining a problem-solution mindset and learning from mistakes to drive meaningful impact. One core principle he highlights is the need to focus on hypotheses that yield significant results, comparing them to squeezing fruit for juice. This strategy involves exploring promising hypotheses further and discarding those with neutral or unimpressive outcomes, a practice essential for effective experimentation, whether in-house or at an agency. The key takeaway is to persevere with ideas, akin to squeezing every drop of insight from hypotheses, and leveraging comprehensive data for a deeper understanding, a competitive edge held by in-house roles in the experimentation field.

Integrating landing pages is crucial for tailored user experiences. Without it, visitors are directed to generic pages lacking personalized messaging. Landing pages are designed to enhance conversion rates and ROI. E-commerce companies benefit by using various types of landing pages, like the classic hero for focused products, listicle hero for storytelling, collection hero for multiple items, and content advertorials for top-funnel content. Mistakes include using one-size-fits-all pages and lacking customer-oriented content. Improve by collaborating with ads teams, using customer voices, and continuous experimentation. Claire More from Splitbase shared insights on optimizing landing pages for effective campaigns.

Don’t test everything. Experimentation isn’t about proving how clever you are. Mistakes in testing ideas from your own head are common; personal bias clouds results. High-performing experimentation needs executive buy-in and integrated roles. Successful teams engage diverse players with vested interest, ensuring hypotheses formation and metric selection are comprehensive. Valuable experimentation saves time by testing what not to do. Rather than just chasing conversion lifts, recognize when improvements won’t make a difference. Cultural shifts, humility, and prioritizing efficient use of time are key components of effective experimentation programs. Test what makes sense.

In this podcast episode, Mark Eltsefon, a senior data scientist and experimentation evangelist, discusses building an experimentation culture in different companies. He shares his journey from starting as a software developer to becoming passionate about data science and experimentation. He contrasts the experimentation culture at TikTok, where data-driven decisions were central and democratization was encouraged, with his current role at an e-commerce company specializing in print-on-demand. He emphasizes the importance of educating stakeholders about data-driven decision-making, and he highlights the need for strong experimentation infrastructure and high-quality hypothesis testing. He believes that while democratization is valuable, it’s important to balance it with maintaining the quality of ideas and statistical rigor.

Juliana Amorim emphasizes that personalization is not a silver bullet and recommends its use when aligned with strong hypotheses and business goals. She co-founded Croct, a personalization experimentation platform, focusing on adapting user journeys to context. Examples include tailoring websites for diverse buyer personas within B2B companies and customizing content for users at different stages, like newcomers or existing customers. AI-based personalization and rule-based personalization can coexist. Brands should prioritize personalizing their websites and consider first-party data support.

Building a personal brand as a conversion rate optimizer is crucial in today’s competitive digital landscape. Your personal brand serves as a powerful tool to establish credibility, showcase expertise, and attract new opportunities. By cultivating a strong brand identity, you position yourself as an authority in the field, gaining trust from potential clients or employers. A well-developed personal brand helps differentiate you from others, highlighting your unique approach and methodologies. It opens doors to collaborations, speaking engagements, and thought leadership opportunities. Additionally, a strong personal brand creates a lasting impression, leading to increased visibility, a larger professional network, and ultimately, a steady stream of clients or job offers.

By implementing ideation strategies and fostering a culture of experimentation, companies can drive innovation and stay ahead of the curve. Building an experimentation culture is crucial for companies that want to innovate and stay ahead of the competition. It starts with creating an environment where experimentation is encouraged and failure is seen as an opportunity to learn and improve. Teams should be given the freedom to try new ideas and test them quickly using small-scale experiments. It’s important to collect and analyze data from these experiments to make informed decisions about what works and what doesn’t. Lucas Vos interviews Lucia van den Brink in this great AMA session.

Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.

David Mannheim, author and expert on personalization, discusses the importance of personalization in marketing. He emphasizes the need for brands to focus on building relationships with customers based on trust and care. Mannheim distinguishes personalization from segmentation, highlighting personalization as a communication principle rather than just a marketing tactic. He also mentions the challenges faced in implementing personalization, such as the pressure for immediate returns and the balance of metrics.

We sat down with Jason Chappel, founder of Defiant, a specialist Customer Value Optimization agency, to discuss what sets CVO apart from CRO. Jason shares his insight on the importance of understanding and optimizing the entire customer journey, from acquisition to retention, and how CVO can help businesses increase customer lifetime value and drive sustainable growth in your business strategy.

We all know that humans are very complex and are hard to predict how they will behave. This is why we want to do use multiple methodologies in tandem to understand our customers.

To have any real influence or buy-in, you need to be able to tell a good story. Khalil recently interviewed Priya Bhatia to get their advice on how to tell a good story.

Optimizing forms during checkout is crucial for a seamless user experience. By simplifying and streamlining the process, businesses reduce cart abandonment rates and increase conversions. Shortening form length, utilizing autofill options, and providing clear instructions enhance efficiency, minimize user frustration, and ultimately boost customer satisfaction, loyalty, and revenue.

Nicoleta walks us through a case-study where she explores a 1 column form vs a 3 column form in checkout.

Learn A/B Testing Statistics, how to use Overall Evaluation Criterion, Twyman’s law, how much traffic do you need to run A/B tests, and other statistical concepts all Experimenters must know.

Are you looking to replace Google Optimize? On Experiment Nation’s recent podcast, Charlotte April Bomford chats with Kenneth Kutyn about replacing Google Optimize, how to select a new vendor, and whether it’s worth building your own Experimentation Platform (spoiler alert: the answer is usually no).

Tim Thijsse, a senior CRO (Conversion Rate Optimization) consultant at Orange Valley, discusses AI and the future of CRO, and the importance of incorporating qualitative research into company-wide processes. He emphasizes the challenge of linking user behavior and mindset to knowledge and information. Tim shares his background, starting as an interaction designer and later transitioning to UX design and CRO. He highlights the evolution of CRO, the need for a culture of experimentation, and the role of qualitative insights in optimization strategies. Tim also discusses the challenges companies face in adopting an experimentation mindset, such as aligning strategy, operations, and technical aspects, and the importance of having a common metric and collaboration among teams.

Ayat Shukairy emphasizes the importance of addressing the functional, social, and emotional aspects of customer needs when designing websites. She shares seven lessons learned from her experience in conversion rate optimization.

Georgi Georgiev performed a meta-analysis of over 1000 A/B tests and has a great deal to share about:
– What you should expect your win rate to be
– The benefits of sequential testing
– Why you should report on confidence intervals

Let’s face it, experimentation is completely misunderstood. The power and the benefit and potential of it is to innovate and develop a business. And when you think about it, that’s what experimentation has always been in relation to the scientific method. If you think about spaceflight, medicine, anything great that’s happened in the history of modern civilization, this has all happened as a result of the careful application of the scientific method.

Experimenters obsess about many A/B Testing Statistics concepts – but not all of them are worth worrying about. Statsig’s Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels – some of his thoughts may surprise you.