BMO's Claire Yeh: Experimentation is key to Personalization success Experiment Nation

A conversation with BMO’s Claire Yeh about Experimentation

Personalization and Experimentation may seem like very unrelated practices?—?but this can’t be further from the truth. I recently spoke to Claire about how she approaches Personalization, how she succeeds at it, and the critical role that Experimentation plays.

Rommil: Hi Claire, how have you been? I hope you’ve been doing well. Thanks for chatting today!

Claire: Hi Rommil! Thanks so much for the opportunity to chat and share my thoughts! I’m doing well, adjusting to working from home and enjoying the last bit of summer. How are you doing?

I’m doing well. Same as you, adjusting, adjusting. But when you work in Experimentation and Personalization as we do, I feel it’s par for the course.

So Claire, could you share with our readers what is it that you do and how you got to where you are today?

I am the Web Personalization and Experimentation manager at Bank of Montreal. I lead a team of rock star strategists managing Personalization practices across various portfolios for BMO and BMO Harris (US). Along with a team of dedicated experience designers, developers, QA, and Marketing Technology partners, we aim to deliver best-in-class experiences. Our key mandate is to improve prospective and existing customer online experiences while removing user friction and driving digital sales.

I started my career in CRM and email marketing, where I realized my interest in delivering a seamless personalized journey through quantitative analysis, UX research, and experimentation. From there, I moved into web personalization and experimentation. Before BMO, I worked on the eCommerce team at Home Depot.

Very cool. I’ve noticed over the last couple of years that Testing and Personalization practices are often put together?—?why do you think that is?

Testing and Personalization come in a pair. Starting with the technology, I like to describe the targeting tool as a traffic controller. You can specify experiences for different segments but can also split the traffic randomly to compare user reactions to variation in experiences. Testing should be table stakes during all user touchpoints. It helps evaluate how users are responding to changes in experience and environment. At the end of testing, we conduct a look back analysis to evaluate how each key segment reacts to an A/B test. This not only helps learn about users but also provides a roadmap for future iterations of the experience. This cycle continues: Analysis?—?AB testing?—?analysis?—?personalization?—?analysis?—?AB testing.

“Personalization isn’t about creepily recording your personal information, but making it easier for you to evaluate, navigate, and conduct tasks throughout each touchpoint.”

100%. It’s very interesting how they blend together so naturally. As you know, Personalization is such a hot topic these days. In a nutshell, what is Personalization and why is it important?

Businesses are changing and continuing to shift towards being more customer-centric.

Companies are finding value in providing a personalized experience that is relevant, timely, and seamless to customers. Customers, on the other hand, also expect easy to use, personalized journeys. Personalization isn’t about creepily recording your personal information, but making it easier for you to evaluate, navigate, and conduct tasks throughout each touchpoint.

I’m with you. Like Experimentation, Personalization is about addressing user needs. Please, go on.

Personalization is not a new concept. We have come a long way from communication that starts with “Hi Rommil…” Take suppression for example. As a customer, you won’t enjoy being served an ad showing products you already own. Not only that but in an offer you are not qualified for. This is a low-hanging fruit for companies, but that can avoid negative feedback from your most valuable customers. Personalization should remove pain points and frictions. Instead of showcasing generic content for every visitor to your website, consider the typical customer journey. Then find the biggest gap and indicators for each phase of the journey to customize relevant information that will help users achieve their goals easily.

Customer behaviour has shifted during this challenging time towards digital adoption. It’s more important now than ever to serve up personalized and frictionless digital journeys.

BMO is a large company with a lot of stakeholders and communication channels. What does a typical Personalization workflow look like in this kind of environment?

We work in a 5-phase flow: research/analysis, ideation/alignment, setup, launch, and learning. With a dedicated strategist working on a portfolio, they act as the Personalization SME. Across the portfolios, strategists also work closely with each other and our Digital, Analytics, UX, Content, Marketing, and Acquisition partners. This allows for a collaborative knowledge sharing that elevates each portfolio.

How do you deal with experience collisions?

We keep a cross-portfolio calendar, as well have a due diligence check prior to setting up any test to avoid experience collisions. The advantage of having a team of strategists working closely is collaboration in planning and learning. Sometimes the solution could mean combining activities or excluding certain audiences to validate AB testing results.

Personalization comes in a few flavours. 2 major ones being rules-based and machine learning-based. Is one better than the other?—?or how do you leverage the strengths of both?

We combine both approaches organically to create efficiency and maximize impact when it comes to leveraging customer data and delivering personalized experiences. As each customer has a long-term relationship with the bank, they don’t always have singular objectives at each given touchpoint. ML insights feed into both rule-based and unsupervised ML implementations.

Rule-based is also easier to grasp conceptually. This is something that most partners can get behind as an evergreen strategy. Segments from ML could be challenging to categorized and interpreted by business partners. We run a lot of rule-based personalization as proof of concept to show quantifiable business impact while ML tends to be designed with automation in mind. There is value leveraging both because at the end of the day we are a business delivering value through actionable learnings.

How do you leverage Experimentation when it comes to Personalization?

Personalization won’t be successful without experimentation. It helps to evaluate the impact. A lot of times what we (the practitioners) think is cool and innovative might not even be noticed by the users. Based on how often we are surprised by test results, we can never be sure if an experience has a direct impact to KPIs. Working in Testing and Optimization, we know not to rely on assumptions. This impact analysis helps advocate the program with senior leaders.

I totally agree. I also consider personalization as an extension to Experimentation as well. Where early Experiments optimize for the general public, Personalization optimizes for segments and people.

“Do your homework?—?instead of throwing mud on the wall to see what sticks, form a hypothesis with quantitative and qualitative support.”

What is the most overlooked aspect of running a Personalization program?

Do your homework?—?instead of throwing mud on the wall to see what sticks, form a hypothesis with quantitative and qualitative support
Constant learning?—?we hire for curiosity and willingness to learn, not only learning about our users, but also continuously learning from different fields such as UX, Analytics, Data Science, and Digital Marketing

Finally, it’s time for the Lightning Round!

Rules-based or Machine Learning Personalization?

Both.

Bayesian or Frequentist?

Bayesian.

What does BMO stand for and why is it in Toronto?

Good question?—?Bank of Montreal, with locations in North America and Internationally, is Canada’s oldest incorporated bank. We have central offices in Montreal, Toronto, Chicago, and many more! The company’s purpose statement reads Boldly Grow the Good, in Business and life. BMO focuses on building, investing, and transforming how we work to drive performance and create an exceptional customer experience. This aligns with what we do every day in Personalization.

Very nice. If you couldn’t work in Experimentation, what would you do?

Opening up a bistro?—?I love to entertain and cook!

With Personalized service, of course.

And lastly, describe Claire in 5 words or less.

Methodically curious and resourceful.

Claire, thank you so much for joining the conversation!

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.