Succeeding at Experimentation and Personalization depends on more than just tools

Leila Sayah

A Conversion Conversation with TELUS Digital’s Leila Sayah

No two companies are the same. They have different goals, tech stacks, employees and goals. But when it comes to Experimentation and Personalization — one thing remains constant across the board: the need for the right mindset. I recently had a great conversation with Leila Sayah about the importance of the right mindset, understanding the customer, and Toronto’s cuisine.


Rommil: Hi Leila, how are you today? How’s the new year treating you?

Leila: Hi Rommil, Happy New Year! I’m doing great, I actually just started my maternity leave. So, this New Year is also a new milestone in life for my family.

What a great way to start off the year! Congratulations!

This might be an awkward segue, but how about we start with a little bit about yourself and what you do over at TELUS Digital?

First of all, thank you so much for reaching out, I’m always super excited to talk anything digital, especially when it comes to personalization and experimentation.

I’m a little Frenchie who landed in Toronto almost 9 years ago, and I’ve been in the digital space for a few years now. I would say that I really developed this mindset and love of Experimentation and Personalization a few years ago during my time at Microsoft. I use the term “mindset” because, to me, Experimentation and Personalization are more than just tools available to CRO or Insights teams. They shape how I approach digital strategy and creating digital experiences, regardless of the team I am on.

“…Experimentation and Personalization are more than just tools available to CRO or Insights teams…”

Love it. I couldn’t agree more. You can teach anyone how to use tools. But if you teach them the mindset of Experimentation — they can achieve great things. So where are you now?

Currently, at TELUS, I’m a Senior Product Owner overseeing the Personalization strategy for the entire Mobility division, as well as optimizing the end-to-end journey of our existing customers. In plain English, my job is to make it easier for our customers to do things such as renewing their plan or adding a family member to their account from the moment they land on the site. Personalization plays a vital role in contextualizing these experiences, creating omnichannel consistency while also helping us learn more about our customers, drive loyalty, and ensure our messages are always relevant.

Very cool. I have so many questions for you! When you say “Personalization”, what does that mean to you?

To me, I see Personalization as a key component in creating any digital customer experience as it ensures we are displaying the right message to the right person at the right time, regardless of the channel they’re coming from (SEO, email, SMS, etc.).

Classic marketing, am I right? So, how do you approach it?

At TELUS, right now we’re approaching personalization more from a segmentation angle rather than 1:1, but I have no doubt that we’ll soon get there.

I also think it’s equally important to define what Personalization is not. It’s not some magic tool that is going to suddenly increase your engagement and conversion KPIs if you have not done the work of understanding who your customer is, what problem(s) you are trying to solve, and how your customer currently behaves when trying to accomplish their “job(s) to be done”.

I couldn’t agree more. Understanding the customer is such an overlooked, yet fundamental thing to do.

Going back to this idea of mindset rather than just a tool, it’s also important for me to keep my mind open to experiment Personalization tactics, embrace business priorities changes and advocate for the practice across the organization.

Scalability is also a big challenge. Customer journeys are not linear and getting the most out of personalization, in my case, is dependent on creating consistency across the journey. No single page or banner can be viewed in isolation. For example, if you ask a customer to take a certain action on one page and then dump them into a flow that doesn’t clearly reference that action, you risk confusing the customer and increasing risks of fallout. This means that you need to have the team(s) and the tools in place that support scaling personalized experiences across the entire journey. Tackling these scalability challenges will be one of the main focus in 2020 for our teams.

“Customer journeys are not linear…”

I like to call that “continuity”. Break it, and you confuse people.

Along the same theme, what kinds of data, in your opinion, are the best types to leverage for Personalization?

You have to know your customers and be clear on what are their potential “Jobs to be done” on your site, without that information you are shooting in the dark and hoping for the best. Not every company has a large team of data scientists and/or AI-powered platforms, and can quite easily fall into the trap of slicing and dicing audiences all over the place, targeting the wrong group with the wrong message at the wrong time,…

Yup.

Being very clear on what you are trying to drive in terms of actions and KPIs, leveraging analytics and user research to create a profile of your audience, not only to know who they are (psycho-demographic data, lifecycle stage,..) but how do they behave on your site (traffic source, transactional data, on-site behaviour,..) is for me the starting point.

Additionally, understanding what are the differences between the customers that are successful in accomplishing the actions you want them to accomplish and the customers that aren’t can be extremely valuable in helping identify personalized journey and content opportunities

Start small, test and learn!


Advertisement


Here’s a more tactical question. What’s your opinion on business-rules vs Machine Learning in terms of Personalization?

Oh, the eternal debate hahaha I’m only going to speak about my own experience and for which it isn’t that black or white, but rather being successful in balancing both to drive good customer experience and business goals.

In my industry, there is value in leveraging both because at the end of the day I’m part of a business that needs to follow specific rules to stay profitable, AND at the same time I need to be able to leverage the machine learning approach to debunk some of the business assumptions. Market and digital customer behaviours change, our personalization approach needs to follow these changes and therefore listen to what our customers are telling us and not our own business based biases.

“Market and digital customer behaviours change, our personalization approach need to follow these changes…”

That’s interesting. In that case, what kinds of Experiments would you suggest to our readers in order to improve Personalization performance?

Let me start by repeating myself, a testing mentality is key to a successful personalization strategy, they both work like peanut butter and jelly in my opinion. Some of the different experiments we run on a regular basis at TELUS to help our personalization performance are:

  • Copy and journey testing — we regularly test different messaging for the same audiences and/or journeys on the site.
  • Segment testing — we always try to test different audiences to fine-tune our targeting strategy. As we evolve our platform capabilities we have started to leverage Adobe Automated personalization capabilities and test Ai powered vs business rule targeting.
  • Setting your personalization experiences as an AB test with a control group. This has allowed us to determine that indeed our personalized experience performs better than our default experience on an ongoing basis. This can be extremely useful when you have skeptical stakeholders that doubt the performance of your ongoing personalization strategies.

Moving on to learnings. How do you go about ensuring that learnings are shared with everyone? And how do the learnings from Experimentation get combined with those from Analytics Insights and User Research?

Ensuring that learnings are shared with everyone is a bit of a work in progress as we surely have room for improvement, however, we do have a few rituals and practices at TELUS that work well for us:

  • We leverage our teams and cross teams showcases to present some of our test results. What I also started to do at the end of 2019 and wished I had more time to keep doing was creating engagement by inviting the larger team to guess which variation of a test won. I used Slack to share the hypothesis and the variations via a poll and communicated that the results would be shared at the next showcase. This helped bring some fun to the experimentation process, allowed all team members (content manager, devs, POs,…) to participate and increase showcase attendance.
  • Experimentation newsletter which features key tests from different teams across the organization
  • A repo/tracker of the different tests that we have run or are running with links to a Test card and a Learning card. The Test card provides more info on the test itself, including the hypothesis, and the Learning card provides the test results as well as next steps. This allows teams to self-serve, get visibility and learn from each other.
  • Last but not least ambassadors! You have to have people in teams that evangelize the practice as not everybody has the same level of knowledge on Experimentation or even sees the value of it (yes, even in 2020). I often sound like a broken record, going from team to team, meeting to meeting with my test results, but it does really help to not only share the results but also build the practice knowledge.
  • When it comes to combining Experimentation learnings with Analytics Insights and User Research I would say that they are linked, to not say dependent on each other. Leveraging analytics insights to identify a problem and come up with a hypothesis is key to a data-driven Experimentation process. User research insights often bring the more qualitative approach, while Experimentation will bring you the more quantitative approach, they can be both combined to determine the performance of a product/experience by leveraging qualitative prototype usability testing and then quantitative AB testing data. Plus, user research insights are a very valuable mine of ideas to help your hypothesis process which is key to Experimentation.

I think the sharing of learnings is something everyone is working on. At Loblaw Digital we are doing pretty much all you’ve mentioned and we’re still looking for ways to do better. Someone I recently chatted with proposed short videos. While another proposed infographics. I’ll definitely keep you in the loop if we find a magic bullet.

Moving on to grittiness. Experimentation, for some, can be a slog — especially when you aren’t finding those wins. How do you keep the motivation up at TELUS?

It’s all about developing a culture of experimentation across the organization and not just making the tools available for use to the digital team.

Creating a space where we embrace both failures and successes, understanding that there are learnings to take in all “unsuccessful” tests is key to me. Understanding that Experimentation is not an exact science, you take a bet (ideally supported by a minimum of data), learn from it, and iterate. Allowing team members to go through this process really helps keep the motivation up as it takes away the pressure of associating failure with a waste of time.

Cross-team sharing of experimentation successes and failures is also very important. Showing to team members that they are not alone going through the process, and enabling brainstorming/ cross-team collaboration has been very helpful. One team’s success can inspire you, so as learnings from other people’s failures.

Involving stakeholders in the hypothesis and ideation process. By ensuring you have the buy-in of the different stakeholders/teams you work with really set the team up for success as you feel supported in your practice regardless of the outcome as it was a TEAM effort.

As a product owner, we are tasked with balancing business needs and customer needs and get hit with everybody’s opinions of what the right experience or product should look like. I leverage Experimentation as a tool to step back from the opinions and let our customers tell us what works best for them, and that as a PO can be a real life-saver when it comes to stakeholder management and somehow helps drive the practice and keep the motivation up.

At a company like Telus, how do you organize yourselves to support experimentation? Is each Experimenter responsible for their own tests — soup to nuts?

We have an Enablement team that supports the experimentation practice across the organization from a technology perspective, as well as drive education and key rituals ( community of practice, regular grooming meetings,..)

I’m part of what we call an Outcome team, which leverages the technology from the Enablement team to “enable” the site experiences. Outcomes team drives the research, the strategy and execution of the digital experiences. We are currently experimenting different models where some outcome teams have small dedicated Optimization teams that drove all experimentations for their larger team portfolio, owning their test from soup to nuts, while other teams do not have dedicated optimization teams and each PO is responsible for optimizing their own portfolio and leverage experimentation as part of their CRO process.

That’s interesting. We have a Center of Excellence for Experimentation at Loblaw Digital which is very similar to your Enablement team concept — except the depth in which the team goes into the ritual depends on the team it is supporting.

Well, look at the time. It’s time for the Lightning Round!

Bonjour ou Hi?

Hi, if Is ay Bonjour I might sound a little too uptight haha

So, this isn’t good?

What do you miss most about France?

FOOD!!!

LOL — what are you saying about Toronto cuisine 😉

Finally, describe yourself in 3 words.

Problem-solver, Motivated, Down-to-earth.

And with that, congratulations again, Leila! And thank you.



You may also like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s