Cloudflare’s Scott Olivares: To get personalization right—have good manners and experiment

Scott Olivares

A Conversion Conversation with Cloudflare’s Scott Olivares

I recently caught up with an old co-worker of mine from Autodesk, Scott Olivares, who’s now rocking it at Cloudflare. On top of sharing some ultra-cute pictures of his family, he shared with me how to approach personalization the right way-by making it more contextual and way less creepy.


Rommil: Hey buddy! Always nice to catch up with a former ‘desker, how have you been?

Scott: It’s been an interesting ride since my Autodesk days. I moved to Orange County from San Francisco when I found out I was going to be a dad. Both my family and my wife’s family are near there, so we wanted to be closer to them while we raised our first baby. I worked at SapientNitro (now Publicis Sapient) for a couple of years, but eventually received an offer to join LinkedIn, which brought us back to the Bay Area.

LinkedIn was an awesome place to work, and I spent 3 years there. However, I always had an itch to start my own business, and in 2017 I had that chance. I joined forces with Nabler, an analytics agency in Bangalore, to build a UX Research and Experimentation Agency based in San Jose.

Oh wow – that’s so cool. I definitely want to hear more about this one day. Maybe during another Conversion Conversation! How was running your own business?

That was a lot of fun, rewarding, and stressful all at the same time. I learned a lot about a lot during that time, and especially learned a lot about myself. After almost a 2 year roller coaster, I decided that leading a small services company wasn’t for me. I found myself consumed by so many things outside of the services we provided and really missed being able to focus on user experience and building creative ways to drive growth.

Totally get it. I felt that way when I had my own company years ago. What happened after you closed shop?

So now I’m at Cloudflare, where we are contributing to building a better Internet for everyone. So far I’m really enjoying it.

Here are my crazy kids:

The official Christmas card photo 🙂

Aww, how cute are they?! So Scott, beyond having a lovely family, what else can you tell us about what you do?

I’m a Product Manager for Cloudflare’s website, www.cloudflare.com, where I lead our efforts to constantly improve the user experience and improve our experimentation and personalization capabilities in a secure and private way.

I had an almost immediate impact upon joining back in February, and it’s been great to see the organization embrace experimentation and data-driven UX enhancements. I’m lucky to have a nice support system of smart designers, engineers, and leaders that make it possible to get things done.

“If you’re not careful, personalization can exhaust your design and content team’s resources quickly.”

Personalization is such a hot space right now. How do you define personalization and why is it important?

Personalization is simply presenting relevant content to someone based on what they want to see.

It’s important because people only want to pay attention to something that means something to them. Most of the time they ignore everything else. Personalization creates an experience that’s more meaningful to people, and therefore should be better than just showing everyone the same content. Usually, when you provide a relevant, meaningful experience, people can make more informed decisions to questions such as “should I read this?” or “is this product what I need?”

“Just like in person-to-person interactions, manners matter on the Internet as well.”

That’s true, but how do you walk the line between a delightful personalized experience and something creepy?

I think it comes down to manners and whether something feels appropriate given the situation. Just like in person-to-person interactions, manners matter on the Internet as well.

In the real world, if I visit a massage therapist with a stiff neck and can barely turn my head, I’d welcome his suggestion to work out whatever issue he thinks I have. But if I’m dropping my son off at school and my neck kind of hurts, it wouldn’t be cool for that same therapist, that I may not even know, to come up and offer to start working on my neck.

Similarly, if I’m on a website browsing various products in a similar category, it would be appropriate for that website to suggest a product in that same category that I may not have seen. But if I’m at home talking to my wife about Ted’s red boots in How I Met Your Mother, I don’t think it’s cool to see an ad for red boots on my Instagram feed. That’s too creepy.

Source: https://marketoonist.com/2014/11/personalization.html

The number of times that has happened to me, yeah that’s way too creepy. Can you give us some examples of a personalized experience that we may have seen but may not have realized were personalized?

This is a hard question because whether or not someone realizes they are in a personalized experience depends on that person’s awareness of the situation and what’s actually possible. I don’t think any of the personalization work I do is too discreet or hidden, but perhaps that’s because I think about this stuff so much. Other people may not realize they are in one of my personalized experiences.

Some examples include:

  • Showing visitors on a SaaS website a “try now” button and customers a “Log In” button.
  • Introducing products based on their browsing activity across the website.
  • Defaulting to a plan type on a plans comparison page based on their browsing activity.
  • Presenting a solution to a problem based on a keyword that drove the person to the website.
  • When a person is browsing a website from a company’s network, showing industry-specific content based on their IP address

Lots of companies attempt to personalize, not many get it right. What are the key elements to getting personalization right?

A while back I wrote a white paper called “Personalization Pitfalls and How to Avoid Them”, and I still think all of the suggestions hold true.

Some things to highlight include the following:

  1. Get the audience segments right. Start with the audience you have.
    When I was at LinkedIn, the product marketing team for one of the company’s most successful products spent a couple of months, and a good amount of money, to define the right audience segments. Then they spent more time to create content for each of those segments. Finally, we launched a personalization campaign on the website. It was very exciting. 
    After a couple of weeks, we analyzed the personalization data and learned that the vast majority of visitors didn’t fall into any of the segments they had defined, and practically everyone was getting the default content. It was a big “aha” moment and a great learning experience for all of us. The lesson learned was that we should start with the audience we have, not the audience we want or the “ideal” audience. You can do that by looking into your web visitor data and creating audiences based on actionable attributes you have. Maybe you won’t know the percentage of visitors that work at a company with over 500 people, but you will know the percentage of visitors that view your enterprise-specific white-papers!
  2. Start small and avoid the Content Monster
    If you’re not careful, personalization can exhaust your design and content team’s resources quickly. I call this the Content Monster. For example, if you have a single homepage banner, then you only need to create one banner at a time. If you decide to target a different banner to 4 different segments, then you just multiplied the number of banners that need to be created by 4. If you’re going to run an A/B test to ensure you’re making a positive impact, then you need 8 banners… 8x the work. That’s just for one page. This can get out of hand quickly. Because of this, it’s wise to start small so that you can gain some success and demonstrate a positive ROI impact. Start with only 2–4 audience segments, and limit the places you personalize to only the highest trafficked pages. Once you demonstrate a positive impact, it will make sense to invest more in content creation.
  3. Experiment your way to success
    I still meet marketers and product managers that create personalized experiences and show it to 100% of users that qualify for it. When I ask how they know if the personalization is making a positive impact, they often don’t know, or point to overall growth. They assume that because they are personalizing the experience, the result is positive. However, this is often not true. The growth they are seeing could be do to a number of reasons, including the company’s overall growth rate. In fact, they could be hindering growth and they wouldn’t know it! You should always have a holdout group, for each segment, that doesn’t receive the personalized experience. This will allow you to compare the conversion rates for the targeted visitors and the non-targeted visitors. Frankly, most of the time, attempts at personalizing an experience have no impact. If this is the case, you need to try different content, or perhaps change your segments. Because if you’re doing all that work to personalize experiences, and it has no impact, you shouldn’t be doing it. Your time would be better spent on a different project. Don’t get me wrong, a personalized experience always works better than a non-personalized experience… but only when you get it right.

“The lesson learned was that we should start with the audience we have, not the audience we want or the “ideal” audience.”

Well y’know I’m totally with you on that one. As I always say, “You should probably test that haha!” That said, How do you see personalization and experimentation working together?

They are like peanut butter and jelly. Actually no, because you can have jelly on its own and it’s still good. Personalization without experimentation is more like high stakes gambling.

Without experimenting and validating that your personalization efforts work better than not personalizing, then you’re just assuming you got it right. Assuming you’re right is a bad move when you’re using company resources to do all this stuff and your objective is to drive growth or create a better experience.

Your personalization campaigns should be within an A/B test. This is the only way to clearly understand the impact it has.

It’s time for the Lightning Round – I hope you’re ready! What are the top 3 ingredients that make up a culture of experimentation?

  1. Good test planning
  2. Communicating results and learnings broadly
  3. Making it easy to run experiments

What test do you wish people would stop running?

Landing page experiments for one-off emails. They rarely get enough traffic to make it worthwhile. I’m all for testing landing pages that get constant traffic, but given how low open and click rates are from email marketing, it’s rare to get enough visitors from a single email to give you a strong signal.

Also, testing minor things below the fold. Be bold and make sure what your testing is visible.

Definitely. If I had a dollar for every test I see run that never gets enough traffic… What do most people forget to do when developing testing plans?

To gauge whether an experiment is likely to have a business impact. Contrary to many others, I don’t believe you should test everything. Testing takes an investment of time and resources, therefore I think you should test things that

  1. get enough attention (e.g. a lot of users see it) and
  2. are significant enough to matter (i.e. changing a single line in a paragraph isn’t going to do anything).

Nice. I’m with you on that. I hear way too much said about testing everything. I’m a pragmatist – focus on what will impact your business, right? And finally, Frequentist or Bayesian?

If I had to pick one, I guess it would be Bayesian because it’s more pragmatic. I think it’s silly for businesses to ignore elements that are likely to affect an experiment, in the name of data purity. Ultimately, most of us are not mathematicians in academia. Experimenting is a means to an end. My job is to introduce new features and enhancements that people like or drive growth. Experiments simply indicate whether or not I got it right.

To that end, sometimes you might not be able to conduct an A/B test experiment, and that’s okay! Again, the goal, for many of us, is not to get the purest data – it’s to launch something that has a positive impact. If you can only do some user testing or other validation research to tell you whether you’re on the right track, then that’s okay. It’s not as strong as an actual A/B test, but it’s much better than just assuming you’re right and launching something.

Thanks so much for chatting with me. It’s been so great to catch-up!



Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

Rommil Santiago