Hi Tim, thanks for chatting with me today! How have you been?
Things are going well – Got a 2nd kiddo on the way so lots of prepping for that!
Congrats, I hope you’re enjoying your sleep now! So, let’s start with a bit about you. Tell us about what you do.
I’ve been working in marketing strategy for most of my career. My main focus is on experimentation. I have run and built CRO practices at multiple agencies and I’m currently a Program Manager for Experimentation on Microsoft Digital Stores.
Very cool. As someone who’s been in this space for a while, how do you balance qualitative research and quantitative methods in delivering value? And what role does frustration play?
My general philosophy is using qualitative and quantitative methods to feed off each other. Use both to find where issues exist (analytics review or other user data), and use qualitative methods to figure out why the problem is happening. Then, once you have some potential solutions, use quantitative methods (AB Testing) to validate if your solution solved the problem or not.
Frustration is probably the most powerful emotion in regards to how it influences user behavior. If you deliver a frustrating experience, users will immediately gravitate toward alternative solutions. If you create an experience with minimal/no frustrations, then you are at par for the course. You can also use your platform to relieve existing frustrations a user has (at the product level or purchasing experience level). This is a key approach to consider when developing features to test.
Frustration also exists in every touch point of experimentation. You are looking for the current frustrations your users are experiencing. You are alleviating the frustration of stakeholders by targeting their business problems and finding ways to use experimentation to help them. Ideally, the optimizer’s job is to find frustration, understand it, and eliminate it.
What is your perspective on how to build and grow great Optimization teams?
The most important thing to have in an optimization team is diversity. You need people from different backgrounds to help everybody else keep an open mind and deliver a product that has had a wide breadth of experience and perspectives in it.
100%. Couldn’t agree more. Go on.
If you’re growing a team, I’d suggest looking for people who are exceptional at the things that the program is currently missing. This includes people with the following experience (this is not a full list, just some examples):
- Copywriting/Content Strategy
- UX Research
- Design
- Web Development
- Analytics
- Project/Program Management
Note here that the 2 most important roles here for velocity are the front end developer and the program manager. You can get away with using shared resources for things like design and copy, but to keep the program moving you most likely need a dedicated developer and PM.
I feel this. So often, experimentation will be offered a time-share of a developer. Never works.
How do you measure the performance of an Experimentation/Optimization team? And how do you nurture an Experimentation culture?
Measuring the success of a program depends on the maturity of it. If you have a program structure that ensures the quality of test ideas, then velocity (number of tests) is a valid measurement tool. Although, a lot of programs do this backwards: They make this metric their primary KPI before they’ve evaluated the quality of their ideation process.
Larger programs can look at “Potential revenue gain” and “Potential revenue loss avoided” as these translate best with business. All in all, the value of experimentation is not always measurable. Testing can lead to discovery of unknown user problems and product opportunities that don’t show up on the metric sheets. Nurturing a culture to think like this is something that’s built together by guiding people through the lifecycle of tests and helping them understand what’s happening.
How do you figure out where to start in terms of Optimization and Experimentation?
Start with the user problem. Then branch out from there.
- Where is the user problem happening?
- Why is this user problem happening?
- How does the user problem impact their behavior?
- How does that behavior impact business KPIs?
- What solutions currently exist?
- What solutions could we build?
- How will we validate this solution (a user research project, an AB test, etc.)?
- etc.
You’ve worked with a long list of impressive clients. What so far, has been your favourite experiment thus far?
Larger prices products ($1k or higher) are always the toughest to move the needle on. Working with Microsoft Surface, we don’t frequently see tests that positively impact all funnel metrics. My favorite experiment was the one that did. We changed the formatting of how users select their product specs and accessories to be a progressive form, instead of one long page. This had the largest development cost into a single test since I’ve worked here, and it feels good that it paid off.
And what have been some of your most common obstacles? How do tackle those?
A very common obstacle is ownership. Whether you’re in-house or at an agency, there’s always a reliability on other people to give the right approvals or feedback needed. For example- At Microsoft, test concepts go through design approval, production feasibility checks, business approval, etc.
A way to have less friction here is to have a corporate sponsor. Someone high enough in leadership that can pull the right levers and create influence for the program.
Totally. Where I work, we recently made it a point to create a RACI with folks we work with. Just makes things easier.
Where do you see this industry in 5 years?
Hard to say as cookies become more of an obstacle. I think more people will start to do it right. Meaning that they see experimentation as a tool to improve everything: the product, the purchase experience, the internal systems, the backend development, everything.
The term CRO. What are your thoughts?
The practice has always been defined as experimentation or the scientific method. It of course picked up this catchy nickname since it was initially adopted online as a marketing/sales tool (which is only a small part of what it can do).
It’s time for the Lightning round! Bayesian or Frequentist?
Frequentist.
Who do you consider as leaders in the business Experimentation space?
It’s hard to say because there’s so many flavors of experimentation backgrounds. I found that books about communication and product management have been super helpful as an experimentation PM.
If you couldn’t work in Experimentation, what would you do?
Not really sure! A professor maybe?
Describe Tim in 5 words or less
An avocado a day…
Ha! That, by far, is the most original answer I’ve heard thus far.
Thank you, Tim, for joining the conversation!
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.
Related posts:
