Exploring organizational design for Experimentation with Alek Toumert
A Conversion Conversation with Optimizely’s Alek Toumert
Setting up and operating an Experimentation program involves many moving parts. On top of bucketing users properly, ensuring interaction is minimized, double-checking that the statistics were done correctly, sharing learnings, and designing the Experiments themselves, one has to structure the organization. Alek has seen companies of all sizes grow their programs. Today, we chat about what he’s seen work, the importance of hiring passionate people, and how LeBron is better than Jordan.
Rommil: Hey Alek, let’s get right to it. How about you share with our readers what you do at Optimizely and a bit of how your career got you to where you are today?
Alek: I’m the lead on Optimizely’s Strategy team, which is part of our Services group. Our Services team works with our customers to help them implement Optimizely, learn the right methodologies, and run their first experiments. Our Strategy team really focuses on educating our customers on how to effectively ideate, design, and analyze experiments. But in the past couple years an additional focus has been how to we help customers operationalize and grow the practices of experimentation across their business.
Prior to joining Optimizely I actually was a customer of Optimizely’s at the American Medical Association. By the end of my time at the AMA I was managing the Digital Analytics and Optimization team, where we used Optimizely across many of our products to teach our stakeholders on how to more effectively use the data available to them to make decisions.
Optimizely is quite popular in the industry. Could you share a bit about how it’s different from other platforms?
Optimizely is the most comprehensive experimentation platform in the industry: client-side and server-side testing, feature flags, a statistical model and results page, personalization and recommendations, a roadmap repository just to name the big ones. No matter your role in your organization or your use case, Optimizely can serve it.
I’d also say a differentiator is the people building and helping our customers with our products. You have a large number of practitioners who have lived the customer challenges they are solving for. It brings a relatable perspective to helping our customers.
I was searching for best practices on how to structure Experimentation teams, I ran across your article, “Enabling Experimentation at Your Organization: Determining Your Team Structure”. How did you come up with the idea for this article?
As I mentioned on what our Strategy team works on helping customers organize and grow their experimentation programs. Over the past 3+ years I’ve helped dozens of organizations do just this so I had a lot of examples and anecdotes on what worked and didn’t. This spans different industries, team sizes, and who ran the practice.
Since it was always a question we were hearing from customers I thought it would be good to give our strategy team’s view more broadly. There’s no perfect way or fit for every organization, but these are good starting points to grow from.
Did anything surprise you as you were researching it?
What’s surprised me the most since joining Optimizely as a customer is that the primary challenges organizations face when growing experimentation are the same no matter the industry, size of the company, or who is running the program. There’s always resource challenges. There’s always buy-in challenges. There’s always challenges in consistency. There’s always challenges in increasing your speed and frequency of execution. At the end of the day the biggest way to solve most of these is the type of people you hire to lead your practice. Who is passionate about experimentation? Who can make change happen?
“At the end of the day the biggest way to solve most of these [problems] is the type of people you hire to lead your practice.”
Something you touch on is “program measurement”. From what I’ve seen, this is easier said than done. Can you explain a bit about how you’ve seen the best-in-class companies measure their program?
The big one is always going to be ROI. Optimizely rolled out a model last year that should make this easier. There’s definitely an opportunity to alter this depending on how your organization views ROI calculations for software.
The other program measurement I referenced is a bit easier and something I feel every organization should be doing. A lot of organizations tell me it’s not as important as they start out, but I argue it’s as important as measuring dollars because it can help you identify inefficiencies in your organization. If you fix these issues you can impact dollars more quickly.
Ah, so like, measuring an Experiment conversion funnel of sorts. Like from Idea to Launch to close?
For example: how quickly you move tests through each stage of your workflow. Or which high-level strategy (e.g. removing distractions) generates your most wins? These are easy to measure from your backlog of ideas with little set-up work. You’ll be happy with yourself if you start measuring program measures sooner than later.
“…sharing learnings is a practice that needs constant testing and iteration on.”
A common aspect of the models you mention is sharing learnings. As you can imagine at startups as well as at large corporations, there’s a lot going on, a lot of information is thrown around. What are some of the best ways you’ve seen companies share learnings?—?and how do they select what kinds of learnings to share?
Our strategy team gets questions on this a lot and I’ve never seen a silver bullet. Much like experimentation in general, sharing learnings is a practice that needs constant testing and iteration on. But some things I’ve seen to be more successful than not are putting learnings into already established internal communications (e.g. an intranet), having the stakeholder of an experiment idea be in charge in sharing the results to the organization, and finding opportunity to gamify results sharing.
I’d advocate that all learnings are good to share. You never know which may prompt someone new to participate in your program or which may be useful to another aspect of your business. But if you have a lot and need to focus, maybe focus on the biggest wins and most surprising results?—?learnings that show we were completely wrong on what we thought we knew about our customers.
“But if you have a lot and need to focus, maybe focus on the biggest wins and most surprising results?—?learnings that show we were completely wrong on what we thought we knew about our customers.”
What are some of the obstacles companies should expect as they evolve how they organize?
The new shiny object. Optimizely may not always be the most exciting software you have in the organization (and even less likely to your executive). So be able to find new ways to gain back interest and excitement.
Experimentation isn’t going to be everyone else’s day job. And you’ll be short on resources. So how can you make experimentation something valuable to their day job?
Measurement on impact will be difficult. It’s best to agree on an approach earlier on and debate how to improve upon it later.
“Measurement on impact will be difficult. It’s best to agree on an approach earlier on and debate how to improve upon it later.”
Finally, time for the Lighting Round! Bayesian or Frequentist?
I’m going to say a little of both so…Stats Engine!
I should have gussed LOL
If you couldn’t be in Experimentation?—?what would you be doing today?
Teaching. A lot of my job is educating today and that’s what I enjoy the most. How can I enable someone to make change at their organization and in turn be personally successful in their career.
If you won a billion dollars today, what are the top 3 things you’d do?
Help my family, help my city, then you probably wouldn’t hear from me again as I travel the rest of my days.
That sounds like a great plan. You’re a good guy, Alek.
Onto more important topics: Cubs or Sox?
Indians. As a Clevelander the Cubs have given me one of the worst heartaches of my life. So even the Sox are in the Indians’ division I dislike them less than the Cubs.
LOL?—?but how do you really feel?
OK….Who’s the greatest of all time? Jordan or Lebron?
LeBron. As a Clevelander LeBron has given me the happiest moment of my life.
That’s awesome. Alek, thank you.