Always start by testing your best variation with Zach Lebovics
AI-Generated Summary
Zach Lebovics shares insights on experimentation and product management. He discusses the importance of documenting assumptions, testing the best-case scenario first, and understanding that experiment failure isn't solely tied to the outcome. Zach also shares personal experiences and learnings from working in companies like Zynga, Ritual, and Tonal, providing valuable tips for product managers. Episode links Zach's Etsy shop link: www.mindfulmix.co Zach's Team Hub on Coda link: https://coda.io/@zach-lebovics/tonals-product-team-hub
Audio
AI-Generated Transcript
The recommendation here is and this is what I take with me going forward is is always document your assumptions test your best case scenario first because at the end of the day experimentation is all about getting the most amount of learning in the shortest amount of time hi my name is Rommil Santiago and I'm the founder of experiment Nation on today's podcast we have Zach Lebovics he is the staff product manager at Tonal
Today Zach shares with us while experiments are powerful we don't always need to run them how you should always test your best scenario first and experiment failure isn't always tied to its outcome we hope you enjoy the episode hi everyone uh welcome to another episode of experiment Nation I am your host R of Santiago I'm the founder of experiment nation and today I have with me Zach an old friend of mine uh we we go how many years a few years
Now I think at the scor is when I first met you um yeah I don't I don't want to steal your thunder um welcome to the show and I'd love for you to introduce yourself to our audience yeah thank you RL first of all thank you for having me really excited to be here really excited to talk about experimentation um I think it was it must have been seven years ago that we aging yeah yeah and then in my head I was trying to remember I'm like I
Remember overlapping with you at the score and then we had a brief overlap at ritual too yeah um but that was right before I departed but just just a little little bit there I remember when I we overlapped at the score I was I was watching you present some I don't know how much we could talk about but some of the work that you were doing and I was like holy crap this this guy is like killing it and I I I was like I'm gonna keep track of this person and and I'm
Happy we we've been able to maintain a relationship but uh I sorry to cut you off kind um yeah let me get into it I mean I'll start with the the fun stuff before getting into to um who I am and what I do but I like to say that I am a an Optimizer um an Optimizer of all things that's really I'm I'm ultimately looking at um doing things twice as fast or uh drop the effort in half for literally everything uh that I look at I may be in laziness but that's just
Laziness I say makes me makes me I would say a ID product manager maybe a mildly annoying husband when I tell my wife she an app because uh her recipe uh process isn't efficient or she's got to move her notes over to some digital product um but that's yeah that's that's who I am I've been a product manager um when RL was just referencing that presentation that was right at the start of my journey as a product manager I I broke into product
Management about a decade ago found the craft through uh an internship at my college I went to the University of watero and uh currently I'm a Staff pm at tonel which is a digital Fitness product makes a really cool machine mounts on your wall and uh we'll talk a little bit about that later but uh this is the new way of working out for me and hopefully for for others in the future outside of work um um I also like to Tinker with things like to
Optimize and so I mentioned I like to try to make things easier um so I built two products with help of family and friends one was making workout planning a little bit easier you can see I remember that one yeah yeah that that work actually started while I was at the score um shout out to Amish um nice he he helped he helped build the the back in for that and uh the most recent thing that I've been taking aning with and building with my wife actually
Is focus on making uh mindfulness breaks or taking mindfulness breaks throughout the day a little bit easier so we've usually I have like the deck of cards that I can easily reference but we we designed a deck of cards and on each card you just have a quick two-minute activity um either breathing one meditation one um or movement which is stretching and yeah start a Kickstarter campaign with that and and have just been kind of cruising in cruise control
Mode uh selling that on Etsy as well as to um whoever is interested so links we will definitely include it in in the show yeah yeah yeah I can definitely provide that nice um and when I'm not on my laptop doing some optimization work or product management work uh you'll find me on the ice playing hockey you'll find me on the golf course um it used to be out in the gym and now it's working out in my basement on my tonal so being Physically Active is a
Core part of of who you are I'm starting to see a pattern here yeah it's weird there's there's that laziness that drives me to want to do things quicker or more your entire day with working hard yeah yeah yeah just replace that with activity so clearly you're a product person at heart an Optimizer at heart um but you started years ago in product and you found your way into optimizing things I'd love to hear about your first
Experience experience sorry uh with experimentation um is it with product or was it with your life or I want to uh you know what it was right before you and I worked together at the score I was working with threez at Singa oh wow and that name in a few years yeah yeah um at Zinga for those that aren't familiar with it all I need to say is is Farmville and people are like oh yeah Zing okay okay in Toronto um which is where I'm
Based the Zinga company um or or the Zinga representation within Toronto they owned a few different games and so is you can think of it as like a portfolio company with a bunch of different games uh while I was there I was working on a chess app called chess with friends and what was really interesting about the app was uh it hadn't been touched in about a year or two when I joined um but the team noticed something which was the day 365 retention so the
People that are still playing the game A year later was higher than most games that they were actively working on um the issue was they didn't have that many new users on boarding but if the retention is strong that might prove to be a fruitful opportunity to work on it some more and see if we can get uh more users into the system because clearly something's working now at the end of the day as with all businesses the Topline metric is is revenue right and
So you can only justify spending time on this game if this game is making enough money to um offset the working capital that goes into it Y and so we were looking at the game um we specifically wanted to pay attention to early retention we figured if late stage retention is that tight how do we focus on an earlier stage in the funnel and and improve that and so um when we were looking at retention um it's very tightly tied to revenue um the more that
We get people to play the more they will see ads which is how we made most of our money I really going to ask you what was the model It's ADS is there in app purchases at all or is mostly ODS at the time there weren't so uh there were two vehicles of monetization one was um it was a Premium app so you either had the free experience and you watch an ad after every turn that you took um or you would pay for the adree experience which we both know not a whole lot of people
Do you make the bulk of your revenue from people watching ads yeah um and so it is important to understand that because um the way that this app is structured is you're playing with friends so if I would have you as a friend r on Facebook I would challenge you to chess game I would play a move move likely that would um be the wrong move because at the time I knew nothing about Chess and then you would rece you're the PM but you know chess is you
Kind of get it but not really yeah we didn't even have a research team at the time and so when I was looking at uh features we had a chat um portion of the app and so I would challenge people at the top of the list which were also High skill level but the only way to talk to them was to send them an invite yeah and so my record was like three in 80 when I first started you get the three mine would be don't worry about it lucky they probably just walked away
From their phone and I I uh won just out of um they just loved you yeah yeah uh so the yeah the monetization model um I send you a move or I I play a move you get a push notification you hop into the app you play your move right after you play your move you see an ad and so the more ads that people see or the more games that they play the more turns that they play um the more money the the um app makes and so uh when we
Were when we were looking at well how do we get people to play more one thing that we were seeing was people wanted to play more they just had friends that or friends that weren't responsive or they didn't have enough friends to play with um story of my life so sorry story story of my life so it's really sad when I Frame It That Way um and so we were thinking how do we get people then to connect and so we introduced this feature called
Playing now and all that it would be was when you jump into the app we would surface other people who are right now in the app who are at your skill level that you can challenge and we tested that on on a subset of users and we saw a 20% lift in um their number of turns started which led to that's uh pretty much one to one with a 20% lift in revenue and that was the first time that I saw an idea come to life in the form of a product
Experience where I saw very clear not only impact to the experience from a member's perspective or a user's perspective but also to the business so that that experiment went very smooth and and nothing went wrong right uh you know what that was actually maybe I was sheltered from what went wrong but my memory seems to to uh recollect only a very smooth experience but I have had many issues beyond that okay I I know that every time I run an
Experiment there's always this uh well well we we screwed up we have to roll that one back um yeah well that's clearly because you're the better PM than I am um but over the years now you've you you spent um building thing things and optimizing things I'd love to kind of take a different approach to this interview because usually we go oh what are some of the best practices here from a positive sense I'd love to hear about
Some of the mistakes that you've made um because we all have learning during our lives um that that you've had from all these uh all these experiments listen experiments are extremely valuable at um deriving impact and and understanding what type of influence you are having but they're not easy to run um everyone runs into issues or um areas where they run an experiment and they're just like man that could have gone better the there are three things that come to mind
Um and for each one of these I'm going to tie a specific experience to it the first one is after my early experience with experimentation I thought experimentation was not only the goal standard it was the tool for everything when you have a hammer when you have experimentation everything looks like an experiment I learned very quickly that experiments are powerful but they're not always
Needed um and there there are certain situations that uh make them less valuable than others m the second one is this was a learning from ritual I spent about I would say in grand total five months running a sequence of experiments and uh my one takeaway from that one um was always test running the best case scenario first and then the last one I'm GNA tie to my experience at tonal um experiment failure isn't tied to the
Outcome meaning the result or the Delta okay yeah that that sounds great let's I love to hear okay so yeah just jump right into it cool cool um let me start with the first one so experiments aren't always as valuable as we think they are um this this hurts me like right there but I'd love to hear this most times they are um but there there are certain circumstances where they are either Roi neutral or negative and
It's important to recognize when you're in one of those environments and so the situation for me was coming off the back of millions of uh da daily active users at Zinga and um hundreds of thousands at the score I jumped into a startup I'm like I'm ready to spread my wings I know everything there is to know about product all five users no yeah I I I know everything there is to know about experimentation and product management
Three years into the Craft um Let me let me see what I can bring to the table for this startup um you can tell I say that factiously because looking back I knew very very little but I was ready for the challenge and um I was maybe not caught off guard but um intrigued by the fact that the user base was so small because that showed um that there was a lot of potential to grow the app um but when I say small it was it was really small like we're talking
Maybe a 100 da wow yeah yeah and in this in this environment um I I just knew that experimentation if you wanted to measure your impact you have to have a control you have to have a variant um maybe we can bend like statistical significance ever so slightly but that was the way of doing things that's how we should be operating so your size at 100 oh my God so it was the I mean the variant here we were running I we we had
To do 5050 um and we were talking about real estate that not everyone visited either and so the basically the sample size would have required at least two and a half months of the experiment running yeah was that and we kicked off the experiment and um at the end of it we didn't see much of an impact in two and a half months in the grand scheme of things doesn't sound like a whole lot of time but when you're talking about a startup yeah um the
Opportunity cost of not that we weren't doing anything else on the side but the opportunity cost of waiting that long for results is just simply too high and so the learning here is there's a cost to experimentation before I didn't I kind of thought of experimentation is yeah it's a tool that is valuable and and should be used pretty much every launch that you do but the cost is there's a cost of building the experience there's a cost
Of doing the analysis um there's time which is also a cost digging into segments or generating insights and so I started I did some research and started thinking I'm like I need to know when experimentation is not needed and so um three situations where I think you'd be better off not running an experiment yeah number one is what I went through it just simply takes too long to get actionable results uh number two is the downside of
The uh the downside risk of the change is low and so if you're running an experiment and um control variant either way like you're the variant isn't going to make a big difference um you may want to question like do we just make the change instead of a bug fix sometimes yeah yeah yeah like like a bug fix or a general um you can even say like a a feature optimization where it's something that a lot of users ask for you're not
Disrupting the experience it's always great to measure the impact but if that comes at the expense of of delivering value sooner you just have to weigh those just jump in there I think that's something that a lot of product people face and not so much marketing growth well a little bit of growth or C conversion rate optimizers where a product manager always has to think about the opportunity cost of their work you're keeping devs away from a certain
Feature or you're you're holding something back from from Market uh and that all comes AC cost as as you as you mentioned I think that's that's a great takeaway go on sorry yeah yeah and the last one seems so intuitive but I've made this mistake before is is when the outcome won't actually change your next set of decisions and so if you are running an experiment because it's part of a grander initiative or because um it is something that you know you need to
Do then uh in general an experiment May it again it depends on your speed of learning but especially in an environment where it's taxing both on the cost side as well as on the time side um then if your decision is going to be the same no matter what you're doing uh do you really need that data in order to move forward because at the end of the day um experiments are more than just understanding your impact they're
About learning and informing future decision making and that that's kind of a a different perspec perspective and some places we've worked together where nothing went out without an experiment and you had to report it and had it analyze by data science and but I like approach where it's pragmatic because I remember in my history when I when I pitched experimentation to product managers I like the first question they gave me was well we don't
Have to test everything right because I have a road map so yeah I I to empathize with that yeah there's there's a balance in um these situations are they could be dangerous if misinterpreted like you don't always need an experiment doesn't mean never run an experiment or like only at uh in very uh slim circumstances or infrequent cadences do you run experiments this is more so experiments for the large part
Should be quite standardized but there are some maybe 10% of the time where you're better off skipping yeah I think like framing it as all PMS do frame their decisions with context you know as you as present them to to the stakeholders it's kind of like well in this case we won't because XYZ rather than just not running it and that's a slippy slope where they're like okay well we could speed up velocity um if we just like stop testing stuff right
Yeah yeah yeah okay so that's that's great how about what's the second learning that you have yeah so the second learning is test your best case scenario first I learned this one at ritual where I mentioned spent multiple months on an experiment which probably should have taken a few weeks too maybe a month and so context for this experiment first of all for those that are less familiar with ritual um ritual is a food ordering
App you go on your phone you place an order you walk to the restaurant you pick up your food you skip the line um that is uh the main value prop and so we were going through a massive while I was there we were going through a massive period of growth because we were running a bunch of campaigns and the growth was great the issue was the cost of acquisition was too high and so next step was okay we haven't figured out growth but growth is in a good spot how
Do we ensure that we are growing a little bit more sustainably or economically and that was measured by um reducing the acquisition cost of these new users and so uh how does acquisition cost at ritual work well um when you on board on ritual you're given credit to spend at any restaurant and so we were looking at that credit where it was spent and realizing that in certain sit situations you users were spending their credit at restaurants that were
Basically paying ritual to acquire new users and so what was going on there was if a user had $10 to spend and they were spending all of that at a restaurant that was paying $5 do to acquire them it only cost ritual $ five instead of the full 10 and so we looked at that and we said huh that's that's really interesting what if we can get everyone to spend their credit at these restaurants that are paying to acquire new
Customers um you can imagine that that would drastically cut down the acquisition cost and in the end it did um we got to a point where we transformed the credits into vouchers those vouchers can only be spent at certain spots and um we reduced the acquisition cost this is Romo Santiago from experiment Nation every week we share interviews with and Conference sessions by our favorite conversion rate
Optimizers from around the world so if you like this video smash that like button and consider subscribing it helped us a bunch um the issue is or not issue but the lesson learned here is we could have made that jump that was where we wanted to go but we were scared of what we could do to early user attention and so with credit we first made it restrictive you can only spend it at certain spots and then we change the value of how much you can spend and then
We introduce vouchers and then with vouchers we introduce more restrictions and so we were basically like taking small steps towards this goo when in the reality that end goal retention was still in a good spot and so the learning here is experiments are rooted in strong hypotheses but sometimes we let our own assumptions slow us down and so whenever you're running an an experiment and you're thinking there's
No way someone would do this or I don't think this is what a user wants those are assumptions and um that means that they're also right for Learning and the recommendation here is and this is what I take with me going forward is is always document your assumptions test your best case scenario first because at the end of the day experimentation is all about getting the most amount of learning in the shortest amount of time and it's
Always quicker to test the best case scenario pullback than to inch your way forward towards it yeah I like that the the way I view that is if you're trying to find a a Maxima you need to test the extremes right yes if you test small then you're like oh that helped test bigger well that helped but that's going to take you forever to get to um but there is the side where you go okay well I can't go that extreme extreme because my leadership will kill me um so you
Need to have that that kind of balance of test the best scenario that they'll accept and not to defend you too much but the environment that we were in was a little bit risk averse so I you know I get why you got there um okay so that's number two what what's number three uh number three in this might sound like a controversial one but I'll I'll explain why think this is the case experiment failure isn't tied to strictly outcome okay um we on an
Experiment because we want to change a certain metric we want to increase conversion we want to get more people ordering we want to get more people um sending starting chess matches with their friends and so that's that's a big piece of it however um there's much more to experimentation is a tool within a broader toolkit experimentation um should be used alongside research should be used
Alongside surveys and I mentioned that experimentation is really all about learning in the shortest amount of time possible and so when I was well when at tonel um I was running an experiment this was early on and it was a pretty meaty one to setup we were just uh we saw that those people that joined programs multi-week programs on tonel they tend to retain better and so what we wanted to do was we wanted to surface suggestions for them to jump
Into right off the bet when they first got their tonal and we ran this experiment we saw at the end of the day um we saw like a slight lift in this metric that we were looking at which is how many people enrolling programs in their first week and how many people stay engaged with in programs in the long term and so if you look at the strictly the cost of the time that went into building the experiment and the True
Result one could argue that it was Roi negative um however because we ran the experiment we were able to learn a few things that we hadn't previously known um people were overwhelmed when presented with three or more options um people needed context in order to feel confident in a recommendation they couldn't just see here here's a great program you should try it they want to know why um people need time to experiment with coaches with content
Before they feel like they can commit to a multi-week program so all of these things are learnings that we didn't previously have in key learnings that are going to make Building Product and features in the future easier um and so the learning here is that people again people tend to associate the success of an experiment with the overall impact that it has had and while impact is really nice and you should be chasing that there's still an opportunity to
Learn when you don't hit the metrics or even there's more of an opportunity to learn when you actually uh tarnish the experience when it's a negative experiment right you see a hit in the metric you didn't want to see it g um that's I don't like programs that only focus on win rates or this this concept of positive winners because at the end of the day experimentation is about learning and the test may not itself generate the learning directly it might
Force you to dig in deeper and you find something um but obviously harder to measure and I I get why people don't you know report on those sort of things yeah yeah and and so that's where I think the success of an experiment I mean the uh perfect scenario the golden scenario is you had a strong hypothesis you tested it you've validated your hypothesis um or invalidated it and you know why yeah um I'd argue that like I've seen experiments run in the past
And I've even um been a culprit of this where you run an experiment you see the results you wanted to or at least close enough and you kind of wipe your hands clean and you move on to the next thing but you don't analyze yeah well what's going on under the hood why are people making these decisions um because that's where the true nuggets are that will help you become a better product manager and inform future decision making yeah now these were these were all great tips
And you know just to summarize reflect what I what I heard um so you mentioned three things the first one is you don't always need to run a test you know be smart about it be pragmatic about it uh test your best case scenario first and and finally the outcome of the experiment um doesn't necessarily mean it's a fail like as long as you get a learning out of it uh that that's that's what you need um but I want to give you an opportunity to share with our
Audience um something the things that are going on in your world that you want the world to know yeah yeah appreciate that um I did mention that uh the product that my wife and I are working on uh R I'll give you the link for that one it's called mindful mix nice uh little project that we we built on the side the other thing that I will gently promote is uh you those for those that may or may not be familiar with the tool called Koda it's the next evolution of
Docs it's it's a great great productivity tool it takes docs sheets really all of your notes and combines it into one Okay the reason I'm bringing that up is because um I created a template on Kota that Kota is uh featuring on email campaigns and it's designed for product managers or really anyone that's in a core functional team it's a template that helps organize all of your team artifacts so for a product manager it' be your road map your Sprint
Planning your team syncs your prds I can provide a link to that template as well definitely right there yeah yeah but uh that is I believe this is being filmed on March 28th that is being announced today and shared from Kota um later today oh dude that's amazing congratulations that's everyone everyone definitely has to check that out I look forward to downloading that myself um yeah it's it's been great catching up with you um don't be a
Stranger good luck with the family life and all that um congrats to you and your wife um yeah thank you for being on the show and um yeah take care thanks so much for having me Romo it's been a ton of fun this is Rommil Santiago from experiment Nation every week we share interviews with and Conference sessions by our favorite conversion rate optimizers from around the world so if you like this video smash that like button and consider subscribing it helps us a bunch
If you liked this post, sign up for Experiment Nation's newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK