AI-Generated Summary

Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game: - Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning! - Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks. - Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success. - Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving. - Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!

https://youtu.be/O2V2LRncM58

Audio

AI-Generated Transcript

(00:00) and I've hated this when I've been in house is when an ecommer like a con a conversion rate company an agency comes in and says oh we have loads of e-commerce experience we're going to run all these e-commerce tests for you and you're like no no that doesn't work do you even know anything about my customers like we've we've had these all

(00:15) these conversations we're spending all this money with you and you don't know anything about my customers hey you're tuning in for interviews with experimenters I'm Anna you're your host and today we're speaking with Ellie Hughes hello everyone and welcome to another experiment Nation interview and uh in this episode we have

(00:39) a great opportunity to learn from someone who is not only at the top of their game but who's also Reinventing rewriting the rule book as she goes um and I think this is a great reason to stay and listen to to the episode because there is no better way to learn than standing on the shoulders of giants and um Ellie you have been

(01:07) awarded experimentor of the year and you are also uh in top 20 uh must meet cro people and so besides that how would you introduce yourself and what would be some of the what would be some advice that you would give for someone who is aspiring to be in top 20 zero people thank you for the amazing intro

(01:38) an um so so yes I'm Ellie I'm the head Consulting at a company called Eclipse I just joined Eclipse I before that I was working for um just over two years at a compal and digital in both places I've been doing experimentation helping clients um but for years before that I was working um in house I think in terms

(02:00) of inspiring people to um to get into the industry I think it's about starting small like most of the people I know who've ended up in experimentation didn't say when they were like 22 I know what I'll do I'll go and become a cro they might not have even known what a cro was so I think it's a case of you might have come in

(02:26) from multiple routs but it's about kind of learning as you go picking up some of these fields like does something about working with data really excite you or does working with customers really excite you or is there something about getting into the nitty-gritty of the engineering and the process and like how

(02:43) do we deliver this better and a higher quality I think those kind of questions get you excited um and kind of can get you motivated into actually learning more I think the other one is never kind of lose your curiosity I think a lot of us start off you know at University maybe with a lot of curiosity we're very

(03:02) openminded and I think as we get older maybe we lose that so I think it's about keeping that openness to failure keeping that Curiosity like why does this happen this way and why is this wrong and and just kind of continuing to kind of keep learning and keeping that openness of of mindset I think subject matter expertise

(03:21) sometimes matters less than I've got a question and I want to answer my question and I'm going to go and keep answering that question so and you have to learn to live with frustration yeah a little bit a little bit yeah that can be really hard so um or realizing that you know sometimes frustration is an opportunity for

(03:44) learning you know sometimes in our career we think our career has to be this like straight line and then when you look back and it hasn't been a straight L like why hasn't it been or actually I went off and did this thing and this taught me this thing and even though it felt quite frustrating at the

(03:57) time sometimes those moments in your career and your moments in your current job where you're trying to run experiments they can feel like they're probably frustrating but actually they're not really so yeah actually you are in reality learning and growing and going forward so yeah yeah exactly we should all be aware of

(04:15) that um staying in the same uh topic for a little while I also want to ask you to give us some tips on let's say time organization or how does your uh workday look like in terms of um you know dividing the tasks in terms of creating hypothesis evaluating tests um looking into data and all of the all of the

(04:43) phases that go into experimentation program um so I've always been someone who's a bit more on the ideation and data side so it's been a few years since I actually last did the whole of the endtoend process of delivery experiment myself and I think that speaks more to the fact that I've realized what

(05:02) strengths I need to play to and to allow others to be stronger at something so I've worked in programs before where I was very very in the end to end and I would have to chunk up my time you know for different clients between this is like 10 years ago some of the days of the week I was focused on analysis of

(05:17) experiments and others I was building and developing experiments um I realized early on that engineering although I was good at it I wasn't as good as other people so I thought why don't I leave successful engineering to other people let them grow in that space and be really successful at it and I'm a bit better at

(05:35) explaining to clients why has this gone wrong um so I would say in terms of how I chunk up my time probably I don't know 30% to half of my day is either explaining to clients why something's gone wrong or working out why something's gone wrong or anticipating something that might go wrong and suggesting something before it

(05:54) goes WR so that I'm kind of thinking through what are the things that could happen and what are these edge cases that that might happen within the program I would say the rest of my time is focused on things like um maybe not specifically doing the analysis of certain experiments but looking at the analysis of the program like how do we

(06:12) look at the number of winners number of significant or non- significant tests and kind of I'll often build like little visualizations or little tools that will help us to um track kind of oh right this is the velocity of our experiments that is the pace this is our number of winners this is our value of the program

(06:31) so it's kind of taking what we've taken from analytical insights which is another area of M that I like working in and actually deploying something more end to endend to look at the whole program so I think you know as my role as I've gotten older as I've gotten more senior I spend a lot less time actually

(06:48) physically building the tests myself um but I definitely think there are some bits around like ideation for example I'd probably spend and depending on the client because again I'm talking about this from the client work perspective rather than in house when I was in house I probably spent at least a day a week just

(07:06) thinking about experiment ideas and ideating I think now that I'm working with lots of different clients that's a bit less per week in terms of how I chunk up my time but if I'm fully running and embedded in a program I'm working with lots of different product managers trying to help them think up their first experiment their next

(07:22) experiment that they want to run and looking at the data and ideating so I would say that's where I've drifted to in terms of my strengths and how I chunk up my time um I wouldn't say I was that organized not always sometimes when I I want to be organized and sometimes when you have those moments where ideas

(07:42) really strike you that are really good you sometimes have to bend the rules a little bit and and actually go outside of the shape of of of how you're meant to to work but um but yeah I think it's about nailing like what does client value mean what does value mean you know can you prioritize your time around

(08:01) what's valuable for you to work on a client and I'd argue me spending ages writing Juro tickets for someone else is a waste of their time and mine but someone else would be much better at doing it than me versus actually me thinking about is there a value here that we're missing out on is there something that we're not seeing in the

(08:21) data and my skill really is like doing that holistic synthesis of of of lots of different typ of data and finding that kind of diamond in the rough idea that we wouldn't have otherwise spotted I think that's where I find my skills apply best and I think the other area is probably particularly working with product teams helping them

(08:42) to capture everything they need to about their experiment before they start working on it so making sure they've actually got their variables listed correctly have they done their thought about their tracking what extra metrics they might need to track um using something like this which I've got pinned up behind me which is like a

(09:00) canvas that helps them fill in all the information they need to so that then they don't get to writing the J ticket or even building the experiment go oh yeah there's like a whole segment of people we've not included and we should have done so get stuff yeah like that there fair amount yeah yeah exactly exactly so yeah

(09:18) that was a really round about answer to your question sorry no you actually opened up so many topics that I want to uh touch upon in the interview uh you mentioned that you're explaining uh to clients why something isn't working and what went wrong there so I want to uh ask you about I want to connect this to

(09:44) peeking uh I know this this might seem not connected at the moment but I want to uh see uh what is the right way to report to clients what is going on with their test while it is running uh and we know we shouldn't PE so how would you tie these two together and what are what can you report uh I think there's almost

(10:09) two there's almost two difficult client conversations there one is a macro one and one is a micro one so the macro one is you've got to the end of the experiment and whether it's come out as a winner non-significant or negative the client's still questioning why the performance of The Experiment was what

(10:28) it turned out to be and then I think there's an earlier conversation which sometimes people avoid having clients which is just talking about what peaking is so by kind of agreeing with them I I think there's always a kind of back and forth in client relationships where you have to agree with that client this is

(10:47) how frequently we're going to look at the experiment results the results will be accurate in the following way and you will get them at this frequency and I think if you establish that early on in your relationship with the client they kind of know that's how often they're going to get their results and so they

(11:04) don't have alternative expectations you know it's a little bit like if you reply to emails when you've got your out of office on you're kind of like denying the purpose of having an out of office so in the same way if you send results more often than the pre-agreed timeline that you're going to talk to the client about

(11:20) results you're setting a precedent where you would give daily reporting on an experiment for example I worked in house someone where where we would have our dashboards refresh daily so we could look at the experiment results reporting daily um but actually it didn't really help you know we'd have product managers

(11:39) come and say oh it's there's only four days of data in the experiment and this happening to this the confidence level and I was like this is irrelevant just just leave it alone just stop looking at it so you know um and one of the things that we would often do that was when I was in House at photo box one of of the

(11:57) things we would often do is report experiments we wouldn't report the result unless it was significant so we would just say in any reporting it's non-significant right now therefore there's no result right right um which which did it drives another Behavior that's bad which is making people like aspire to watch the

(12:18) confidence graph go up and up and up and they hope it's going to be good but um it's less of a bad behavior than looking every day and going why is it gone up why is it gone down why has it gone up why has it gone down and then having to explain that actually they're peing um so yeah there's a few there's a few

(12:35) different coping mechanisms in there I think but setting those expectations and then knowing yourself right I'm going to look at about a week unless there's a big error so you could personally from a QA perspective look more often but just don't tell the client you know monitor it for the first couple of days to make

(12:51) sure that there isn't an issue if it goes really you know right into the red and you know it's really negative that is an issue it's worth reporting to the client but um right I think otherwise you can secretly privately peig but don't tell client than you doing good tip good tip yeah and how would you go about

(13:12) explaining to them what went wrong and before that uh what is the process uh of uh evaluating the test and finding the bugs and errors that crept up along the way this is Romo Santiago from experiment Nation every week we share interviews with and Conference sessions by our favorite conversion rate optimizers from around the world so if

(13:38) you like this video smash that like button and consider subscribing it helps us a bunch I think a lot of people at that point are like well who do we blame for the fact that it went wrong and I think um that can be quite hard but I think the best thing to do is to actually do your own root cause analysis

(13:52) and say look it wasn't actually us there was another piece of code that went live in a release at the same time as the experiment that we just weren't cognizant of clearly there's a process flow issue here we need to be connected to that team so that they're cognizant of what we're doing and this won't

(14:08) happen again and I think experimentation the process itself of doing it is always iterative so you always learn things as you go along that could be things that the first time you run a type of experiment coming from a particular traffic Source or that plays with a certain page you discover extra QA

(14:24) scenarios you didn't think you had or extra people you need to talk to to get approvals and sign offs and maybe that one time you didn't know that that person existed and it's nobody fault you just Were Meant to have involved them in the process and I think making sure that you and the client are both happy with

(14:40) the fact that the process needs to evolve so that then you know in a six months or a year's time you know your process will be better you're not going to have a perfect process from day one um so I think it's just about kind of being open to that and then again setting the client expectations that you

(14:55) will need to change the process that you'll start with something simple and then as you find extra edge cases that need to be more complicated you kind of build them into your process as you go and creating opportunity for them to always give you feedback as well I think that's a really important right of

(15:11) course of course of course and yeah I think that uh a lot of it lies in the I will say educating the client but I don't to come off as you know Patron or anything so they definitely I don't think that's ping I think a lot of clients would turn around and say I don't know anything you're the expert that's why I've hired you so so yeah I

(15:37) think you're right it is about education and it is yeah you can even use that word outright you know you could be like oh we're going to enable you we're going to equip you but sometimes they outright want training or they want to know how to do something because they really don't know how to do it so right um but

(15:52) I think yeah you're right Common Ground has to be kind of established so they know not to expect you know uplifts from day one that will just keep happening yeah man managing those expectations as well like if it's a client side uh experimentation program versus service side serice side you often take a little

(16:12) bit longer to get going because you're getting into the code base I always try and set their expectations that it's at minimum six weeks before you'll even get your first experiment live if you're working service side and I'm not saying that that's high or low but I'm saying that it's kind of of average compared to

(16:28) other programs that I've worked in um I know other businesses would be faster but I think when you're first getting started you don't know what you don't know when you're working with a client's code base so I think setting that expectation the first experiment won't go live for like six weeks and then like

(16:42) oh well at minimum will then for run for three to four weeks and they're suddenly going oh wait you only been working with us for like two months before we get our first winner yes and then the penny kind of drops a little bit for them um so yeah kind of yeah challenging them bit that's that's one of the

(17:02) conversations that needs to happen but uh we were also talking about this earlier before the interview where we talked about how to increase the power of your test and now we are getting into more statistical side of it so you mentioned there are several approaches that you can do um and you mentioned one of them let's say try

(17:31) small changes or then on the totally opposite side try something totally radical um what would you say are other approaches in this sense um so I think there's there's two things that that you can do there's actually approaching it statistically which is like how can I make it really rigorously good so that

(17:51) then I know I'm definitely going to reach power I'm going to reach confidence and Power want in my experiment result and then there's also like the what you just mentioned which is the size of the ideas the quality of the ideas going in so that kind of you know power helps you correct from Miss opportunity rate and so when um you're

(18:10) kind of looking to grow to make your your website better to grow your customer base to increase Revenue you don't you especially if you're starting small you don't want to miss out on those opportunities so what can you do to increase the size of the ideas that you're testing so that then actually you're more likely to detect something

(18:31) sooner so you don't need as much power so if you imagine a small change might need four weeks for you to detect it a large change might only need two weeks for it to be a fully powered experiment that's reach sample size so if you had two big Ideas you'd get them done in two weeks it's counterintuitive people think

(18:47) oh it's a big idea so it's harder to build will take longer to build and not longer to run it's like no actually because it's so much bigger it will take less time to run for you to get a result so actually then if you've got another idea prepped straight away it can launch and you'll get two highly powered

(19:02) experiments that are opportunities for growth rather than a smaller experiment that will take a lot longer to run and so I think right right that is another client education thing which is a nightmare it's really difficult but um I think just I what I do in that instance is I just spend a lot of time with the

(19:21) person to try and help them understand the fact that it's a bit counterintuitive that when it's a big idea it's easier to detect when it's frequented statistics and when it's a small idea it's harder to detect and I think the question I always get back from clients and product managers when I'm in house equally is how do I know

(19:37) it's a big idea before I run it you know that kind of minimum detectable effect thing well how do I know what the effect size is when I'm sample sizing you can go well this is the industry best standard do you really think your experiment idea is big enough to be bigger or lower than that do you think

(19:51) your experiment is big enough to achieve 30% lift or something like that or is is it you know more on the 5% or 2% left end and then the client turns around and says this is a really small change I'm like well and you kind of leave it up to them because you have to give them the tools to Think Through what's the what's the size of

(20:10) this really so right do you do you explain to them the concept of minimum detectable effect and do you use it when it comes to evaluating the size of an idea and when it comes to priority iing um what I tend to do instead and actually this is one of the other ways in which I use this this canvas behind

(20:35) me is I get the client to think about um given the page you're on what's the number of your new visitors who are hitting that page and actually for some clients they don't always know that you know I had I was running a workshop a few weeks ago where we were looking at just general sample sizes we weren't

(20:54) even looking at minimum detectable effect we weren't even actually sample sizing we were just looking at the number of unique visitors on Sample pages and they were like oh wait this page that we wanted to launch over here that's in this funnel it's actually really small it's only getting like a million unique visitors a year or

(21:10) something whereas this page over here is getting like 300 million a year or something you know much much bigger in terms of the scale and sometimes you can lead them on a journey where they discover that themselves and they suddenly realize oh right this thing you've been going on about I now get it

(21:26) because I need more people to be on the page for me to be able to actually serve this to them and then once you've kind of got them hooked on am I showing this to enough people so I think if you were doing like rice prioritization that we reach you know that size the number of people who are actually visiting the

(21:42) page and then after that I might use um effect size if we have enough information so like when I was at photo box we'd been testing for long enough like for you know a year and a half of experiments if someone said to me that they had were running a particular experiment on a particular product page

(22:03) on a particular metric I could go back and look at multiple experiments that have run in the past even at the same time of year and say right the Run rate will be about 4.5 weeks and it became really really accurate because we'd built up a list of other experiments that had previously run and we re could

(22:19) say oh well actually the exact time that that took to run on that exact page for that metrics was 36 days and actually the experiment that we then when we reran it it was about 30 days so we knew we were in the right sort of ballpark with our with our sample sizing um whether I exclusively use it for

(22:36) prioritization or not maybe I ha the answer it depends but I think it depends on the client I think if they right if if they're more capable of thinking in like Impact versus effort that I might use that instead like is this a big idea rather than minimum detectable effect so the Impact versus the effort and just get them to kind of

(23:00) focus on like oh could we make the idea bigger then it will run for less time and kind of focus on it in that kind so but but no it's a tricky one and I don't think there's a single one siiz fits all solution I don't know if you've experienced that but I think different clients It's Different Strokes for

(23:18) different folks isn't it so of course of course I mean for some of them it's just easier to to get them on this sta iCal side of things and um explain how the cookie crumbles but um I think that some of them just still expect um constant uplifts from cro and uh not you know incremental growth and slow steady changes

(23:47) also few of them really want to experiment uh in order to to learn and get insights it's mostly just for you know increasing the the conversion rate on the website but it definitely takes time I would say an experience um and also sharing the the learnings with them um in the way that it is explaining to them why a negative

(24:15) result is actually good for you because now you know something that you didn't know before and you can include it in the next test so I I always find the tests we tend to interrogate the most are the negative ones because when something's positive it's like confirmation bias like if something's a winner you're like yeah I expected that

(24:36) to win and then when it's a loser you didn't expect it to win and you end up diving into there's very few experiments um where I've ended up doing a lot of statistical analysis in the results when it was a winner and I can only think of like two or three where one of them was because we needed a really rigorous

(24:52) result so we just needed to be able to report the ROI on the experiment very clearly and so we did a lot of statistical analysis afterwards and then the other two they were winners where they won for the wrong reason like they won at like a global level but didn't win on another metric that we were expecting them to so we were confused

(25:10) right and it drove us to go in a bit deeper but most of the time it's those losers where it's like a total conundrum where you're like H why did this lose it should have won like what and then you realize that you know we're all here to fail and to learn yeah then you start looking for reasons to see why it's not working from

(25:29) the technical side and not from the ideation side or yeah ex something similar something along those lines yes but um um can you can you give me an example of your recent aha moment so to say uh what what what what was it that you recently learned and that you want to share um um you mean from an experiment or from life

(25:59) you know experimenting in yeah um gosh uh so I mean so having only just started on my new company I haven't actually personally run my first experiment yet in my new role but in my previous role um I was working with a couple of different clients in like the last sort of six months that we were running

(26:21) experiments with them and and trying to um test out different things we could do I think actually one of the AHA moments was really more on the ideation side and we we found like a group of users we didn't know were having a terrible time and it sort of triggered the process of coming up with some experiment ideas that were

(26:44) focused on that theme of experimentation and it was something that the client had kind of not considered before and the way that we triggered that process was actually we literally took hot Java batim put it up on a slide with 20 people in the room and 20 people on the call dialed in and said look this is

(27:02) what real users have said and they've said that basically your developers are idiots like there was a real user quote that was like your developers are terrible they don't know what they're doing this website experience is horrible and like people were literally turning around and going it was it was a

(27:15) shock it was a bit of a shock to everybody involved and I just ended up telling to the client say is this stuff you've heard before does this and they were like we do know this but actually not everyone knows this so um yeah it was a how did you how did you discover that was it um via uh user survey that

(27:35) you just put up during the time of the test or um um it yeah um no no it's um what did we do exactly so we had I think access to the clients like entire hot jar backlog that had come from like the previous year when they had been running AB tests and launching features and doing different things on the site so

(28:04) that particular piece of feedback came from one product area where they've been running certain things and so that led to us kind of having that bit of insight that was quite focused on a particular group of users that had a certain experience and so it was almost like they were just an underserved group that

(28:23) had not bubbled up in the quantitive data but had bubbled up in the qualitative data so getting that pairing between the Quant and the qual and that's often where I'm I'm more of a Quant person personally but I'm often Quant person reaching into the qual to try and grab you know or what I do is I I analyze qualitative data at scale so

(28:46) they gave us access to 26,000 rows of hot jile feedback which I then analyzed using like net promoter score scale but also tagging up a lot of the feedback so that we could collate it at scale and go right of the stuff where we've had written feedback so that it's not just a star review but it's written feedback um

(29:06) you've had 3,000 reviews that relate to this topic most of them are negative do you want to do something about that um so I think you know that's off in the way so that's where you you went uh in looking for Clues but yeah kind of moving away from topic but definitely negative reviews are goldmines yeah absolutely this is where

(29:31) you feel the sentiment and get them out talk to the fears uncertainties and doubts on the on the page um but I also wanted to ask you something uh about the difference uh in experimentation when it comes to different types of uh services and and products so uh what are the processes when you are trying to experiment and build tests for

(30:02) typical e-commerce products and let's say for the uh services like a B2B SX product or or something like that exactly yeah um so I've had a mixture of working at working with different clients from lots of different Industries but also when I've been in house I was in house somewhere that was like a B2B

(30:26) um Hotel bed bank and so we could not run classic at scale experiments with our with our clients we didn't have the numbers and then I've also worked places like Photo box where we have the numbers but the type of um e-commerce platform it is isn't a normal e-commerce platform because it's a creation experience you

(30:45) go in and you build something and an individual customer might spend seven days building a photo book or they might SP spend seven months so it's about catching them at the right moment for their for their conversion so I think the most important thing is and I've hated this when I've been in house is

(31:05) when an ecommer like a con a conversion rate company an agency comes in and says oh we have loads of e-commerce experience we're going to run all these e-commerce tests for you and you're like no no that doesn't work do you even know anything about my customers like we've we've had these all these conversations

(31:20) we're spending all this money with you and you don't know anything about my customers and I can think of a specific example photo box where there were some ideas floated that were very e-commerce and actually didn't work for the customer they were very driven by certain things high up in the funnel that actually isn't the important part

(31:38) of the conversion Journey the important part of the conversion journey is much much deeper when someone's actually building their photo book and then going to the checkout um as a flip side when I was working at the B2B B bank that I was talking about before um we couldn't do classic at scale experiments but what when you're a

(31:55) Bea be like SAS provider you know somewhere like think of a jira or atlassian or something you know who all your customers are or even you know your optimizely you know who all your customers are they have to sign in in order to use your product like there's converting people as in getting them to sign up to use your product which is

(32:12) probably your primy metric in those cases but once they're using the product you know everything about them you know their size you know when when I was working at this hotel bed Bank we knew we could actually divvy up into cohorts groups of clients who similar to each other and actually run experiments where

(32:28) they were smaller scale but we gave half the clients the beta program and half the clients the current program and then just watch what their conversion rate was like over an extended period of time and we we'd sort of chosen them to be similar size and similar client profile you know the ICP was very similar um and

(32:45) so it was a different approach to experimenting it was more like a cohort analysis where you watch what happened to these two cohorts over a longer period of time like a longitudinal study almost but it led to us being able to have a different type of insight for our experiments so so yeah I think there's

(33:02) possibilities when you're that smaller B2B company because you kind of know so much about your customers it's amazing you know you can really get in there and go oh that's why they want this feature and that's where they're struggling to be able to convert and that sort of thing so it's so interesting yeah so

(33:18) interesting to to get to know them like that and then uh personalize their experience in that way experiment yeah um and also I'm also kind of um I've been thinking about the lifetime of the the customers let's say when it comes to a service when they are signed in and then there's this period of time when they are just

(33:45) uh you know experimenting with experimenting uh with the service and then they have to become uh the paid customer mhm so so how do you um calculate that time or kind of predict how long they need before actually convert and and become a paid customer um is is there is there a way to to know something like this I think

(34:13) well I guess it depends on what type of services and what um conversion events are important to you if I think over some clients I worked with there was a client I used to work with who were a bit like so they offered like a product management um product um so with them they would think about things like what's the number of

(34:34) active users and are they a paid active user are they an inactive you know like a non-paid active user that sort of thing and having those as their conversion points um if I think back over what we used to do at photobox although it's not B2B in the same way you have these like long conversion life cycles where you have to kind of measure

(34:52) and I think it's about collecting enough data so that you can say all right after about this amount of time about 80% of people who were first shown the product on say day one of the month by this number of days most of them will have converted or not so that's how long that life cycle should be should we make a

(35:12) tradeoff between having a a very trustworthy experience trustworthy test or uh should we lower the significance and then not miss that we can find in the testing I think um I think personally I prefer not to lower the confidence rate you're already introducing a false Discovery rate of like one in 10 when you have a 90%

(35:47) confidence level so I think um there's kind of two approaches that I would normally pick I would either propose slightly bigger ideas and I know we already covered that a little bit but kind of focusing on maybe doing fewer more high value tests and allowing them to just run for the amount of time that

(36:08) you need to especially if it's something really risky and kind of the second approach is thinking about when is a test not appropriate so are there times when instead of the test being the right thing to do is it actually better to do something smaller or different or or can you combine it with other kinds of

(36:28) research or testing that mean that actually you're not dependent on a single data point so instead of being focused on I put everything into this one AB test are there kind of six types of evidence you could collect at the same time including maybe an AB test where you prove that it's non- significant

(36:45) maybe because you don't have enough sample but you have other types of evidence that suggest that you could use those you could run that experiment so I think um personally I I think you either need need to make it a test or not a test you know it's either an experiment where you're going to get statistical rigor

(37:05) for um Roi purposes so if it's an experiment where you really really need the ROI to be really good um then and really clear and you can report it back to the board for example that you made a certain amount of money from an experiment then yes have that be that kind of more structured more rigorous

(37:22) experiment but if it was more of a sort of learning experiment expent where I'm trying to work out what to do and make sure something's not a bad change maybe you're less bothered by statistical significance and like you don't want to hold yourself to that same set set of rigor not that I'm saying not use the

(37:38) statistics that are available to you but just kind of know in the back of your mind like what's your shipping criteria what's your decision you're going to make off the back of the experiment is it I would have shipped this anyway so I'm just making sure or I don't know what customers like so I'm going to try a few different

(37:57) things and see what resonates and see which ones come back positive and negative or the experiment was nonsignificant and this was something I just needed to be non-significant there was no big change there was no you know I just needed to make sure it wasn't sort of um it was something I could didn't have any specific result either

(38:14) way or actually know I really need this to be a winner like I really need to know really clearly that this was a big winner and how much money it made me and that sort of thing and just I almost um so I have a section on here which is uh at the bottom here which is about what's your shipping criteria so

(38:32) what's your decision that you'll make off the back of the experiment and I asked product managers to specify before they run the experiment what they'll choose to do when the experiment finishes so that instead of that kind of decision paralysis when the experiment finishes we're all going oh did we

(38:47) decide to roll this out if if it was a winner uh or are we going to iterate it or you know some other decision that we haven't yet made sort of thing so I think it's a bit of a tricky one um but yeah just having that kind of alignment basically yeah and these are all great tips and it's so U refreshing to you

(39:09) know to know that there are so many different types of experiments that and different purposes to run them so that you can adjust with the situation and of course have the team learn about those and also implement the learnings um so this was a great interview Ellie thank you so much I think I learned a lot this

(39:32) time so I really want to thank you again and uh I hope that that you also had a great time but um yeah thank you very much and um also thanks RAM for creating the platform for both of us to be able to chat like this this is Romo Santiago from experiment Nation every week we share interviews with and Conference sessions by our

(39:57) favorite conversion rate optimizers from around the world so if you like this video smash that like button and consider subscribing

If you liked this post, sign up for Experiment Nation's newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK