Adventures in Experimentation: A deep-dive into CRO tools Experiment Nation

Home / Adventures in Experimentation / Adventures in Experimentation: A deep-dive into CRO tools

Podcast transcript

Experiment Nation 0:09
From Experiment Nation, time Rommil, and this is Adventures in Experimentation. Our panel of CRO professionals leverage their years of experience in UX and conversion copywriting, to field common CRO questions. If you are new to the field or even if you are a veteran, you’ll always learn something new.

Positive John 0:40
How’s it going, everyone? It’s positive. John here taking the mic on today’s Experiment Nation podcast. For the ones who don’t know me. I’m a data enthusiast in a long relationship with normal distributions. And currently head of growth experiments in iTech, where a coach teams on how to be strategic and mindful about their experiments to grow different products in our portfolio. A lot of fun, I must say. I’m talking with you from sunny Wroc?aw in Poland. And today we have a new team member joining our Adventures in Experimentation podcast my own Welcome to the adventure.

Marion Ravel 1:16
Hi, everyone. Hi, John. Thanks for listening. My name is Marion I have been working in CRO for the past six years specializing in ecommerce. I am working full time for SAS company called Sage. In my job, I focus on optimizing customer journey and personalized experience on ecommerce websites. I am passionate about what I do. And I’m here because I’m looking to learn more and share my knowledge with whoever is interested.

Positive John 1:51
Learning knowledge and sharing. This is all what we’re here for. All right, we selected three topics for today. And by the end of this episode episode, expected to confirm or add a couple extra free tools for your CRO toolbox. And to get exposure to a use case of balancing experiment portfolios. Today’s topics come from experimentation, community discussions, and some fresh work experiences that we thought it to be valuable to share with y’all. So let’s dive in. I’d like to start today’s talk with a short list of free tools available to CRO professionals. But let’s do a couple steps back for context. I always like to think on the distribution of effort and budget between people processes and tools. So how much importance you give to tools? How does that usually work for you? What’s your experience and recommendation for this split? So I feel like this is a relevant discussion, no matter the setting. You know, if you’re in the agency setup, or in the in-house team, there should be a clear allocation. So Marion, I’m sure a bit of your experience with us on how you think about this distribution between people, processes and tools when considering bringing new tools for your car. Oh, practice there?

Marion Ravel 3:21
Well, I would like to first to say that there is a very, very large number of tools available in the market, which is great, because we have a wide range of choice, which is also confusing, because you don’t know which one to go for. And I think that’s why this topic is very interesting. Yeah, the way I usually go about choosing a tool is to obviously benchmark the market very quickly and see online what’s available for what I’m looking for. For instance, if I want to record session, because I’m trying to see what the users are doing on my website, I will check for session recording tools, and read on a couple of blogs and books and online obviously. And then I’ll check for kind of price of it it’s free for how long or what’s in the free package, for instance. But what I tried to do is to choose the tool that will ensure my need so if it’s session recording session recording if I’m trying to understand what’s my form, how to how to improve a form for instance, I will I will try to look for specifically a tool for forms. And it’s not only on to improve my analysis, but it’s also to see what’s on the market and always try to find a better one or one that will be easier to implement, etc.

Positive John 5:04
I like what you what you mentioned the very beginning. Kind of a disclaimer. There’s so many tools available today. Right? Yeah, I remember in my like seeing like, if not mistaken, CXL has those maps of tools for recording session are tools for product analytics. And they met how crowded the scenario and the market is today. Back to the initial question here, just just so we covered this context, you know, whenever I’m thinking about an optimization team, being a cross functional, or maybe a single player in conversion optimization, I usually consider that okay, thinking about the tools is 10% of the process, you know, having the right people for solving the right business problems, accounts for 60% of like the importance of my conversion optimization program setup, if you will, and the processes itself with a comfort 30%. So I feel like this is a little bit of me remembering what Craig Sullivan had in this CRO agency masterclass. I don’t think it’s available on CXL anymore, but it was a blessing of the material. So he recommended something like 60/30/10 for this split of consider people processes and tools. I like to think about this analogy of, you know, if you don’t have the right people, it doesn’t matter having the best tool available, right. And the analogy to play with is if you don’t have good drivers, or if they don’t have driver licenses, there’s no point of giving them a Ferrari to play with, right? Maybe it’s not the best analogy, but still common sense. So Marion, how would you split like the importance of considering the tools in a conversion program? Would you say like, it’s 10% of your think time? It’s 20%? How much is that before we dive deeper into the specifics of the tools.

Marion Ravel 7:04
You’re totally, right. I mean, it’s, it’s not much of my thinking, actually, I because, and I think it’s a personal point of view, I really like to try new interface and see how they are also because I’m working with SAS. So I think it’s interesting to to, for me to check houses kind of to get the inscription, the monthly price, the free thing, etc. But I also, so I like to try new tool. But I think the actual thing of recruiting a new tool, it’s about 10% of my work. If it’s not less.

Positive John 7:43
I see recruiting a new tool, and like how you position this, because it’s it’s usually a hiring process, right, selecting the right tool, understanding our requirements behind for really delivering on the premise and what the team needs is quite a work. I agree with you, for us to deep dive on the tools. So Marion, you brought us a short list of free tools available for CRO professionals. You mentioned a lot of tools available there. So I’d like to understand you you mentioned you’ve been working with SAS. So are those tools that you’re going to recommend and talk a little bit more about more towards people working in agency setups, like in house? Is there any difference? You know, having this context, I feel it would help listeners to one better the options they have? So here we can get a little bit into the specifics of tools you recommend? And what were your experience? Okay, you were playing on SAS side on in house teams on agency? And what do you think about those tools? Let’s maybe go one by one there.

Marion Ravel 8:47
Okay. So usually, I think as in house CRO, so I very often don’t have the perspective of an agency, but most of the time, I also think as in house and very small ecommerce or website because I think CRO today is still very, not not not enough accessible for smaller company and small website, I think it’s something that is last on the list of what ecommerce website owner think of doing to improve their their website, at least in France. Yeah. This is my own point of view, because of, you know, knowing different people and working with different industry. So what I think about Hotjar, for instance, is that it’s it’s a great tool for small websites, or beginner ecommerce websites at the beginning, because it’s limited in terms of recording but not in terms of time. So basically, if you have smaller websites small vision recording to do or small Heat Map here and there, you can still use this tool in the long term, not having to register in another one in couple of months because your subscription is finished, I think so I think that’s something that can help you from the beginning in the long term. And even if you grow, then you can still use Hotjar, because you can get a more expensive plan and still keep using it. Zuko is a tool specialize in analyzing form. And it’s really great because it’s very easy to implement. And it’s free for I think, a month, and you have people who are there to help you totally for free, even on your free membership, on chat on call available with available within 15 minutes to help you implement it on your website, try to start to get results on your form and how to analyze the results, etc. So it’s like a free tool. with actual support. It’s amazing crew and a specialized on forums. So this one is really good, Crazy Egg. I haven’t tried, but I’ve heard a lot of good thing. I’m planning on trying it. And VWO which I discovered recently because I thought it was this amazing tool, which wasn’t accessible to everyone. Because of its price, it’s actually free for 30 days. So I’ve I’ve tried it recently. And I found it’s like very complete, because you have basically everything you need as a CRO. So it’s not a tool to start with. But once you’ve actually grown in your CRO strategy, and you said it at the beginning, you said people process and tool, I think if you don’t have a lot of people in your team at the beginning, and you only have a tool that one you need to go for because it can I think automatize a lot of things and is obviously not the same as having someone in your team, but maybe unload some work, you would have to, to do by yourself. So that’s very interesting tool that helps you recording heat map. But also, prioritization of AB test roadmap, actual AB test can be made on the platform, obviously, you don’t have to set for instance, the page where you want to start recording this tool is you haven’t you have just to install the tag and immediately start recording every single page and get a heat map and score map on every single page of your website can analyze form and helps you to create survey, images rate funnels, conversion funnels, also without having to reinstalled something else or add tagging or anything. Everything does it by itself. So it’s very all in one. The only thing that would miss i think if i was very, very picky with this tool is it doesn’t include internal search insight, that would have been amazing.

Positive John 13:21
Internal search. Yes. Interesting. Yeah, on the topic of internal search, I’m here thinking, because it’s nice that you bring an ecommerce or heavy commerce background. And on the flip side, I can complement with, you know, growth agency environment, how tools are managed in that side, but also cross functional teams for larger product, SaaS subscription business models. So we have like a wide variety of context in this conversation. I usually like to classify the tools in quantitative and qualitative. And mostly because of the the biased experience that I have with the Google right. So Google Analytics, Google Search Console, Google ads. I’ve been I’ve been using a lot of Google Trends lately for some market research as well. Google optimized Google segment. Those are free tools. And they’re free tools for a reason, right? Google wants you to understand a couple insights, page level insights, mostly. So you keep spending on Google ads, which is their bread and butter. Yeah. So those are those are free tools. And I love the connection and integration between them. All right. And as you mentioned, the search internal search insights is something that it’s one configuration away from Google Analytics. In your basic setup, you don’t even need to go Google Analytics 360, which is their paid plan. I feel like the biggest restriction with Google in the free formats, at least for quantitative is that once you start doing some more crazy segmentation you do have the problem of sampl data, which can, you know, annoy some analysts out there. But hey, it’s the price to pay. So on the quantitative side, I’m very, you know, prone Google set of software. On the qualitative side, I must agree with you Hotjar it’s generally my go to, the only thing is that Hotjar has been changing so much their monetization. One of the things that I really got sad in a way when Hotjar sunset, their form analytics, so I don’t know if you remember, but Hotjar had form analytics build in their software that they sunset, maybe like some six, eight months ago. And you mentioned about Zuko in this amazing support with basically analysts on the chat on the call to help you figure out insights. Yeah, so Zuko is definitely one that it’s getting on my list here.

Marion Ravel 15:53
Yeah. It’s very interesting what you said about Google Analytics. And it’s, you’re right, I mean, you have everything about internal search on Google Analytics. And it’s pretty precise.

Positive John 16:04
Yeah, I like I really like the tool. Like, even though right now, I’ve been using a little different set of tools. You know, I moved from Google Analytics to Adobe analytics, and I’m getting more in love with that tool to be honest, instead of Hotjar, using SessionCam, which delivers on the premise, I still just preferred the look and feel of Hotjar instead of Google Optimize, I’ve been using Sitespect. So those three tools, they’re paid tools for larger enterprises, but they deliver very well. I like the flexibility of Sitespect for setting up experiments, by the way. Yeah. Another thing that I had on my list here, it’s calculators, right, I feel like for people running experiments, you know, the AB test calculators is usually something that we use, I really like see CXL AB test calculator, I generally do my pre test planning using CXL calculators. And sometimes when they need to go like simplistic Bayesian, I do my test analysis using their Analysis tab, as well, you find this you know Googling CXL called AB test calculator, you find on the most updated version of the tool that they deliver six sell calculators, right, usually, like my bread and butter, I have used generally like the right inputs to start there. And it’s a free resource. It’s really something that I use all the time.

Marion Ravel 17:30
I’m just checking it at the moment. And it’s exactly the reason why I’m doing this kind of podcast, I had no idea that exists. And it’s amazing.

Positive John 17:40
So we’re both learning.

Marion Ravel 17:42
Exactly. No, usually I just do the calculation in my head, like very quickly, just because I know how many people are coming the page with the months. But it’s like, the very, very, very average average. And people are like, so when we will, when we have the results of the test. You know, in my company, I have like, obviously, a lot of people asking for us to do many tests. And sometimes they’re like, okay, we’ll we have the results by, I don’t know, q3 or whatever. And I’m like, well, considering we’re doing it on the form page, and there is that amount of people coming in it, and I’m gonna make the test really by two weeks. It’s just a hassel, you know, and like, giving some numbers and, and usually they like, oh, okay, so it’s gonna be like, say, 2nd April. And you’re like no no no no is not an exact date.

Positive John 18:37
Yeah, and this is usually your, your best guess. Yeah, because the most problem that I have, like, now I have the calculator in front of me. And you actually like the less input, like the set of inputs is your weekly traffic, either in terms of session or users, your baseline conversion rate, confidence level and statistical power, which by by design standard of the industry is 95, and 80. And then you put the number of variants, including control. So if you’re running an AB, you’re going to put two there. And then this calculator will run some Markov chains, I would not remember the proper statistics that they run behind. But that’s why you have this breakdown of Okay, the number of weeks running test and the optimal minimum detectable effect that you can actually get. And the problem here with this calculators that, you know, when you start an experiment, let’s see you’re running something on the homepage. How can you actually know what is the impact that your change will having that page because like, you know, the mathematics behind everything that we do in AB testing, is, you know, under the laws of statistics, and everything that we’re trying to do is basically to calculate the uncertainty right? And this guess game that you start your experiment with that? I feel like I’m going to have 5% impact. This is really trying your best to calculate the uncertainty of the best effort that you put in that variant? I feel like this is where I bring most of the statistical nerd inside of me, Marion.

Marion Ravel 20:11
No, no, it’s very interesting. You’re right.

Positive John 20:14
So moving for our second topic, I know we discussed mostly like free sources. You know, in tools here, what, as your operation grows, you can expect to start paying for some of these tools, right, such as Hotjar we mentioned, like the business plan for a proper integration with Google Optimize, which is a nice combo. Or maybe your you need to start paying for those tools, just to have more capacity on the analysis, as most tools will cap you at some point. So I wanted to understand from our experience, did you ever had this moment where you were trying to negotiate with clients for that are resistant for paying for a tool that you needed? I feel like this is more tailored towards listeners in the agency environment. But did you ever had this experience?

Marion Ravel 21:02
Yeah, all the time, all the time. Even internally, internally in my company, I see because usually they want to centralize spend and use, for instance, if they have Adobe, then all the tool from Adobe, or if they have Microsoft, and we are using a lot of Microsoft tools in at Sage. So then we want to have the whole tout have software they have, at some time, it’s just not what I need. And in time of CRO is some time it’s very precise tool that you need. And it’s not something that is already in the map of a huge software, SaaS companies such as Adobe, at least they were the last one.

Positive John 21:54
One of the things I wanted to share is like, when you are in this agency setup, I remember having this discussion of why a client should pay for it too. Instead of the agency being agency cost. And long story short, I just feel like this is relevant to share this one trick that I had to get clients to pay for their own tools and the conversation, we usually go this way. So me as the experiment owner comes to the client, so Okay, client, you need to start paying for this tier of the product. So we can do this one awesome thing that aligns with our strategy. And it’s only available in the pay tool, and the client answers, but I expect this to be in the agency costs. Why are you asking me to pay this? And the way I usually answer that wins the game. So well, we can pay for the tool. But let’s imagine this scenario, at one point, we expect you to build your own in house team, right and outgrow the services of our agency, which is just a natural cycle of agencies. Yeah. And if we pay for the tool, we own your data. And if you leave your data is still ours, how do you feel about that. So they’re never good, they never feel good about that. And they usually pay for to do so this is just a quick trick to use. If you’re an agency setup, and you’re having this friction between, you know, you needed to have a Hotjar business tier or something along those lines. A usabilityhub.com, that you’re just trying to run some usability tests, a survey monkey panel, and the clients view is, you know, generating friction, for getting that extra access, this might help you. So let’s imagine a scenario that we have all those tools running Marion, you know, we have a clear business objective in mind, a set of customer problems that we’re trying to prioritize, to solve, that will impact the growth model and add value to the bottom line of the business. So on the operations side, we usually have a set of initiatives right that boop, break down into testing hypothesis or one off corrections. And this usually makes our our backlog. So I wanted to know from you, Marion, I have you managed an experiment or one off corrections backlog before? And how did you manage to keep this backlog diversified? So the actions inside that backlog? were covering a larger portion of templates in your website, or different angles of testing? Did you think that diversification was necessary? I understand that some times we’re trying to solve a very specific user problem. We’re focused on the target is better. Yeah. But I wanted you to share this experience of managing a backlog and keeping it diversified with testing hypothesis and one off corrections, anything that you could share with us.

Marion Ravel 24:47
Okay, first of all, I think my backlog is not often very diversify because I usually go after one KPI at a time. So all my hypotheses are driven to try to improve that KPI, for instance, if for a quarter, or a year, I have to target to increase the number of free trial demands on my SaaS software, then I’ll have a number of hypotheses, which are all related to increases number of free trial. So sadly, for like, couple of weeks or months, that’s the only kind of hypothesis that I have. But even though I’m trying to solve this, or help this KPI to grow, I still ask myself, where I can get ideas. So usually I go on the web, and I think it’s I don’t know if it’s French, or it can be in English or not. But this is a website called Convertize smart persuasion is giving 250 AB test ideas. And they are categorized between principle like, do you call this psychological principle ecological principle, or you have another category, which is type of websites, or you can, you know, check, like, retail while you’re working on SaaS company just to get inspiration, and then type of page. So that’s another source of idea, and brand branding. I also tried to ask myself, what would happen in real life? So it’s very simple exercise. But I’m just trying to think, while looking for ideas. What if, at this stage, my customer was in front of a salesperson. So if I want to improve the free trial, for instance, I tried to imagine my customer getting into the store, trying to find an accounting software, which is what I’m sitting at Sage, and talking to a salesperson, what do they say? What is he looking for in the store window? Where does he go first in the in the store? Does he go to the person that and just imagining this scenario helped me to find ideas, because I know how to certainly I know how to help having a great experience, even though the guy is in his behind this computer, basically,

Positive John 27:26
Yeah, I love this exercise of what would happen in the real life, I’m honestly taking note of this. And adding to my knowledge base, we’re also going to make sure to add the links, Marion a just mentioned is 250 AB test suggestions. Let’s make sure we bring those links to our listeners in the show notes. But what I wanted to share and I feel like it goes aligned with what you’re saying is that just before I left brainly, we had as a team, like this realization, while refining the backlog, to allow work continuity for the next months for the team, that most of our tactics and testing hypothesis, we’re targeting, adding things on top of the current user experience for a small set of templates available in the website. And if I translate now thinking, what would happen in the real life, I can imagine like this student, right, bring these Education Technology giant coming to store, let’s say, and we’re adding so many things, well, this student is trying to walk around the shop and explore a little bit. And we’re just asking things and you know, adding to the experience. So it really makes sense, making this a good heuristic, what would happen in the real life. And on my end, it felt like we were putting most of our eggs in the same basket, you know, limiting our learning potential, and possibly reducing our chances to find wins elsewhere. Especially considering the traffic volume that brainly has we’re talking about 350 million monthly active users and growing, as I like to say, bring these the real deal of experimentation. What contributed to me getting triggered was once again, I mentioned before a mentioning again, this book, trustworthy online experiments. It’s really a must read for all experimentation folks out there. There’s one chapter in the book dedicated to research methodologies, right and explores their trade offs superficially. And this passage talks to me and here I’m just reading through one of the passages from the book. In general, we recommend always having a portfolio of ideas. Most should be investments in attempting to optimize near the current location, but a few radical ideas should be tried to see whether those jumps lead to a bigger hilll. So I would like to understand them. Marion, based on what you said, I understand that you have like this one single KPI. Yeah. But would you say that based on your activity based on these AB test ideas that you collect online, you would be able to balance your testing in your experiments in having like, Okay, I have a couple of them who are targeting optimizing templates. And I have a couple of them, which are like very big ideas, trying to generate innovation in the way we do things. How do you feel about that?

Marion Ravel 30:30
I think I’m gonna take again, my real life example. Because basically, when I imagine this customer in the real life world, trying to having the same experience, but in real life, when you do that, you can see emerging all the friction, that would happen on your website, because the customer doesn’t have this real life experience doesn’t have someone directly talking to him. And every time he has a question, you can just ask it and get answer. Or every time he wants to touch something, you can just rub it with his hand. So that’s, I think, solve the friction part, to me, at least part of them, and analysis. The tools are session recording everything we talked about during this meeting. And about the innovation part, it’s kind of the same exercise, but you have to think of everything, the salesperson in the real life cannot show to the customer and in front of him. Because there is also things which are better online, there is also customers that would have a much better experience online than they have in shop because there’s things they can see, I will take the example of Tesla, who is selling cars online, who knows how to do both. They know how to make the customer feel like is in Tesla showroom, and he’s transported in this luxury experience, or maybe innovation luxury experience, you can see the car they shiny, very nice picture, high resolution, moving, etc. But they also do something that the customer wouldn’t have if it was in the showroom, which is showing something that you cannot see if you’re in the showroom, which is for instance, bringing the car completely where you can see the inside of the car.

Positive John 32:28
Without the covers, yeah,

Marion Ravel 32:29
Without the cover. That’s something you can see online from your computer, you can actually inspect every aspect of the car and see where the airbags are. And the car is like completely dislocated. And this is something you can see from your computer, but you cannot see offline, for instance, and thinking about all these ideas that you could do, because you have your customer in front of his computer is making you think with outside the box and checking also website from others, checking other Industries and yours, like keeping your mind always looking for ideas.

Positive John 33:07
Yeah, I like how you position the Tesla example, Tesla is definitely an innovator in their practices in ecommerce. I feel like it’s an example that fits the bill here of what we’re talking about diversifying the the options, I’m pretty sure that at the very beginning of their ecommerce, they didn’t have all this innovative way of displaying information. And they have to come up with that. And I keep wondering, like, what were the methodologies that they had, because in a way, once again, they were innovators. It’s not like Volkswagen, or Mercedes, they had those experiences in their ecommerce is right, Tesla started selling cars online to start with. And one of the things that I’ve been trying to specialize, it’s like running experiments workshop, to basically bring this set of innovation or to basically diversify the portfolio. And when I think about diversification, I basically tried to classify the experiments that I have, and that I come up with my team across some extra dimensions. And I’ve been mostly following reforge is a portfolio spectrum framework, and it suited me and my team very well as a mental model to make sure that we’re diversifying as we can. I’m gonna link this in the show notes, but try to visualize, you know, our listeners, the spectrum as six different dimensions ranging from solving for optimization all the way to innovation. When an experiment is trying to simplify, enhance or reorder elements and messaging, we considered those low hanging fruits tests, and we’re basically trying to solve for optimization. Yeah, on the flip side, when an experiment is restructuring, adding or reinventing the user experience, so Marion just like mentioned, like, Tesla really reinvented the way that you see specifications of a car online. And those are tests are playing on the innovation attempts. And they’re considered like the big bets of your experiments. It is a very good exercise to go through your current testing backlog there and classify your tests under those dimensions to see how diversified you are. Again, back to the idea of workshops, do you have any any type of workshop that you run with your team members? for generating like ideas? Do you usually do it alone? Do you bring in your team? How does it work for you generating those ideas? I know that you mentioned a couple of resources. But is it more of a team play for you? Are you doing mostly alone, based on your current setting? In Sage?

Marion Ravel 35:46
Usually, we try to do workshop with people working in very different departments, for instance, would have people in my team so someone in development and UI, but also someone from sales who are on the phone all day long, with customers trying to close the deal, and customers support, because at the end of the day, the whole customer journey needs to be improved. And having all the feedback from post buying is extremely important. Because it helps bringing to the customer also visualize what’s going to happen after he purchased. Sometimes we forget that the customer needs to see himself after the purchase, see what’s going to happen. It’s stress not to know what’s next after he gave his credit card number. So after said is very important. Customer Support is very important. So usually we like this kind of team. And sometimes we bring someone from the product team and I try to to bring different people and share the first, for instance draft of of the landing page to the five second test to see what’s coming up from it. The issue with that is that we don’t have people outside of the industry outside of the brand thinking about it. So we all have these Sage mind, which is a bit annoying.

Positive John 37:16
I see. Yeah, I would like to echo actually what you mentioned about the importance of having customer support and sales team present in this ideation session. It’s something that we mentioned in our previous episode actually, on maintaining the feedback loops for ideas. And it’s something that came up in the talk with Eden that, you know, having the conversation with the sales team, and customer support is generally where we have like the best source of ideas of what to improve moving forward. It’s great that you bring that again, it just adds more confirmation on this kind of, you know, concept of talking with people who are talking with your end customers. Yeah, you mentioned something very similar to what I do, basically. So having this cross functional workshops, I’ve been basically doing in a two step framework that I learned from one of the senior product designers in brainly. She’s amazing running workshops. And I learned this, you know, two day workshop, which is somehow a summarized design sprint for running experiments where we cover a little bit of research a little bit of state of the art, a little bit of how competition is executing, we remember ourselves, What is the problem? What’s the customer problem that we’re trying to solve? We get ideas online. And on the second day, we basically put sketches on the paper or everything using a Miro board making very dynamic interactive, we vote more qualitative with with dots. And then it becomes work of the experiment owner to kind of summarize the ideas generated into that workshop into design requirements that will build your ABCD variants, and the process continues from there. One of the things that I heard in this experimentation deep dive specialization from reforge from Adam Grenier. He he was the one leading the course he’s ex-growth at Uber. And he brought this idea of alternative worlds workshop that really got me curious. We exercise in one session with him and some other team members. And it’s something along those lines. Like, we have this cross functional team. Let’s imagine we bring someone from sales, a developer, a copywriter, and every one of us pick a company. So let’s say Marion, you’re your apple and Amazon. The developer is Uber and the all start thinking like if I were Apple if i were, Amazon, if I were Uber. How would I choose to solve this problem that we’re all together trying to solve for and the amount of innovative ideas that come from this workshop. I remember like in this session that we were together, one person decided to be Disney and they had such an innovative and more creative solution for that particular problem that was very, you know, encouraging as a powerful methodology. Yeah, I feel like as CRO professionals were very good on suggesting optimizations and solving for the low hanging fruits. But it’s the more creative side, the big bets, the innovative ideas that it’s hard to come through for us. And this seems to be a powerful workshop as an exercise to get those ideas in the table. We’re going to link all those ideas and references in the show note, I’m also going to share a recent LinkedIn post that I’ve put live that continues this discussion of balancing experiment portfolios from a hedge fund manager perspective, mostly because I’ve been watching billions a little bit too much. But Marion, I feel like we have it. That’s another episode of adventures in experimentation.

This episode was edited by Jin Yang. She is the co founder of Mindset Masters. Mindset Masters works with professionals in all industries to encourage high school and university students to develop a growth mindset. If you are interested in speaking in one of their online workshops, please reach out at any time at contact.mindsetmasters@gmail.com.

Transcribed by https://otter.ai

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.