How to Maximize Hypotheses Testing ft. Eduardo Marconi Pinheiro Lima Experiment Nation

AI-Generated Summary

One of the most impactful lessons learned in the journey of conversion rate optimization (CRO) is the significance of the problem-solution mindset. It emphasizes the importance of always going back to the core problem you aim to solve. Even when dealing with a multitude of data and insights, viewing everything through the lens of the original problem ensures that your testing efforts remain purposeful and aligned. This mindset helps avoid getting sidetracked by various tangents, maintaining a clear focus on the ultimate goal of providing meaningful solutions.
One of the most significant lessons learned is the importance of hypothesis testing in the context of experimentation. Just like squeezing a fruit to extract its juice, the first test on a hypothesis often reveals sensitivity to customer behavior. If the results are notably good or bad, it indicates that something important has been discovered. However, if subsequent tests yield neutral or inconclusive results, it’s essential to discard that hypothesis and explore new ones. This iterative approach helps refine experimentation strategies and focus on what truly matters to drive meaningful outcomes.
The key point discussed is the importance of thoroughly analyzing and squeezing hypotheses for more insights and value. It’s emphasized that after running a test, you should always ask, ‘What’s next?’ and not just move on to the next idea. This involves deep diving into the results, looking at different segments and variables, and trying to understand why something worked or didn’t work. The example given involves optimizing filter options for an internal search page, where the initial test showed positive results, but a deeper analysis revealed that the impact varied by product category. This led to refining the filters by category, improving conversion, and sales.

Video

Audio

AI-Generated Transcript

Eduardo Marconi Pinheiro Lima 0:01

So, when you do a first test, and you get a really good result, or a really bad result as well, it means you hit something that’s sensitive to the customer says. And then from that point on, you should focus on that, and try to get the best out of it. And so you don’t have any more results. When the results are similar to neutral or nothing. Then it’s really that I bought this and get a new one.

Anthony Morgan 0:39

Hey, experiment nation. Anthony here from IE NaVi. I will be your podcast host today. I’m excited to be talking to Eduardo, from myglue. As we dive into many things, from his experience, I’ll let him introduce himself because he’ll do it much better than I will. So I thought I’d tell us about yourself.

Eduardo Marconi Pinheiro Lima 0:59

Okay, thank you very much, Anthony for the invite. For the conversation, very pleased to be here. It’s my first time good experimentation. Well, my name is Eduardo Marconi. I am a CRO fracture nude since 2011. I did a bit of everything inside the area. So I worked in an agency. It worked in house for a big company in consulting consultancy as well. And then on 2019, I decided to open my own agency. And then I assumed this agency for two years was called the journey. And then it was acquired by Grupo Maga Lu, which is a big retailer in Brazil. And then, from then on, now I am the CRO director for the whole group. And my team’s experimentation teams that I work with, attend all the brands within the group. Yeah.

Anthony Morgan 2:05

So how did you get into CRO? Like, I know you started in marketing, right? So how did you actually get into experimenting ncrf.

Eduardo Marconi Pinheiro Lima 2:14

The funny story was by chance, like, I never heard of it before. I was an illustrator where worked, was working in London, with a bit of everything over there in agencies. And then when I decided to move back to the zoo, he got to notice for a startup, it was like just a project that was beginning at the time. They were in contact with some tools in Israel, and bringing those tools overs to resume. And my job was to give them methodology that we could work on using those tools. And in some of those tools was an A B testing tool, and a heat map tool. So we started with this, both tools offer me to clients, a service of trying to improve their conversion rate was not even called CRO at the time, had few articles, few documentation about the subject. And then what we found out that the clients that you were talking to were very interested in and knowing more to know more about the the consumers and the users that and that’s how it all started. So we was what we used to say he was he was like looks like he opening a lot of like areas because nobody haven’t haven’t heard of it before. And then but yeah, was was quite interstimulus was good times. They’re trying to make people to understand what you were doing when we even nurse didn’t have a clear idea what the service was

Anthony Morgan 4:03

yet. Yeah, I feel like that’s how CRL in general, just kind of a lot of people can have have a similar experience to that. There’s no like educational background. Especially when you look back 10 plus years ago, like CSL didn’t exist, at least as far as I know, at least not to the scale that it’s at now. And so a lot of it was kind of stumbling in there and then learning as you go, which I think is really cool. Because we’ve seen a lot of people get to the same place in terms of their thought pattern when it comes to CRO we’ve also seen like other differing beliefs on how to approach things. And so it’s really interesting how, how CRO and how that’s changed and shaped over the years. One question I did have was, you might know this, but I’ve noticed that there’s a lot of CRM Oh, experts and agencies in Brazil, it seems to be kind of like a hot industry, I guess it’s in Brazil. And I don’t see that in like other South American countries. Do you know why that it’s

Eduardo Marconi Pinheiro Lima 5:17

not really, to be honest, but like, what I see is that a lot of companies, they actually, they look for to understand where the consumer is, and because price is so big, and you have very different barriers, depending on the region that they are. And then culture as well differences among the population may be, this is something that interests them, especially to grow in specific areas. So what I found over these years is that, yes, over here, the community, the resident community is growing, I think we are hunger for knowledge. Even having a lot of people entering the area, we shouldn’t have here in Brazil, a lot of talks, summits or workshops about CRO, so it’s something that is new. And we have to look up for knowledge for references abroad. Some other reason as well, that maybe we are seeing a lot of people from Brazil coming in to international events or like to signing up for content, and so on, because we lack of this knowledge internally here, over here. And even. And this was something that when we opened the agency was something that we missed a lot, because we don’t have we didn’t have people that knew the area, and also that was interested in doing so. And but even being like one of the pioneers here, I think we didn’t do a good job of trying to, I’m trying to build, like Charlie’s to show like something that see XL XL doing that, like in terms of building content, like opening up like a community talking and so things were always very isolated over here. Nowadays, I started to see some people like already discussing it some groups. So yeah, I’m very happy from where we are going to like I think I can see, like, improvement, a huge improvement on that. But I think we are still, we still miss a little bit of that, you know, like to be closer to the bigger car or international community.

Anthony Morgan 7:56

Yeah, and we, I would say that it’s very much it’s very similar in the US, and just in general in the CRO world. There is these pockets, where it’s very similar the way that they think about CRO so you have that like CRO that CSL pocket where, you know, there’s varying approaches, but it’s very similar the way that they think about CRO what I would call like, true CRO, but then we have like these other pockets of full service agencies or like sometimes even a CRO agency, where they say they do CRO but it’s not really CRO like it lacks the research aspect. It’s mostly just kind of like throwing up ideas and guessing we see that a lot. There’s like this mainstream CRO and then there’s the CSL version of CRO that at least in the e Commerce Industry, and Shopify, where I spend a lot of time there’s a there’s a big difference between the way the merchant talks about CRO and like CSL talks about CRO so it’s really interesting to see something that’s really new, across the board, no matter where you’re at, it’s it’s a, it’s still fairly new. It’s, there’s no like, it’s not like, you know, other you know, marketing. You can go in and you can you can actually get like education and college you can dive into those areas and actually get, you know, university education, you can’t get that with conversion optimization. So it’s so new, like experimentation. All of that is still so very young. It’s gonna be interesting to see how it changes over time. And if that like mainstream that Some ecommerce space starts to shift as it does they become more familiar with the CSL and, and the true CRO approach that is very time consuming, like research is time consuming, it’s a lot easier to throw up ideas that that don’t come out of research that are just kind of like guessing, than it is to spend time doing quantitative and qualitative analysis to find problems, and then, you know, identify solutions through, you know, through more research. And so that’s, that’s one thing that that I’m seeing here in the US, I’m sure you’re probably seeing that in Brazil is like these different pockets that are forming. Because there’s not like a, because it’s so young. And so many people are kind of learning it on their own isolated, it creates these different approaches, which is both good. There’s positives and negatives to it.

Eduardo Marconi Pinheiro Lima 11:02

Yeah, exactly. I think it’s, it’s a process, right, I think we’re at the beginning of the area. I don’t know path. So it’s, I think it’s, it’s common to have a lot of people saying different things, if different levels of knowledge, until we get to a point that people understand what it is about the service, the core of it, and then will be easier for people to to get the knowledge like people to hire as well, the service as well, to understand what they’re very high. I think it changed a lot over days from where we started from now, it’s completely different over in resume. And when we started, we we did didn’t know even what an AP test was, what a heatmap was. So everything was the nowadays, you have to explain in details, what you’re going to do, how you’re going to do your research, what sort of analysis you do, what kind of tools that you need to use. So the questions got deeper, which is good. I think it’s like, it’s awesome for the area, that the the who hires has, like, the knowledge to discuss the service makes it much easier, competitive, increase competition increase, right. But that’s also good, because it means that people are finding it important to, to have the service and, and to discuss their

Anthony Morgan 12:41

right, as you’ve been a part of the learning, right, kind of learning on the job about CRO and like getting, you know, familiarized with those tools and learning how to do them without like a formal education. When you look back? What are some of those, like, biggest mistakes that you made? Or like the biggest mistake that you made where you look back? And you’re like, I can’t believe like, that’s how I thought I should approach it or that, you know, I thought that was true. We’re now you’re like, completely on the opposite end? Is there anything in particular that you can recall?

Eduardo Marconi Pinheiro Lima 13:20

Yeah, we have some funny stories about that. Like, I think the first one was when we we signed the first two, to try to sell over clients. It was something completely new. We understood it we we signed up from a company from Israel, so has a lot of language by yours as well on the discussions. And the way we understood the B testing tool was something that you could personalize.

Rommil Santiago 13:48

This is Rommil Santiago from experiment nation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing it helps us a bunch now back to the episode

Eduardo Marconi Pinheiro Lima 14:02

product, my brother 24/7, you will check like every hour, what was the sales curve of that product, and you could swap it using the two. And then we actually sold like a contract a huge contract here for a big company on that way, like on that matter, like of the contract. And of course we couldn’t deliver that because it’s impossible to do that manually have a lot of people just changing products and evaluating like one hour curves of sales to find the results. So that was a huge mistake. We had to review contract and everything like was was like a mess. But in the end was a big learning of like what was the well, how we could use the tool itself. And I think the other one that at the beginning was very high people didn’t understand as well the two And they asked us to do like bigger changes on websites. So when, when it says over here, ecommerce has been responsive, like to have like, mobile sites apart from a desktop these seven, eight years ago, they the most of the platforms, ecommerce platforms didn’t have a mobile site version of it. And the US clients was asking us to build this inside the EB testing tools, because we could change out the front end. And we actually did a few of those that as well consumed a lot of time, a lot of energy with like, very little results, in terms of steps was just like, for them was an opportunity to do a short cuts in terms of solving a bigger problem that they had. That was a platform being like mobile. So that was another big mistake that if it will go back, I’ll definitely say, Don’t go that you’d have to wait. Because we lost a lot of like, good sleep hours and stress to try to solve that decider up tested to which is definitely not the purpose of the tool.

Anthony Morgan 16:23

Yeah, definitely. What, what’s been the most like impactful thing that you’ve learned in your journey?

Eduardo Marconi Pinheiro Lima 16:33

I think is the methodology, like the problem solution, mindsets, of things. I think that’s very interesting, applies for everything in life, to be honest. But I never thought on that way before entering the CRO where, because and because even now that we have a lot of data, we have a lot of information, and things. We always have to go back to the problem to the origin of everything that we are doing, and to build the best solutions. And I think that this was the most important thing that over the years, we learned, and it’s impressive, how much if the more purpose you have, like, on what you’re doing, like what problem you want to solve, like the premises of these, and like the idea behind it, the more clear they are. Clear they are and then it’s impressive how the arguments to the decision, like, oh, that’s the test that we need to run. And like the results afterwards, closer to get like good results. Like and learnings. It seems like, that’s the key lesson there learned over these.

Anthony Morgan 18:04

Yep. So always go back to the problem. Exactly. Yeah. Yeah, it’s really easy, especially as you do more research to get caught up. And all of the different things that you’re learning and insights from that. But viewing that through the lens of the problem is the way that you can make the most impact from a testing standpoint,

Eduardo Marconi Pinheiro Lima 18:29

right? Exactly. And when you work for big companies, you have to talk to a lot of people to publish something, right? carillonneur has specialized areas. So if we’re gonna talk about shipping, do a B testing for shipping, you go through, you understand what they’re doing, like, what’s the core of the what to do, but then it has to also talk to the design team, you have to talk with the data team, you have to talk with a lot of people branding, sometimes like side, like the copywriting and so on. So the, it also gives you as well, when it’s time to discuss ideas, new data, data that make you change the path of the things that you are doing. So you don’t have like a core like a problem and a purpose of what you’re gonna do. It’s very easy to divert from your original idea and create something completely different, which is not solving any problem for the final. So yeah, that’s something that you have to be careful because it’s very easy to get misled in terms

Anthony Morgan 19:38

of yeah, oh yeah, you can go down so many different tangents and, and, and get inundated with a bunch of different priorities or problems that aren’t even might not even be related to the core problem that you’re trying to solve are your most important priority. already at that given moment, and it can be so easy to then get spread thin across a bunch of things and your research and the value you’re providing ultimately, is as limited, right, because you can only go so deep when you’ve got a bunch of different tangents that are filling up your, your insights backlog or your mind ultimately, that’s so true. It’s that’s, that’s something that I’ve learned over over the years to is, is making sure that you go back to the problem that you’re focusing in on that particular problem and not getting caught up in these other problems or shiny objects that come in. Because it’s just far too easy to do that. So having that priority, that problem that you’re solving is so key. So I want to dive into that glue and some of the work that you do there. So if you could introduce myglue and familiarize them with with kind of who Magu is. And then we can dive in to like McAleese service offering when it comes to trade marketing CRO? And learn a bit a little bit more about that. Yeah,

Eduardo Marconi Pinheiro Lima 21:26

sure. So my god Lu is, is a retailer company has been over 60 years now. Old, it has like woman, more than 1000 and some was 1400 stores all over Brazil. And then it’s the second biggest ecommerce here in the in Brazil. So it’s local, we don’t, we are not an international company we are focused on in Brazil. It started as an electronics home appliance store, and moved now. And we are one of the biggest marketplaces in Brazil. So in terms of companies that are similar, say like, of course, the most famous ecommerce, Amazon, it’s very similar offer that we have there. But Maga Lu also has like a bigger, a very big purpose. I think that they weren’t they always say that they wanted to give access to all what is a privilege to few like he in Brazil, so like how they can offer things, how they can work throughout the whole system ecosystem of the of retailers, sellers clients. And then, which makes like, also accompany for that very big purpose around over here in there, too. And also, one other thing that’s important say that we have like our mothership that we call it, that’s Margalo that’s the biggest brand that’s most famous one. But we also have inside the group, and other brands that are specialized and leaders inside their area. So for instance, we have like naturals that’s the most like the biggest sports ecommerce that has in Brazil, Africa, which is like the same for beauty. And also such with Well, it’s the same for books. And it’s actually which is like a fashion brand as well, over here, like sigh website that sells all about fashion, like a lot of different brands for multi brand store. So we have like the general, like, sells everything. And also the specific websites that are to specific categories, as well, which makes very interesting, this hero job here because you have both words that you can work with internally. And then when Junu Oh, that was my company was acquired. We now are we don’t sell to the markets anymore. We are internal internals there we are in Hungary house like area that work for hope for the audience that’s inside the Mughal group. At that moment, they didn’t have like a very structured experimentation program. They had like some initiatives inside the brands, but was not something spreads throughout the whole group. And then that’s our plan. That’s our strategy here now to build like a program to all the brands all the companies that we have there. And then about trade CRO This is one of the services that we do. We we work with different areas internally. And we have one specific part of our department here. That is the only one that we sell to them. Harkins, that’s their what we call CRO for trade marketing, which is, instead of us being our first place inside our sights and doing tests, we offer to industries to come to Marilou. And then build a experimentation program internally inside magotteaux. So it’s like a third party, being able to do experiments inside our brands. And the idea is that we play that by category. So each department inside the brands, they can have like one captain, which, which is a brand, that will be co responsible, together responsible, together reverse to build an experimentation roadmap to the brand, or the category itself. So depending on the, the program, for instance, like if you have like a type of product, that we don’t sell a lot in our stores, they could hire that to improve the sales of the whole department, from then on to build like a brand’s relation with the users. So it’s something that can be can can be done as well. And then there idea so this captain, they can work for the awareness of the category or the performance of the brand, it’s up to them. And they can do both. And sometimes they do both, like, they work in different strategies internally there to build up their, their brand brands, market share, and so on. This captain, they can define the roadmap together with us in terms of that so and our responsibility is to be like a branch of them being like an outsider besides the model, but also discussing the strategies that they’re doing internally with the Mugello teams as well. So I can nutro area does attend both worlds, that definitely sometimes they collide in terms of them because of different interests. But our plan is to build a win win strategy for the brand, to increase their market share their sales, and us to increase our overall sales, the department share internally as well. So that’s that’s the CIO trade offer, like, up right. So, but to do the experiments, we go back to the problem. So we have to do research as well, for the spread, right? Again, today, what we are building to them is like a half full CRO program, where we will run the researchers for them. So that can be like surveys, they can be like just like pep studies can do can be a lot of things that they will user testing of their pace, we understand how the brand is seen by Maga users, clients, and their opinions, their navigational data. And then from their own, we build strategies for them. And that strategies could be to launch new products, increase market share, or maybe to improve their content or reduce costs. So we had some examples of flat industries that for instance, they wanted us, they wanted to know because they have to build, like six images for their product page. And to build that images, they spend a lot of money in studios. So like, but does it make sense to have six photos? Maybe we for you, they’ll have the same result. Maybe they need 10 images? So what’s the right amount of images that you need? To make more sense in terms of the product. And that’s something that we can help them with. And then inevitably, those costs alone. You’re spending too much because you don’t need five, six, you can do the same job with three. And maybe that’s okay for them. So that’s something that we have seen a lot as well, which is quite interesting because something that we were not used to analyze and to and to define as a priority in terms of a b testing for us.

Anthony Morgan 29:56

So obviously with Magnolia that’s a very unique situation where you’re working with brands that are that are selling through your site to improve their product listings and learn more about their consumers, and ultimately the branch of their consumers that are shopping on Mad glue. But in other contexts, like for other brands that maybe aren’t selling through Magu, or CRO agencies that, that don’t have this type of service. What are what are some things that they can extract from this process, or, you know, some ways that they could utilize some of the research that you do, or insight gathering that you’re able to do that they could then utilize it and, you know, with their brand, or with the brands that they work with?

Eduardo Marconi Pinheiro Lima 30:54

I think it’s important to stress that like this program, the software didn’t start inside Mughal we I was actually offering that when I had my agency before. So I would go to industries, and knock on their doors and look. How much data do you have a fair user inside the point of purchase? Inside the E commerce? I know that you have data from your ecommerce, but what about other places that sell them. And I got surprised that they have almost no data, they don’t have any information. Unless what they receive from this retailers, which usually is not much because the retailers don’t have like big teams to still do the service for the industry internally. So I came up with this idea to build this program. So why don’t we go together and marketing to the retailers and offer them a service that I would play the neutral as like an outside company that have like, not like specific interest in terms of getting to know their data in total things. And I will do the middle manager. So like, I will do the roadmap, do the tests, and then the company, the retailer, could give me access the data. And I don’t have a contract with them, that I only share whatever they let me share in terms of them. So that was very interesting. And he actually worked a lot, I found out that the industry has helped, that helped me to get into a lot of retailers, because of the industry, they already had the contracts, they already have the the relationship with the with the companies, and they were open to do that kind of service. So that was interesting. So for agencies, I think it’s an offering, it’s a different type of offering that can be done for in house departments for CRO is a different point of view. Because when you were an in house, your your your main responsibility is about the platform itself as a CRO, so you were you try to understand how to improve the performance of the platform, sir, looking for navigational paths for content, maybe buttons, maybe like the structure of the pages, and so on. But then, especially on big retailers as mega Lou or other companies that to sell everything, right? It’s not they’re not specifically to one product. You have to sell a lot of things. So you don’t get deep into inside like away a product based perspective. So and that helps a lot because when the industry comes, they’re always passionate about their products. They love what they sell, they come and they know a lot about their products, their features, their benefits, the best angles that you should expose the products and so on. But as as retailers, we don’t look as much as one specific brand or that product. So at the end of the day, when we started to learn those different perspectives, it gave us a completely different overview of things that we could do inside the product page inside our organic search page, you know, so that’s very the sheet and it’s a different kind of AV testing that we do is different kinds of roadmap.

Anthony Morgan 35:00

Yeah, that’s very cool. That’s honestly, that genius idea that you had there to, to take that into to partner those brands with the retailers that they’re selling on, and ultimately work with them to sell more on those retailers. So that’s something that I haven’t seen here in the US at probably exists. And maybe I’m not familiar with it. But really cool concept there, I think, quite interesting. So I’d assume you then at some point in time connected with Matt glue, started doing that with Magu. And then that year, it was like, let’s bring this in house. And that’s kind of how that happened.

Eduardo Marconi Pinheiro Lima 35:49

For Margo itself was not was not the case, I was talking with some companies at the time inside the strain area, but was a different story in terms of like the the acquisition and everything. But it gives you opportunity to get to know a lot of companies, by the industry perspective that that was nice. And that one of the companies from McElroy group, that’s Africa is a beauty company like the beauty category. We were working with an industry from that beauty category, that one of their biggest goals was to do CRO for this specific brand. And handled when I came to Maga Lu. Now of course that makes it easier to contact to talk and so on. But, yeah, it’s it’s an unpredictable opportunity in terms of sets to get to mix both forwards. And whenever you are doing a good job that could happen as well. Maybe sometimes the retailer says, Oh, that’s nice. What you’re doing here for the industry is quite nice. Don’t you want to do it for us as well, I do want to help our team to prove our our strategy scheme. So.

Anthony Morgan 37:17

Yep. Yeah, that makes sense. Yeah, so one of the things that I think is really interested about your experience is the fact that you’ve worked in an agency, you worked as an as a consultant, you ran your own agency, now you’re in house Magalie working with, with multiple brands at a very large company. And so that’s your career press. Really cool. There’s there’s a lot of different roles that

you filled, which I think is there’s a lot of insight that we could pull from that.

In that, you know, both working now in house and as an agency owner, what are some of the biggest lessons you’ve learned? through that?

Eduardo Marconi Pinheiro Lima 38:15

I like to ask a question, because I think it’s important. Can you give me an example of something you did? For the first time in your life? There are two axles on it,

Anthony Morgan 38:27

that I was excellent on it. Not a whole lot.

Eduardo Marconi Pinheiro Lima 38:35

I asked this question to all my prospects clients to everyone. I think the only answer that I felt okay, that’s, that’s fair, was our children, right? Like the children the first time you do them? You accept them the first time? Yeah. But apart from that, it’s very rare for you to excel in something when you do it for the first time. So why do you think we seek the same for our hypothesis? I, I got a view, after all these years, that hypotheses are like fruits while you want to make a juice. So you squeeze the fruit for the first time. Depending on how much juice you get out of the fruit. You keep squeezing until you get to a point that’s no margins on the fruit, then you throw it away. And again, a new fruit. For me, I bonuses have the same principle. So when you do a first test, and you get a really good result, or a really bad result as well, it means you hit something that’s sensitive to the customer says to do it, and then from that point on, you should focus on that and try to get the best out of it. And so you don’t have any more results, when the results are similar to neutral or nothing, then it’s really that I bought this and get a new one. So it took me a long time to understand that in terms of it, and to sell that to my customers as well when I was an agency, because it’s very easy for you to build like a huge roadmap with a lot of things on it, and you don’t open space, to retest it, to go deeper, to understand things. And I did some analysis later on, to try to understand where were my biggest results, my bullets, silver bullets like results, right. And in the end of the day, they were never from their version one. Often hypotheses, they were from the version two, version three, version four sometimes, right? Just that’s the mindset. That’s the incremental logic of it. And you learn from your tests, I think more important than getting like a result is to get the learnings from it. And knowing what to do if that learnings and have room to apply the Learns into the to the platform again, like to roll out again, let’s for the users, for you to get to know whether they were actually true or not in terms of that. So I think at the end of the day, that was the biggest learning that I have throughout the whole process here. Of, of not giving up too easily. On an idea.

Anthony Morgan 41:47

Yep. Yep. That’s good. So hypothesis is like a fruit squeezing, too, you can’t squeeze it no more. No, that’s really good. I think it’s because it’s so easy to Oh, you got a positive way. And let’s move on to the next hypothesis, or this one was really negative, let’s scrap that obviously, we had a terrible idea. And instead of pressing through, and especially with like, bad ideas, bad ideas can be so valuable, because you can you can learn so much you can you can identify something that you added that clearly was detrimental to the user experience, clearly, you know, whatever you did, whether it’s removing or adding something, or changing something that change was was super impactful. And so I guess one of my follow up questions would be when you’re looking at squeezing hypothesis after version one, after you run that test, whether it’s a positive, and it’s statistically significant winner or loser.

What do you do to extract more value? Like, if you’ve got a winner? What can that version to look like? How do you get to version two? How do you get to version three?

Eduardo Marconi Pinheiro Lima 43:04

Yeah, we want to try to do is like deep dive the results from the version one to try to understand. Not exactly the real number of it, like, but let’s try to get some ideas of what might happen in terms of that sometimes, if you see the right number, you don’t have like statistic values to it. Because when you break down the numbers, it starts to get smaller, and it’s hard for you to get in. But that helps you get around at some point of views. So why, why that maybe some times an experiment worked really well, for a specific audience, and didn’t work for another one. Right? So what could have happened there. And then we go back sometimes, to the hypothesis to the to the leak, where the test was developed. Try to experiment it, try to use it like the test again. Think about it the process, maybe sometimes even do the research on top of it say okay, that’s the idea what other research that we can do. What can we ask the client, but but keep poking like that idea, like in terms of what what’s next. I always tried to do that to my team. They said okay, we can go, we can finish an AP test, I can analysis without a what’s next. So that’s and then in we will work from that. So and then they this is something that we need to do, everyone needs to do at the end of the tests. And then when we go back to it, and we we put them back on our roadmap. To decide then we will decide if we will prioritize the version two of the Let’s experiment over another hypothesis, depending on the audience depend on how we grow, how small was the audience that we found the what next, or, or not like to prioritize, but it’s like a mix, we don’t have like one specific way of going to vary it from version one to version two. But the idea has spent time on trying to gather all the information you have. And to do this provocation like to go that slow. Look, there’s something else that needs to be done. Right? Search for references, do another research or deep dive numbers, but try to understand what’s going on because it will never work for everyone always will have like a specific group that could have like a negative impact in terms of the project. So the idea is try to understand that, right?

Anthony Morgan 46:05

Yeah, no, that makes sense. And so going from version one to version two, and squeezing more out of that hypothesis, you’d say, dive into segments, new versus returning, look at channels, what was there any ones that performed way better, or way worse, and then trying to understand why that happened, which can oftentimes draw you back to research and other insights that you gathered, or maybe you have to, to restart the research process or, you know, gather some more qualitative data through through what you’re kind of what you’ve learned in this test. That’s, that’s, that definitely makes sense. We, I have an example, which I’d love to hear an example to, on your end. You have any examples that you can think of where you had a test or hypothesis, you ran a test was a clear winner or as a clear user, loser? And then you squeeze that hypothesis for more and had a version two or version three? Is there any particular examples that you can think of for that?

Eduardo Marconi Pinheiro Lima 47:30

Yeah, we’ve, for instance, we had like, imagine that we have like, an internal search page result that like a product listing. And then what we wanted to do is like we didn’t have sorry, going back, we had the research page. If the user wanted to filter, the options, they had to click in a filter button, and open all the options. But depending on the category, you have a lot of filters, different types of filters. And then we understood that some of the shooters were the most clicked ones for each one of the categories. So what inside was to move from inside the filter area the the filter that was used the most to be like a shortcut on the top of the product lists. So we easier for them to filter. Without going throughout all the steps of clicking, moving, choosing rolling back and down, up and down, and, and so on. So we did this, what we call like was a quick filter, inside the search bar over on the top of the list. So what we did is we test that, using brands, for instance, for all the other products or the other base, because when it’s good in general logo, the brands were the most clicked filter for that specific brand that you were working segment of a mugger. Okay, so we do that. The total results, it was good. We could see improvement on that. But then what we did was okay, but did it work the same for all the departments internally. And then when we started to deep dive the numbers, we checked that for the our most selling products over there. It worked really well. And that’s why the general result was good. But when we look into the longtail The problem is we don’t sell that much depending on the category. The brand didn’t have like a huge effects on the users. So We had a few clicks, we had like, we lost, we had less sales in terms of the total that we saw the categories and so on. So deep dive in that we could understand that we could generate like an increase in sales by generalizing for the brands. But when we deep dive, we discovered out that for some categories, we could do, we could use other filters to be on that position. Like I don’t know sighs like or maybe I wait are high depending on the product. So there are other type of filters that work best. And then we could increase our conversion and our sales by splitting in different different filters by category, instead of making like a general choice on that. So that was an example that in the end of the day, we could just implement it. But study, we found out that we could do more, and then improve even more than ever results,

Anthony Morgan 51:16

you broke up your search the performance of users on your search results by the category or product they’re purchasing. Right. And when you when you did that, you were able to see that brands, the brand filter option didn’t perform as well on some of the other categories, right. And so the new year, we’ll take it a step further and, and squeeze more out of that hypothesis. I love that concept. I think it’s huge, like being able to identify what’s next at the end of a test is, is it’s an acquired skill that only comes in time. You just have to have loads of tests under your belt to even be able to identify what comes next. And there’s, there’s loads of research that’s involved underneath that too, to extract more value out of that hypothesis. So I really love that concept. And I noted it here hypothesis is like fruit. Gonna go to steal that one. I’ll make sure every single time everybody knows that that came from from you. Anything else that that you wanted to kind of highlight about your different roles? When you transition from an agency owner to working in house with Matt glue? Was there anything that looking back, you would have done differently as an agency owner? Now that you’re in house? Like? Has there been things like that, that agency owners or those working at an agency could couldn’t learn from the different roles that you’ve been in? And your experience, both in house and as an agency owner?

Eduardo Marconi Pinheiro Lima 53:06

Yeah, yeah, I think I think that’s something that for me, like, as I love to be a question you about cry like, areas. For me the the most interesting projects were the ones that I had, the more data involved on it, and the data from different levels inside the company strategic data. So as an agency owner, sometimes when you you sign a contract, you don’t have access to a lot of data for the code, maybe they’ll give you like the analytics tool, maybe like a heat map to, but it’s very hard for them to give you access to their bottom line results, for instance, like or even, like, the impact that it had on logistics on costs. And then and then on the I don’t know, payment methods analysis, where they can like lose money, depending on which payment methods that users are using the most. So there are a lot of other variables. When you when you are in a retailer for instance, that like as an agency, you don’t have access to it and and that make you sometimes blind in terms of making the better decisions. So as my advice for for an agency that like is to build this relationship with the sea level or directors or the executive level that you are discussing with and trying to prove them the importance of having this access to the stores. That your goal is not To share that to the market or so on, definitely not, but to improve the decision making. And then, because of that, sometimes because we lacked this, this deep analysis we sometimes don’t understand why that we did a test, it got the good results, and why the retailer sometimes is not implementing it. What happens there, what’s going on in the middle? And then, and it’s not always clear. I love this something local hear if you feel the same in us. But he resumed, I had a lot of this feeling that sometimes like, Oh, my God, what to like, Why? Why the results we have, but we don’t understand the bigger picture of it. Because because you don’t have access to that information. And that makes the whole difference. When you are internal in house, and you have access to it, and you start to connect all the different tools, you do your tests, and get the results and finding out exactly what might be the bottlenecks throughout the whole process from the consumer to bite to delivers at his house. It’s something completely different. So Right.

Anthony Morgan 56:26

Yeah, no, that’s, that’s really good advice. It’s so true that, you know, in running e Nabi, we get limited picture on that ecommerce merchant. And, you know, we get access to their analytics, and we build out some infrastructure for them. So we can grab qualitative data to but we don’t have all of the different data that they have. And oftentimes, the merchant doesn’t even have all the data that they need. And so having that full picture of data, though, can help you better prioritize test ideas that are prioritized hypothesis, better prioritize those insights, where you’re, where you’re ultimately focusing the problems that you see, makes it way better for you to truly be an extension of that company’s team, and, and to provide them the most value when you can have that full picture. Otherwise, you’re just limited into your area. And so that’s huge. That’s, that’s a really good advice there.

Eduardo Marconi Pinheiro Lima 57:40

Exactly. Because I still feel that CRO is an operational area. Yeah, this over here, Brazil, treated a lot as like an operation, not not a strategic area that you could leverage your results massively like in terms of like, look at that, let’s put the biggest ideas of the company to test and not the small ones. You know, like, sometimes what I see a lot over the years is exactly that is like, the small ideas like the button, like the manual, and so on. Okay, that’s okay, you can test. But the big decisions I shoe making directly without an experimentation program, like studying understanding the problem, but the solution, so and it’s important, and I think for the agencies, it’s important to present that like the data, the parties of getting all this data together to collect to make the better decisions, because that could be the difference in terms of getting this contract, or maybe keeping the contracts later on. Because it’s why there’s values, like not changing the color of the button or changing specific content in a landing page. They definitely give gains as well. But in much less, less Can you

Anthony Morgan 59:16

write, right? No, yeah, experimentation can go beyond just the user experience. There’s, there’s many layers to where it can be be utilized and where it should be utilized. So I love that idea of it being more just part of an organization’s operations than like a strategic role, but, you know, it plays a much bigger role and it’s part of operations because it’s, it’s a core part of how the organization operates and thinks about things is through the lens of experimentation. So that’s Really good. Well, I really enjoyed the time that we had together. I think the one thing that I’ll leave our listeners to is always think what’s next. When it comes to experimenting, when a test is done, ask yourself what’s next because like our daughter said, Here, hypothesis is like a fruit and squeeze it until you can’t squeeze any more juice out of it. And so love that concept. Really enjoyed our time together. Thanks again, Eduardo. Thanks. So pleasure talking to you. Again, thanks. Thank you

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.

Join the Directory

Advertisment

How to Maximize Hypotheses Testing ft. Eduardo Marconi Pinheiro Lima

AI-Generated Summary

Video

Audio

AI-Generated Transcript

Connect with Experimenters from around the world

Categories

Related posts: