Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.
Eddie Aguilar 0:00
For the developers that are just getting into the space, or even, let’s say, I know a lot of managers right now are just trying to learn how to code to just so they can up their game. Um, you’re gonna break stuff all the time. It’s, that’s kind of the point.
Charlotte April Bomford 0:29
Hello, everyone. So if this is your first time tuning in, on experiment nation, my name is Charlotte, your podcast host. And I have very special guests with a very, very interesting story with his experimentation experience. His name is Eddie and Eddie, how are you doing today? I’m pretty good. Pretty good. How are you doing? You’re so well. Yeah, I’m doing well. So you know, like, I’m thinking like, what’s a better way to introduce but to introduce yourself? Can I let you take on from here and we can start the conversation? What do you think? Sure, yeah.
Eddie Aguilar 1:11
Eddie Aguilar, I’ve been a developer since I was I would say about 15 years old, I really started developing programs for games actually said during
Charlotte April Bomford 1:27
the dial up periods, and you can hear the sound and all that.
Eddie Aguilar 1:32
Yeah, it, it was. I started developing brackets, basically like tournament brackets for gaming, and then eventually made my way into just the eSports scene for StarCraft specifically. And that was like heavily based on just you know, optimizing your gameplay. For some time, I really felt like there wasn’t good experiences out there that just kind of was user centric. I really felt like there was a lack of that. And that’s what really made me shift more into the optimization space was because I wanted to start being able to introduce some of the technicalities that come from like, you know, all the code that you use, and making it so it’s easier for people that are doing marketing to be able to like, combine the two, right? So marketing technology right now, it is huge. When I worked through funnel MV, or rejoined fondue IV, it was around 2007 teen or so or 2018. Were building personalization tools. Like I was basically helping our AV testing clients leverage our personalization engine. So I rejoined as a solutions engineer, and one of my clients was envision app. They were an awesome client. It’s it was a great opportunity to work with them and learn with them. I ran into issues.
Charlotte April Bomford 3:22
Like what happened? That’s very interesting.
Eddie Aguilar 3:26
Yeah. So envision app itself is a it’s, it’s like a prototyping tool. It helps you and your company be able to like a upload designs, get feedback on the designs across the board. There’s also a version where you can do like live editing kind of like mural. Yeah, so they definitely had like a little mural feature before mural came around. But they essentially did their main, their main focus, prototyping back then. And, and getting companies to be able to, like leverage designers to build prototypes.
Charlotte April Bomford 4:16
Yeah, and collaborate as well. They can do like live edits and all that. What were you trying to do then? So you have what’s the KPI that you were trying to achieve?
Eddie Aguilar 4:28
Yeah, so essentially, I ay. We were looking to optimize revenue and LTV. Those were like our primary KPI for and what we were trying to do was serve up basically upsells
Charlotte April Bomford 4:51
you know, upgrading.
Eddie Aguilar 4:53
Yeah. And what we were doing was essentially switching like how the price I see modules worked based on like the current plan that you were on. And it was on the main website. So nothing like within the product itself, we were grabbing some information from the product side, and then leveraging that on, you know, the client side and showcasing the difference to the users just like, you know, based on their tears, I would say, essentially, you know, you, you start developing this things. And you, you know, as any developer, you’re working with client code, you’re working with a client website. And they’re all different. You’re always learning something.
Charlotte April Bomford 5:42
Exactly. They’re all different, and some of them complicated than others.
Eddie Aguilar 5:46
Charlotte April Bomford 10:08
it up 10 minutes after I’m not saying that it’s a good thing. But there are some times like issues or areas that would be like hours and you’re like, oh, it’s not working. It’s been like three hours, four hours, at least they were able to pick it up for 10 minutes. Yeah,
Eddie Aguilar 10:24
Charlotte April Bomford 11:20
Sometimes these things happen, and sometimes, you know, like the best, I think, like, like, I think I would speak to a lot of optimizers here, where, you know, we’ve tried to QA and went live on production, it didn’t go well as we’ve planned it out. But that kind of like brings up opportunities for improvement, you know, like processes in place, like you mentioned, and more conversation, more collaboration with the company to prevent this in the future. So yeah, it’s interesting. Oh, yeah.
Charlotte April Bomford 11:55
What happened after that? So you’ve had that conversation? What was the what was what’s happened?
Eddie Aguilar 12:03
Charlotte April Bomford 12:39
that was a very interesting experience. Eddie, I think like you were telling me how complicated this test was, and that you’re trying to get data points from different users at the same time just to understand what plan they’re in, so that you can run the test specifically, specifically for that user dependent, again, on the plan that they have, which is very, very complicated and very interesting.
Eddie Aguilar 13:08
Charlotte April Bomford 13:55
Yeah. That that is relevant to that specific user. Yeah.
Eddie Aguilar 14:00
Yeah. So it definitely, it’s definitely was a lot trickier. And, obviously, you know, I worked with the Envision app security team and the air team and the development teams, just to have, you know, a process in place that’s able to like just, if, if I make a change globally on the website, though, no, just you know,
Charlotte April Bomford 14:35
how long did it take? So after passing that test, let’s say or, you know, removing the project out of the script, how long did it take for you to run the test again, like Did it take a few weeks back in the drawing board or,
Eddie Aguilar 14:49
um, I would say, a week, just it we went it turned. We turned it around again in a week.
Charlotte April Bomford 14:58
Yeah, and then let me know Like, what are the results? Because I think the results are incredible.
Eddie Aguilar 15:04
Yeah, so it was 227% revenue lift, Oh, wow. Massive was like 70% LTV lived. Just, you know, making sure that you kept your upgrade, copy and your upgrade pricing consistent with the user and their intention, right. Huge lift. And we were able to showcase that lift. So
Charlotte April Bomford 15:44
amazing. Are they still using it envision apps or evolved it? That test?
Eddie Aguilar 15:53
I’m pretty sure it’s evolved, you know, the marketing manager there, she is very intelligent. And I’m sure it’s evolved to some greater than what we already did and accomplished. And like I said, this was back in like, you know, 2018 or so it’s been a while. So it’s definitely evolved. It’s might have you never know, sometimes in digital landscapes, things go full circle, I feel like you never know, they might just be showing the old pricing.
Charlotte April Bomford 16:31
Just go back to the web. Just kidding. So why are you wondering, that’s, that’s a crazy experience, because I’m pretty sure like a lot of us here especially like, would be like, oh, yeah, that happened to me, too. What are the takeaways? Like what, what would be the advice that you’ll give? Because in all honesty, not a lot of I wouldn’t say, you know, because you’re a programmer turned CRO, you know, and not a lot of CRO specialists have programming background. But they’re probably great project managers. But what would be your takeaway from this? And what would be your, you know, advice from us human beings who are not God’s level? Like?
Eddie Aguilar 17:22
I would say it’s communication in these issues. Being able to develop your skill set in communicating these issues with not just like, the managers you work with, but possibly, you know, VPS C suite. And, yeah, I mean, that’s the biggest, like, that’s the biggest takeaway I got out of this. I, you know, before this, it was a vision app, they still loved me as the developer. It wasn’t like anything that I messed up. It’s a huge learning, right. The test itself proved valuable for them. Even if I may have cost them a little bit in error logging it, it was the return of investment still good. Yeah. Exactly. Yeah, I definitely was interesting time. And envision app definitely was one of my best clients to work with. Like I said, you know, the marketing manager and I, we still talked to this day. She’s currently a clot. She’s at Cloudflare. She’s a project manager, Emily Flannery, I hope she’s okay with me, mentioning her name.
Charlotte April Bomford 19:01
Eddie Aguilar 20:17
Yeah, I would say it’s valuable. And I’ll say it’s valuable. Because imagine you just your small development, your small CRO team, and if you don’t have your developers overwhelmed, you can definitely jump in and help him. And I, you know, I think that’s a big takeaway is, you know, sometimes developers do get overwhelmed, you know, they’re stressed about a specific test, they can’t figure out. You know, it’s, it’s always a learning and I think, you know, understanding at least some of the technicality of it can help develop you as like, the manager when you’re speaking to clients, or when you’re speaking, like I mentioned to VPS, you’re able to speak to get a little more and like, you know, help your developed restaurant
Charlotte April Bomford 21:15
self. Yeah, exactly. Yeah. Yeah. And you, too, yeah.
Eddie Aguilar 21:21
It’s a team effort.
Charlotte April Bomford 21:23
Yeah, exactly. Yeah. You’d be able to prioritize things better know what the limitations are. So yeah, that’s very interesting.
Eddie Aguilar 21:35
Yeah, no, I mean, other than just, for the developers that are just getting into the space, or even, let’s say, I know, a lot of managers right now are just trying to learn how to code to just so they can up their game, um, you’re gonna break stuff all the time. It’s, that’s kind of the point. What we’re doing, we’re gonna, it’s we’re breaking things to make it better, in a sense, right? We currently don’t like how the website functions, because it’s not either converting well, or it’s just not user centric. So we test and we move things and we break things of how the current website works, just to see if it does work. If the statistics tells us it works, right. So
Charlotte April Bomford 22:30
yeah, but yeah, thank you so much, Edie for having a forum for coming over. I mean, and, you know, telling your experience to the experimentation. Community. But yeah, we’re looking forward to having you again in the show, and learning more about CRO and your experience.
Eddie Aguilar 22:55
Yeah. I have many incidents and many.
Charlotte April Bomford 23:00
Yeah. Yes, we’re wanting more. Thanks, Eddie.
Eddie Aguilar 23:07
No point. Then. Thank you have a good one. You too. Have a good day, Eddie.
If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK
Connect with Experimenters from around the world
We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.
- Adventures in Experimentation (9)
- Analysis (2)
- Announcements (48)
- Ask the Experiment Nation (11)
- Certified CROs (193)
- Community (2)
- Conversation Starters (2)
- CR-No (5)
- CRO Handbook (4)
- CRO Memes (18)
- Experiment Nation: The Conference RELOADED (41)
- Experimentation from around the world (2)
- For LinkedIn (121)
- Frameworks (2)
- Growth (14)
- ICYMI (2)
- Interviews with Experimenters (167)
- Management (1)
- Marketing (11)
- Opinion (8)
- Podcasts with Experimenters (15)
- Point/Counterpoint (1)
- Product Experimentation (5)
- Product Management (9)
- Profile (9)
- Question of the week (5)
- Sponsored by Experiment Nation (1)
- The Craft of Experimentation (1)
- The Cultures of Experimentation (1)
- The weekly buzz (13)
- The year in review (1)
- Uncategorized (352)
- Weekly Newsletter (183)
- Where I Started (2)
- Making the case for data-backed design featuring Brian Massey - September 30, 2023
- Announcing the 2023 Conference Awards - September 24, 2023
- How to Maximize Hypotheses Testing ft. Eduardo Marconi Pinheiro Lima - September 9, 2023