Learn from Mistakes: What Eddie Aguilar learned from accidentally taking down InVision

AI-Generated Summary

Eddie Aguilar, a developer and CRO specialist, shares his experience with experimentation and optimization. He discusses how he started as a developer at a young age and eventually shifted to optimization. He recounts a specific incident while working with Envision App, where he made a global change to the website that caused a significant error, resulting in a spike in error logs. Eddie emphasizes the importance of communication and collaboration in such situations. Despite the error, the test yielded impressive results, with a 227% revenue lift and 70% LTV lift.

Video

Audio

AI-Generated Transcript

Eddie Aguilar 0:00
For the developers that are just getting into the space, or even, let’s say, I know a lot of managers right now are just trying to learn how to code to just so they can up their game. Um, you’re gonna break stuff all the time. It’s, that’s kind of the point.

Charlotte April Bomford 0:29
Hello, everyone. So if this is your first time tuning in, on experiment nation, my name is Charlotte, your podcast host. And I have very special guests with a very, very interesting story with his experimentation experience. His name is Eddie and Eddie, how are you doing today? I’m pretty good. Pretty good. How are you doing? You’re so well. Yeah, I’m doing well. So you know, like, I’m thinking like, what’s a better way to introduce but to introduce yourself? Can I let you take on from here and we can start the conversation? What do you think? Sure, yeah.

Eddie Aguilar 1:11
Eddie Aguilar, I’ve been a developer since I was I would say about 15 years old, I really started developing programs for games actually said during

Charlotte April Bomford 1:27
the dial up periods, and you can hear the sound and all that.

Eddie Aguilar 1:32
Yeah, it, it was. I started developing brackets, basically like tournament brackets for gaming, and then eventually made my way into just the eSports scene for StarCraft specifically. And that was like heavily based on just you know, optimizing your gameplay. For some time, I really felt like there wasn’t good experiences out there that just kind of was user centric. I really felt like there was a lack of that. And that’s what really made me shift more into the optimization space was because I wanted to start being able to introduce some of the technicalities that come from like, you know, all the code that you use, and making it so it’s easier for people that are doing marketing to be able to like, combine the two, right? So marketing technology right now, it is huge. When I worked through funnel MV, or rejoined fondue IV, it was around 2007 teen or so or 2018. Were building personalization tools. Like I was basically helping our AV testing clients leverage our personalization engine. So I rejoined as a solutions engineer, and one of my clients was envision app. They were an awesome client. It’s it was a great opportunity to work with them and learn with them. I ran into issues.

Charlotte April Bomford 3:22
Like what happened? That’s very interesting.

Eddie Aguilar 3:26
Yeah. So envision app itself is a it’s, it’s like a prototyping tool. It helps you and your company be able to like a upload designs, get feedback on the designs across the board. There’s also a version where you can do like live editing kind of like mural. Yeah, so they definitely had like a little mural feature before mural came around. But they essentially did their main, their main focus, prototyping back then. And, and getting companies to be able to, like leverage designers to build prototypes.

Charlotte April Bomford 4:16
Yeah, and collaborate as well. They can do like live edits and all that. What were you trying to do then? So you have what’s the KPI that you were trying to achieve?

Eddie Aguilar 4:28
Yeah, so essentially, I ay. We were looking to optimize revenue and LTV. Those were like our primary KPI for and what we were trying to do was serve up basically upsells

Charlotte April Bomford 4:51
you know, upgrading.

Eddie Aguilar 4:53
Yeah. And what we were doing was essentially switching like how the price I see modules worked based on like the current plan that you were on. And it was on the main website. So nothing like within the product itself, we were grabbing some information from the product side, and then leveraging that on, you know, the client side and showcasing the difference to the users just like, you know, based on their tears, I would say, essentially, you know, you, you start developing this things. And you, you know, as any developer, you’re working with client code, you’re working with a client website. And they’re all different. You’re always learning something.

Charlotte April Bomford 5:42
Exactly. They’re all different, and some of them complicated than others.

Eddie Aguilar 5:46
Yeah. And essentially, you know, you also learn as a developer, certain things to just build are baked into your process, right, before you start actually developing at this point. But I was a little comfortable with how like session storages and local storage is worth, what not? I had defined the value in what’s called Project JavaScript in Optimizely. And what project JavaScript is it just lets you put JavaScript globally on the website? For Yeah, yeah. So I mean, you can definitely cause a lot of issues with that if you’re not careful, right. But because I was mainly just trying to create and grab some session storages like, just, you know, information that really the client doesn’t see. It wasn’t, it wasn’t anything that I thought too much about where I was going to need to, like, catch my eye. So in JavaScript, we have a thing called these catches where you can catch errors. And you can essentially, like either do nothing with the errors. So it just looks like nothing is broken. Yep. And that, I mean, honestly, it’s not fine. That’s hard. It’s not like I’ve had, I’ve had projects where I’ve gone crazy looking for errors or something. But what they had done was they catch the error and like, did nothing with the error. So like, they didn’t really like post it to the console. They didn’t, they just did nothing to it. So but that’s another topic. But essentially, you know, I wasn’t leveraging this at the time, because I was, like I said, just using code that was like on the client side, and that they just really don’t see your leverage. But I did not know that envision app had some form of internal console logging. So they were able to see like all that, like they were able to see the air. At one point, I decided to just, you know, save the project JavaScript, and then launched completely globally to across envision that. If the value obviously didn’t exist, it was going to return that there was an error. And that’s what exactly happened. It just started console log started airing out for every user that was on envision app. It basically sent envision apps error logging, like, through the roof. Yeah. Yeah. Like it was just like an immediate reaction after I, I recall, it was like 10 minutes later after me pressing that save button where I had the marketing manager of Envision app just yeah, they’re sending me a message and they’re like, Hey, I don’t know what is going on. But they’re saying something is you know, firing an error on the app. And it’s causing like our air logs to spy which a they’re charged. They were charged her like, you know, log that happened. Oh, crap. Okay. So yeah, so the, you know, the pricing of that at the moment was a very expensive costs. Um, I’m sure they were able to, you know, work it out of that, but at least they picked

Charlotte April Bomford 10:08
it up 10 minutes after I’m not saying that it’s a good thing. But there are some times like issues or areas that would be like hours and you’re like, oh, it’s not working. It’s been like three hours, four hours, at least they were able to pick it up for 10 minutes. Yeah,

Eddie Aguilar 10:24
I would say, I would hope after 10 minutes, especially, I’m certain they, you know, right now, they’re over, like 10 million users or something, they were in the millions back then I’m sure now they’re way over that. And I’m sure, you know, error logging that many users at the same time, which is pretty bad for envision that security team or dev team. But essentially, you know, I reverted what I did, we went back to the drawing board, kind of discussed it out with their team, like, move steps moving forward, you know, set up some process in place. So that way, we’re not just injecting JavaScript across the website.

Charlotte April Bomford 11:20
Sometimes these things happen, and sometimes, you know, like the best, I think, like, like, I think I would speak to a lot of optimizers here, where, you know, we’ve tried to QA and went live on production, it didn’t go well as we’ve planned it out. But that kind of like brings up opportunities for improvement, you know, like processes in place, like you mentioned, and more conversation, more collaboration with the company to prevent this in the future. So yeah, it’s interesting. Oh, yeah.

Charlotte April Bomford 11:55
What happened after that? So you’ve had that conversation? What was the what was what’s happened?

Eddie Aguilar 12:03
Yeah. So they, they do they do these follow ups, you know, to these errors, right. And just basically, you know, what I did in the project JavaScript, essentially, just making sure that if we’re going to do any changes that just affect the global website, it’s going to be like ticketed, just so that way, it’s not just like, marketing is aware that we’re making those changes. It’s also the, you know, development teams that are involved. Yeah. So

Charlotte April Bomford 12:39
that was a very interesting experience. Eddie, I think like you were telling me how complicated this test was, and that you’re trying to get data points from different users at the same time just to understand what plan they’re in, so that you can run the test specifically, specifically for that user dependent, again, on the plan that they have, which is very, very complicated and very interesting.

Eddie Aguilar 13:08
Yeah. And we were, we were not we were touching multiple points of the website. So it was like the homepage, then the plans page. And then if you got into, like, the actual checkout and booking flow, we were also making sure that whatever upgraded plans you saw there were also being updated. So it was, you know, it was a global change. And there was a global test to across the website. So I needed to leverage pot project JavaScript that that to, you know, make sure that I can just be consistent with the data that the user seeing in regard. The just the information.

Charlotte April Bomford 13:55
Yeah. That that is relevant to that specific user. Yeah.

Eddie Aguilar 14:00
Yeah. So it definitely, it’s definitely was a lot trickier. And, obviously, you know, I worked with the Envision app security team and the air team and the development teams, just to have, you know, a process in place that’s able to like just, if, if I make a change globally on the website, though, no, just you know,

Charlotte April Bomford 14:35
how long did it take? So after passing that test, let’s say or, you know, removing the project out of the script, how long did it take for you to run the test again, like Did it take a few weeks back in the drawing board or,

Eddie Aguilar 14:49
um, I would say, a week, just it we went it turned. We turned it around again in a week.

Charlotte April Bomford 14:58
Yeah, and then let me know Like, what are the results? Because I think the results are incredible.

Eddie Aguilar 15:04
Yeah, so it was 227% revenue lift, Oh, wow. Massive was like 70% LTV lived. Just, you know, making sure that you kept your upgrade, copy and your upgrade pricing consistent with the user and their intention, right. Huge lift. And we were able to showcase that lift. So

Charlotte April Bomford 15:44
amazing. Are they still using it envision apps or evolved it? That test?

Eddie Aguilar 15:53
I’m pretty sure it’s evolved, you know, the marketing manager there, she is very intelligent. And I’m sure it’s evolved to some greater than what we already did and accomplished. And like I said, this was back in like, you know, 2018 or so it’s been a while. So it’s definitely evolved. It’s might have you never know, sometimes in digital landscapes, things go full circle, I feel like you never know, they might just be showing the old pricing.

Charlotte April Bomford 16:31
Just go back to the web. Just kidding. So why are you wondering, that’s, that’s a crazy experience, because I’m pretty sure like a lot of us here especially like, would be like, oh, yeah, that happened to me, too. What are the takeaways? Like what, what would be the advice that you’ll give? Because in all honesty, not a lot of I wouldn’t say, you know, because you’re a programmer turned CRO, you know, and not a lot of CRO specialists have programming background. But they’re probably great project managers. But what would be your takeaway from this? And what would be your, you know, advice from us human beings who are not God’s level? Like?

Eddie Aguilar 17:22
I would say it’s communication in these issues. Being able to develop your skill set in communicating these issues with not just like, the managers you work with, but possibly, you know, VPS C suite. And, yeah, I mean, that’s the biggest, like, that’s the biggest takeaway I got out of this. I, you know, before this, it was a vision app, they still loved me as the developer. It wasn’t like anything that I messed up. It’s a huge learning, right. The test itself proved valuable for them. Even if I may have cost them a little bit in error logging it, it was the return of investment still good. Yeah. Exactly. Yeah, I definitely was interesting time. And envision app definitely was one of my best clients to work with. Like I said, you know, the marketing manager and I, we still talked to this day. She’s currently a clot. She’s at Cloudflare. She’s a project manager, Emily Flannery, I hope she’s okay with me, mentioning her name.

Charlotte April Bomford 19:01
Hello, Emily. Sorry, again, about it’s all good, though. That is such an amazing, you know, experience amazing learning and amazing is hearing about what you’ve been through, like that huge envision app experience, and then you’ve the things that you’re able to turn it around anyway, within a week and such an amazing result. I probably have one question that I just you know, are thinking about, like, what do you think it’s an because I myself, like I have an idea of HTML, CSS, JavaScript, I understand you know, session storage, like how do you values in there and stuff? But would you think that as a CRO some specialists are CRO managers? What do you think about them, going into that space and trying to understand or having that knowledge about it? Umm, CSS and JavaScript, and all that, like, would that be an invaluable knowledge that you probably would want to know, in addition to your zero knowledge? If not, then why not? You know? So?

Eddie Aguilar 20:17
Yeah, I would say it’s valuable. And I’ll say it’s valuable. Because imagine you just your small development, your small CRO team, and if you don’t have your developers overwhelmed, you can definitely jump in and help him. And I, you know, I think that’s a big takeaway is, you know, sometimes developers do get overwhelmed, you know, they’re stressed about a specific test, they can’t figure out. You know, it’s, it’s always a learning and I think, you know, understanding at least some of the technicality of it can help develop you as like, the manager when you’re speaking to clients, or when you’re speaking, like I mentioned to VPS, you’re able to speak to get a little more and like, you know, help your developed restaurant

Charlotte April Bomford 21:15
self. Yeah, exactly. Yeah. Yeah. And you, too, yeah.

Eddie Aguilar 21:21
It’s a team effort.

Charlotte April Bomford 21:23
Yeah, exactly. Yeah. You’d be able to prioritize things better know what the limitations are. So yeah, that’s very interesting.

Eddie Aguilar 21:35
Yeah, no, I mean, other than just, for the developers that are just getting into the space, or even, let’s say, I know, a lot of managers right now are just trying to learn how to code to just so they can up their game, um, you’re gonna break stuff all the time. It’s, that’s kind of the point. What we’re doing, we’re gonna, it’s we’re breaking things to make it better, in a sense, right? We currently don’t like how the website functions, because it’s not either converting well, or it’s just not user centric. So we test and we move things and we break things of how the current website works, just to see if it does work. If the statistics tells us it works, right. So

Charlotte April Bomford 22:30
yeah, but yeah, thank you so much, Edie for having a forum for coming over. I mean, and, you know, telling your experience to the experimentation. Community. But yeah, we’re looking forward to having you again in the show, and learning more about CRO and your experience.

Eddie Aguilar 22:55
Yeah. I have many incidents and many.

Charlotte April Bomford 23:00
Yeah. Yes, we’re wanting more. Thanks, Eddie.

Eddie Aguilar 23:07
No point. Then. Thank you have a good one. You too. Have a good day, Eddie.

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK