Maximizing Results: A Comprehensive Guide to Ad Network Experimentation ft. João Marcelo Rodrigues

AI-Generated Summary

Maximizing results in ad network optimization requires a systematic approach. Start with A/B testing on Google Ads and gradually introduce more complex campaigns, balancing technical and creative aspects. Analyze your channel’s performance, ensuring sufficient impressions and conversions for statistically significant results. Shape your product to align with the ad network’s requirements and consider user experience. Prioritize velocity in testing initially, gradually enhancing quality, and encourage collaboration across different platforms for valuable insights in the quest for optimization.

Video

Audio

AI-Generated Transcript

João Marcelo Rodrigues 0:00
And after you have more materiality on your test, you can add new creatives to your campaign. And after you add new creatives to your campaign, you can test for example, different types of campaigns. So I think is a is a process that you start with the most simple one, that would be the optimization using the AB test of the Google ads and then you will start to evolve for mutual campaigns working together, and then start to to improve the insights that you acquire from that. And of course, you cannot use the insights only on the campaign’s you can use after on your products and then combine the experimentation over the campaign with for example, experimentation on a landing page, or now on our app signup process. I think that is the key for for maximize your results.

Charlotte April Bomford 0:56
Hello, and welcome to experiment nations podcasts. I’m your host Charlotte Belford. Today’s an exciting day as we will be having a chat with John about how we could use the optimization process to design experiments on ad networks like Google ads. Very interesting. John is the CRO specialist for rate, pay tackle and has been for two years. John, would you like to tell me more about yourself and your journey to becoming a CRO specialist? Yes. Basically, my background is own software development. I started my career on a public company here in Brazil, the company that supplies water for the population. And after that, I started to work with projects. And after project I started to work with Li and then I became a project manager for a consulting company. And inside this consulting company, one of my projects they involve with our CRO agents that was working with the company to provide some benefits for a customer. And then I started to learn about SEO, what is SEO, how the process involving OCIO in CRO mainly looking for the project inside this customer. And after that I started to study in growth and started to to follow a few growth hackers and understand a little bit more about the experiment programs, and then started my career as a CRO.

That’s amazing. That’s a long journey. But you got there. What amaze you with the CRO process? Is that it? Do you find it like really interesting that you’re like, Nah, I’m not going to do software development and anymore, or project management algo CRO, what made this what made the switch?

Speaker 1 2:50
Basically, what makes me feel most comfortable about the CRO is the fact that you can combine the human behavior studies with the marketing with the technology. I’m a tech guy, I like technology, but I like to provide benefits for the people. I don’t like to work, for example, in banking applications. No, I like to be there developing a product for the people in CRO providing me a lot of insights of how improve this construction of new apps or software’s. And then, I don’t know, I think it was what what makes me choose for this area was this possibility to be working on technical area. But of course with these human backgrounds involved all the time. So I like to say that what we do is art, we stay on a sharp line using the our logical process, but we are still doing art. And I like what I’m most like is about zero is this possibility to be doing this creative process and also using the technology.

Charlotte April Bomford 4:00
I love how you put it because I’m thinking of CRO as when you’re playing music because when you play music, it is kind of like technical and creative at the same time. And CRO for me, it’s kind of like the perfect balance of creative and technical. So I love how you put it. So anyway, the I’m actually curious about the topic that we’re going to talk about today. Can you further expand on how we could use like they have like a duck point system on how we can use design experiments in the optimization process and help ad networks like Google ads or other ad networks like probably Bing or Facebook ads etc.

João Marcelo Rodrigues 4:45
Yes, basically the principles of CRO are the same. You are doing a test inside an ad network but the principles are the same. The difference is that you have a few extra problems to solve for example, you do don’t have control of how the ad network is going to escalate or not your ads, and you don’t, you can, you cannot be so sure of how many people are going to see to be touched by your ad. So you need to try to get a few steps back. And then you start to design your program. I think the most difficult part is that nowadays, everyone is using the AI is to improve the performance of the campaigns. So you don’t have much control of your segmentation is it’s very complicated to, to be inside these kinds of ad networks. Without all that control that we are used to, for example, when you are performing an experiment on your website or your app, you can do the segmentation, you can have a great insight of how many people are going to be there, you know, your know your traffic. So on the edge networks, you don’t know. So we developed a few strategies to be ahead of these problems. And basically, to start trusting us basically, to start your experimentation over these ad networks, I like to split over two kinds of tests, I have a test that you are performing the optimization over an ad that you have already used on the edge network and the new ads. So it’s important to split on these two categories to start your program over where where you can be more productive, where you can have better results. So I like to start improving the ads that you have, you already have data, and you want to have users being touched by dead. So after that, you start to develop your new creatives, you can set up a budget, for example, for new creative is not true, of course to minimize the risks of this operation. And then you can start producing your new your new ads based on the knowledge you acquire performing optimization over the ads you have already running on the ad networks.

Charlotte April Bomford 7:16
So you’re basically creating two campaigns, but the first part is to optimize the existing campaign. And then whatever you learn from the existing campaign, you create another campaign in parallel with the other existing campaign and run it and see which one works better. Is that what your it means,

João Marcelo Rodrigues 7:39
for example, how much money you you are investing on the ad network, sometimes? Yes, sometimes you can do the test inside the same campaign with different creatives inside the same campaign. Of course, the better would be to have a test campaign isolated from your main campaigns to avoid risks to avoid, avoid losing much money when you are only testing your new creatives. But in some cases, the customer doesn’t have a big a bigger budget. So you need to use the same campaign over to test new ads, and to improve your your already running ads.

Charlotte April Bomford 8:23
That’s a good point. So if in case let’s say I’m an optimizer or a Google Ads specialist, what would be the advice like for example, it’s my first time testing? How would you explain to me the optimization process and how I can do the AB testing on the Google ad platform? Let’s say, Okay,

João Marcelo Rodrigues 8:43
first you need to look for, for your previously ads, I don’t have an historical previous previous ads. Okay. So we will start from zero, you will start to run your campaign. And then you start, you look for the creatives that have more most impact over your customers, because it’s impossible to change the way the channel works. So you need to work in the way the channel is working for you. So you need to look for your ads, and try to estimate you try to estimate how many how many ads you can test how many variants or features you can test during to this to that escalator you have with this budget you are testing. So you need to look for your results. And then you can estimate how many tests you can perform and the time window for the test to be conclusive. So first you need to run the campaigns and try to estimate what we call the the key factor of the campaign. That is the elasticity of the campaign of how much money you can you need to support and how many users are being impacted with this ads because it varies from business from it. It kind of go have pieces now you’ll have different bids, different bids. So you need to understand first how your channel is working. And after you understand how your channel is working, it’s much easier for you to visualize where you can perform the optimization. And this first optimization would be using the A B tests of the Google platform, you can use the B tests to optimize the the creatives that you have most successful with. So I think to start with, this is the best to start your experimentation. And after you have more materiality on your test, you can add new creatives to your campaign. And after you add new creatives to your campaign, you can test for example, different types of campaigns. So I think is a is a process that you start with the most simple one, that would be the optimization using the AB test of the Google ads, and then you will start to evolve for mutual campaigns working together, and then start to to improve the insights that you acquire from that. And of course, you cannot use the insights only on the campaigns. You can use the after on your products, and then combine the experimentation over the campaign with for example, experimentation on a landing page, or now on our app signup process. I think that is the key for for maximize your results.

Rommil Santiago 11:27
This is Rommil Santiago from experiment nation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing it helps us a bunch. Now back to the episode.

Charlotte April Bomford 11:43
So after the the, let’s say experimentation, how would obviously like with a B testing, you’d be able to know based on the number of conversions, right? And so if I’m again, like for example, I’m you’re talking to someone who doesn’t know about or, you know, I have Google ads, or experience, but I haven’t done a B testing. And you have gone through this optimization process? How would I analyze the data? And say that, okay, this conversions are good, but are these good conversions? Or are these quality leads, etc? Like, what would your advice be?

João Marcelo Rodrigues 12:27
Great, great question. Basically, when you are performing a CRO experimentation on our website, you evaluate your kreski in your conversion. On the ads campaign, you need to evaluate the impressions that you have on your on your creative. And then the conversion. Of course, you cannot look for all the metrics. Sometimes I saw teams that were getting mess, because they are trying to look for all the metrics. They are let’s use the metric is key. And inside this metric tickets, they have retention retention, over 30 days retention over 90 days. First deposit first conversion of a lot of metrics. No, I think it’s better for you to use only two metrics, metrics that we will define the success of the experiment, that pretty much will be the conversions over the impressions. And then our secondary metric is that you need to to evaluate the I call trade off metric, a metric that you don’t have to be improving these metrics. But it’s good to look for a void for example, the the false conversions. So you need to look, for example, I started to look, for example, the first metric, the metric that gonna define the success will be the conversion. And the second metric will be the retention over 30 days, for example, it’s according to the Bs is to how your product is. So basically, I look to these two metrics want to define the success of the experiment, and the other only to control and to avoid this misunderstanding with the fake fake conversions. And sometimes you induce the users to, to perform something to perform a sign up process, but they are the good users for your product, they aren’t the best fit for your product. So you need to look for two metrics, well metric to define the success of the experiment. Another metric or it’s a trade off metric, is surfing that you don’t want to improve is not the main goal of your optimization, but you are checking to avoid these kind of situations. But only two metrics, I think more than two metrics. I’ve seen cases that people were using 50 metrics, and 50 metrics. Of course, when you’re performing a test, one of these metrics will be will have a negative impact. And it’s sometimes it’s make harder for you to take the decisions

Charlotte April Bomford 15:00
I agree on that one, define at least the three KPIs or at least two to three, and then base your decision on there. Now, the question I have is, for example, the Google Ads person has gone through this lengthy process of optimization, doing a B test analysis know that. And if in case the AB the variation one, which is usually the first variation, one, it may be a huge change from one campaign to another. I’m not sure because I haven’t I have done a B testing with Google ads before, but that was around a year and a half ago, thinks we have change. So if in case, another, like creatives sets of, you know, different sets of headlines and all that, would I, if I’m the Google Ad specialist, do I have to recreate another campaign to reflect the winning variation?

João Marcelo Rodrigues 16:06
It’s not necessary. For example, from what I understand or for you said, you have different results with different types of campaigns. And sometimes you need to define what what’s the best campaign for your business. On the past, we use it a lot, for example, the awareness campaigns. But nowadays, as I said, with the AI optimization of the campaign’s, it’s hard to have a profit over awareness campaign. So you need to first before you start your experiment process, you need to define what are the best campaign types for your business. And sometimes, for example, when you use awareness awareness campaign, the conversion criterias will be different. So you need to perform your test evaluating did the criteria conversions off of that specific campaigns. So using that kind of way to think you avoid, for example, this kind of problem that you have a creative that was performing amazing good on a kind of campaign, when you change, it’s not the best, the best ad for the kind of campaign. So I like to focus on a few kinds of campaigns, and then perform the optimization inside that campaigns. So you cannot you don’t will won’t be the you won’t be changing the way that campaigns according to your ads. No, you first define your campaign structure, and then you perform the test. So you minimize this effect of changing the structure. And of course, different channels also are important too. Sometimes you test something for Facebook, that doesn’t work on Google ads. And what I think it’s valuable to understand this is really focused on where your ad is being displayed. Think a lot about the user experience sometimes is an ad that your user going to spend at least five seconds looking for the ads. And sometimes No, sometimes you’ll have more gap when user needs to, for example, row by string and pass for your ad. So all these kinds of details of the user experience of where your ads are being displayed, I think helps you to understand what works better for each channel and for each campaign structure.

Charlotte April Bomford 18:33
That’s a good explanation. Really, it is. It saves all the hard work, you know, the other question I have is, if I want to do a B testing, or with Google ads, are there any criterias that I have to look for before I can say like, oh, yeah, we’re ready for a B testing?

João Marcelo Rodrigues 18:53
Yes, first, you need to to perform the project of your tests. And you need to evaluate, for example, the how many impressions you are having on each kind of ads you have inside your campaign. And after you do this first analysis, then you can start to to understand where are the creatives where you can be successful testing and how many times you’re going to need to spend testing or ever on each one of these creatives. So it’s a work that you have to do first, you need to analyze all that your campaign structure and sometimes you need to change your budget to distribution in order to to help the test experimentation probe process to have more velocity for example. And then after you do this setup, you you are prepared for actually running the the test if you don’t do this kind of analysis before. What happens that you have most of your tests as inconclusive tests. So it’s an effort that doesn’t provide results for you or for your team. So you need first, to assure that you have the minimum impressions on each ad and the minimum of conversions to have a statistically significant result. This is what you have to do first, and then after you have this structure, you start the tests.

Charlotte April Bomford 20:25
What’s the minimum impression? And what’s the minimum conversions? Is there any like threshold or minimum minimum number that you have? That you would advise? We

João Marcelo Rodrigues 20:37
don’t have a specific number, because it depends off your for example, your BS is basically when you are working with b2c, I can say for my less customers. When you are working with b2c, you need to look for your conversion rate. And then you can estimate how many impressions as you do the estimate trip estimate use for traffic that you need for moving a specific metric, you need to perform this analysis to understand inside the channels, but the metric is you are looking at the conversions and impressions. And then you can define out according to offer in your bed, you can define how many impressions you need to you must have in order to have a result. But for example, using Google ads, owner conversion rate baseline of 10%, we can have, we need to have at least 50,000 impressions over your head to have a conclusive result on each variant. You need to have 50,000 impressions to have a visa is a guess, based on this baseline of 10% on these B’s is on b2c basis. So you need to perform this analysis before you start testing.

Charlotte April Bomford 22:00
So what’s where can you start in terms of performing a note and analysis?

João Marcelo Rodrigues 22:06
Basically, this analysis is to look for example, to define where you’re gonna, you’re gonna be at the RS statistic is parameters, for example, I like to use statistical significance of 90%. So according to that you can be you can get back and then estimate how many impressions you’re gonna have. And of course, I’ve seen large campaigns that you can have nine 9.99%, for example, in it’s, it’s okay, because the impressions are so big that allows you to have this flexibility over the statistics, in in time window, in a window of time of basically of one week, for example, in one NICU one week, you can find this kind of significance on your tests, but it depends a lot of your case of your scenario. So you need to make this analysis based on your parameters, your initial parameters, how, what the significant statistical significance we’re going to use. And then you can estimate how many impressions you can you need to have to in order to have a good result.

Charlotte April Bomford 23:24
That’s great explanation. So yeah, I don’t have any further questions unless you want to provide a conclusion to all of our viewers.

João Marcelo Rodrigues 23:36
Oh, my conclusion. It’s about replicate the CRO knowledge and inside that your channel. And of course, sometimes you you, you have an experimental program in your company, but not only ad networks. So you can of course, use the insights that were had already have, use the historical understand what is performing good inside your product, and then bring some insights for your campaign. Of course, you need to pay attention on the user experience is most very different from web page from a LinkedIn page. And then you can of course, bring the main, for example, and narratives but the main cop is the working side your product to the campaigns. And of course, you need to perform a linear storytelling, you need to understand that your customer will be the first contact with your brand with your product on your ads. And then further, you’re gonna have the second third, the third, third, so you can provide different information different product benefits, according to the channel, not the other way. Sometimes we try to shape the channel as you’re our product. We tried to change the campaign structure change the events that we use. Before optimization, and that’s not the solution, you need to shape your product to the ad network and how you do that. Understanding the user experiment, understanding where your ads are being displayed, how they are being displaying what platforms in what kinds of media, sometimes it’s a video, it’s a movie, it’s a video. So you need to have an edge that combined with that platform, so it specifically, even inside Google, you have multiple possibilities of displays. So we need to understand what works better to your product and not the other way, not try to the network to to be good jepara know, your product needs to be featured on the ad network requirements. And after you do this kind of thing, the overview thing you can do, you can shake your roadmap to experimentation over the ad networks in the same way you do do the roadmap for our web page for our landing page for an app. But first you need to understand these kinds of different on these kinds of channels. And then you you can provide to your you can start the program of tests, which sometimes means many person many persons ask it, what’s the best hour? What’s the velocity I need to have? How many tests should I should I perform? No, that’s not a one answer for that. Oh, should I focus on quality over amount of tests? No, I think you have to focus of course, on the best of the two words, you have to have the last over your tests multiple tests, where you have to have quality for if you can choose only one I like to start with the velocity, I like to first understand where you can test where you have draft your where you have impressions enough to test and after you make these arrangements and they start running your test program, you start to optimizing then you first understand what are the problems are we are having a lot of inconclusive tests why? So probably we are our sample eight is not okay, we need to do to improve to let the campaign’s rank for more than one knee we two weeks, or sometimes entire mouth to have conclusive results. So after you start at your maximum velocity that you estimate, of course, you start to improving the quality and sometimes the hypothesis or not. Okay, we are Yeah, we are testing only copied. And it’s not providing any effects over our metric. So let’s start different ad shapes, let us video let’s use gifts now. So we have many possibilities to test. But before we start to improving the quality of the tests, I like to work on the maximum velocity. So I think the that’s kind of my way to think to improve the results. And of course, to bring the results for the company sooner.

Charlotte April Bomford 28:17
I agree. And I think like having experimentation on one platform or on the ad networks kind of helped us, you know, people who optimize on the website, and they kind of like work hand in hand. It’s like, okay, what’s winning on your so it’s winning on mine, let’s combine them together. And let’s make it better. And yeah, I feel like some companies might lack that cooperation or collaboration between platforms. But it is kind of like good to get learnings from different channels to apply it to the specific channels that you’re tasked with. So yeah, it’s really good insights. John, we really enjoyed the conversation, and really learned a lot from your experience. And yeah, we’re looking forward to having you again with another topic with experimentation. Oh,

João Marcelo Rodrigues 29:18
it would be a pleasure. I’m a great fan of experiment nation. I’ve been following Eero mu and all the team more than two years now. And I I’m really happy to be here and talk with me to another

Charlotte April Bomford 29:36
awesome thank you so much.

João Marcelo Rodrigues 29:39
Bye bye.

Rommil Santiago 29:42
This is Rommil Santiago from experiment nation. Every week we share interviews with and conference sessions by our favorite conversion rate optimizers from around the world. So if you liked this video, smash that like button and consider subscribing it helps us a bunch. Now back to the episode.

If you liked this post, sign up for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

---