Twilio’s Justin Coons shares what you need to know about SEO Experimentation

0:00
You’re listening to Experiment Nation: The Podcast with this guy. Hi, I’m Rommil. On this show we interview experimenters from around the world. We share their stories, their lessons and advice with you, our listeners.

Rommil Santiago 0:37
Welco me to the show. Today, we have Justin Coons from Twilio. Hello!

Justin Coons 0:43
Hi! Thanks so much for having me.

Rommil Santiago 0:45
Yeah, for sure. We’re so happy to have you on. So to kick it off, I’d love to hear a bit about yourself and how you got to where you are today.

Justin Coons 0:54
Yeah, so, it’s it’s been a career path with a few different kind of curves in it over the years. right out of college, I was pretty interested in being a sports writer. So I had moved to San Antonio, and works on there for the Express news, and covered some sports for the Associated Press. This happened to be right around 2007 2008. And most of the folks I knew in newspapers at the time, were like get out now…. to be in newspapers. And then the people started getting laid off. And so I’m thinking okay, well, you know, there’s no time like the present, I think I need to make a career change. And I actually got hired on with a small eCommerce shop that sold sporting equipment in San Antonio, and got my cert there as a content writer, and then that kind of naturally lead into SEO. And from there, it’s been kind of agency roles. And kind of moving around to my current position right now, which is working as a product manager at Twilio.

Rommil Santiago 2:01
In at Twilio, is are you still touching SEO or is it something related?

Justin Coons 2:06
Yeah, still touching SEO somewhat. So I’m working on the marketing website. And specifically, it’s globalization efforts. So we’re looking to publish content in a number of different countries. And then, with SEO being such a big part of that, that’s, yeah, happy to kind of do that. Right there.

Rommil Santiago 2:26
Awesome. So you did mention that you’re a sports writer in San Antonio, and I’m from Toronto. And if you haven’t heard, you know, we won a championship a couple years back with, with this, this fellow named Kawhii. I was wondering what your thoughts were about the whole situation with Kawhii and him leaving, etc.

Justin Coons 2:47
Oh, gosh. Yeah. So the, when I covered the the Spurs for the Associated Press, I kind of did it as a junior reporter. And so I would usually get to go to cover games. You know, like the Tuesday night games, the 9pm games were like, you know, half the arena leaves by halftime. And this was in the era where it was like the Twilight of the big three. So those guys wouldn’t even be getting any minutes in these games. And so we get to watch like Tiago Splitter, Danny Green, and Kawhii and Kawhii seemed like, you know, such a diamond in the rough in San Antonio. And I’m really glad that he was able to get out into an environment where he could kind of they could build a team around him, and he could be successful.

Rommil Santiago 3:37
Yeah, we’re, we were happy to hold them for the months that he stayed in Toronto. It’s a shame he didn’t, you know, stick around, we tried. But you know, we’ll take what we can. He’s definitely loved here still. So that’s very interesting. Not not to deviate too far. So that right now, this is usually where I have the section called in the news. We try to be a show here. So I was searching the interwebs. And I stumbled on a article from Entrepreneur and the title of it was, “Selling online. Here are four things you need to know about SEO – and one of them that they one of the points that they mentioned, is that one should relentlessly experiment in terms of driving SEO. What are your thoughts with that? Because, you know, often hear and maybe it’s because of, you know, I’m not really in the SEO space, but you often don’t hear people talking about experimenting, like crazy in SEO.

Justin Coons 4:39
Yeah, it’s kind of wild, because that is really like the heart and soul of SEO is experimentation. Because when you get down to what search is at its core is, for example, like the Google search algorithm that has maybe 300 or 400 individual components to it. And we know A handful of them. And we really don’t know how much what is weighted. And when it is it is weighted more than in certain other contexts. And so really, experimentation is the only way to find things out when you’re you’re trying to drive your SEO. Yeah, so let’s start there.

Rommil Santiago 5:18
Oh, yeah, you touched on that word, Google does change there. They have a lot of algorithms, but they also change them a lot. My understanding is that they used to announce these sort of algorithm changes, and now they don’t do that as much. So how do you how do you know that your experiments that you’ve run, are still valid in like, a quarter a year or two years?

Justin Coons 5:38
Yeah, it’s, I don’t know if it’s really even possible, really, when it comes to experimentation that you do on your SEO, you’d want to look at a nearer time horizon. So and I’d say want to keep it a lot more disciplined, and and kind of, like narrow focused, then you might do with some more like broader usability testing, for example.

Rommil Santiago 6:03
Mm hmm. And when you mean, when you say, near focus, I probably miss miss quoted you there. What’s that period? Like? Is it a week, two weeks a month?

Justin Coons 6:16
Yeah. So it’s something that in most cases, you’d be looking at maybe two to three months. And really, what you’d want to do is, is formulate your experiments around testing against a specific component of the algorithm that you’d like to impact. So I think the simplest example is on a page level, ie, the kind of the most important content that is on the page is actually your title tag. So updating things like title tags, you should be able to test that component of the algorithm and, you know, kind of look at that that two to three month time horizon, because anything beyond that, you there’s so many unknowns and different variables that can come in and affect it. And then again, you’re kind of the observer biasing the test at that point, I think.

Rommil Santiago 7:09
And and when you do these tests, what are some of the metrics that you look at? Is it? Well, I’ll let you answer that?

Justin Coons 7:16
Yeah. So really, kind of the the big one is keyword ranking.

Rommil Santiago 7:22
And oh, sorry, just to interject there, like how would you monitor something like that?

Justin Coons 7:27
Yeah, what I’d recommend using is actually Google Search Console instead of anything else. The difference between Google Search Console and some of the other tools like like Moz, or SEMrush, is Search Console shows you your actual observed rankings over a period of time. So your average rank in that time period, versus kind of a less accurate version that comes from the SEMrushes and Mozes of the world. Because what they do is, they essentially just scrape search results pages. So what you’re seeing in Moz is kind of the snapshot of a single event in time, while search console is kind of your aggregation of people who actually saw your results over that period of time.

Rommil Santiago 8:20
Now, we’ve spoken a lot about Google. What about all the other other search engines like just optimizing for Ask Jeeves? Is that sort of thing?

Justin Coons 8:30
No. Yeah, so optimizing for other search engines is something you should pay attention to if it’s relevant to your market. So a good example, domestically in Canada and the United States is, we see a lot more DuckDuckGo usage on websites that cater to the developer community. So rising to DuckDuckGo is important there. If you have a big presence in China, for example, it’s necessary to optimize to Baidu. But other than that, like, Google’s kind of the king of the castle. So with that, I think is really the way to go.

Rommil Santiago 9:13
Would you say in general, they have like, what is it about 80% of the market? Are we talking about much higher?

Justin Coons 9:20
Yeah, I think in most areas I’ve seen it’s kind of in the 60 to 80 range. In places like Germany, it goes even higher from what I remember.

Rommil Santiago 9:31
Oh, okay. So, obviously, SEO is is very important. Now, how is it that folks like myself really haven’t? Like, I’ve never ran an experiment in Seo? I we’ve talked before prior to this conversation about running experiments, and that’s how I got the idea to bring you onto the show. But why aren’t more people doing this? Or am I in the Am I just blind?

Justin Coons 9:56
I think it’s tough. Seo really isn’t all that important for a lot of businesses where like, it’s, you’re, you’re focusing your marketing dollar is better spent elsewhere. So SEO just kind of becomes like a thing that you do and not necessarily a focus. But I think in a lot of cases, it can help to optimize tool and experiment with it. But it’s definitely not a magic bullet for a lot of businesses.

Rommil Santiago 10:27
What could I do wrong to assume that if you work on the SEO, it would improve other other things like SEM?

Justin Coons 10:37
Kinda

Rommil Santiago 10:40
Was that a tough question?

Justin Coons 10:43
It’s interesting. There’s this interplay that you see between sem and SEO that when one does well, the other one typically does well. And a lot of times it feels kind of counterintuitive, where you’re like, Okay, if I’m buying ads, and I’m in the top three, that’s going to hurt my my SEO, right? Because people are just going to click my ad. And in most cases, what you’ll see is that having that ad up there in one of the top three slots actually improves your click through rate on on your organic result.

Rommil Santiago 11:15
Interesting. Yeah,

Justin Coons 11:17
you end up getting an aggregate more traffic.

Rommil Santiago 11:19
So that, in a sense, saves you money. Because you’re not doing that, like there. If I click on the organic links, then that’s cheaper for us os ot not?

Justin Coons 11:28
Exactly. Yeah. It’s, it’s a, like a cool feedback loop.

Rommil Santiago 11:33
Hmm, this might be something I bring to my own workplace a little bit more. So, so let’s just say, you know, you decide or someone decides to run an SEO experiment. Now, like with all experiments, there’s always these things to watch out for these gotchas that some would call it, are there things that we should keep in mind as we start to run SEO experience experiments? Sorry.

Justin Coons 11:59
Yeah. So I would experiment directly in your production environment. So it’s, it’s kind of hairy and it seems kind of scary to talk about. But anytime you get into a situation where you’re using an optimization tool, like an Optimizely, or Adobe target, to update page content, or change the value of a page, that’s coming onto the page via JavaScript, and the way Googlebot handles JavaScript is kind of nuanced. And so you can end up getting kind of inaccurate or inconsistent results using, like a third party tool to do that. So I would recommend just starting out directly optimizing into your production environment. And then kind of, as I mentioned earlier, really keeping your focus very narrow on the experiments that you run. So you know, starting with a very clearly defined hypothesis, and then an expectation on the result that you would see. And then typically, within that two to three month time horizon, you should see like, either the result you were expecting or nothing. And if it’s the latter, that’s where you can revert or kind of optimize your test from that point.

Rommil Santiago 13:13
So if I understand this correctly, if we want to start doing SEO experiments, we should start have strengthening our relationships with developers, essentially, because they’re going to have to make the changes.

Justin Coons 13:25
Yes, absolutely. That’s regarding developer buy in for SEO experiments is critical.

Rommil Santiago 13:32
Well, I think that’s true for a lot of experiments. But that’s a good flag for us to to keep in mind. Now. Now, we’ve talked about the importance of it. We’ve talked about some of the gotchas, how exactly do you structure one of these things where now you’ve mentioned that we don’t have a front end tool necessarily. And search engines don’t crawl our sites consistently? How would you structure an experiment for SEO?

Justin Coons 13:57
Yeah, it’s not as cut and dry as a typical usability test. So really, it’s again, kind of defining what you’re influencing and what you expect to happen. And then from there, you can the kind of, yeah, formulate a test as a result of that.

Rommil Santiago 14:18
Okay, so what kinds of things would you test? Let’s say you’re brand new to an organization? What what are the top things that you would suggest? Or would you would yourself run tests on?

Justin Coons 14:32
Yeah, so the quick wins that I typically look at with a new organization or client are title tags are usually the easiest one. So as I mentioned earlier, this is the most critical piece of content that’s on any page on your website. And so optimizing title tags or reviewing them, is typically a pretty easy, quick win. I often like I can’t remember many times. were tested new title tags and things haven’t worked the way I expected. And then with a test like that, the two metrics you’d be looking at would be keyword ranking and click through rate. So how you write the title tags is pretty important. And even if you’re ranking in keywords doesn’t go up, if your click through rate goes up, then obviously you get more traffic. And then the second test that I usually like to run for SEO is actually looking at interlinking. So using a tool, like Screaming Frog or Botify, you can kind of count the aggregate linking towards certain pages on the website. Oftentimes, depending on how the website was structured, you’ll find that a lot of the higher value pages on the site are the ones that the business intends to be, like high value landing pages just aren’t linked as much as they should be. So that kind of presents opportunities for you to go in and kind of rearrange how this linking is structured. And that’s one where like, again, there’s there’s no easy tool to do this. So you kind of have to formulate a plan and a roadmap, take it to the engineering team. And then yeah, develop it and run it on the fly really.

Rommil Santiago 16:23
So. So you’ve talked about things we should test? Are there things we should not test?

Justin Coons 16:33
No.

Rommil Santiago 16:34
Wow!

Justin Coons 16:37
What’s what’s kind of interesting when I was thinking about, you know, the concept of testing in SEO, and there’s an SEO that I really enjoy following named Oliver HG Mason, and he kind of gets really out there, he tests some pretty interesting stuff. And he has been able to kind of work out some pretty cool solutions as a result of that. And so like, I feel like thinking that there there are any guardrails or boundaries to this maybe isn’t the best way to think of it. Because, you know, none of us outside of engineers at Google, like truly understand the search algorithm completely. So we should really kind of test your heart’s content, like, you know, within the bounds of risk, I’d say,

Rommil Santiago 17:26
That’s fair. I mean, I guess you wouldn’t be what it is you you probably could test the fonts and spelling if you were so inclined.

Justin Coons 17:35
Yeah, like, one of the recent tests that I saw, Oliver Mason had done was within Google Search Console that has this tool where you can look up information on like any page on your website, so you type in a URL, and it’ll tell you the last time it was crawled, what user agent, it was crawled with how it looked when it rendered, you know, canonical chosen and so on. But what he had found is if you create essentially a 301 redirect from any URL on your website, to a URL on a different website, you can actually spy on other other URLs out there. So like, you can check out Google bots activity on your competitors website. I thought that was kind of a fascinating test that he ran that, but you know, he publicized it, so you can’t do this anymore. But it was cool.

Rommil Santiago 18:33
Okay, so it’s time for that part of the show, which I like to call the lightning round. So this part of the show is where we ask you pretty much very fast questions and your answers are the ones that come up in your mind right away. I don’t think about it too long, you know, this more of like a knee jerk reaction. So first question, Bayesian or frequentist?

Justin Coons 18:57
Frequentists. Keep it simple.

Rommil Santiago 18:59
Interesting. Often, you hear often you hear Bayesian and, and I like to get the the frequentist supporters and the Bayesian supporters in the room, and I like to have them have a conversation and just stand back. It gets pretty. They get very, very passionate. Okay, so if you couldn’t use Google, if you couldn’t use Google, which search engine would you use?

Justin Coons 19:22
I’d say for its privacy capabilities. DuckDuckGo.

Rommil Santiago 19:27
I’m gonna have to check this out. Long Form versus short term, sorry, long form or short form content.

Justin Coons 19:34
Long Form, it’s almost anytime you need to rank for an important keyword get ready to write 4000 words.

Rommil Santiago 19:43
Oh, 4000 words.

Justin Coons 19:45
Maybe not that many. But yeah.

Rommil Santiago 19:48
That’s actually a that’s an interesting point to just jump off there because you often hear you know, just market market like, what do they call the those who watch football games, but sit In an armchair the armchair quarterback. Those types of marketers say that, you know, there should be only three clicks and content no one reads on the internet. So you’re saying long form content could be very beneficial.

Justin Coons 20:13
Yeah, I think in a lot of cases, long form content typically wins out over short form content.

Rommil Santiago 20:22
Okay, moving on to something less SEO, your favorite basketball team and why is it the Toronto Raptors? No, but honestly, which team are you rooting for?

Justin Coons 20:36
Oh, gosh, it’s it’s the Denver Nuggets.

Rommil Santiago 20:40
And I respect that.

Justin Coons 20:41
It’s been rough.

Rommil Santiago 20:44
They got a couple of folks there who are pretty good, but yeah, they need a little bit of support. Yeah, I follow them and they’ve had better days – but so have the Raptors, so it’s gonna be one of those seasons. If you were a webpage, what terms would you optimize for?

Justin Coons 21:04
Cool guy.

Rommil Santiago 21:07
I want around 4000 words on cool guy. Yeah. Anyway, you. With that said. Thank you for coming on the show. A I want to give you an opportunity to plug in plug anything that you want to? Is there anything that you’d like our listeners to know?

Justin Coons 21:23
Yeah, nothing major. Check us out at Twilio.

Rommil Santiago 21:26
I’ve used them before us, and it’s a very good service. Alright, thanks a lot. Have a good one.

Justin Coons 21:31
Thank you.

Rommil Santiago 21:38
And that’s our show. What did we learn today, we learned that not only can you run experiments for SEO, you should run experiments for SEO. In fact, experiments are at the heart of it. We also learned not to believe in the armchair quarterbacks, your content doesn’t always have to be short. And finally, we learned that Justin’s favorite teams is the Raptors, or maybe it was the Nuggets, but who’s keeping track? So if you liked this episode, and you think we deserve it, please consider subscribing and visiting us at experimentnation.com. And with that, thank you for listening. Until next time.

Transcribed by https://otter.ai