How to be good at coding A/B tests with Jeroen Wiersma

Also available in audio format:

Transcript

Jeroen Wiersma 0:00
What I see a lot of companies doing is that specifically, the more really complicated tests where you have a lot of data and it’s really surface side is fine. But on 10, they also run blindside test next to that, that are less complicated and simply more cost effective to implement it yourself. So even as a mature company, why not go the cost effective route? So I definitely both both have their ways. And I really think it depends on the company. It’s a client side testing is simply more cost effective and server side testing is more expensive with a little bit higher quality, a little bit lower risk.

Richard Joe 0:43
Hey, it’s Richard here from the experiment nation podcast. And today I’m going to talk to Jereon Wieresema from Klein, NL going to talk about how to code your own AB test itself, the benefits and learnings from it, and some of the common tests that falls and hints and tips that can make you a good coder with testing. So buckle up your seat belt. Let’s get into it.

Richard Joe 1:07
Hello, people. it’s Richard here from Experiment Nation. And today I’ve got Jereon Wieresema who is from Klien NL Klien’s, an ecommerce website based in the Netherlands. He is a product owner. But a few years ago, he started off as a front end billables I believe it was a junior front end web div, then you move into a CRO role. And then you finally made it into a product manager role. So welcome to the show you Jereon.

Richard Joe 1:40
Well, thanks. Thanks. Happy to be here. Awesome to meet you and talk about CRO. Yeah. So look, it’s great to connect with you. Finally, it’s, you know, you’re definitely a very technical kind of person. But it’s good to have a technical person on the show. So tell me how you got, you know, started in CRO, what how you evolve from being a front end web dev at Klien and then evolved into being a CRO and in how did you get involved in doing your own ABTS? Because, you know, quite often it’s, it’s jumping in the deep end to go and sit up tests, let alone, you know, rely on other people and agencies or other divs to to develop tests for you and you just have a strategy. That’s usually the situation and quite a few companies. So yeah, tell me your evolution and how you how you go from there from there?

Jeroen Wiersma 2:41
Well, when I started the line, there wasn’t really any CRO or there. They had done a couple of AV tests in the past. And I was just finishing up my study. So I just started with the I did a thesis about mobile usability. And there was some there was an advice in that thesis of doing some tests, but there was nobody to do it. So I had some front end developer skills. So I was like, okay, maybe I can just try it myself. It were the best test, by the way, babies chasing better colors.

Jeroen Wiersma 3:18
But it’s just just to get started. So yeah, and from then I just, I did for like three years. So three years, just running on and on tests getting better at it, and

Jeroen Wiersma 3:33
just learning how to implement test better as well.

Jeroen Wiersma 3:38
Because there was wasn’t any other specific zero version around. I had to do, from research to implementation to evaluation. Yeah, so the whole process was with me, I did have a little bit of support from an outside agency once they helped me a little bit with a one day course, like, Hey, this is how you do CRO. Yeah, so it was like a, like a workshop like, this is the end to end process from, you know, looking at the data, qualitative and quantitative data, then developing hypotheses and doing it doing it tastes, you know, says prioritization and roadmap and those sort of things. It was like, it wasn’t like a crash course. And CRO you fine? Yeah. Yeah, pretty much it helped me with with the most important foundations of CRO and the things you should look at and the mistakes, the common pitfalls you don’t want to fall in. And of course, zero is such a big subject. Yeah, you can learn it in a day. It definitely helped me get started. Yeah. What’s What’s some of the

Richard Joe 4:47
common pitfalls that you made? I mean, because I’m thinking a lot of our readers or listeners here would be interested in knowing about that because it’s quite common. Well, it’s quite it’s it’s a new

Richard Joe 5:00
easier for someone to both be doing strategy and execution at the same time. So how did you? How do you do that? How did you deal with it? And yeah, what what are what are some of the common things that you what issues that you came across? Well, one pitfall is don’t do strategy and implementation on your own.

Richard Joe 5:22
Objective isn’t that it’s hard to give the high level and get into the weeds

Jeroen Wiersma 5:27
of doing things. It just, it just isn’t optimal. Just you can’t you can’t run quality tests time after time, every two to three weeks while you’re trying to do research, implementation evaluation all on your own. You need a team. And that’s also what pipeline personnel we, I advise them, okay, I’m gonna go as a product owner, I’m still going to be involved with the building of the application. It’s, I’m gonna say, Okay, let’s get an agency.

Jeroen Wiersma 6:01
Specifically, this one was online dialogue from Phil messily. Yeah. And, and they, they are doing the full process now. And I think it’s going a lot better now. So

Richard Joe 6:13
a bit pitfalls. Don’t do it all on your own. And I think that’s a big one. Yeah. So you look good. Back to coding? What are the sort of because you know, you’ve done coding yourself on the test that you also strategized for a while. So what’s the main benefits you think CRA would would have from having some basic coding knowledge? Not necessarily that they that they would be full on coding during tests? But yeah, what are the some what are some benefits from having some coding skills? I think.

Jeroen Wiersma 6:47
I mean, the Deaf markets, currently, everyone’s looking for developers, is a problem with lack of developers. And what I experienced within my company, is that a little bit of experience in coding goes a long way, because it enables you to run a lot of the tests yourself, I think,

Jeroen Wiersma 7:09
with if you know, a good knowledge of HTML, CSS, and JavaScript, should be able to do 80 to 90% of the AB test on your own. And this means that if you are able to implement these tests, you will be able to run more tests, when you run those tests, the development team only has to make it when it’s a winning test. Yeah, that’s definitely an interesting one. So it’s very cost effective, it’s very independent, you will be able to, especially in a small team, you will be able to do more, you don’t need to wait for development to put on the new builds. And okay, this is now on the website. And oh, wait, the test is broken. But we need to wait another week, because the next build is there really makes you flexible. It really puts CRO more on the marketing side of it. And next to that, if you know a little bit of coding, it goes beyond AV testing. Because everything these days, especially if, for me, I’m working on my retainer, it’s all about working with a webshop. So you probably have to work maybe with Tag Manager or some other fancy tool.

Jeroen Wiersma 8:27
And, and then you also know those basics. So it goes

Jeroen Wiersma 8:32
just goes a long way to know a little bit of front end skills. When you’re working with websites. Definitely. Yeah, I completely agree. gear review, I mean, both in terms of,

Richard Joe 8:44
you know, not having the DMS as a bottleneck, especially in like, organizations where CRO is quite new. And you’re you’re the only CRO you don’t have a fancy team with an analytics guy, a dev, a dev guy on your team, like it’s just you. And I’ve been in those situations where you, you’re waiting on the dev, that’s not their job, they’re busy building the website or doing other things. And then your thing is like maybe a little favor that they’re doing for you. And it’s something you’re waiting for, like, you know, you could be waiting weeks or months or things to be put into production,

Richard Joe 9:23
let alone implementing a winning test. So being able to go there and jump into the code, and maybe only using the developer of I don’t know you’re stuck or you need someone to QA it or implementer winning tests, and that that puts a lot less pressure on them.

Richard Joe 9:41
But yeah, I think I agree with you on the other parts of it’s not just the AV testing, but it’s has cross sort of

Richard Joe 9:50
on a cross platform or has, you can sort of go laterally into like other things like GTM because, you know knowing

Richard Joe 9:58
knowing how to like

Richard Joe 10:00
I don’t know,

Richard Joe 10:02
track a button clicked by AD ID or class like knowing those sorts of things, it’s requires a little bit of Frontier knowledge. Right. So

Richard Joe 10:11
like, completely agree and it sets sets you up for a good sort of technical foundation of actually known how websites work?

Jeroen Wiersma 10:21
Yeah, definitely. I would say, I wouldn’t say that.

Jeroen Wiersma 10:26
Developers wouldn’t be able to do a better job than Oh, yeah, I, I think that that is definitely sometimes the case. But then you would need someone that is specifically assigned to building those steps, I think that is very, that would be the end goal. That was my end goal when working with line, just introducing Sierra or getting those first ask me was first when getting people excited. And then it’s, will be okay, maybe we need to take that next step and make it a little bit more professional, get a dedicated, test implementer, or work with an agency, I think that’s definitely the goal you’re working towards, if you’re introducing CRO, and, and doing a little bit of coding definitely helps you with reaching something like that.

Richard Joe 11:14
And I guess I guess it’s sort of, part of it’s the acting as an evangelist, because if you can go and

Richard Joe 11:21
do some strategy and code or code, a few tests here and there, get a few wins, then that shows a competency internally in our organization. And you can kind of you can

Richard Joe 11:33
be in a it’s easier to be an evangelist in that sense, because you’ve done it all. But you’ve done all that stuff, you know.

Richard Joe 11:40
So you can create can go a long way. And, you know, obviously, for you, you’ve already moved into a product manager role. And look, just as a side topic, you

Richard Joe 11:53
as a product manager, I believe they do some product managers do a B testing and get into those side of things. Is that something that your store evolved in a hands on basis, or just sort of an overview?

Jeroen Wiersma 12:07
Currently,

Jeroen Wiersma 12:09
since the agency took over, we want someone else a junior person in that role, and someone else’s managing that person, but I’m still available for expertise, in that sense, and also if people need help implementing, and specifically, when tests are one, they gotta be building the product roadmap. So So in that sense, I am definitely still

Jeroen Wiersma 12:39
still in the process.

Jeroen Wiersma 12:41
Because it I mean, a product owner should work with a CRO person. Because, yeah, that roadmap of, hey, where we’re gonna go with our products should also be based on research. So I’m still heavily involved in that sense, but not like with the hands on stuff and not the hands on coding stuff. Yeah, yeah. What’s, what’s the interest? Just going back to, you know, the benefits you found with coding? What What’s the main sort of?

Richard Joe 13:09
What would you say the benefit of custom coding a test is compared to using and relying on the WYSIWYG. Now, for our listeners out there

Richard Joe 13:19
was the wig is just basically

Richard Joe 13:22
basically the, the visual component of, of the website where you can

Richard Joe 13:29
vote much coding knowledge, you can just, you know, just change like, the CTA button or whatever, using a tool like VWO Optimizely. I think the acronym stands for What You See Is What You Get, right? Yeah, that’s correct. Yeah. Yeah. Yeah. So good. Yeah. What’s the what’s the main benefits you found?

Richard Joe 13:51
Actually going there and custom coding on our platform like me, who are optimized to optimize Google Optimize compared to rely on the WYSIWYG? Okay, well, I actually, when I really started, I also used it, which is what you get editor, myself. And

Jeroen Wiersma 14:12
what I have happened is that a lot of times, the more complicated tests, I would not be able to implement them with only this piece of tool. Or

Jeroen Wiersma 14:24
if you would have, like a dynamic webpage, and you would want to change it with a simple visual editor would not be possible either. And stuff would break. So I would implement the test and in the Visual Editor, it looks fine. I’m like, okay, ready to go. And no, it doesn’t work in on the real website, because it’s changing. It’s,

Jeroen Wiersma 14:49
I mean, sometimes it might work, but it’s very limited.

Richard Joe 14:53
Yeah, I don’t know technically, why it breaks but I’ve encountered this because I was

Richard Joe 14:59
always I do

Richard Joe 15:00
I was putting in some sticky elements on a website was making it sticky, but using the WYSIWYG. And it looks fine. Like, you know, when you go to Preview mode, but then when you go to like, say, You go incognito in Chrome and you you go back on the video variation page, it’s like all distorted and, you know, different browsers make sure you made sure the variation differently. So

Richard Joe 15:25
would you send a learning lesson here is the WYSIWYG is good for very basic tests, like you want to change

Richard Joe 15:32
the CTA button or an image or a headline, but not for

Jeroen Wiersma 15:38
more complicated tests and dynamic websites? Yeah, I think so I think for four days in Texas grades and maybe also some little bit of CSS, you can also use it. And I think if you have knowledge of coding, you are able to use this WYSIWYG better, because there’s also some, some nuances in when things trigger and when you should implement something. So

Jeroen Wiersma 16:01
this also helps you understand that a bit, definitely for the for the more basic test, you can use that if you want to get a little bit more complicated, maybe put in a specific rule or it’s dynamic webpage where the page might change depending on where it is, you definitely want to go, which

Richard Joe 16:20
now that’s that’s, that’s that’s a good point there. How would someone who’s new to coding

Richard Joe 16:27
that’s listening to this podcasts? Or maybe they’d been a strategist for a few years? And they don’t really think they need to know coding or know very little? Like, how, what’s a good way for them to learn basic HTML, CSS and JavaScript? Like, do you know, do you know any websites that

Richard Joe 16:44
you’d recommend?

Jeroen Wiersma 16:47
Well, if you really want to if you really have no knowledge of HTML, CSS, and JavaScript, I, when I was in school, I always used the book from John Beckett. HTML CSS is awesome. It also has you have to, but this is not specifically cater to AV testing, right. In general, you also have W three schools. Yeah, it’s also one, which is nice. But I think the most important one, if you really want to learn it, you need to actually try to implement it with something and use it because otherwise, you just remember. So if you really want to test it, maybe how I learned it is making your own portfolio websites with just HTML, CSS, and looking up some resources in there, you actually get something out of it. Let’s just move on to like sort of hints and tips we can talk about in regards to building out building our tests. And, you know, we talked about flickering and how to avoid it. So could you explain a bit a bit more about that?

Jeroen Wiersma 17:50
I think

Jeroen Wiersma 17:52
in order to avoid flickering, specifically, if you would do it with JavaScript, there’s always a specific way of doing that. And it’s called, I’ll try not to get too technical, it’s called a mutation observer. So don’t get scared. And mutation observer is simply like, a robot looking at a webpage. And he’s just waiting for that specific button to load. And then as soon as it loads, bam, it runs the code. So that’s a mutation observer. So this in this way, you make sure when something loads, you instantly change it. Usually, it’s even faster than before it’s visible. And this way, the user does not see flickering, the mutation observer itself is a little bit tricky. To get a hang off, in my course, I make an entire chapter about how to actually use it, and how to change the

Richard Joe 18:42
parameters in it. But if you master that little thing, you should be able to run tests on any website without flickering. So just to clarify, the mutation observer is just injecting this light the code into the DOM, like, as soon as an event happens, or

Jeroen Wiersma 19:02
it well, it’s pretty much in the name. It’s observing the loading of a website. And did you tell you tell it hey, I want to add to when this specific element comes up, yeah, I want you to run code or do something and you can write that yourself. You can. I mean, that’s usually where I wrap the changes in.

Jeroen Wiersma 19:27
So if I would want to change the color of a button, for example, I would have one line that does that. But I would always wrap it with a mutation observer, because then it runs when the elements appears. Otherwise, what usually happens is a whole web page loads. Yeah, then the experiment applies. And then then you get this flickering effect and then to the delight sort of effect where it’s sort of like you like in the observer just sort of knows, like, half a second something’s not quite right. Yeah, exactly.

Jeroen Wiersma 20:00
If it will just make sure that

Jeroen Wiersma 20:04
your user doesn’t see both variants. So it’s a way of it.

Jeroen Wiersma 20:10
Very important way of preventing flickering or specifically with blind sight.

Richard Joe 20:17
And what about things like test pollution? I did see on your course description you talked about there, can you can you please explain to the audience what test pollution is and how you can avoid that through for coding?

Jeroen Wiersma 20:32
Well,

Jeroen Wiersma 20:34
so test pollution, specifically is, say you’re running a test on the product page, and you’re making a change on the product page. You want to test something when a user presses the app department? Yeah. So if you’re, if you would run that. So if you would track every user that would go on that product page, and then you would look at the results. And

Jeroen Wiersma 21:01
that only only the sessions where they press the Add to Cart are relevant.

Jeroen Wiersma 21:06
That’s what you’re testing with. Sometimes you don’t press the add to cart, but you’re still checking it. So you’re getting a little bit of test pollution, or maybe not pollution dilution. So

Jeroen Wiersma 21:17
this makes your the impact that you need

Jeroen Wiersma 21:21
a lot. You need more, because you haven’t, you have more sessions. So you need to make more of an impact. Because your, your your actual change isn’t with every session that is in the test. So you’ve diluted and you’re there for your experiment be elongated, the generation will be elongated. Yeah. Yeah, exactly. Yeah. And with coding specifically, where I would use it for and which is also my work.

Jeroen Wiersma 21:50
Because you can use JavaScript specifically in Tag Manager, you are able to activate an experiment exactly at the time that you want. So what I would do is I would write some code in with Google Tag Manager, I would say, hey, when someone presses an Add to cart on the product page, then start to experiment. So this way, your test isn’t diluted by all those other sessions that are not relevant to text. And this is something you could very easily do if you do some JavaScript.

Jeroen Wiersma 22:23
And you don’t have to do with the Tag Manager, but anywhere where you can just run a little bit of JavaScript before the test runs.

Richard Joe 22:30
And you say you, you do explain this in your course. Yeah, yeah, I have

Richard Joe 22:38
over a whole lesson about specifically debts, and they’re there I do with the Tag Manager, okay, just just tell people to how to activate an experiment based on that’s very useful. And look, lastly, just talking about, you know, server side versus client side testing, I suicides become the whole rage Now, last few years, it’s not something I’m personally familiar with, you know, we’re more familiar with clients, it seemed like most listeners here, you know, like VWO, Google Optimize, we just basically just eat up the DOM, comments on suicide, any pointers from guides to coding,

Jeroen Wiersma 23:18
things you got to be aware of most tools these days, usually also have a surface site, just look in their documentation, they will have a way to implement the certify test, because all those because it’s the heart rate, they want to get into that, but I think Surfside testing, and we were talking about specifically flickering and how to avoid it with client side. It is patient observer, okay? Server side testing is really,

Jeroen Wiersma 23:48
you don’t have any flickering, because you’re running the test from the server. So

Jeroen Wiersma 23:54
to explain it, non technically, you have a browser, it loads the page, and it makes changes. So what happens with Surfside St.

Jeroen Wiersma 24:06
before it goes to the browser, it’s already changed. So you can never see the flickering. So

Jeroen Wiersma 24:13
that is one of the main upsides of server side testing.

Jeroen Wiersma 24:18
The thing I do not enjoy, and why client side testing is very much relevant. And a lot of people do with more people and Surfside is, only a developer can really do run service like this, because you run from the back end probably need to be a back end developer as well. And yeah, work together with the team. But I think in like bigger organizations with a big team that wants to run multiple tests, and I think building a sort of check tool or using one and implementing all tests that way. It’s expensive, but it’s definitely good for your test. The quality of your test increases. Yeah. And would you say that that’s something that no

Richard Joe 25:00
organization would do

Richard Joe 25:02
as they mature over over the years, like, I mean, you think quiet cyber be something you do, like when you’re first starting out, because it’s so much more accessible and easy to do, even if you’re not, you don’t know how to code you can just use it wasn’t work, whereas suicide you like it sounds like it’s something it’s kind of like for more mature

Richard Joe 25:24
CRO organizations, you agree with it?

Jeroen Wiersma 25:29
It’s usually what happens. I think what goes in tandem with that is that it moves.

Jeroen Wiersma 25:37
It’s a little bit of professionalism in the developer side, because when you run it surface side, it usually gets, I mean, we talked before about the downfall of using a developer, you need to wait for them in production. But if that developer actually has the time for that, and is well integrated in the dev team is working with zero team, and it can go along with that velocity of testing. Yeah, it really works very well. And I think what you’re saying with the maturity of the company, with the maturity of a CRO team, I think that’s definitely a way to go. I wouldn’t say that client side testing isn’t something

Jeroen Wiersma 26:18
mature companies would not do. Because

Jeroen Wiersma 26:22
what I see a lot of companies doing is that specifically, the more really complicated tests, where you have a lot of data, and it’s really surface side is fine. But on 10, they also run blind sight test.

Jeroen Wiersma 26:36
Next thing that that are less complicated, and simply

Jeroen Wiersma 26:40
more cost effective to implement it yourself. So even as a mature company, why not go to cost effective route?

Jeroen Wiersma 26:50
So I definitely both both have their ways. And I really think it depends on the company, it’s a client side testing is simply more cost effective. And surface side testing is more expensive, with a little bit higher quality, a little bit lower risk.

Richard Joe 27:07
So maybe service side is like driving a Ferrari, whereas, you know, client side might be like driving a Toyota Corolla, or like it does a job and

Jeroen Wiersma 27:18
it’s fine. It’s not, you know, there’s not going to drive, it’s not going to go from zero to 100. In like three seconds. Yeah, exactly. I mean, it’s obviously it’s better if you have the resources and money for it. And yeah, the actual changes that, like the, with the T Excel course, it talked about how you can predict how much revenue as revenue you can get. I mean, if that supersedes the extra cost you need as a development team to run such a process. And do it, I mean, it’s probably better if you’re really big organization, but if you just have come to the benchmark of oh, this is gonna be viable, you might want to start out with blind sight testing with a little bit of more, less expensive to and just get started with some of the more basic tests, and then maybe there are some coding, and then get a little bit more advanced and get your first wins, and then move on to making it better and better. And like you said, like, you know, starting off with doing easy tests, using maybe free tools like Google Optimize, you can sell the CRM program better to the organization, then once you get those early wins, hopefully, you know, we’ll be able to

Richard Joe 28:29
initiate some change management further down the line to

Richard Joe 28:33
move to suicide, if it’s something that suits you.

Richard Joe 28:37
Cool. So look, it’s yeah, we’ve we’ve had a good chat about

Richard Joe 28:42
how you started to doing

Richard Joe 28:45
abtc. And as a CRO, some of the common pitfalls, when we get involved when when they start out doing coding. And you know, some of the hints and tips including server side and

Richard Joe 29:00
how to prevent flickering and those sorts of things. Listen to your own. How can people contact you off

Richard Joe 29:06
out there in the big wide world?

Jeroen Wiersma 29:11
LinkedIn, definitely just type my name, usually via SMA, or LinkedIn, you’ll find me you can connect me with me, you can also go to my website. Yep. Which is a little bit hard to pronounce with your good wiersma.com. And I’ll make sure to put it in the comments somewhere if it’s posted. And

Jeroen Wiersma 29:31
you can contact me via there, or

Jeroen Wiersma 29:34
that’s usually the way that people just send me an email

Richard Joe 29:40
for a conversation. That’s cool. And a little bit of a shameless plug.

Richard Joe 29:47
You got a course here. I think it’s really good. Oh, actually, I’ve been meaning to like pay for the course myself. I’ll do it straight after this podcast. But yeah, can you tell our listeners Yeah, you little You

Richard Joe 30:00
what your course is made and how it’s going to benefit them?

Jeroen Wiersma 30:04
Well, I think

Jeroen Wiersma 30:05
my course is specifically for a conversion optimizer that’s maybe running in a marketing team. And it’s just simply wants to run their own tests. So basically, the person that is now using the Visual Editor, and you just want to be able to do more, by simply

Jeroen Wiersma 30:23
following my course, it’s not even that long, you’ll get the basics of using a Yahveh script with HTML, CSS. And you’ll make, make sure to find those common pitfalls of preventing flickering and making sure to activate the tests in the right way. And they will also get you a very basic understanding of how to use JavaScript. So it will just level up your game in online technical marketing, specifically, and I’m sure that you don’t really need a lot of technical experience. If you love a little bit of HTML, you can follow my course you should be able to run a bunch of awesome AV tests that you couldn’t do before. So it’s my course in a nutshell. Yeah. Awesome. Awesome. So yeah. Let’s see Ron, thanks for coming to the show. And

Richard Joe 31:16
yeah, we’ll see you next time, guys. Thanks.

If you liked this post, signup for Experiment Nation’s newsletter to receive more great interviews like this, memes, editorials, and conference sessions in your inbox: https://bit.ly/3HOKCTK

---