Rommil from Experiment Nation: Could you introduce yourself, where you work, and where you are from?

Luka Nikolic: Hi! I’m Luka and I work as a CRO analyst in a Danish iGaming company called Better Collective. I live and work in a city called Niš which is located in southern Serbia where I am from.

I've never been to Serbia, Google Photos suggests that it is a beautiful! That said, tell us a bit about your career path and how you got to where you are today.

Having studied business at university, I spent the first few years of my career doing sales and marketing in the manufacturing industry. I then discovered and fell in love with UX design and decided to fully transition into the world of digital. My first role in that space was a junior product management position in Horisen, a Swiss B2B SaaS company in the SMS marketing niche. This was really where I cut my teeth, so to speak, doing a lot of stuff like UX writing for really complex software products and the like. During that time I also found out about web analytics, CRO and experimentation. The multidisciplinary nature of the field is what initially drew me to it and I haven’t looked back since.

As an analyst, how do you go about deciding where to run an Experiment?

Typically, the pages I work on have a lot of paid or organic traffic but relatively low conversion rates. These are often the top candidates for optimization, though some of the more recent experiments I’ve run were more geared towards learning than increasing conversions per se (e.g. testing video as a content medium purely to understand how visitors interact with it on certain page types, how it affects their behavior etc.). It really depends on the scope of the project but as a general rule of thumb, wherever we can obtain the most scalable insights on a product portfolio level, that’s where we’ll run an experiment.

Qualitative and quantitative research are very important to Experimentation. How do you recommend to our readers that they leverage them?

In combination, of course :) Not triangulating one type of data with another is a huge waste of opportunity given that they’re so complementary in nature. I know that what I’m saying is nothing revolutionary but somehow I still come across these online articles along the lines of “Should you do quantitative or qualitative research?” every once in a while. It’s a false dichotomy and you should always do both because that’s the only real way you can uncover any meaningful insights.

100%. So many companies don't actively try to connect the dots. Such a missed opportunity.

At the heart of Experimentation, lies strong hypotheses. How do you define a strong hypothesis?

Everything I’ve learned in this industry, I only know because I’m standing on the shoulders of giants. People like Craig Sullivan, Lukas Vermeer and Molly Stevens who have been in the trenches for years are all to thank for the fact that we now have a very practical framework for creating strong hypotheses. It’s the soundest and most comprehensive approach that I have come across just yet, which is why this particular format best fits my personal definition of what it takes to build a strong hypothesis.

What do you include in your communications to maximize the learnings from Experimentation?

Following up on test results with recommendations for next steps is really important in this regard. I find that especially in organizations where experimentation is a relatively new concept, there is a tendency to stop iterating immediately after testing the first version of the hypothesis and just move on to the next thing. If your variation won, why not try and go beyond to make it perform even better? If it lost, why not try to uncover what caused it to perform worse by testing a better hypothesis than the original one? So it’s important to include any information necessary to keep this cycle of experimentation going as opposed to giving up on the very first try.

"...in organizations where experimentation is a relatively new concept, there is a tendency to stop iterating immediately after testing the first version of the hypothesis and just move on to the next thing." - Luka Nikolic

Tweet

You recently posted about obliquity, could you explain what that is and how it relates to Experimentation?

It’s a concept by John Kay, a distinguished British economist who defined obliquity as the best way to achieve complex goals, which is indirectly. How this is done is by going in the opposite direction than the one you want to go in, as he puts it. Relating this back to experimentation, if you want to boost your conversion rate or improve your product, the best way to do this in the long run is to “fail” (quotation marks left intentionally) a lot in the short run so as to maximize the number of learnings that you will later capitalize on. But it is impossible to get to this later stage without having gone through all the painful insights gathering first. “Winning through failing” would probably be the best way to put it, and while it’s not a novel idea, I feel like it’s worth reiterating in the world of CRO where there is often a focus on short-term results at the expense of long-term business health. And ultimately, experimentation should be a way of making business decisions, not merely boosting online sales because that would just be selling it short.

You recently mentioned that one of the best things of being a CRO is being specialist-generalist could you explain that to our audience?

Of course, what I meant by that was that it allows people like me who have a severe case of “work ADHD” to work in a number of disciplines that all fall under the same umbrella. Whereas most disciplines reward specialization, a good CRO has to be adept at a number of skills to be able to do their job well. In that sense, I’m in love with the multifaceted nature of the job and I enjoy the fact there’s always something more to learn, whether it be copywriting, statistics, user research or something else. As a career, it’s the proverbial gift that keeps on giving, if you will.

It's time to change gears. It's time for the Lightning Round!

Describe yourself in 5 words or less.

Curious, driven, thorough, empathetic, sociable.

Frequentist or Bayesian?

Depends on the case at hand but I haven’t had much exposure to Bayesian so far so I guess I’ll have to go with Frequentist.

If you couldn’t work in Experimentation, what would you be doing today?

Touring the world in a rock band, without a doubt. As soon as COVID has passed, that is.

What do you have going on that you feel our audience should know about?

I’m compiling a book of all the amazing CRO and Experimentation posts I see on LinkedIn every day. Unfortunately, that is a joke but I do wish such a book existed or that many of the experts that I follow online would compile all their daily insights into PDF summaries or something of the sort.

Well, that's an interesting idea! I hope one of our readers can figure something out.

Who should we interview next and why?

Dennis Meisner from product experimentation at Facebook! I find his writing style very clear and engaging and it would be a pleasure to hear more from him in Experiment Nation.

Awesome! Thanks for chatting with me today!