Concrete tips from CRO veteran Nermin Canik Experiment Nation

A Conversation with Nermin Canik about Experimentation

I usually chat with Experimenters at a relatively high-level, but that wasn’t the case during my conversation with Nermin. We dove deep into tactics and her best advice on how to approach common Experimentation problems and strategy development. Oh, she also shared a view of her incredible workspace?—?I know I’m a bit jealous.

Rommil: Hi Nermin, thank you for taking the time to chat! How are you?

Nermin: Hey Rommil, I’m very well. Experimenters from different countries share their passions thanks to you! I’m glad to chat with you!

Thank you for saying that?—?I’m happy to chat with you too. So let’s start. I’d love to learn a bit about you and how did you get to where you are today.

I have been working as a Conversion Optimization Consultant to improve conversion rate and maximize ROI since 2015.

Back in my career story; I graduated from the Department of Computer Engineering, Ege University in 2013. After graduation, I started to work at a bank as an IT Business Analyst. While I was working with developers & business teams to create user-friendly products. Then I decided to improve myself to create successful products. So I’ve taken dozens of different online courses after work. I will never forget one of the courses! I took the “Human-Computer Interaction” course on Coursera.

Bingo! It is the first time I’ve heard of user experience (UX) and conversion rate terms. I decided to change my career journey from IT to UX.

While I was working at a UX design agency as a project manager, we’ve been getting a lot of revision requests from clients about designs. So my ex-boss created a conversion optimization team and he offered me to the leadership of this team. I was also a conversion optimization expert because of my technical background (computer science/engineering). Thanks to this now we could say “Let’s test the idea” when a client says “I think … should be”.

Then, I started to work at a digital marketing agency as a CRO expert. I’ve planned to combine different approaches of UX design & digital marketing agencies.

Finally, I have been working as an independent consultant and trainer since 2018. I founded my own company. 🙂

Very nice. I’m always fascinated with how people end up in Experimentation or CRO. Folks seem to come from all sorts of background, but the one thing that they have in common is that they are knowledgeable about multiple disciplines. Very interesting.

So Nermin, having worked in Experimentation for quite some time now, what are the most questions that clients have?

What’s the ideal …. rate?
Which platforms should we use? How to validate the A/B testing platform quality before investment?
How long should we run an experiment?
How many visitors do I need to reach to finish an experiment?
We have a problem with the page flicker effect. How can we solve this problem?
When should we analyze the results? After 1 week?
We’re going to start a new campaign or TV ads. Does this affect this experiment’s results?

Ha. I think I get asked most of those everyday. I hear you.

How do you decide where to start in terms of Optimization? Do you have any tips?

Building a dashboard that includes micro and macro KPIs ( Using heatmaps on tables, creating custom alerts).
Weekly, monthly, quarterly roadmaps.
Creating a prioritization framework/model to score impact, cost etc. https://effectiveexperiments.com/blog/the-practical-guide-to-a-b-test-prioritisation/
Communicating with all stakeholders! Voting optimization ideas!

One of the biggest complaints that Experimenters have is that tests take too long to run?—?what suggestions do you have for folks looking to shorten this duration?

Be patient to reach statistical significance and ideal duration.

Don’t forget that invalid (false) results you get in a short time will cost you more in the end.

But we can always use these tips to optimize our test duration;

Less variation. If you test more variations in an experiment, you will need more traffic. And more traffic means more test time.
Choose the right KPIs (Especially, primarily goal). 2 years ago, a publisher told me that their A/B tests never end up clearly. I did an audit of their analytics & A/B testing platforms. While I was checking experiment settings on A/B testing platforms, I’ve discovered the problem. They chose “session” or minor KPIs as a primary goal!
Big changes for getting low-traffic websites. You’ll have to make big/radical changes instead of a minor change if your website is getting low-traffic.
Seasonal changes. You may get more traffic temporarily but be careful! Please consider paid marketing channels’ efficiency. (For example; Visitors who are coming from Google Ads GDN haven’t same motivations to buy a product like Google Ads Shopping, etc)
Check experiment’s filters

Great tips. I’m a big fan of testing big things. The number of times I’ve seen teams willing to release a big feature without testing, but yet, only want to test minor changes blows my mind.

Moving on. Another common project many companies have is redesigning their homepage?—?do you have any suggestions on how best to tackle something like this with Experimentation?

The Present Homepage (Old Version) = Playground
If you’re planning to redesign your homepage (or any page or funnel), the present homepage (original version) will be your playground. For example; Are you planning to add a module in the new version? You can test it by adding a new module in the present version (original version)
Don’t publish the new version directly. Test it! Do it step by step. Show the new version of homepage to %10 of total visitors for example. Collect data about user behaviour! Discover potential bugs and users’ problems then fix them! After you complete the fix, expand the percentage of visitors who can see the new version.
Be careful on redirect tests for SEO!
Testing platforms recommend defining the canonical URLs for variants. This ensures that your redirect page variants won’t have an impact on your original page’s SEO rankings. Communicate with the SEO team.

That’s a great tip regarding canonical URLs when working with redirects. Love it.

There are so many skills involved with CRO and Experimentation, what is the hardest part of becoming proficient at Experimentation?

I think the hardest part of our process, building a culture at the company! CRO consultants or experimenters should have good communication skills. For example; I often discover great optimization opportunities or A/B testing ideas while talking with a call center department.

And platform/technology is changing quickly. You always need to update yourself! I believe specialization in an area is very important but also you should learn a bit about other expertises in digital marketing. For example; Google has announced to remove Google Shopping from Google results in Turkey because of legal competition law. As a CRO consultant, I don’t manage paid marketing but I have to know details of this update. Because users’ journeys will be changed and it will affect conversion rates.

Finally, it’s time for the Lightning round!

Bayesian or Frequentist?

Bayesian

How do you say Experiment in Turkish?

“Deney”

If you couldn’t work in Experimentation, what would you do?

I have a data-driven lifestyle. I always collect different data about myself. For example; sleeping/walking time (Mi Fit app), working time (Toggl extension/app), daily calories (Yazio app), networking activities?—?last meeting time/new connections (Google Spreadsheet), daily emotion score (diary notebooks), financial dashboards (Google Data Studio, Parasut). I’m a data addict!

If I couldn’t work in experimentation, my alternative job will still be related to optimization. I may be part of the optimization team in IT, Finance, Sales, Marketing, HR or Call Center department, etc.

“You should be a lecturer/teacher”?—?Some of the ex-coworkers and my training participants/clients often advise that because I love teaching! Everyone has a tactic for learning. Mine is teaching.

The methods I use when I want to learn a new technology/technique in the fastest and most permanent way:

First step: To start a project
Second step: To train someone else about the topic.

If you’ve ever taught a class on Experimentation?—?you’ll appreciate this diagram.

What tips do you have for those interested in getting into Experimentation?

Start now, not tomorrow 🙂
Measure results on both Analytics and Experimentation platform
Be careful of wholesale sales results for measuring conversion rate!
Don’t make radical changes on the website while an experiment is running
Learn Analytics, UX Research & Design, Coding, Data Visualization etc
Specialization!
Communicate with other departments (digital marketing?—?paid marketing, SEO, programmatic, content, IT, call center, sales etc) for great optimization ideas.
Following sector professionals on Twitter/Linkedin/Medium etc.
Join Slack channels & meet experimenters to motivate yourself
Set your goals (For example; online courses, blog posts)

Why am I not surprised that an Experimenter has goal-setting as a tip? 😉

Describe Nermin in 5 words or less.

Organized, Hard-working, Energetic, Passionate, #whereisnermintoday,

We all can be a little jealous of Nermin

Thank you, Nermin for joining the conversation!

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.