How Indeed’s Abhishek Rathore runs the world’s biggest job site’s Experimentation Program Experiment Nation

A Conversion Conversation with Indeed’s Abhishek Rathore

As more companies adopt Experimentation, they have to make tough choices in terms of technology, statistical approaches, and incentives. Today I got the chance to chat with Abhishek to learn about how he runs Indeed’s program?—?how he measures its performance and promotes a culture of Experimentation.

Rommil: Hey Abhishek, thanks for chatting with me today!

Let’s start things with you sharing a bit about your journey and what you do today?

Abhishek: I was the first Product Manager in the eCommerce business of Snapdeal responsible for building the Buyer Platform including Search and Discovery. I, later on, worked on Advertising and several other initiatives at Snapdeal and it was a fulfilling journey of seeing a startup grow from nothing to at one point of time 6 Billion USD+ valuation, 7K Employees and myself leading a team of over 30+ PMs and Analysts. I then moved to Rakuten to lead the ECommerce Search Platform?—?working with multi-country teams in Paris, Boston and Tokyo.

At Indeed, I lead the Experimentation Science and Platform team making sure we validate thousands of hypotheses quickly and reliably to ultimately “Help People Get Jobs”!!

So, what does the Experimentation Science and Platform team oversee at Indeed.com?

We are responsible for everything related to Experimentation at Indeed. We divide our roles into 3 pillars: People, Process and Product.

People: We ensure that the right set of training and learnings are available for anyone who runs experiments at Indeed which includes PM’s, Engineers, Analysts and Digital Marketing specialists.
Process: We ensure that the right set of processes are followed while creating and managing experiments
Product: We have our in-house tools to ensure quality decisions are made faster through experimentation

Very cool. That’s a lot like what the Experimentation Team does where I work

I noticed that you have moved from India to Japan. Do you notice any difference in mentality towards Experimentation between the two cultures?

Japan has always believed in the concept of “Kaizen”?—?continuous improvement. So, in that sense experimentation is much more ingrained in the way we work here. However, overall it’s the maturity levels of the organizations that differ. In India, where I worked, the internet companies were still in its infancy and in the 0–1 mode, hence it was all about building foundational stuff and making sure we have a business.

A common question for companies is whether to buy or build an Experimentation platform. What do you folks do? What prompted that decision?

All our Experimentation infrastructure is in-house. It’s something that is ingrained in how we have built our company and allows maximum flexibility in how we operate. No third party solution was meeting all the criteria to help solve our needs because of the scale at which we operate.

Changing gears to results analysis. What’s your opinion on Frequentist vs Bayesian? Which approach do you prefer?

We use Frequentist at Indeed and have not seen a major reason to shift our thinking. Bayesian has its set of advantages, however, I do not believe that it can be a big value add for us at this point.

Indeed.com is a very popular resource for job seekers and employers, so I imagine there is a good number of Experiments going on at any given time. How do you go about managing Experimentation interaction?

Frankly, we do not at this moment!! We assume the SUTVA assumption holds true for our experiments and analyze accordingly. There are some systems like Advertising experiments, where the SUTVA assumption hypothetically gets violated and we are still working towards understanding the magnitude of the problem.

I keep forgetting what SUTVA stands for. https://en.wikipedia.org/wiki/Rubin_causal_model

What’s your approach to accounting for Novelty Bias?

We have recently defined a minimum analysis window of 1 week to account for the novelty bias. We encourage experimenters to analyze experiments only after a week has passed after the experiment has started and building this as part of the platform as well.

My company also has a minimum of 1 week. This is mostly for seasonality, but it’s also partly to account for novelty.

What’s your philosophy around defining KPIs for Experiments and your Experiment program?

This is one question that I want to learn from other Experimentation leads a lot. As of now, we have defined Experiment Velocity and Quality KPI’s and focus a lot on improving them. However, my philosophy is to somehow link my KPI’s to the value generated from Experiments. I am still struggling to define good KPIs around that

I hear you. I see it as trying to measure the impact of copywriting. You know it’s contributing?—?but attributing hard numbers to it is always challenging.

With that said, our program’s performance is measured on Culture, Process, and ROI. Measurement is a bit of a slog but it’s working so far?—?however, like the rest of the company, we’re constantly trying to find better ways to do things.

What do you consider a culture of Experimentation and what do you do at Indeed to promote it?

At Indeed, we have been blessed in a way that the entire organization has an experimentation mindset. So, we have the top management support in how we operate. We highlight experimenters following good experimentation practices in an “Experimenter of the Month” program which helps people to know about interesting experiments being run by fellow experimenters.

That’s pretty awesome. Finally, it’s time for the Lightning Round!

What excites you the most about the future of Experimentation?

It’s still Day 0 in the field of Experimentation!!

If you couldn’t work in Experimentation what would you do?

I am a Problem Solver!!

Describe Abhishek in less than 5 words.

High Impact Problem Solver

Abhishek, it’s been a pleasure to chat with you! Thank you for joining the conversation.

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.