How many tests should you run per month?

Collin Crowel recently asked the community, “Is test velocity overrated?“. Where test velocity is the number of tests launched over a period of time. About 70% of respondents said “False” and I’m here to tell you that I’m firmly in the “it depends” camp”.

Generally speaking, the goal of any Experimentation program should be to learn. I encourage you to question those who say otherwise. Understanding a problem space, helps you ask better questions, which helps you run better experiments, which helps you iterate faster, and which helps you achieve your goal more efficiently. One way to think about this is to pretend like your problem space is a math equation, and your job is to figure out the answer – except you don’t know what the equation is. Unless you start learning what’s the right question to focus on, you’re only taking stabs in the dark. With that said, yes. You can brute force it. You can test quickly, but it is almost always the least efficient way to achieve your goal. 

The other problem with focusing on test velocity is people can easily game it, especially if they’re incentivized to. If you goal someone on test velocity, test quality almost always drops. This is when you get into testing button colours and other low-priority nonsense. Test velocity is a vanity metric. No scientist looks at a problem and glorifies how many tests they ran – nor should you….usually.

Now, if you recall, I started off by saying “it depends”. This is because there is an exception to this rule. When you’re starting off a program you have other considerations. When folks are not familiar with A/B testing, or research, or analyzing results you have to put in the reps. Just like when exercising or learning a new sport. You have to build that mental muscle memory so that you can focus on the more important things, rather than worrying if you got the mechanics right. It’s only at this very early stage, does velocity matter. You have to run a bunch of tests to see if the pipes are working, to practice your math, and to get used to the data. Once you start to feel comfortable, that’s when you should quickly move to focusing on something other than test velocity.

To add, focusing on test velocity at the beginning of a program is also beneficial when trying to show progress to senior leadership. Test velocity can demonstrate effort and maybe some early results – but that honeymoon phase ends pretty quickly when people start asking about what learnings you generated that the rest of the company can leverage.

Note that I’m not saying to not iterate quickly. You definitely should do that. However, what I am saying is to celebrate the outcomes, not just the output. Now go out there, and learn something.

What do you think? Is test velocity overrated? Let us know.

Good luck and see you in 2 weeks!

Rommil Santiago
Founder, Experiment Nation