Test before you build anything or you’ll waste time course-correcting with Rommil Santiago

There’s a lot of talk about how to find things to test, how to analyze tests, and the importance of iterating on tests. This week, I was thinking about how I could add something a little different to the conversation and decided to share my thoughts about when to run a test – particularly when building products. 

Building quality products, digital or otherwise, is generally not a trivial exercise – namely for larger companies. Doing so requires devs, product managers, researchers, designers, QA, and analysts, just to name a few – in other words, it isn’t cheap and it takes a significant amount of time. Because of this heavy investment, it’s helpful to have an idea of whether all the effort was worth it before any work is done. Sure, one could conduct surveys and user research. It’s important work and it certainly gets you into the ballpark of what could resonate with your market. However, there’s a major risk: what people say they do is often not what they actually do. Despite the fact that running experiments on your actual users and customers will get you a more reliable read on how something will perform, many companies opt for running tests very late in the development process with fairly high fidelity. This is a huge missed opportunity. By running tests earlier in the build process – in fact, ideally, before you build anything net new – one can save weeks if not months of effort (and in many cases thousands and thousands of dollars).

Here’s an illustration. Let’s say you and your friends wanted to walk from Toronto to San Francisco – because it was a nice day. Before you head out, you get a quick read of what you need to do by looking at a printed map (play along with me here). After a few moments, you gather that San Fran is southwest of Silicon Valley North, and approximately 5000 kilometers away. So you point yourselves towards that direction (as best as a group of friends could) and start walking – but during this entire journey, you keep your phones in your pockets. But what you don’t realize is that your heading is off course by 1 degree. 

Days and weeks go by and eventually, you and your entourage decide you’ve walked far enough and take your phones out again to reassess the situation just to discover you’re actually 100 km east of the friendly city – all because you walked about 1 degree off course. 100 km off is not that far off relatively speaking – but course-correcting this late in the game will cost you a day of walking. Just imagine if you were 2 degrees off, or 2, or 10. It’s easy to see how this could get a lot worse quickly.

Coming back to building products. Surveys and user research are like printed maps. They give you a great feel of the environment and roughly what direction to go – but aren’t so great at telling you precisely how off you are. Running experiments help you course-correct and the sooner your run them, the more you are sure that you are working in the right direction. So the next time you are considering building something new, consider very early-stage testing. Consider testing concepts on things that already exist, and even painted door experiments to get an early read on things and iterating until you feel confident of your path. Otherwise, you’ll waste time fixing and undoing things that could have been completely avoided in the first place.

Good luck and see you in 2 weeks!

Rommil Santiago
Founder, Experiment Nation