Ask the Experiment Nation - Should we build our own Experimentation Platform? Experiment Nation

Q: Should we build our own Experimentation Platform?

Recently, Davis Treybig posted an article about the gap between the best Experimentation Platforms vs. those the rest of us have. In short, he writes about how Experimentation tooling hasn’t kept up with the industry. This got me thinking. Does this mean you should build your own Experimentation Platform in-house? Despite all the great points that Davis raised in his article, my answer is a firm, it depends.

Great cultures build great tools – not vice versa
I will not deny that as a company matures, that their technical needs will naturally grow. However, having a best-in-class Experimentation tool is meaningless without the right Experimentation culture, people, and processes in place. A company has never become evidence-driven by building a great tool. It’s more often the case that evidence-driven cultures build tools to keep up with their needs. People and process first. The rest will follow.

You’re better served by focusing on your strengths
There is something to be said about the expertise required to develop testing tools. Not all companies are strong at developing tooling like this. We are not all like Netflix – and that’s not going to change overnight. Sometimes the opportunity cost of building something poorly doesn’t make a ton of sense. (Mostly because you won’t get it right the time.) Beyond that, not only do these platforms have to be maintained, but major industry shifts can have serious impacts to home-grown solutions (E.g. Our cookie-less future, the industry shift away from DMPs and towards CDPs, etc.). Unless your platform is adequately staffed, you’ll need to steal/borrow/beg for resources to address these issues – constantly. You’ll end up spending more time trying to fix bugs than answering hypotheses.

Sometimes, simple is enough
Not all business models require sophisticated solutions. If you’re the owner of a local car repair shop, chances are you shouldn’t invest in a multi-channel, sequential, experimentation platform. There are many valid experiment designs that don’t require fancy tools. Just because Netflix does it, doesn’t mean it makes sense for you.

With all that said, you’d think I hated David’s post. On the contrary, I think it’s great and a wonderful overview of how complex Experimentation Platforms have to be for today’s leading digital companies. I just want to highlight, that it’s the poor musician that blames their instrument. You don’t need a Stradivarius to play a wonderful song – just the right mindset and skill.

See you in 2 weeks,

Rommil Santiago
Founder, Experiment Nation

Continue the conversation on our Slack channel