Be more than just data-driven Experiment Nation

When someone says that they are data-driven, it’s often because they want to imply that they only do things that are strongly supported by data and not on any gut-feelings. But is being purely data-driven really a good thing all the time? I’d argue that it isn’t.

“Facts are stubborn, but statistics are more pliable”?—?Mark Twain

We often interchange the terms: facts, data, and insights?—?but there are definitely distinct differences. While facts represent the truth or reality about something, data is biased information collected to help us uncover the truth. Furthermore, insights are biased opinions of what the truth is based on the data. How can data and insights be biased? The answer is simple. Humans.

When trying to use data to verify something, many decisions have to be made and each decision introduces some form of bias. For example, say you are trying to understand whether a cup of coffee is hot or not. Decisions have to be made around how to measure the coffee’s “hotness”. Assuming you decide that the coffee’s temperature is the data you should collect— how will you collect that data? With your finger? A thermometer? Where would you take your measurement? At the surface? In the center? How long should you measure the temperature for? How many samples? And once you have your data, you have to interpret it. What temperature is considered “hot”? Who decides that threshold? It is at this stage that things get sticky.

Using the example above, what defines hot? What is hot for one person, may not be hot for another? Hot in one context may not be hot in another. How do we arrive at an agreed upon definition of hot? Do you take an average of everyone’s opinion? Do you take the median? Do you listen to the loudest person in the room? How many people do you ask? Did someone else answer this elsewhere already? At the end of the day, you literally can decide that any temperature is hot. Or in other words, you can interpret data to mean almost anything.

“Do not miss your chance to blow”?—?Eminem

Some ways to reduce the impact of bias is to align on how to interpret data up front, analyze a situation from multiple angles, and to collect a lot of data. With less bias, comes more confidence in insights, and quicker decisions can be made. But all this comes at an expense: Time.

Interestingly, to reduce the risk of data bias, we increase the risks associated with taking too long to analyze data. One major risk is a change in the market. In fast moving industries with a lot of competition, changes in the market occur regularly. For example, competitors release new features monthly if not weekly if not daily, government regulations change, Google changes its algorithms, so on and so forth. Take too long to generate insights, and you risk making business decisions that aren’t relevant in the current environment.

Being data-driven is more suited to short-term, iterative strategies?—?i.e. optimization. However, what being data-driven is bad for is vision. I see vision as the direction you head towards with the goal of maximizing the success of your company. While optimization maximizes for your current situation, it won’t necessarily result in your company being all it can be.

“Skate where the puck is going.”?—?Wayne Gretzky

Being data-informed, on the other hand, is making decisions based on limited data. Having limited data can be due to various reasons such as literally not measuring something for long enough, or more often, the data doesn’t even exist.

Vision lives in the land of limited data. More often than not, you need to forge your own path towards a position that you can grow and defend based on a bet. All the data and analysis in the world will never tell you if you’re making the right bet when you go into uncharted waters. Being a fast-follower works for some companies, but it is a tough route to take since you still won’t know if copying another company will work for yours. But taking these educated risks are the only way to find a so-called global maximum, i.e., to fully realize the maximum potential of your company.

Now, I’m not saying you should run a company completely based on guesses and gut instinct. What I am proposing, however, is a balanced approach. One that embraces being both data-driven and data-informed.

“Be as water, my friend.”?—?Bruce Lee

Don’t be rigid in your mindset. There is a time and place for being data-driven or data-informed. It’s less about what approach your espouse, and more about what you are trying to achieve. Flow between the two approaches as needed. Know when to be driven by data and make sound decisions to optimize what you have. And know when to be informed by data, and make calculated risks with the goal of maximizing long-term potential.

The two mindsets feed into each other. Taking a data-driven approach should lead to a data-informed approach. As you try to maximize your current world using a data-driven approach, look at where else the data is informing you to experiment. Take data-informed and calculated risks (aka moonshots) to help shape your future. As you take your moonshots, iterate (or sunset) based on a data-driven approach. And the cycle continues.

So the next time someone asks you whether you are data-driven or not, simply tell them you’re as water. (You may want to paraphrase.)

With that, I leave you this.

Connect with Experimenters from around the world

We’ll highlight our latest members throughout our site, shout them out on LinkedIn, and for those who are interested, include them in an upcoming profile feature on our site.