Why experiment?

Many important business questions require distinguishing between cause and effect. Consider the following examples: 

• How does a recommender system change purchases and revenues?
• What is the influence of a marketing campaign on product choice?
• How do Facebook Likes affect sales?
• Which salesperson compensation system leads to higher customer repeat purchase?

These questions consider a cause (a recommender system, marketing campaign, Likes, etc.), and link it to an effect (purchases, revenues, product choice, sales, etc.).

We call such questions causal questions. They are notoriously hard to answer. 

What can go wrong?

A lot, unfortunately!

In most cases, using existing data may confound the cause and the effect. Consider as an example the question of how Facebook Likes question affect  sales. While sales and Likes are correlated, we cannot infer the direction of causality.

It is possible that higher sales lead to customers Liking the product more. Or it could be that sales and likes are both a result of a common cause. Customers may “like” Harry Potter or Goethe’s Faust page and buy the books. But that is only because they are both great books for their target audience.

Alternative explanations such as this one, and other statistical problems, are hard to rule out. And if you cannot rule them out, your results might be misleading (scientific lingo for: "wrong"). This makes it hard to identify causal effects.

As a result, in most cases it is not sufficient to collect existing data and do an analysis. 

Why experiment?

Good decisions are often based on counterfactual thinking.

Experiments answer the question of "What would have happened otherwise?", for example "How many books would Faust have sold without a single Like?", or “How many products would we have sold had we not advertised”. We call this the counterfactual: inferring the outcome of a situation that never happened.

An experiment allows us to simulate what would happen with, and without the “cause” we are studying (Likes, advertising, price changes, etc.), and compare these two states of the world.  Without an experiment, we often have no idea what "would have happened otherwise", and we cannot make causal claims.  

Because causal questions are so central in business, most companies experiment extensively.

What's next?

If you want to know how experiments can help your organization, sign up for DED 2017.

If you have questions, send us an Email!