Better Decisions With Data: Asking the Right Question

3. Ask a ‘counterfactual’ question

If they ask a factual question, such as, “Who is most likely to cancel?” and then target those customers, they would get an answer that could be useful in many ways—but not helpful in informing the decision about whom to target. It ignores the possibility that incentives might only work for certain customers. The question HP really needs to ask is a counterfactual one: What effect will incentives likely have on specific customers? Answering it requires collecting and analyzing new data.

Stefano Puntoni and Bart De Langhe are the authors of Decision-Driven Analytics: Leveraging Human Intelligence to Unlock the Power of Data (Wharton School Press, 2024).

The best approach is a randomized experiment. HP would divide a group of customers randomly into two groups, offer an incentive to one, and monitor the attrition rates for the two groups. If there was a difference, HP could conclude that the incentive could be effective, and then look further at which customers responded. This randomized experiment is a dramatic departure from the “best practice” of focusing only on customers at the highest risk of canceling their subscriptions. Instead, it reveals whether incentives could work to reduce cancellations, and if so, which type of customer to target with those incentives.

Ask a “counterfactual” question when you need to compare outcomes with and without a particular intervention. These questions involve hypothetical scenarios and demand causal inference, making them inherently more intricate than factual questions. For example, during the 2012 U.S. presidential campaign, Barack Obama’s team gained a crucial edge against Mitt Romney by asking, “Who is most susceptible to being persuaded to vote for Obama?” This counterfactual question allowed the team to identify swing voters most inclined to change their voting behavior when visited by a campaign worker, rather than targeting those most likely to vote for Obama or hard-core Romney supporters, which would have squandered valuable resources. Their approach revolutionized the way political campaigns were conducted, emphasizing precision and efficiency in mobilizing support and persuading undecided voters.

How an organization might use it

Most businesses rightly see data as a source for making better decisions. But the conventional data-driven approach often falls short because of common errors made by the decision-makers. Many leaders rely excessively on existing data that might not always address the issue at hand, or they pass key decisions to data scientists who don’t understand the business dilemma they’re trying to solve. Decision-makers are also prone to lead with a preference, arrive at a solution, and then find data to back it up.

When determining the decisions to be made, focus on options within your control and pertinent to your organizational position. Because most decision-makers gravitate toward familiar options, broaden yours by engaging with diverse perspectives. Ask trusted colleagues or gather your team to get further clarification. The choice set should include options that are both feasible and effective, and that can genuinely enhance desired business outcomes. It should exclude options with prohibitive costs or risks.

1. Clarify the decision

During this critical first step, data—especially existing data—should not be a consideration. If analytics start from the data that are available instead of the decision to be made, this increases the chances of asking the wrong type of question. That’s not to say that data mining and exploratory data analysis have no role in business. But decision-driven analytics assume that the first responsibility of business executives is to have a razor-sharp focus on their job and deliver on their key responsibilities. To do that, it works better to start from the decision at hand and work backward toward the data needed. Data mining and exploratory data analysis are valuable, but they serve a different goal.

Published May 28, 2024, by Knowledge at Wharton.


Alternatively, decision-driven analytics puts decision-makers at the center, resolving the common mismatch between analytics and actual business decisions. It starts from the decision that needs to be made and works backward toward the data needed. But it also requires more from leaders, who must shift the focus from getting answers to asking the right questions. This approach highlights the strategic importance of what we don’t know, underscoring the importance of intellectual humility. In fact, crafting the right questions is an essential and foundational step in the data-analysis process.

Action steps

Ask a “factual” question when you’re seeking a prediction. Businesses need to answer this type of question frequently. For example, the smooth running of manufacturing operations rests on predictions about when a machine is likely to start failing. That information can then be used to create a maintenance plan. For retailers, understanding the cost of dealing with returns is similarly important: It would be incredibly valuable to identify which products have a higher likelihood of being returned. With this knowledge, companies can proactively strategize either by adjusting the product pricing, considering additional measures to reduce returns, or even deciding against offering such products entirely.

One of Hewlett Packard’s business strategies is the “Instant Ink” subscription that lets customers pay a monthly fee to receive printer ink delivered to their home (and allows them to cancel the subscription at any time). Imagine that HP wants to use data to proactively address customer attrition, intervening with incentives such as discounts, in hopes of stopping customers from canceling. But because they can’t offer those incentives to everyone, they need to know whom to target.

2. Ask a ‘factual’ question