[Source: The 5 Biggest Biases That Affect Decision-Making, The NeuroLeadership Institute]
A product’s success largely depends on establishing a market that is willing to buy the product that you’re building. When developing a new product or iterating on an existing one, product teams often rely on qualitative research (in the form of customer interviews, surveys, or support tickets or interaction between customers and the customer support team) to guide this product development process.
However, I’ve often noticed a seeming gap between what customers say versus what they do which leads to suboptimal prioritization and decision making by product teams. As teams strive to receive and incorporate customer feedback in a more rigorous way, a basic understanding of cognitive psychology strongly affects how product teams get this information; and helps demystify what customers actually want (vs what they think they want).
1. Avoid cherry-picking feedback that confirms your pre-existing beliefs.
Creating a defensible hypothesis as the first step to solving a complex problem is one of the superskills I was encouraged to learn early on in my consulting career. However, when you’re working on building a product and have a hypothesis for how to do it, it is really easy to succumb to confirmation bias when you’re seeking feedback from customers.
For example, if I have a belief that my product should be packaged as a monthly subscription instead of as a consumptive “pay-as-you-go” offer, I might be more likely to look for customer feedback that supports that belief and ignore evidence to the contrary. The first step to counteract this tendency is to ensure that the framing of your research questions itself does not nudge customers towards one direction. Second, it’s always helpful to have someone who is not invested in the same hypothesis as you (perhaps a stakeholder from your broader cross-functional team) accompany you to the conversation and play devil’s advocate to help analyze the data objectively. Lastly, always triangulate your data sources to inform your decision-making process.
2. Top of mind should not necessarily be top priority - avoid the squeaky wheel syndrome when prioritizing.
When building a product, it is imperative to keep your ear to the ground and listen to what your customers want. However, building a new feature every time some of your users ask for it can result in a product roadmap that follows no consistent direction.
For example, if a product team recently sat in on a sales call or received a large number of support tickets about a specific feature, they may be more likely to prioritize fixing that feature, even if it may not be the most critical issue for the product. Cindy Alvarez, author of Lean Customer Development calls this the “squeaky wheel syndrome,” where the loudest voices are the unhappiest customers. These customers might have a lot to say, while several relatively happy customers might keep their thoughts to themselves.
To fight this bias, balancing out the quality of customer feedback with the quantity helps ensure that every product decision can then be evaluated against a coherent product strategy instead of just against any recent evidence, or the most squeaky wheel. Extracting patterns from a breadth of feedback and insight you receive from customers can help surface shared pains that your product can solve.
3. Consider what you don’t see - survivorship bias obliterates critical insights that could help improve your product.
We all draw inspiration from the fabled stories of the 1% - the businesses that succeed, the entrepreneurs who become billionaires and the leaps of faith that work out. In doing so, we succumb to survivorship bias - a cognitive shortcut that leads us to overlook failure in pursuit of a compelling narrative of success.
When gathering customer feedback, it is easy to mistake the “successful” subgroup as the entire group, due to the invisibility of the “failure” subgroup. In the typical process of gathering feedback from only the active, regular users of a product, you lose valuable insight from a segment of users who sign up to use a product but never really get started. These customers were interested in your product to begin with but then churned - this probably means your product is not meeting their needs and there is room for improvement. Understanding why these customers are leaving can give you valuable feedback that will help you improve your product and retain future customers.
In order to avoid survivorship bias, data from product telemetry can be a key source for understanding how customers interact with a product beyond simply interviews and customer perception through self-reporting. Further, when observing a pattern from a dataset, step back and ask yourself whether this pattern represents the general user segment or only a specific user segment; to better contextualize the patterns and not overgeneralize the relevant conclusion for your product. Lastly, a well-designed offboarding process can help provide insight from often overlooked customers, even during their departure. Understanding why they are leaving, what might be done to keep them and getting feedback to help improve the product for future customers can help reduce customer churn and increase the customer’s lifetime value.
(Hustle Fuel represents my own personal views. I am speaking for myself and not on behalf of my employer, Microsoft Corporation.)