Data trap for new product managers

Ackbar knows.

Product managers are expected to be driven by data. The results of their decisions must be supported by metrics that are examined over time to determine if the goal was reached or not. However, there is a trap in the numbers that can easily give Product Managers a false impression of success: vanity metrics.
The concept of vanity metrics was introduced by Eric Ries, author of The Lean Startup, to define measurements that look good on paper and to the media, but provide little insight on to what’s the cause and what to do next. You can see examples of vanity metrics every day such as an increase in visits, number of downloads or user growth. However, achieving 1,000,000 downloads can be the result of a Product Manager’s job, but also from the sales effort, or customer support, or a combination of all, or even none of these. Moreover, what will you do next? How can you replicate this behavior? can it be improved?
To avoid falling into measuring vanity metrics, Ries proposes three criteria to test “good” metrics:


The metrics you decide to measure must have a clear cause-effect relationship. This will allow you to know exactly what to do to if you want to replicate or improve results and learn from every action you take.


Metrics are nothing if they are not understood by the people whose decision they must influence. As a Product Manager, you need every team that you work with to understand what it means that the churn rate went from 5% to 10%. There are two options to make this happen: simplify the language with which you present your results, or train everyone in the technical terms that need to be used.


Good metrics are unbiased. Anyone measuring them should be able to come to the same conclusion. This will allow you to build trust from the teams you work with and avoid finger-pointing when things don’t go as expected. An auditable metric should be able to stand the test of real customer discussion: what your customers say validate what your reports do.


As a new Product Manager, vanity metrics might seem attractive but provide little long-term value to you and your product. Avoid falling into that trap by using actionable, accessible and auditable metrics. The best Product Managers understand the difference between the two, so following these simple guidelines will set you off to a great start in your new career path.