The State of Product Analytics

Editor’s note: The following is a guest post from Mixpanel. If you’re interested in writing a piece for us, contact gaby@productschool.com

Does Silicon Valley really have the best product managers? Do product teams in different regions care about different metrics? And what’s the real cost of innovation without data? 

These are just some of the questions we set out to answer in Mixpanel’s first-ever State of Product Analytics report in partnership with Product School, released this week. In it, we surveyed more than 450 product professionals around the globe—from SMBs to Fortune 100 enterprises—about what metrics matter most to them, how they use data to drive product innovation, and the barriers and challenges they face to becoming data-driven. 

Here’s what we found. 

#1: From novices to experts and in between, product managers are all over the world

You can throw out the myth that tech hubs like Silicon Valley, New York, and London are the only places  data-centric product innovators call home. On the contrary, our survey data showed that sophisticated product teams can be found everywhere (from Tel Aviv to Sweden), and they’re leveraging all kinds of data to drive innovation—including from surveys, customer conversations, competitive intelligence, and product analytics. 

And while we learned that product teams’ level of sophistication with data is not influenced by geography, it does inform the type of data they rely on most to make decisions. 

As product teams move up in sophistication, the primary source of their data also shifts from qualitative (via surveys and customer conversations), to quantitative (from an intelligence or analytics solution).

Distribution of sophistication with data by city and country
Usage of data type across data sophistication

And while different types of data provide insights that are key to driving product-led growth and innovation, the quantitative insights harnessed by the most sophisticated product teams help them understand—and plan for—things like: 

  • Behavioral trend data, in aggregate, across customers as user bases change and grow (no matter how small or large)
  • How users really feel about their product (with less risk of confirmation bias, or the “counterfeit yes”).
  • How to prioritize features that create value for users, rather than those that are just keeping up with the competition.

To be sure, both are essential. There has always been space for qualitative data to inform product decisions, most notably by helping those who may be disconnected from customers day-to-day (like engineers and managers) build empathy for users. But, hard data from quantitative research keeps decisions rooted in reality by concretely showing what conflicts exist (e.g. the numbers of users not completing “x”).

Yoav Yechiam, Product Strategy Advisor at Isreal-based Y-Perspective, says that in the past year he’s seen a shift in mindset with companies realizing that front-end data management is more effective and valuable than backend transactions.

“We see many companies shift their emphasis to implementing third party analytics solutions, as opposed to relying solely on their own data scientists and BI experts. I expect this trend to continue with the new capabilities, and scalability options of cloud based Analytics platforms,”

#2: Product teams in the U.S. have a larger focus on revenue than their European counterparts

Across the globe, the top three metrics that product teams care about are “retention”, “engagement”, and “conversion”. But when we look closer at the U.S. and Europe, some interesting differences emerge.

While U.S. product teams prioritize “retention”, “engagement”, and “revenue” in that order, their European peers place “revenue” near the bottom, above only “NPS”—and their top metrics are all centered around product KPIs like “active usage”, and “activation” (in addition to the global top three).

Top 3 metrics for product teams
Us & Canada vs Europe

Although the data doesn’t say for certain, one hypothesis we landed on is that in a highly competitive market like the U.S., top-line metrics win out due to product teams’ laser focus on growth and expansion, and an overarching belief that delivering on customer value equates to revenue.

European product teams’ interest in product KPIs rather than top-line metrics could signal an approach to product development that looks at customer value a little differently. Whether we see this realized with ever more innovative, customer-centric products coming from Europe’s tech community will be an evolving story—we’ll report back in 2021.

#3 The real cost of innovation without data

If there’s one universal takeaway from our research, it’s this:

Despite the abundance of available data, product teams still struggle to measure the metrics that matter. 

Through our research, we learned that only 10% of product teams are able to validate their most important product decisions with data. In fact, half of all product teams aren’t confident they have the tools they need to get the right answers (this number shoots up to 74% for novice product professionals).

And not being able to measure the metrics that matter comes at a cost. The data here is sobering:

  • 67% aren’t confident they understand their conversion rates and know where and why users drop-off.
  • 51% aren’t confident they know what brings users back to their product.
  • 48% aren’t confident they understand well their customer journeys through their product.
  • 47% aren’t confident they have the ability to measure the impact of their feature launches.

If product decisions are made in a black box, without teams fully understanding their users, then things are more likely to get built based on assumptions and gut feel—a gamble that could negatively impact the customer experience.

Wondering if all of this has changed in the time of COVID-19? There’s no better time than now, according to Blend product manager Clement Kao, to start taking a cue from the most sophisticated product teams.

“Now is a fantastic time to double down on instrumentation and quantitative data collection, especially as you continue to learn about the new needs of your customers and end users,” Kao explained to Mixpanel. “That way, as soon as you’ve shipped new functionally to address the pains that you’ve discovered, you’ll be able to refine and iterate much faster than your competitors can.” 

Even in the best of times—and I think we can agree that these are not the best of times—product teams must be able to pinpoint what drives conversions, engagement, and retention with complete confidence.

Now more than ever, teams are able to merge business needs, user feedback and business objectives all while proving ROI will be best equipped to delight users in the future—whatever it may hold.

Make sure to download the full report, The State of Product Analytics, here.

Telegram Channel banner

Enjoyed the article? You may like this too: