Evidence Driven Development Strategy to Make Better Data-Driven Decisions

You probably have dealt with stakeholders in the past, sometimes they want you to follow a specific strategy. Nevertheless, it is ok to be skeptical if you don’t share the same vision. If this happens to you, don’t panic! Josh Vincent, Sr. Product Manager at Xero, faced a similar situation and along with his team, they started putting pieces together to build their own strategy. As a final result, they came up with the Evidence Drive Design strategy, inspired in the lean methodology.

Evidence Driven Design (EDD) Strategy: What Is It and Why?

  • Evidence Driven Development is an effective product development strategy where decisions are made based on data. In this methodology, a problem is clearly defined in terms of various factors, an EDD score is calculated, and experiments are conducted to validate the data and assumptions being made.
  • Using EDD helps ensure we have conversations around value to build features that have the greatest relative impact.

EDD Vision: Make decisions with data 

EDD Process Overview

1. The Product Description Sheet — Overview of the product idea

High-level Product/Feature Description: Create a Product Description Sheet with answers to relevant questions of the problem statement like mentioned below:

    • Customer: Who is this for? For whom are we creating value? 
    • Problem: What is the top problem (or 3) facing this customer segment?
    • Unique Value Proposition: How are we uniquely solving their need? A compelling message that states why we are worth buying.
    • Solution: Top ways we will solve the problems
    • Key Business Success Metrics (KPIs): How do we measure success?
    • Early adopter: Who is the potential early user of the solution?
    • Existing Alternatives: How are users solving this today?
    • Channels: How are you going to get, keep and grow customers? Get -> How will we drive awareness, usage, interest? Keep users? Keep -> How will we keep customers back? 
    • Key Stakeholders: Whose buy-in do we need? Who are the key influences to these stakeholders?
    • Key Resources/Partners: Critical internal and external resources need to deliver the solution.
    • Revenue/Business Value: What’s the value to the business for delivering the product?
    • Cost structure: Resources/expenses for building and maintaining the product. Which key activities are most expensive – product development, marketing, customer support? Which key resources are most expensive? 

2. Score  Calculating ROI of a product

Create an EDD Score Spreadsheet for calculating the ROI of a product.

    • What is an EDD Score? It’s a relative indicator of ROI for a feature/product over the long term in $/effort. The EDD Score relies on assumptions so it is not perfect or precise, it is also used as a metric to compare initiatives side by side. We can see it as a way to prioritize by generating a value that can be used to rank features.

6 metrics to calculate the EDD Score 

    • Impact Per User
    • Customer Acquisition Cost
    • Churn Rate
    • Cost to Serve
    • Cost to Build
    • Months to Release

Calculating the EDD score doesn’t have to take a lot of time, invest only enough to make the next decision.

  1. Very Simple exercise – this takes around 5 minutes where you rate questions on a scale of 1 – 5. For example, are the customers going to be happy with our product?
  2. Moderate exercise – this takes around 10 minutes where you make estimates for the impact per user (monthly), Customer Acquisition Cost, Staff needed, Churn Rate, etc.
  3. Detailed exercise – this will help you fully comprehend the project compared to other initiatives. Consider the customer growth and apply an attach rate for each year to the user numbers.

At the end of this, compare the EDD Scores of all the products.

Evidence-Driven-Development-Strategy

Can you trust the EDD Scores? Keep in mind that these numbers are only as accurate as the fictitious data, ludicrous assumptions and wishful thinking they’re based upon!

3. Roadmap — Activities needed to build and ship the product.

Turning Your Gut Feel into Data: create a Product Experiment Roadmap to gain confidence in the numbers you have placed in your EDD Score.

With the EDD score sheet and product description complete, build your case.

    1. Create experiments for each metric.
    2. Scope each experiment.
    3. Do the ones with the best yield first.

And now, you should validate your assumptions with data:

Evidence-Driven-Development-Strategy

    1. Hypothesis – Target your metrics. For example, users will value and pay for a bookkeeper on demand.
    2. Experiment – Create and run experiments; ask 50 customers, bookkeepers, and partners if they would be willing to pay for this service.
    3. Results – Document & discuss your findings; results 27 of 50 respondents said they would be willing to pay for online support from a bookkeeper for up to 60 minutes.
    4. Conclusions – Confirm/change your assumptions and publish findings; now that we know a realistic attach rate, we can better calculate the ROI on this effort.

Anatomy of an Experiment and Description:

    • Critical Question → What is <our cost to build, revenue per user, etc.>. For example, to answer what we will do to conduct the experiment, the answer would be, “We will research competitors Ys pricing offering for feature X.”
    • Experiment → We will <describe experiment> – This is the data that we will gather and how we will analyze it.
    • Observe – Then we will see if <Hypothesis> – Then, we will see if our planned pricing is less expensive.
    • Hypothesize → We believe <Predict Result> – What we expect the data to show – We believe our planned pricing is less than theirs…

Result → This means … and we can be more / less confident in our critical question – How we will interpret the data if it is what we expect – and this means that customers will pay $1 per for transaction so we can be more confident in our monthly revenue per user critical number.

Pricing/costing experiments in Units

Evidence-Driven-Development-Strategy

Experiment Process

Evidence-Driven-Development-Strategy

Document, Debate, Share

    1. Results – What was the data you generated?
    2. Experiment Quality – Did you gain any useful data?
    3. Budget – Did it take you as long as you expected?
    4. Confidence Increase – Did your confidence go up? How much?
    5. Share the results on company intranet if interesting or insightful
    6. Did the model change?

Have you applied the EDD methodology yet? Reach out to us at @productschool


Meet Joshuah Vincent

Joshuah VincentJosh Vincent has been building software since his computer science days at University. He is a seasoned Product Manager, Designer, and Entrepreneur. He spent almost a decade shipping products at Microsoft, followed by a few startups in Silicon Valley. Most recently he shipped a major Payment Network for businesses and employee users of Xero. He has driven the shipment of over 20 internationally recognized products, including SaaS platforms, hardware, and software.

Enjoyed the article? You may like this too: