How to Drive Product Growth with A/B Testing

In this talk, Mark Marcello explains several key topics in Product Management, including why growth experimentation is important, what things a company should do to make money, best practices, interactive case studies, and the power of intuition.

Let’s take a look at A/B testing:

Meet Mark Marcello

Mark Marcello is a Product Manager in CROmetrics where he is responsible to facilitate A/B Testing for the company’s clients, including Airbnb and Viking Cruises by leveraging the tool Optimizely.

He is an e-commerce product management expert who holds over a decade of progressive professional experience and has demonstrated the ability to combine strategy, creativity, and strong business acumen with well-developed technical talents that position companies for success. 

How A/B Testing Can Unlock the Full Potential of a Growth Strategy

Definition of A/B Testing

‘Do the Thing That Makes the Money’

  • Let data tell you what customers want 
  • Make testing and experimentation a part of the business strategy
  • Identify and focus on the singular, most important, metric
  • Accelerate throughput to increase ROI

Consistent Growth is a Continuous Cycle of Experimentation

  • ABT (Always Be Thinking)
    • Ideate: Bring strong ideas to the clients
    • Plan: Strategise how to go forth with the idea
    • Build: Develop. Input client feedback
    • Launch
    • Analyse: Assess how a product can be made better

Amazon Views Experimentation as Critical to Growth

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.

Jeff Bezos

Facebook

The strategy of Facebook is to learn as quickly as possible what our community wants us to do – and that requires a culture that encourages people to try things and test things and fail, and it also requires building infrastructure to enable people to do that.

Mark Zuckerberg

Why Don’t Companies ‘Do the Thing That Makes the Money’?

  • Many Companies aren’t embracing experimentation as quickly as competitors
  • Testing and research has historically been difficult
    • Expensive
    • Time consuming
    • Unreliable
    • Competitively complicated
  • Experimentation program success requires speed and infrastructure
  • Cultural blockers can encumber testing strategy
Man working at desk

“Old” Approach to Satisfying Customers: Not Data Driven

  • Prior experience and company “Folklore”
  • “This is how it’s always been done”
  • “Mailing 20% off coupons is what works”
  • “Hunches” & HiPPOs/opinions

The Creative Function Should Be Data-Driven

  • “Brand guidelines”
  • “Creative know-how”
  • Leverage creative talent informed by customer data 

Testing as a Strategy: Real Estate Pop Quiz

Can Intuition and Experience Beat Data-Driven Experimentation? 

A/B Testing graph variation A
A/B Testing Graph variation B

Result:

Mark explains that variation A wins because it’s far easier for the customer to figure out what each option does, when it’s clearly labeled, as opposed to the fact that when it’s just represented through icons. Hence, once should never ignore their intuition even in this data-driven age.

How to Develop Strategic Customer Insights that Drives Growth

1. Heatmaps Show Us Where People Actually Click

2. Analytics Offer Insights on User Behavior
3. Customer Feedback Yields Winning Copy Ideas
4. Tailored Experiences Based on Inbound Traffic Source Drive Increased Efficiency Normal Site PPC Landing Page

PPC vs Normal Site graph

5. Experiment Results Yield New Ideas to Test

Experiment results yield new ideas to test

Kickstarting a Testing Strategy That Creates a Culture of Experimentation

Identify the One Thing That Makes the Money

  • Facebook: Daily Active Users
  • Uber/Lyft: Weekly rides
  • Optimizely: Monthly experiment starts
  • Airbnb: Nights booked
  • CROmetrics: Weekly experiments run

Prioritize Tests to Maximize ROI

  • Reach
  • Impact
  • Confidence
  • Ease

Develop Key Competencies for Program Success

  • Drive culture of experimentation through engagement
  • Track strategic insights and develop playbook
  • Augment internal staff
    • Increase experimentation velocity
    • Unify cross functional teams
  • Easiest task to outsource
    • Dedicated Engineering team

Testing as a Strategy: Final Exam 

Can Intuition and Experience Beat Data-Driven Experimentation?

Graph Pay rate
Graph note from manager

Result:

Test results
Test results graph

Mark explains that though the results for both the variations were similar for completed applications, the Note from Manager added a personal touch and hence led to an increased down-funnel metric. The note also mentions first come, first serve basis, which triggers a sense of urgency in the customers.

Do the Thing That Makes the Money 

  • Make testing and experimentation, a strategy
  • Develop strategic customer data
  • Kickstart your program with or without a partner
    • Identify one, most important, metric
    • Prioritize experiments
    • Accelerate throughput to increase ROI

Mark shows that one should never ignore their intuition and always go with the data. He also emphasizes that a long copy always isn’t necessarily a bad thing. It might prove to be more effective when used in the right way and has appropriate content.

Product Mindset CTA

Enjoyed the article? You may like this too: