The Beauty of A/B Testing And Why PMs Should Do It
Engineers, designers, gallery managers, business people… The list goes on. Product Managers come from various backgrounds, and there isn’t a story that is exactly alike with another one. Some people found their way into product early, and for some, it took a little bit longer.
However, they found their spots in the industry, and we are happy to have them with us. Here is yet another PM who took his own unique road into product and Optimizely. He shared his thoughts in a webinar on A/B testing for Product Managers and answered a few questions from our followers.
Background and product management
The Product Manager of Optimizely, Byron Jones, has been with Optimizely for a little over four years which is as he points out a long time when speaking of companies in Silicon Valley. Talking about different road into product, before Optimizely he worked at NASA’s Jet Propulsion Lab for ten years but got tired of working for the government and wanted a break.
At first, at Optimizely he was working on a success team where he could take on roles outside of the success team because the company was quite small. This role changing helped him transition into product.
He mentions two other key takeaways that helped him further in his transition. The first one was having some technical experience even though Aerospace is not the same as the software technology and the second one was his experience in the tech space.
A/B testing for Product Managers
“As a Product Manager, you own the success or failure of your product,” Byron says. He gives two primary ways to be successful. The first one is about the metrics and moving them the desired direction through experimentation. Everybody has goals that they’re aiming for, and those goals can be reached by experimenting.
The second is releasing features in a measured and controlled way so that you can get better insights into the product with the control group. This way you also get user feedback early on which helps you be more successful in the long term of your product release.
The beauty of testing
He quotes Mike Wolfe from Trunk Club because of his powerful statement. “Testing fits in almost everywhere at Trunk Club. We test everything from box sizes and packaging to taglines on our homepage. We’re constantly trying to improve our service, and we validate success through qualitative feedback and data.”
The design team works closely with the product team to make the user experience as great as possible and to drive new business. Especially at Trunk Club, they invest in personalized experiences. Sometimes it is good to challenge the norms and run an uncertain experiment.
The good thing about experiments is that they’re not permanent and through data, you can see quite quickly if it’s working or not.
Five stages of A/B testing
The process starts with analyzing the qualitative or quantitative data that you have available about the product, the site, the users or the customers. Next, you identify the opportunities for the experiment and come up with a hypothesis. When you have a hypothesis, you start building the experiment, measuring the metrics and define a key metric.
When all this has been set and done, you will run the tests. The time to put into this varies because the important part is that the test reaches statistical significance and that you don’t end up and having a false positive.
The last stage in the process is measuring the results. Sometimes tests provide a negative impact, but it’s okay because the most important thing is running tests and learning while doing it.
Questions from followers
Could you speak a little bit about the technical skills that helped you transition into product management?
How does A/B testing change by industry?
I’d say that it shifts the type of metrics that you’re looking for. In tech, there are different sides of it. We work a lot with retailers who are very straightforward types of industry to do A/B testing on. You’re looking for, for example, how do you take users through a signup funnel going from online advertising into the site to make a conversion.
I’d say you can use that kind of generic template to any industry whether it’s travel, insurance or FinTech. What you’re thinking about is what is the customer journey from how they find out about you all the way to conversion that you’re trying to achieve. You map that out; you find the points along the journey that are going to be important to you to move the metrics for.
From an e-commerce point of view, it’s about going from that initial sign up to putting something in the cart for purchasing. There’s an equivalent of that to all the other industries out there.
How would you decide what to split test or basically how do you prioritize what to A/B test?
Prioritization is a hard thing as a product manager in general. I think about A/B testing in two ways which are; what’s the level of effort that’s involved in this type of A/B tests and what’s the potential impact of the test if it’s successful. You can plot them on the wall and give each score and then find the test that might have the lowest level of effort but the highest potential impact and start from there.
That’s typically how we’ll prioritize our tests here. We use different tools that are out there to manage the testing process. If you are using that type of scoring it can help you prioritize your work.
As somebody who was in data science and got pulled into a PM position what are some of the qualitative aspects of being a PM?
That’s a great path to product management, I think. Being analytical is a huge plus. On the other side, I think leadership is a huge part of it and communication is the other. I think you need to be able to read through influence rather than authority.
As a Product Manager, you’re not necessarily a manager to any of the people that you’re working with on a day-to-day basis, but you’re trying to influence them that your idea for this product is going to be the best. So the question is how do you do that in a way that you know is charismatic and inspiring to the others. I think those are two areas to think about when coming to a Product Manager’s role.
Only by experimenting will you know if your ideas work or not and that can save you a lot of time and money. If it turns out not working, don’t worry. Start working on something else and that might work better. You won’t know unless you try. According to Optimizely’s own words “Every great invention was born from experimentation.”
Have any comments? Tweet us @ProductSchool
We teach product management courses in San Francisco, Santa Clara, Los Angeles, and New York. To learn more about our upcoming courses and how to apply click over to our course page.