All the Ways That A/B Testing Sucks by Yammer PM
A/B testing can be Product Manager’s best friend. What is A/B testing and are the pros and cons of using it? It’s false thinking that A/B testing means creating a new experience, giving that to all of your users and then checking what happens afterward. This is not how it works. A/B testing is a diagnostic tool, it’s not a cure. It’s only as good as the questions you bring to it, and you don’t give it to all your users. But it’s not perfect, and at times it really sucks.
Anna Marie Clifton
Product Manager at Yammer for about a year. Prior to Yammer worked as a PM at Asana. Before breaking into product she was a gallery manager in New York. Co-hosts Clearly Product Podcast that publishes new content once a month.
What is A/B testing?
A/B testing is something that is a big part of being a Product Manager. To put it simply it’s a statistically valid way to see how good your future ideas are. You take an experience, create a comparable experience and launch that to half of your users. By releasing two versions at the same time, you can measure what works better and what affects the users. Then you follow the metrics and choose the one that did better.
The reason behind launching the new feature only to 50% of your users lies in the fact that you can compare the results to the other 50% and see which one did better. You also have to have the both experiments running at the same time to be able to define what events e.g. holidays affected the results.
The rosy side of A/B testing
After spending so much time on your test, it feels great at the end when you can choose one and say that it did better. By A/B testing features you get lots of valuable data about what to do and what the users prefer. You can make your product better and your users happier.
Why A/B testing sucks from the engineering point of view
It can be tough for Product Managers to run A/B tests but they’re not the only ones it’s hard for. The engineers also might find it tough because writing code for the experiment takes longer than coding for shipping. They need to build an experience that works as well as a control experience, and that takes a lot longer than just building and shipping.
They also can’t make changes when the experiment is running. If a bug was found in the test in the worst case the experiment has to be stopped, the bug needs to be fixed, and then the test can be started again. In some cases, the bug may have to be left there and not interrupt the experiment, but sometimes the bug can cause a negative user experience.
Why A/B testing sucks for PM’s
As much as A/B testing brings joy to everyone in the team when it’s finished during the process, it can be a pain in the a**. It sucks for PM’s because it takes a lot of time. You want to find a retention to measure, but retention takes a lot of time to measure. You need a ton of users on both sides of the test, but that doesn’t happen in one day, so you have to wait for that. Users also don’t like change and sometimes they don’t act the way you’d want or expect them to, and so again you have to wait.
It’s also a lot of work beforehand. You need to define the questions you want answers to because A/B testing is not going to give you an answer to a question that you don’t ask. That can take a while. A lot of work is required after the experiment as well because the results tell you what but not why.
Questions from the audience
How do you get comfortable with the possibility of things going wrong?
It’s experience. It’s failing and watching yourself not get fired and repeating the same thing all over again. I talked about this to my boss’ boss and about three-four months after I started at Yammer. I said that I’m scared of being fired all the time and that I’m messing something up. He responded that it took him about a year to get over the fear of thinking that he will get fired every day.
As product managers, we have a tendency to be more risk tolerant and to deal with that. I’d encourage you to do two things. Think about the times that you failed and how bad it really was. It’s usually worse in your mind than it actually was in reality. Look for other people that have failed and how they’re still doing because no one is in jail for messing up an A/B test.
You will fail, and you need to own that. You have to come to terms with that, and that will help you figure it out. Mentors will help you as well.
Are there any other types of testing than hypothesis driven testing?
I’ve never worked in an environment that didn’t. I think people assume that you build an experiment and then you look at the data and then you learn something. There’s that pre-work of establishing what you plan to learn before you build the experiment and then you look at the data see did you learn it was it true or not.
It drives for really pure hypothesis and pure product development process and it helps you make really effective trade-offs around is this necessary in order to learn the thing we want to learn or not which is one of the most helpful things about it.
Some can be more intuition driven rather than hypothesis and testing driven but if you’re going to do an A/B test you’re really going to want to have a really solid hypothesis before you kick one off.
How does one land a job at Yammer?
There’s a lot written about this on Medium and other sources. My experience was that I came from Asana and the PM at Asana had worked with the PM at Yammer before, and we got a coffee, and it all turned into a beginning of that conversation.
We have a product homework, so it’s a very product exercise driven process. I’ve never once looked at a resume of someone that has applied at Yammer. Our recruiter will do that and a minor screening. Then we have a product challenge which is a home assignment about what people think about product, testing, and hypothesis and that’s the only thing we look at. That’s the resume to determine whether to interview someone or not.
After all the testing and time consumed in it the sad part is that you still don’t know for 100% certainty that the thing you saw happen happened because of what you did. You just have to trust the test and see where it takes you. A/B testing totally sucks but it’s still worth it.