Uber Product Manager, Randy Edgar, reveals how to crack the PM interview
Job interview. Is there anything more scary than that? You walk into a room, talk about yourself and your abilities and answer their questions the best way you. When it’s finished, you wait for them to contact you and hope for the best. We’ve all been there but fear it no more. Here are the keys on how to rule a job interview.
Former Facebook Product Manager of Enterprise Product, Randy Edgar, has done hundreds of job interviews. He does not lack experience in that field. Previously he worked six years at Facebook and is now the Group Product Manager at Uber. He studied Business Administration and Management and ended up in product about ten years ago.
In Product School’s event, he shared his insights on how to crack the PM interview. The main thing to remember is “never say no to an interview because you never know how it’s going to work out.” The key points that he went through in his presentation were what are Product Management Managers looking for when they are hiring Product Managers and when they ask a question you know what they’re looking for and what the signals for what they’re looking for are. He also shared some tips for how things work out and what guidelines to follow with some questions.
Goals & Metrics focus
In an interview, they will ask you questions about a product. They might ask, for example, why are you building it. The right way to answer to this is not to say that you want to do it for the consumers but to specify what exactly are the goals and metrics that you are after. On the other hand, if they ask you what you would like to build for their company the response has to be tied to the company’s goals and missions.
The perfect answer according to Randy would start with “I want to build product X for your company because the company’s goal is Y.” The sentence would continue with “And how I would measure that is with the key metric Z.” If this is unsure to you then Randy suggests “answering a question with a question to find out what they’re looking for and that’s how you nail an interview.”
Importance of Data and Innovation
Decision making in product management has to be data-driven. Always. The reason that the product you want to build “is cool” is not good enough. The argument needs to be supported by data and you need to mention this fact in the interview.
Every hiring manager values innovation. What they want to know is what you are bringing into the company and to the team. When asked what you would want to build for their company all you need is “one good idea.” Randy’s advise is not to be afraid of getting up and using the white board for brainstorming and innovating on site. The person interviewing you might even join you for brainstorming!
Depending on the company you are applying for, the level of the technicality of the job varies a lot. If you apply for Google, you’ll know that the job will be technically tough and, therefore so is the interview, but if it’s some smaller company and you’re not sure about it, ask. The reason behind this is that you want to know how high the bar is set. It’s useful not only for the applicant to know but also for the company.
A protip that Randy gives is to do your research on the company before the interview. When you do it, you know what to expect. Also you will know what or who has succeeded and what or who hasn’t, and you know the things to avoid from saying in an interview.
Do you have the abilities to execute plans and lead as well as vision?
What is essential for a product manager is to have the ability to be a leader in the team, have a vision on what direction the product is going and be able to execute plans. “What does this product look like in two years?” is one of the questions that a product manager needs to be able to answer at all times. He needs to have a somewhat clear vision in his mind about the product.
Also a part of leadership is the ability to get along with the designers and engineers in the team. This is a crucial part of getting things done, and in an interview, you need to be able to prove that you can do it. Randy’s suggestion is to make a three-minute speech that proves that point. An important note is that it doesn’t necessarily mean that you can get things done but that you can get your team to execute your vision.
“Never talk for longer than two minutes without asking questions!”, Randy says. The most asked question in interviews is “tell us about yourself.” Plan ahead what to say in two minutes and practice it. Nobody wants to listen to you talk about yourself for 15 minutes. Companies want communication and to know that you have the skills it requires. They want to hire someone they like so the most important thing is to communicate and smile. For anyone interested he recommends a book called “Blank” by Malcolm Gladwell on this matter.
Communication within the team is very important for a product manager, whether it’s about getting along with the designers or the engineers. You need to make compromises and be collaborative. It’s a two-way street and requires mutual respect.
Typical interview questions
– Tell me about yourself
– How do you measure the success of a product?
– Product X’s usage is down, what would you do next?
– Pick a product and tell me a feature they should build next. Why?
– What would be your process for prioritizing features on your roadmap?
– Tell me about a product that’s well designed, or that’s not well designed
If you practice answering the questions above you can’t go wrong. Also the main thing is like Randy said figuring out what they’re looking for. That is the key to any interview, and when you have that figured out, nothing can stop you.
Being a Product Design Scientist
Chris Abad from UserTesting tells us what it’s like being a Product Design Scientist
A generally known fact is that it’s not always easy for Product Managers to gain the trust of the designers and engineers in their team. Especially if the PM is new to the field, proving the team that he’s reliable and worth their trust can be challenging. But what if the Product Manager is a designer himself?
Chris Abad is the Head of Product and Design at UserTesting. He has 15 years of experience in product and has been working at UserTesting now for about a year. He began his career by starting a bootstrap company with some friends after college. The company did social media advertising.
His second startup was a venture-backed company which did online classifieds and social gaming on top of other things. Later on, he sold the company. Even though most of the things he did in the past with his companies were product management, he hadn’t realized this fact until later. The point where he started thinking of himself as a Product Manager was when he joined larger companies and had to define what he had done and how he would fit in the team.
In a recent webinar by Product School, Chris talked about what it’s like being a Product Design Scientist. He studied art and graphic design back in school, and now his work combines both design and product. According to his words to him “modern product management combines a scientific approach with a designer’s toolkit & perspective.” What he means is that in his opinion modern product management overlaps a lot with design.
A lot of people think that design is just about the way things look. To Chris, it’s also about how they work. “Design is having a perspective on the world. It is an approach to moving from where we are now to where we want to go to”, he says.
Chris also compares design to product management and agrees with David Kelley, the founder of IDEO, that the both of them have the same goal which is “to have empathy for the people you’re trying to design or make the enhancements for.” “Product management tries to understand users, design tries to understand people it’s designing for,” David Kelley has said.
How does science fit into all this?
Chris also talks about product saying that even though it might not sound exactly like design, it is starting to sound a lot like science. To prove this point he quotes Marty Cagan, a partner at SV Product Group, and says “the general principle of product discovery is to find the fastest and cheapest way to validate our hypothesis.” A product manager has to be able to think like a scientist, have scientific approaches to form a hypothesis and to test it.
In modern product management, a lot of experiments are run, and that’s where the “experiment loop” comes in handy. The loop consists of four phases. The first is having a hypothesis, the second is building a prototype, the third is testing it, and the fourth is to learn & decide. Chris says that working like this will help product managers to think scientifically.
Being a Product Management Scientist
Chris argues that the reason why they call it being a Product Management Scientist is that they believe those are the elements that gather it together to make good product management practices. It can make companies more effective if they combined design techniques to the scientifical approach.
At the end of his presentation, he recommends a great book for product managers to read on how to solve customers’ problems through a product very quickly. It’s called “Sprint” by the Google Ventures Team. It includes a five-day process to form the initial opinion on solving the problem. The five stages will be repeated throughout the 6-week process so that on the first week the stages are defined, and during the other five weeks, the pace is maintained to get into a loop of working.
Questions from the listeners
How would this work in a scenario without the luxury of a lot of customers?
Having a lot of customers can be an advantage. When we worked with Salesforce, we were lucky enough to not only have a lot of customers to rely on, but we also had a research team that helped us with recruiting because it’s quite difficult to get people lined up every single week.
There are a couple of things that you can do. If you have an early stage startup and that’s the reason you don’t have customers you still certainly have target customers. You have people that you’re trying to go after and you need to figure out how you’re going to find those people. When you have a product, you’re going to have to solve this problem and figure out how to find them. They might be online, in physical spaces or maybe in conferences they attend. You need to go looking for them.
For a consumer product, it may be a little bit easier. You might find customers in certain stores or other locations that you go to. You can try to grab people off the street. Being here at UserTesting this is one of the things that we think about a lot. We hear all the time that finding people is one of the hardest things to do when it comes to doing this sort of research. That’s a need that we try to fill. We try to make that easy for people. We have a large panel of people, over a million, at our disposal and we build software to help people to get easy access to that.
On weekly sprint how do you keep a steady pace or flow for the design team throughout the sprint?
I’m going to interpret the question in a particular way. There are a couple of things that come to my mind. One of them, and I would say this isn’t just about the design team, is in general how you keep the pace. You have to be incredibly focused. One of the biggest problems that I see and that I’ve had with some of my teams is that you’re trying to do too many things at the same time. You can’t maintain that type of pace and that type of progression if you’re juggling different things at once.
What we did one time is that we reserved a room and we physically moved everyone’s desks in there. We were not allowed to leave. That became our war room, and we stayed in that room for six weeks. We didn’t work on any other projects which was a big factor and what allowed us to keep that focus and that pace.
That’s one part of what I think the question might’ve been about. The other part is how to keep the design team focused. Let’s think of a situation where the design team is somewhere separate from that activity, and you have to keep an eye on them to keep them moving forward. At the same time, you’re trying to drive the project. I think it is critical that design is a partner in that process.
I strongly believe that a lot of it is about communicating how to do product discovery. I’m also a strong believer of the fact that product discovery can’t happen unless you have the right people in the room. Design is in a critical role in that practice so ideally what I think is if it’s done right it’s not really a question of how I keep the design team focused because the team is sitting right there with you. They have the same focus in the same activities and the same priorities as you have.
How do you convince stakeholders that you need to run more experiments when they believe that what we have is currently good enough to ship and we should just ship it?
I’ve certainly spent a lot of time in my career having to fight the battle of convincing people that these sort of activities are useful. I honestly think that there are a couple of things that I’ve done that I would say have been the most effective. One of them is to try to shift the conversation towards the outcomes and the results because quite frankly if you’re shipping products without doing this and they are having a great impact on the company, and you’re acquiring users they’re not needed.
The whole point of this is that if we ship things that we think are going to be amazing and have a great impact on people we expect them to work. When they don’t work we backtrack and say “well okay that didn’t work well. How are we going to do it differently?” Learning to push that focus away from shipping and instead highlighting the impact is important.
That has been my go-to. What we want to do is save ourselves a headache and embarrassment of shipping something that nobody wants and that doesn’t work. We try to get ahead of it and convince people that it’s most likely not going to do the thing that we think it will so we should do experiments.
Sometimes that doesn’t work. I’ve been in lots of situations where you have to let it happen. Then you retrospectively go back and say “Well we shipped this thing. We tried it that way. Here’s the impact that it had. That’s not quite what we were looking for. Now that we see that let’s try to do it a different way.” Sometimes you have to feel the pain internally within your company before there’s enough momentum and desire to go and try to do things differently.
Do you have any advice to those who want to break into the industry of product management?
I think just to tie it back to what I’ve been talking about I would say I have a pretty particular perspective on how it has worked for me. I think good product management is about context. What has worked for me is the type of products that I’ve worked on and the customers that I’ve tried to build products for but it isn’t the only type of management there is. For lots of other scenarios, there’s definitely more of a business mindset that makes a lot of sense and they need more of a technical mindset.
It’s not really a silver bullet or one-size-fits-all model. I would say what I think is important is to be clear about what you’re most interested in and what fits your personality the most. Play to your strengths and build a career in product management around that. There are certainly different tracks out there and this is the track that I’m on. This is the type of Product Manager I look for because it works for us.
There are definitely different ways to get into product management. Some may be more difficult than others, but it’s important to make the journey and not give up if it’s what one really wants. Don’t give away your visions and personality. There is room for everyone.
Jessica Uelmen is a Product School graduate. She recently got a new Product Manager job at a company called Fitbit. We took her out for a coffee to congratulate her. While out we asked her a few questions about the work, how she got the job and how did Product School help her in the process. Sneak peek for all of the fitbitters out there; she will be improving the product for you!
Laura: To start with, tell us a little bit about your progression into product management.
Jessica: I suppose it’s a little non-traditional as they say all the time. I came from an engineering background; I’m an electrical engineer. I worked for a small micro controlling robotics company where I moved my way up in that organization by first being the Engineering Manager into being a Director. I did, for example, a lot of business deals.
I didn’t realize until later on at Product school that a lot of things that I was doing at the time were mainly product management. Then, though, I didn’t even know what a Product Manager was or if it was a proper job description.
I moved to Silicon Valley and started working for a start-up called Udacity. At Udacity I did program management and later on transferred to the product and the growth team. Then I found Product School and wanted to get started. I hadn’t heard of product management and didn’t know how one can do it. That’s me in a nutshell.
Laura:Did you dive into the course with the hope of getting a new Product Manager job or were you just looking to learn and grow?
Jessica: I was looking to learn and grow and to get a more official definition of what Product Managers do. I think everybody had their definitions. I heard a lot of different things from many different people about product management, and I wanted to focus on an actual process of it instead of hearing just bits and pieces.
Laura: I don’t think it’s uncommon that people land on product management roles even though they didn’t study product management in college or receive formal training. I’m glad that you were able to get that training from Product School and now you’re at Fitbit. Can you tell us about your position? How did you get the job? What was the hiring process like?
Jessica: I’m joining the devices team and mainly what I’ll be focusing on at first is UI and UX and how your device connects to your mobile app or your desktop app. Specifically, I’ll be focusing on onboarding it and making the first experience with a new app a little bit better hopefully.
As for the interview process, it was one of the first things where I came across with the job description. I think it was LinkedIn that sent me a “you might be fit for this” sort of notification. I went to the website and submitted my application but I didn’t think that I’d be hearing back from them and that I was just putting it into the void.
However, I got a call later from the recruiter about a couple of weeks later. I did a phone screening with the recruiter which went well and I also did a phone screening with a person who’s now my manager. I also had an on-site talk with about five or six people where I had to talk about product management principles and ideas, my technical background, what I had worked on and what I hadn’t. They asked pretty standard interview questions, I think.
Laura:To unpack the onset a little further today, what do you have on the whiteboard? Are you playing with data or..?
Jessica: Not so much. There were pictures of a new feature component. There were also things planned for the users like “what do you like about your Fitbit,” “what do you not like” and “what do you think we can improve.” There were a lot of product insight questions. There was a lot about working in teams, how would I interview engineers, how do I get things done and what are my general working processes as well.
Laura: You must’ve crushed the interview since you got the job! What skills do you feel contributed to your success in getting the job?
Jessica: I think it was the work that I had done in the past related to product management. I come from hard work background. I’m an electrical engineer and I had done a lot of product related things even though I wasn’t a Product Manager. In my previous job I had been defining what the company was going to work on, what the new development was going to be and why we were doing it.
I think I was able to find the right experiences and samples of my past work. I dug deeper into how I wrote specifications, gathered requirements and justified the reasons of how I made decisions in the past. We talked about the times when it was hard to get engineers and others on board with an idea and how I went through that
Laura: You’re right. It’s important for those who are out there looking for jobs to find the right fit. It’s not just about what’s on your resume but about the fact that you’re targeted by jobs in companies whose problems you can solve based on what you’ve done in the past. You’ve been in the new role now for 90 days.
Do you have any advice for the first 90 days as a Product Manager?
Jessica: Yes. Ask a lot of questions. That’s really been helping me. I’ve gone up to people saying “Hi! I’m new, I’m dumb, tell me things.” It helps you to get to know the basics, the plans and the people you’ll be working with, and helps you to make better decisions later. That’s where I am right now. I’m starting to develop my own ideas validated by talking to people. I’m asking a lot of questions and people are always happy to answer. It’s a little scary to ask them sometimes, but they are glad to answer.
Laura: Let’s go to your experiences at Product School. How did Product School help you prepare for this transition?
Jessica: Honestly, I think, in a lot of ways. It gave me confidence.It gave me some structure to look at what product management is as a role. I was able to form past experiences and say “I’ve done pieces of this and that” even though it might not have been an entire project or an entire life cycle of a product. Product School helped me realize that I belong in the field. It also gave me the practice to look at things from an outside perspective of a company and confidence to say to myself: “I got this.”
Laura:What advice do you have for our aspiring Product Managers out there?
Jessica: I know this is going to sound silly, but I want to go back to reminding people to ask a lot of questions. It helps. Also, have confidence in yourself. Tie things to personal experience as much as you can. If there’s something in what you’re learning now that relates to something you’ve done in the past go back and journal about it and reflect on it. That will build your confidence and help you in interviews and in the future.
Changing careers or starting something new can put your confidence to a test. It can be challenging and you may feel like you don’t want to do it. However, having the confidence to believe in yourself and the willpower to do it will reward you in the end. Product Management courses help aspiring Product Managers succeed in an interview and give useful tools for the working world.
We teach product management courses in San Francisco, Santa Clara, Los Angeles, and New York. To learn more about our upcoming courses and how to apply click over to our course page.
Growth Explained by Facebook’s Core Product Manager
Yaron Fidler from Facebook shares his keys to growth.
What is growth? Why is growth important to the product? Is it just about the financial benefit? Is it more important to gain more users or to keep the old ones happy? To keep the company blooming, the product needs to be pushed the right direction. Understanding growth is vital for the process.
Yaron Fidler is Facebook’s Core Product Manager. He started as a data engineer before moving to Product Management and now has nearly nine years of experience managing global data operations, the Internet industry and leading teams. At Facebook, he worked on growth for a year and a half. At the moment he is working on ads. Before joining the Facebook team, he was employed by Ebay. He worked under different titles mainly focusing on SEO and structured data.
In the recent event, he talked about the key points about growth and why it is important. For every existing company, the purpose of its existence is to do well and to grow. Growth answers whether there is product market fit for the product or not.
Many people would think that the main reason for growing is profit. However, Yaron says that it’s not the case. For non-profit companies, the purpose is to grow their market and to get more users, not to gain more money. Uber is a great example of this. The company is not making profit but it’s growing all the time to new countries.
Choose the right growth metric for the product
For every product, there should be one growth metric set to reflect the product market fit and to show future financial outcomes. That metric is chosen at the earliest state possible, and it shouldn’t be changed unless for some reason it is the wrong one. It can be challenging identifying the metric at first but once done the company should stick with it.
For Facebook, for example, the growth metric is the amount of daily or monthly users. Each month they can see in their data exactly how the number of those users increased or decreased. They can discover, not only how many people joined that month and how many stopped using it, but also how many active users, people that use the product more than once a month, they have.
For other companies, of course, the metrics are different. For Ebay, for example, the growth metric is about the gross merchandise value. For them, it’s not about the number of users or the amount of money they make but instead how much money goes through their product. They need buyers as well as sellers, and their product connects these two groups.
Create the required data for growth
Growth accounting, according to Yaron, consists of three types of user profiles. In the first group are the new users. They are the ones that have just discovered the product and have started to use it. The second group includes resurrected users that haven’t used the product in a while but have returned now. The users that have stopped using the product altogether create the last group.
With these three factors, Yaron creates a formula for measuring growth easily. The formula counts together the amount of new users and the resurrected users. After adding the two together the number of users that stopped using the product is deducted. This formula creates clear data about the users in percentages that can be used later for evaluation. “Growth products must be data-driven,” Yaron says. “Companies that manage to grow massively, like Facebook and Google, are companies that are very data-driven.”
How to get users and keep them?
The goal, however, is not only getting more users but also to get the already existing users to come back and to use the product again. Yaron lists different tools and tactics that will help companies to get more people discover their product, to convince those people to use the product and to keep the active users coming back.
The things that will help products get discovered are, for example, SEO, Google ads and invites. For converting the people the most important thing is to remove friction and concentrate on the product’s layout. Already active users, on the other hand, want to find value in the product and to keep getting new user experiences. “Users already using the product have found value in it, and they’re easier to retain. What is difficult is acquiring a new one.”
Growth is about balancing between trying to keep the old users but at the same time acquiring new ones. Nobody thought when Facebook came out that it would reach 1 billion users some day. Now its goal is to get 2 billion. Growth can be surprising but when it’s done well it creates super companies.
We teach product management courses in San Francisco, Santa Clara, Los Angeles, and New York. To learn more about our upcoming courses and how to apply click over to our course page.
The Thing about A/B tests – To Learn or To Ship?
There is no such thing as good or bad idea. In the beginning, there are simply ideas. To find out if those ideas would work in the future they need to be A/B tested but what is A/B testing? What is the difference between “to ship” and “to learn” testing? What do Product Managers have to keep in mind when doing A/B testing? Anna Marie Clifton from Yammer will answer these questions.
Anna Marie Clifton is the Product Manager for Yammer and also the co-host for the Clearly Product Book Club Podcast that comes out monthly. She started by managing a gallery in New York City but later switched to Product Management. Yammer is a collaboration tool that was acquired by Microsoft in 2012. It is a data-driven company which is one of the things that has made it strong over the years. Anna Marie has been doing projects on iOS, Android, The Web and is currently leading the notifications initiative and working on some algorithms and search based projects.
In the recent Webinar she talked about what A/B testing is, how to do it effectively and the best things you can do as Product Manager when it comes to completing the tests. “A/B testing, also called split testing, is in its simplest form a statistically valid way to go about seeing how good or bad the future ideas are. In A/B testing the idea is to release two different versions of a feature to a random set of users and then to measure what those users do relative to each other. To complete it you need thousands of users doing that particular thing in order to split test the idea.”
“The thing that is extremely important is that the two large groups have alternate versions of the feature at the same time. The reason for this is that there are a number of things that can happen that are out of the person’s control. However, you want to do your best in trying to control those factors. To avoid any bias based on the time series event, it has to happen at the same time with randomly selected users and in large sample sizes.”
She talked about two fundamental types of A/B testing which are “to ship” or “to learn.” “A Product Manager should try to determine which type of test they’re working on as soon as possible in the development process so that they can use that to make the trade-off calls and drive development faster. The purpose behind “to ship” type of testing is that they usually fix something or they are done for product completeness.”
On the other hand, the “to learn” type of test might not be a complete experience at all. “It might be only a front-end test without any back-end connected to it. Once the test type has been decided, the baseline for how to make trade-off calls needs to be established. As a Product Manager, you want to make sure that when you’ve decided the test type, you communicate it clearly to your engineers and designers and that you’re making decisions that are backed up by what you’re planning to do.”
“The best thing a Product Manager can do for his team during A/B testing to make people move faster is to remove ambiguity and help clear the path for the team to work. In A/B testing environments, people are normally testing twice as many things as they ever end up shipping. On average you can expect four or five of your tests to fail for every four or five tests that succeed which is a high percentage of tests that you expect to lose. They are expected to be bad ideas on purpose because you are working on new creative and innovative ideas. However, you can’t always run tests “to learn.” At times you have to run tests and ship it if it wins and move on.”
She also gives Product Managers a pro tip when she says that they should keep in mind what their main objective is in the company. “Even though you might feel like you’re doing a lot of unnecessary “to learn” testing or iteration your engineers and designers don’t think so. They think about, for example, the technical debt and visual inconsistencies which are what you should think about as well but don’t forget that the most important thing for you as a Product Manager is to keep the company afloat and that is really what the team is tasked to do.”
Questions from viewers:
What do you typically do when there are no definitive results from your test?
There’s almost never a very clear-cut case. You follow which metrics move on the global level, and those will be your top-line metrics. At the end of the test, you’ll see which metrics moved on and which didn’t. Typically they move very little and they’re very hard to affect which is why they are your top-line metrics.
Secondly, you’re looking at the local metrics. There can also be a mismatch meaning that the local metrics are doing what the global metrics were expected to do and if that doesn’t create growth you need to stop it. It grows into a lively conversation trying to define why the metrics moved in that particular way. From those little changes the Product Manager has to make direct conclusions and decide which way the project will move.
How do you overcome the challenge of explaining a product that’s not yet built?
I haven’t done a lot of this in the past year and a half but a Product Manager has to be capable of storytelling about data to tie the results together and to be able to tell how the metrics moved when they changed so little. Therefore, a Product Manager needs to be able to do a little storytelling about the product itself as well. A good point to start with, would be figuring out who you’re communicating with, understand what the product would mean to them and what their needs are.
You mentioned that for A/B testing you should really have thousands of users. Do you have any recommendations for Product Managers with 10 to 30 enterprise clients as opposed to thousands of consumers?
One of the things about Yammer is that we’re an enterprise product. We don’t test on enterprise clients, but instead, we test on their users. For example you may have 10 to 30 enterprises with several hundred or several thousand end-users each and you can carry out tests on these end-users. You don’t have to do it on the network level.
At Yammer most of the things that we test, we can isolate user-to-user. If you don’t have thousands of end-users that you can test in the A/B way then there’s also a lot of customer driven development that you can do.
What are your thoughts about using session replay tools when doing A/B testing?
We don’t do a lot of sessions at Yammer so what I think you’re probably asking is, how to map out the number of users that do things and in what order they do them. The reason why we don’t do many sessions is because session tracking is complicated. At Yammer we’ve built our core metrics to be robust enough that we don’t need to measure sessions to find out why people are doing something.
As mentioned earlier, it’s the Product Manager’s job to tie together the global metrics and the local metrics but session tracking is a layer between the two and it can confuse them. It’s also technically difficult to perform and I wouldn’t rely on it. There are third-party tools that can do that for you but they tend to be broad definitions, and you don’t have direct control over what the experience will be like for the user.
How do you prioritize features that should go in an A/B test? I understand it’s mostly about prioritizing and not related to A/B test but what feature types have you found that have worked for an A/B test?
I’ll give an example. We have a project based on being able to edit posts and we are testing with end-users in big networks. If you imagine that one user created a post and then edited the post afterward, and another user that was not in the experiment and doesn’t have the possibility to edit posts or see edited posts went to see it, they would only ever see the first version of this post. However, the first person would think that everyone can see his edited post. This would break the user experience across the board. This is one of the situations you have to keep in mind and then decide whether you want the users to be able to interact with each other or does the feature need to be available for everyone.
Can you share your advice or insights for those that are aspiring Product Managers and want to break into the field?
Before I was a Product Manager I was managing an art gallery and knew that product management was what I wanted to do but I didn’t have the background to do it. I want to be very encouraging because when you change careers you’re going to get more rejections than you ever get acceptance. Be emotionally prepared to be rejected and don’t take it personally. You have to handle that because even when you get a job you will get a lot of rejection from the customers, engineers and designers. The best way to get into it is to do it on the side working with someone building a little project. You need to be hanging out with people who are making those kinds of things. There are a lot of free tools available that you can use to help you.
If you want to get into product management, the best way to learn is to ask questions. Ask as often as possible. That’s why we’ve started weekly AMA sessions in our Slack Community for aspiring product managers or current PM’s who are curious about the processes of others.
This past week we had the chance to ask any question imaginable about product management to one of the most influential PM’s in the industry, Shiva Rajaraman, a product guru who has worked as:
– VP of Product at Spotify
– Director of Product Management at Google
– Senior Product Manager at Twitter
– Senior Product Manager at YouTube
Shiva Rajaraman during his talk at Product School New York.
We chatted about everything from finding key metrics, overcoming the biggest product launch challenges, to the most sought after skills for a future product manager. Take a look below:
Product Management Skills & Qualifications:
What do you think are the minimum core qualifications or competencies (academically speaking) on both the technical and management side to be a successful software PM?
1) Be technically/analytically competent in something. That may be financial modeling or design; chemical engineering or CS. A lot of what we do in product management requires you to be adept in data analysis and especially interpretation of data so being weak there is a non-starter.
2) “Management” side – I prefer great storytellers and succinct communicators. For example, I and my peers often employ interviews that involve delivering a presentation to a group. Those present in the room may critique the underlying assumptions or go in a bit deeper to get to your underlying rationale for a decision or how you framed the options. Accordingly, you need to be great at explaining things and keeping your cool while you answer smart (and often not so smart questions).
Beyond that, it’s helpful to have end-to-end responsibility for a major project, company, volunteer gig, whatever where you were the accountable decision maker, and you were also the most passionate stakeholder. That doesn’t have to be in a professional capacity, but it’s helpful to have experience, where you had to run the show.
What qualities in a Software Engineer do you need to be a good product manager and what qualities he/she needs to develop? Also, how can he/she make a smooth transition to product manager role?
Generally for new PMs here are some tips:
1) Understand a clear story about “what success looks like” for your entire company
2) Start to understand how you would measure that and how your product or area impacts that directly
3) Focus on having the most impact against that top level goal – don’t get too caught up in tons of details – be the person that steps back and says are we on the right path or not?
4) be objective and own it – if your team or your tactics aren’t having an impact be the voice of reason that you should revisit your hypothesis or try a new approach
I often hear from other PMs that you don’t have to “know” code to be PM, however, in my experience as a Web Producer, it’s often very helpful to at least know how to read code (i.e., speaking with developers, troubleshooting your product, etc). How would you balance this skills?
The analogy for me is a building architect vs. someone who handles construction. You want to have a good sense as a PM what are the limits. Sometimes it’s physics or the speed of light. Sometimes it’s just that there are design/engineering norms you should understand. So what I ask is that PMs do a 1 day “understand the stack” as they get started on new efforts. They should be technical enough to comprehend what that means w/o having to code. The risk at times with being too technical is that you really become a tactical project manager vs. helping to set vision and progress against that vision – that is usually where people fall short.
On preparing for a career in product management:
What would you recommend to a new PM to learn good behaviors/habits who is working on a small/young product team at a startup?
Great habit is to look at what other people are launching and shipping and ask yourself, what do you think they thought they were doing and what would success look like them? 10K people using the feature to do what? Have more engagement for a type of user? Attract a new type of user? In other words, reverse engineer their goal and how they counter-position against the norm. The world is a series of experiments going on at any time simultaneously. Start guessing what they are trying to test or achieve.
What advice would you give to a traditional project manager looking to move into a more software/technical product management role?
Project management gives you the attention to detail that is a potential foundation for Product Management but one of the best things you can do is to start understanding (or helping formulate) the “why behind the what” which is key for product awesomeness. In many cases, you can start to make that a part of the stories or briefs behind your projects and you as a project manager can remind teams to measure or talk about goals or progress against goals that were part of the “why”. That’s great practice to make the transition clean.
Insights from past work at Spotify
What was the most important thing you kept in mind while building the Spotify app?
Overall, a lot of what we do in product during a growth phase is balancing differentiation (getting people into the funnel) with retention (engaging people in ways that make them come back). Differentiation at Spotify or YouTube was often about differentiated content experiences (unique creators only on YouTube, unique content such as the behind the lyrics Genius cards associated with top tracks, or unique use cases such as using Spotify to keep you on pace for a run with music that is synced to a specific beat). Engagement is often about personalization (Discover Weekly or Daily Mix on Spotify or taste-based recommendations on YouTube)
Here we use the Spotify model of Tribes/Squads/Chapters/Guilds/etc., where does `customer/user journey` fits in this “feature centric” structure? How does Spotify deal with it (besides having the Glue team)?
Tribes own missions at Spotify and must own clear customer/user journeys. Used this model at YouTube as well where you have a series of constituent-oriented groups – Viewer, Creator, etc. that have multiple teams within them. The value of constituent-oriented groups is that you ensure you have a champion for a clear user types/journeys in addition to good organizations beneath them to ideate and execute on various goals. Over time you may not need “customer-oriented” organization structures as various teams stay focused on end-to-end tasks and appreciate multiple points of view as they mature. However, often it’s helpful to do this when you have an imbalance in attention or perspective you need to correct.
Decision making & development processes:
I am a big believer in agile and lean but have never done agile at the scale as at Spotify. What sort of challenges a Product Manager faces with agile culture in large scale, how do you deal with them?
Agile important but great to have missions that are org wide.
Good recipe for product excellence overall:
1) Craft missions that everyone in org believes in and tie to company goals
2) Don’t let everyone solve the same mission but ensure anyone who owns a mission can act in any way to make an impact – don’t silo them to a specific code base, platform, or feature/view set. Otherwise, they will try to solve the goal but only with resources at disposal, or only by exerting pixels they can control.
3) Organize agile teams around missions – they inherit goals but can focus on unique tactics. However, you don’t really want a bunch of teams coming up with a cacophony of goals. Companies need to have priorities. Similarly, you don’t want a lot of top-down “here’s what you need to do” – that stifles innovation.
4) Finally, if you don’t have clear goals and each team invents their own goals you have a disaster so remember to spend time ensuring people understand the goals and if need be pin those goals up and over-emphasize them at all hands and in written or video communication.
How quickly do you turn customer feedback into feature requests? Do you have a process in place for placing this information to be reviewed and then determined if a change of action in Product Roadmap is needed?
Who can make these decisions?
Key thing is to focus on those top level goals and missions. If people have clear missions/goals (increase retention X%), then you need less top-down approval. People can demonstrate that they are moving the needle against that mission. If you are sloppy with creating true missions and goals against them you need a lot more oversight which gets frustrating for everyone involved.
How do you find a balance between building features vs. building technology?
Goes hand in hand. Think about YouTube – sometimes you have a set of features that are about rights management – that is what Content ID was in the early days. Just protect copyright. But you start to realize this is a much more interesting capability not just a set of rights tools. You are scanning 150 years of video every day and matching it with massive sets of video that have been ingested before. That is a capability built on scalable, revolutionary technology. A good lesson is to ask yourself, “when I am building this feature do I need or am I also building a great capability?” and if the answer is yes, ask yourself what other features could you support. If the investment in tech gives you option value to launch many more cool and impactful features core or complementary to your mission, you have a good idea you might want to invest in the tech and not just bolt on a feature. Content ID at YouTube. AWS at Amazon. On-demand delivery of anything at Uber.
Challenges on the job:
What are biggest challenges during the launch of a new product that you have encountered and how you resolved?
Everything you thought you knew ahead of the launch is typically thrown out the day you launch 🙂 The key thing is to remember (I know I sound like a broken record) is what you were trying to accomplish. Now you have real data. So you want to go deep and understand what happened in actuality given you had a hypothesis that led you to build something.
a) You launched something new – do you people like your proposition? Do they understand it? Are you attracting the people you thought you were? Do they try it out?
b) Assuming you have a great group you’ve attracted, and they are now using the product, start understanding how they use it (you may be surprised overtime) and live and breathe the lowlights and highlights. # of Tweets is not a metric you want to stick with for too long as fun as it is watching that timeline post-launch. Specifically, understand what usage correlates with retention – e.g. what do people who stick around 30 days after launch do in your product? And/or understand what tasks does someone complete and what to they never touch?
c) Step back after a month or so and ask yourself, “did any of my assumptions leading to this product change?” for the better? for the worse? anything new you should act on? anything obvious you need to tweak? Learn what worked but also question your fundamentals – maybe you have learned something exceptional you need to act on for good or for bad.
What were some of the challenges you faced becoming a PM? Did you become a PM at a company you’d already been working at?
Biggest challenge I faced being a PM was often being in situations where I didn’t have a lot of context. I didn’t see the biggest picture and assumed someone had one but I didn’t ask. The other challenge was finding some space to make my own bets. They didn’t have to be big ones, but I was happiest when I could also have my own canvas to try something out, however, minor and increasingly that grew in scope. This challenge is often a tradeoff – you won’t be the source of all the great ideas your company should pursue. However, finding some headspace and time to try new things is key to learning and feeling 100% utilized.
Many people talk about creating value, not features. But in practice, people usually focus on features. How do you manage that?
Remember you are trying to achieve a result. Features are a way to get there and typically in our world getting that result does involve shipping something 🙂 It’s important to stay focused on that result, but great PMs can’t be too abstract. I love PMs who obsess about the details of flows, logic, content/community, etc. but I also want to make sure you at some point you ask yourself: hey? Is this working? Why not? Do we need to tweak the feature or go back to the root of the decision tree and try again? If that means you should move on to new ideas, all good. Better to learn that fast.
How do you increase trust with the rest of the people in the company, whether stakeholders or devs, to help you with your product vision or any changes that you want to voice?
Seriously, it’s important to have cultural bonds with people – hang out and get to know each other – don’t just meet for the first time when you are negotiating a roadmap. At YouTube, we had the whole Creator Product, and Engineering team go make videos for example with the participation of xfn peers in content management/licensing/etc. We also had them interface directly with creators for a few days and understand their pain points. I’m not the biggest believer in empathy for the sake of it. However, understanding someone’s life around your product gives you great context. At Spotify, the team that built a great product for runners brought treadmills into the office. Ultimately, the best incarnation of this is to think of your “team” as not just product, design, engineering but also marketing, content, and support (+more). Define your teams from day 1 with this mentality, and you’ll bridge the org quickly.
How far into the future did you dare to look with your product roadmap? and How often did you work on product strategy?
I like to look at goals at least 6 months to a year out. For product roadmaps and associated tactics to achieve those goals, I like to stay within 6 months. So put another way, maybe you want to hit $100M in revenue a year from now. You’d plan several efforts/bets over several cycles to get there, ideally all less than 6 months in duration to prove their efficacy (e.g. a roadmap). Truly hard things take more time so those are maybe exceptions, but for the most part you should have a healthy stack of things that let you launch and learn. Also, I’ve received great advice from my mentors to focus first on the “biggest” things – e.g. don’t fill up your glass with little pebbles; fill it up with big rocks and then squeeze in the pebbles. In other words, look at next 6 months and think about your biggest efforts you need to try and then squeeze in around that if you can. Ultimately your job as a PM is to manage a portfolio of lower risk and higher risk efforts that affect your key goals as a company. Make sure you don’t fill that portfolio solely with obvious but low impact efforts.
Which is more important- user data from analytics or direct feedback from users?
It’s not either/or and given there are a few questions on research let me try to summarize a few thoughts:
1) Direct feedback is often great for first impressions. It’s hard to understand, especially for new users, why those who leave or bounce quickly do so. I believe this is the stage you just need to ask questions: what is this app about? What does it let you do? Do you know how to do it? Show me? You will lose a lot of users because they don’t get past screen 1 or day 1. You live or die by those first impressions. Logs won’t tell you much – ask them what was their impression in lab studies, in-app surveys, and/or various other research methods that involve direct feedback.
2) Instrumented behavior is great in cases where you have usage and you want to correlate that w/ levels of engagement, degree of churn, etc. Even if someone *tells* you they love the app you want to see if they actually use it and/or in many cases find out what usage correlates most with retention. This part is hard to do with in person or diary based user studies. People aren’t always great at describing what they really do or what they will do. They tend to be very good at articulating whether they love/hate something (perceived value) and whether they understand something (cognition of what they can do) but what they actually do is often best done through direct instrumentation.
What are most important metrics you suggest product owners to always keep eyes on?
A few questions on metrics so I’ll group some answers here.
1) Ultimately a lot of apps or services are about engagement (time spent) and/or tasks completed (jobs done). I would first understand your top level metric and make sure things add up to that.
2) WARNING! Don’t just measure what is easy to measure. That is the biggest trap. Proxies are fine but only if they truly map to the higher level metric. For example, at YouTube “views” are not the same as “watch time”. You can game a bunch of clicks but it’s hard to keep people watching. Remember to focus on the true measure of your service’s quality. Similarly, page visits aren’t equivalent to things bought. Maybe you only link to commerce activity but that doesn’t mean you can get away without measuring it – you need to find a real proxy. It’s easy to get people to open up a screen. You can make it bigger, flashier, or just make it the easiest thing to click on everywhere. You have to measure the ROI of every pixel so spend time understanding what your goal function really is and how to instrument it and/or the best true proxy.
3) Once you have your high-level metric, break it down. For example, engagement time is often about # of users (growth/marketing acquisition), # of visits per user (things you help them do or experience and levers to get them back), time spent per visit (better experiences and more things to do when your users come back). Each of these components may be what you truly act on. But always ensure these roll up cleanly to your top level goal so you can keep an eye on one metric cannibalizing another and/or you can ensure you understand how much impact that lever truly has.
4) Often this is very very obvious and should be obvious because these goals map to easily understood behavior. Sometimes, however, you realize it’s not. On some services, for example, engagement may not matter as you might pay for something and just keep paying even if you don’t use it. However, you likely still need to work on things to get users to come back (differentiated experiences, content, access to data, etc) which ties to user acquisition. In some cases, it is about engagement but it’s very crowded so you need to spend a lot of time with motifs, emails, or re-engagement marketing to get people into a ritual and until it’s cemented into their habits it’s all or nothing. Tactics vary tremendously by product and service even if the goals are simple.
Ultimately, your metrics should map to staple goals for your company. If they aren’t common sense linkages you may have an issue or you many have complicated things a bit too much with your cleverness 🙂
Final thoughts for breaking into product:
1) If right out the gate it’s tough to land immediately in a PM role, find a role where you can interact with product and offer unique insights that help product managers succeed and no matter what help craft hypotheses jointly with your product partners.
2) Find a domain you love and learn more about what’s going on there and live and breathe more than just product. Understand the underlying problems people are trying to solve and what assumptions are being destroyed. Note: I generally believe that PMs are generalists and great ones can work in many domains. But to start off or bridge into a role, having a good grasp and passion for a vertical helps make you an essential complement and often is key to developing great products though you can’t be brainwashed into thinking established assumptions are immutable.