A/B Testing for Product Managers: How to Optimize Your Product
If you’re getting into product management from a marketing or UX design background, you likely have experience with A/B testing and optimization. But don’t cross these responsibilities off your list just yet! The two are valuable tools for Product Managers to understand their users.
In his experience designing and building products from the ground up for more than a decade, Senior Product Manager Sean Pardo has learned a lot about optimization and A/B testing on the job. He found his way into product management while working and collaborating with designers and engineers at e-commerce startups — a niche that relies heavily on optimization for customer conversion.
Sean currently teaches a Product Gym class on A/B testing, so he was a natural go-to for learning how to successfully use optimization and A/B testing for Product Managers.
Read on for his expertise on:
- How to optimize your product as a Product Manager
- A/B testing for product management
- The hardest part about learning A/B testing for Product Managers
- Differences in product optimization across industries
Here’s what we took away from our interview with him.
How to Optimize Your Product as a Product Manager
Optimization can be applied across any part of the user funnel, and this is where product management overlaps with marketing. Marketing optimization happens at the beginning of the customer journey where it is responsible for capturing the audience at the top of the funnel. These efforts are concentrated on the awareness about the product and help to attract leads that may convert into customers.
Product optimization comes in as you get deeper into the user journey, i.e., after registration and into the actual usage of the journey. Sean summarizes it in one phrase:
Optimization is just prioritization of what is valuable.
Prioritization may be another buzzword in product marketing, but the two are essentially synonyms in the product lifecycle stage that PMs are concerned with.
“Take an example of a product like AirBnB,” Sean explains. “Marketing’s job is to get you the user on the platform. But once they are on it and are trying to set up their next trip, it’s all about prioritization of what is valuable. ‘What do you show them or not?’, ‘What are you trying to get them to do first?’. What you prioritize in the user journey is all part of optimization.”
Put simply, “The beginning of optimization is marketing, the middle is product management, and the end is passed on to operations and customer support or customer success, which are just as important allies,” he says.
A/B Testing for Product Management
According to Sean, A/B testing is the effort of trying to understand the relationship between what you’re building and how your users are responding to it.
“It’s really about being able to evaluate what you’re contributing and what you’re not contributing. Ask yourself, ‘If I change this button or change its color, what does that effort really mean? What does it equate to? Is it better or is it worse?’ A/B testing is one of the best ways to understand the relationship between your product and users, and see if it is good or not,” he explains.
But there’s a catch. One of the major pitfalls of A/B testing is that it can be limited to the design of your product. “You can have fantastic A/B testing, but you need to have a fantastic design as well. If the candidates [features] that you’re testing against each other aren’t well thought design-wise, your testing will not be as effective because you’re going to be pitting weak candidates against each other. Success, in this case, will be very local,” he says.
Sean advises using other product analytics tools in conjunction with A/B testing to understand the user’s relationship with your product and the value it is giving them.
How can you make sure you are doing the right thing, and more importantly, how do you decide what to test?
“To figure out what to test, always start with the user,” Sean says. “User research is prominent and really important at the beginning phase of the candidate selection process. You’re going to get a lot of information on user pain points through customer research methods like usability testing and formal interviews. I always tell my students to start at the basics, or main product use case, before expanding their testing to the tertiary spaces on the platform.”
Like with all other aspects of product management, find out where your users are spending their time and focus on that.
The Hardest Part of A/B Testing for Product Managers
So from the above advice you’ve identified all the items that you need to test, it’s time to put them against each other, right? Not quite. It might seem like a good idea to test A/B/C/D, instead of A/B, all at once, but it’s not beneficial in terms of resources.
“It requires a lot of time, money, and effort from the design and engineering teams to create all four candidates. It’s almost like creating a product four times. It also takes a lot more time to get enough people to test the variables to make a conclusion,” Sean says.
To prevent such a scenario from happening, Sean advises Product Managers to understand where they will get the most value. “Always look at the easiest, most productive and most efficient way to get value. A/B testing can get costly in so many ways. You have to resist the urge to test everything and hone in on what you’re hearing from your customer,” he says.
Differences in Product Optimization Across Industries
Having a varied experience in product management, spanning e-commerce to financial services, Sean has noticed that the way products are designed and optimized is heavily reliant on the industry.
“E-commerce is very much driven by the data, where marketing acquisition takes a front seat as opposed to product-first conversion or acquisition. On the other hand, industries like finance allow you to talk to users directly in a way that’s not possible with e-commerce. You can understand pain points and important features of a product better when you have the allowance for activities like one-on-one interviews with the customer,” he explains.
Watch Sean talk about what surprised him about product management in the financial sector, and all the interesting ways he and his team explored A/B testing and optimization for financial products in the full interview video below:
“Anyone Can Be a Product Manager”
Just like with product optimization, Sean believes that passion and curiosity can make anyone a great Product Manager. “If you have the curiosity and passion about this role and look into it, you will be successful. Find people who are going to support you, keep them close, and ask yourself, ‘If anyone could do this job. Why not me? And if, why not me then? Why not now?’” he says.
If you want to learn more about Sean’s course, or about the other product management courses offered at Product Gym, schedule a free call with our career coaches. We’d love to answer all your questions!