Measuring Product Managers (in Swedish or English)
I’m back from a week of product management workshops and seminars in Sweden, including a Product Leadership event hosted by Tolpagorni’s Magnus Billgren.
In a half-dozen discussions with the heads of product management groups, I was struck by how familiar their concerns are. We could have been in Sunnyvale rather than in Stockholm. Topics that came up repeatedly:
- What metrics do we use for evaluating product managers, and how can we tell if they are doing a good job? Are there PM KPIs*?
- Our agile development teams tell us that roadmaps are no longer needed, but our customers and sales teams still demand firm commitments.
- How should we organize our product teams? Senior/junior, technology vs. market segment, product owner vs. product manager?
- What do career paths look like?
These were discussions about how organizations and real people interact in a technical environment. They reflect what execs running product management groups worry about: boosting effectiveness, building relationships, and relating the business of product management to the company’s overarching business.
Some product geography
Sweden’s tech industry is heavily weighted toward B2B and telecom, with many firms clustered around Ericsson in Stockholm’s northern suburbs. Product managers stay with their companies (and in their current positions) much longer than we’re used to in Silicon Valley. And since telecom customers have long implementation cycles, product releases are methodically timed. Internally agile development teams are faced with un-agile, contract-driven shipment dates. In search of cost reduction and talent, most teams are split geographically. In other words, not very different from B2B tech firms back home.
With limited time in each workshop, I used an old agilist technique of soliciting concerns, then having attendees rank them. Voila! A backlog of product management issues that we time-boxed and attacked in order. (And a handy topic list for my next few posts.)
Here’s a summary of discussions about the first topic above – metrics for evaluating product managers, or “PM KPIs” - which drew heavily on an assessment tool and post that Scott Gilbert and I created in 2010.
What should we measure?
There’s a lot of confusion between metrics about products, assessments of product management teams, and scorecards for individual PM job performance. Taking each in turn:
[1] Product metrics are an embodiment of our market and financial goals, such as “units shipped” or “incremental revenue” or “gross margin in our reseller channel” or “market share in Asia Pacific.” Results depend by an infinite set of external and internal factors (sales compensation plans, competitive price shifts, PR, on-time release, weather) that are mostly out of a product manager’s control.
Product KPIs are surface indicators that should trigger deeper questions. Why are margins higher in Latin America than Europe? Can we see any impact from our competitor’s social/community efforts? Which segments should we target for next quarter’s whiz-bang features? What internal sales education techniques work the best? As problem solvers, we take these as challenges.
[2] Department-level assessments help us improve the business of getting great products shipped. Success is a cross-functional team sport, so these tend to be a mix of objective and subjective scores. “Are requirements complete and reflecting market needs?” is paired with “does engineering agree that PM is the primary proxy for customers?”. “Customers understand our core benefits/features” goes hand-in-hand with “Sales is eager for product management to meet prospects.”
Again, we use these metrics or assessments for a problem-solving approach. If messaging is lost in the handoff from product to marketing to sales to channel to customer, where are we getting confused? If late-arriving requirements are frustrating our developers, how can we anticipate better (or convince Engineering to relax a little, or keep Sales from promising futures)? KPIs should be weathervanes, not career bludgeons.
[3] Finally, the hard question that my Swedish colleagues wanted answers to: how to quantitatively evaluate individual product managers? Jag vet inte. (That’s Swedish for “Honestly, I don’t know.“)
In my experience, the best product managers throw themselves against unanticipated product-specific market and organizational issues. They practice higher-order problem solving, making progress fiendishly hard to measure. Great PMs fill the organizational gaps, work intramurally, and often hang back when credit is being awarded. Their success can be invisible: the absence of confusion and unhealthy politicking.
I’m hesitant to assign numeric goals to my individual product managers. If I give bonuses for perfect requirements, I’ll get 100-page MRDs (instead of time spent with customers). If I reward pure revenue performance, my folks might spend all of their time on the road (and neglect next quarter’s planning). Etc.
My best indicators of a product manager’s success are subjective:
- Does she have a roadmap agreed to by Engineering and Marketing (as a proxy for consensus building)?
- Does Engineering consider her a core part of their team (and not a technical lightweight)?
- Do Marketing and Sales see her as the source of product facts, strategy, and market intelligence?
- Do I trust her judgment, objectivity, whole product thinking, and personal integrity?
- Does the team show more focus (and less panic) over time?
- Are other departments trying to recruit or borrow her?
Not as scientific an answer as I’d like, but the basis for discussion here and in Stockholm. If you have more quantitative approaches, please share.
Sound Byte
Measuring product success is easy; gauging departmental success a little harder. I’m still stumped for general, quantitative metrics for individual product managers.
* Key performance indicators (KPIs) are quantitative definitions of success, which are then used to measure progress or score results.
8 thoughts on “Measuring Product Managers (in Swedish or English)”
Rich,
Great post. Particularly the part: ” Their success can be invisible: the absence of confusion and unhealthy politicking.”. This is so true, that when it is working the machine is well oiled. Lose a good Product Manager, and suddenly the friction goes way up, and people are wondering what happened.
It is difficult for my superiors to understand the soft metrics, and it can be very frustrating when your annual objectives are things that really are antithetical to product management (one year, I was told I needed to be on 6 or more sales calls a month. With the sales person, closing business. Sales Engineer cum Product Manager.)
Glad to see you post!
Thanks Rich, good topic for discussion.
You are right that measuring individual performance for product managers is hard. You can make a legit argument that it takes up to 24 months to gauge if a new PM is good or not. When I hire a new PM, I might send them into the market to learn for 3 months, then they will spend 3 months writing some requirements, then engineering will build something for 6 months, then we have a 6 month sales cycle, then we have to spend time evaluating results. Hopefully we’d be more agile than that but it’s possible it would take this long. But as a manager of product management, you can’t wait 18-24 months to assess an individual’s performance – that’s horribly inefficient.
To make it worse, in my experience most companies judge their product managers individual metrics on what you describe as product metrics above: product revenue, margin, or customer sat. A product manager might have (indirect) control of these metrics, but there are a lot of factors that impact them way outside of the PM’s control. What if the PM is measured on revenue and halfway through the year the company makes an acquisition and redirects Sales resourcing? Huge impact to the PM’s metrics through no fault of the PM him/herself.
What I have settled on for measuring individual performance is to measure the activities required for a PM to be market-driven. Namely: 1) a quota for market visits (10/quarter is a good start IMO), 2) being able to defend an updated business plan in front of me and their peers every quarter, without using the phrases “I think,” or “In my opinion” (which requires higher order thought and research), and 3) keeping their finger on the pulse of the business by defining and measuring the right product level metrics and communicating them in the form of a dashboard report monthly. I like these metrics because they require a PM to get out from behind their desk and be outside-in focused, and the PM can control them, as opposed to revenue, margin or CSAT which the PM cannot control.
All things being equal, I’d prefer to measure outcomes as opposed to activities, but as you put it, it’s wicked hard to separate and quantify the inputs of a PM vis-a-vi all the other variables that go into making a product fly. So until we crack that code, I’ll use activities as a proxy.
Thanks for such a thoughtful set of suggestions. I especially like (3), since it puts the metrics onus back on each product manager.
I’ve really evolved into the “decouple performance reviews / compensation from metrics” point of view.
When you measure someone on something specific (unambiguous, easy to measure, atomic, etc), they become biased towards doing the thing that most effectively improves that metric.
When you focus on the metrics that really matter, not only is there too much (time) lag in the system before you get useful information back, but you can’t isolate the variables.
If someone is not spending enough time in the field (easy to measure) what it probably means is that they don’t have sufficient insight into the problems that are important to customers (easy to query, subjective to evaluate, impossible to quantify). I prefer to have that conversation as part of an ongoing dialog, not a performance review (or at least it is only a recap of past discussions during the review).
There really shouldn’t be any surprises during the annual-review meeting (except the $ amount of the raise/bonus).
The best way to manage product managers, imho, is the same as the best way to manage any knowledge worker – through ongoing dialog, coaching and mentoring – NOT to measure them with an assured-to-be-inaccurate ruler.
I loved this “Our agile development teams tell us that roadmaps are no longer needed, but our customers and sales teams still demand firm commitments.”
Great Article!
In an agile product organization, the roadmaps for customers and sales should concentrate on the problems to be solved, rather than the way they will be solved. This establishes a vision to guide the product evolution without overly prescribing it.
Greg – agreed. I’ll add that the same is true for any organization – waterfall or agile. The commitments to customers (and partners, and sales, and customer service, and the rest of your org) should look like
“In release X, we will address problem Y, and improve the (customer experience | value proposition) for Z”
They should not be “We will implement widget M” or even “feature N” or “capability O” (although you might need to talk about capabilities, from an outbound standpoint in a B2C situation – you can manage your sales force to handle this “contextual translation” in B2B or B2B2C scenarios).
Also – Rich, yet another great article!
Nice post–although I would love to have more hard metrics to hang my hat on, I agree that we just can’t put a number on some things (and roles). Unfortunately, I’ve seen management at a couple of different companies that don’t get the soft metrics part and think that product management is expendable:(