I love building new products. Ever since I was building junky web apps as a geeky high schooler, I always get excited the first time something actually works. It has always felt like magic. Now that I’m older, I increasingly feel the pressure of showing my impact. After the initial euphoria passes, I now immediately measure the metrics that represent success. Something that has been bothering me lately is that regardless of your methodology (waterfall, agile, scrum, burndown, trello anarchy, etc), I never hear others talk enough about product success metrics.
When I joined HubSpot I learned from many others about behavioral analytics. Sadly, I find myself constantly fighting responses when I speak with friends in the industry such as:
- “We forgot to add tracking”
- “We want to ship it and see how it does”
- “We don’t have any specific goals for this release other than to improve the design”
- “What should we measure?”
- “We can’t afford to use behavioral analytics, it’s too expensive”
This is how I want to react every time I hear one of those answers:
Just kidding. I am always asking questions to understand the rationale so I can try to help add perspective.
These are the tough questions I want to ask in response:
- What’s more expensive? A behavioral analytics system or shipping the wrong features / wasting the time of your product and engineering team?
- If you hear feedback from a couple of customers, is that representative of all users?
- How do you know that the users are actually doing what they say they’re doing?
- Do you think you’ll get a team’s best work if the only goal is to release their work?
- What do you think will garner more resources in the future? “We improved the experience, just look at it!” vs. “I increased conversion rates of signup to value by 10%, with an expected lift in revenue of Y”.
I don’t think you need to spend weeks off in a corner crunching numbers to come up with the answers to these questions. My suggestion is to spend 30 minutes thinking about a goal, why you’re working on something, and then a simple mechanism to measure success.
I push teams to answer these questions:
- What represents success for this release/feature?
- What is the current baseline?
- What is the hypothetical ceiling of improvement?
- Given the baseline and ceiling, how much do you think you can improve the metric?
- What will be the mechanism to track success/failure?
- When should you evaluate progress?
You don’t have to be super fancy and build Excel models, but at least spend 15-30 minutes thinking through the basics for a new feature. Regardless if you’re building something brand new or iterating on an old feature, I always think it’s worth considering the above questions.
As the saying goes, “if you can’t measure it, you can’t improve it”.
Leave a Reply