We had a dashboard metric that looked perfect. It climbed every week. We put it in board updates. We mentioned it in standups. We felt good.
Then we lost three customers in the same month, and every one of them had been "healthy" according to that number.
The metric was not lying. It was just measuring something that had nothing to do with whether customers were making progress. We had picked a counter — something our product could increment — and confused it with evidence that people were getting value.
This is the story of how a single number made us worse at building, and what we replaced it with.
The appeal of one metric is obvious. It simplifies decisions. "Will this move the number?" becomes a filter for every feature request, every sprint priority, every design choice. Alignment feels effortless.
But alignment around the wrong signal is just coordinated drift.
Here is what happened: the metric rewarded activity, not outcomes. High activity could mean a customer was thriving. It could also mean they were thrashing — retrying things that should have worked the first time, clicking around looking for an answer they could not find. We could not tell the difference, and we did not try, because the line kept going up.
Worse, we started shaping the product to feed the metric. Features that would have reduced unnecessary steps got deprioritized because they threatened the number. We were building for the dashboard, not for the person on the other side of it.
This is how a north star becomes a ceiling. You optimize so hard for one signal that you stop asking whether the signal still means what you think it means.
A vanity metric has a specific quality: it increases naturally as your customer base grows, regardless of whether you do anything well. Counts of events, total accounts, cumulative anything. The product equivalent of measuring a child's age and calling it health.
The danger is not that these numbers are useless. Some are fine to track. The danger is that they absorb the attention budget. A team that stares at one big number every morning stops noticing the smaller signals — harder to collect, but closer to truth.
We were in that trap. Our weekly review started and ended with the counter. Questions like "are customers reaching the outcome they signed up for?" did not have a row in the spreadsheet.
We did not replace one metric with another single metric. That would have been the same mistake in a new outfit.
Instead, we defined three outcome signals that described real customer progress. Not what our system counted, but what our customers accomplished. The distinction matters.
An activity metric says: "They used the product." An outcome metric says: "They got the result they came for."
To find ours, we asked one question per customer segment: "What does success look like for this person within their first thirty days?" Then: "Can we observe that success without asking them to self-report?" If yes, it became a signal. If no, we built the observation point.
This was harder than tracking a counter. Outcome signals require you to understand what your customers are actually trying to do — which means talking to them, watching them, and accepting that their goals might not map neatly to your feature set.
Three things happened after we made the switch.
Product decisions got sharper. When you measure whether a customer reached an outcome, you stop debating features in the abstract. You either moved the success rate or you did not. Some ideas we had been excited about turned out to be irrelevant. Some we had dismissed turned out to matter enormously.
We caught problems earlier. Customers who were active but not progressing now stood out. Previously they blended in because the counter did not distinguish between productive work and frustrated clicking.
The team got more honest. A vanity metric is comfortable because it almost always goes up. Outcome metrics are uncomfortable because they expose gaps. That discomfort is the point. If your dashboard never makes you nervous, it is probably not measuring anything important.
If you are staring at a number that only goes up, ask yourself: does this tell me whether my customers are making progress, or does it tell me that my product is being touched?
Those are different questions, and they lead to different products.
The metric we retired was not wrong. It was a mirror pointed at us instead of a window pointed at our customers. We needed the window.
Measure what your customers accomplish. Not what your product counts.
Be the first to comment.
0 comments
Loading comments...