There was this concept of objectives and counter objectives which was meant to combat Goodhart’s law. The idea was that you define what you want to measure and then define another metric which acts as regularization for the first. A good example is if you define a metric of WAU only, you might be motivated to bring in as many new users each week as possible. Setting a counter metric of retention makes you take into consideration that it’s important they stay.
I don’t remember who suggested this - I vaguely remember it was someone at a16z, but can’t find the original place I read it
There was this concept of objectives and counter objectives which was meant to combat Goodhart’s law.
Metrics and counter-metrics, IIRC originally coined by Julie Zhuo.
It’s a great framework for making sure that you consider Goodhart’s law, but it’s also only as good as the person thinking about it.
The problem I have seen in practice is that the same person whose job it is to think of the counter-metric (usually the PM) often looks better and gets promoted/paid more when the real costs of chasing the KPI stay hidden. PM leadership doesn’t have the bandwidth to make sure every PM is being rigorous.
I’ve felt this pressure as a PM (I don’t think I gave into the temptation but that’s not for me to judge) and as a developer I’ve seen my PM invest only nominal time on the counter-metric, where arguably you should be spending more time thinking about it than the metric itself. In practice, that resulted in things like security-through-annoyance-and-unreliability.
I’ve been in those shoes, so I know defining metrics to track business goals is really hard. But I do think people on the ground can tell when it’s a matter of difficulty as opposed to a matter of cutting corners or storytelling spin.
I don’t remember who suggested this - I vaguely remember it was someone at a16z, but can’t find the original place I read it