Discussion about this post

User's avatar
Aravind Iyengar's avatar

While this technically and logically makes sense, the distillation of the core set of metrics that drive a particular KPI for an enterprise usually becomes a contentious matter. This is perhaps the most objective in optimizing ads, where there is a very direct ("linear") relation between what you control and how it drives revenue, and how the granular items aggregate ("add up") to more coarser, macroscopic KPIs like revenue. In most other areas, there are complexities, non-linearities, non-additive ways in which the granular metrics combine to form KPIs of interest to be optimized for. In the face of these complexities, human minds are unable to reason in simplistic ways, and the "trust" in core metrics becomes subjective -- a matter of one's ability to comprehend and therefore believe in a mathematical model, or become a skeptic. Think of all the paradoxes and counter-intuitive strategies like the Monty Hall problem (factoring stochasticity is not always straightforward) and Braess' paradox (optimizing with constraints is not always what you expect). In most real-world cases, we are faced with exactly these kinds of situations -- you need to factor in stochasticity of events, and optimize with constraints. Furthermore, establishing causality is a completely different dimension of complexity altogether. Unless you agree with the mathematical model for a situation and comprehend the logic behind why the optimal solutions for it are optimal, you will be a skeptic.

Expand full comment
Rui Diao's avatar

This really resonates! The idea of distilling complexity down to a core set of 'canonical' metrics is powerful, especially when you consider the cognitive overhead of tracking everything.

I remember a similar situation working on Google Ads where we had an explosion of CTR metrics—every specific click target had its own CTR, leading to dozens of aggregated versions. Establishing a canonical set that truly mattered for strategic decisions was a game-changer; the constant debates over which CTR was 'the right one' evaporated.

It totally validates your argument: the long tail is crucial for deep debugging (like checking that one specific edge case event), but it shouldn't clutter the main narrative.

Expand full comment

No posts

Ready for more?