Vanity Metrics

1. Overview

Vanity metrics are superficial measurements that appear impressive on the surface but fail to provide meaningful insights into a business’s performance or inform future strategies. These metrics are often easy to measure and gratifying to report, creating a false sense of success and progress. The core purpose of understanding vanity metrics is, paradoxically, to avoid them. By recognizing which metrics are superficial, organizations can shift their focus to more substantive, actionable data that reflects true business health and drives strategic decision-making. The problem that the concept of vanity metrics solves is the widespread tendency for startups and established companies alike to chase misleading indicators of growth. This can lead to a misallocation of resources, a failure to address underlying problems, and an inability to optimize for long-term value creation. The term was popularized by Eric Ries in his book, The Lean Startup, where he contrasts them with “actionable metrics”—data that demonstrates clear cause and effect and can guide concrete business actions.

In the context of commons-aligned value creation, the focus on vanity metrics is particularly problematic. Commons-based projects and enterprises are oriented toward creating and sustaining shared resources and community well-being, rather than simply maximizing private profit. Vanity metrics, such as website pageviews or the number of social media followers, often measure audience size or superficial engagement without capturing the depth of community participation, the health of the shared resource, or the actual value being co-created. For a commons-oriented project, success is not just about scale but about the quality of interaction, the equity of distribution, and the resilience of the ecosystem. Relying on vanity metrics can steer a commons project away from its core mission, encouraging it to pursue growth for growth’s sake rather than fostering a thriving, sustainable community. Therefore, a critical understanding of vanity metrics is essential for any organization committed to commons principles, as it enables a more authentic and mission-aligned approach to measuring and understanding impact.

2. Core Principles

1. Actionability over Affirmation: The primary principle is to prioritize metrics that lead to concrete actions and informed decisions rather than those that merely affirm existing biases or create a superficial sense of accomplishment. A metric is only valuable if it helps you decide what to do next.
2. Correlation is not Causation: Vanity metrics often highlight correlations without revealing the underlying causal relationships. A core principle is to dig deeper to understand the ‘why’ behind the numbers and to identify metrics that show a clear cause-and-effect relationship between actions and outcomes.
3. Segmentation and Cohort Analysis: Aggregate data can be misleading. A fundamental principle is to segment data and analyze user cohorts to understand how different groups of users behave over time. This provides a much clearer picture of user engagement and retention than a single, cumulative number.
4. Alignment with Strategic Goals: Metrics should be directly tied to the organization’s strategic objectives. If a metric does not help you measure progress toward a specific, meaningful goal, it is likely a vanity metric. This principle ensures that measurement efforts are focused and relevant.
5. Reproducibility and Controllability: A good metric should be reproducible, meaning that you can understand the factors that influence it and can take actions to intentionally and predictably affect it. If a metric is subject to wild, unpredictable fluctuations or external factors beyond your control, its utility is limited.
6. Focus on Ratios and Rates: Raw, cumulative numbers are often vanity metrics. The principle of focusing on ratios and rates (e.g., conversion rate, churn rate, ratio of daily active users to monthly active users) provides a more nuanced and comparable view of performance over time.

3. Key Practices

1. Define Key Business Objectives First: Before selecting metrics, clearly define the most important business objectives. This ensures that the chosen metrics are relevant and aligned with what truly matters to the organization’s success.
2. Use the “So What?” Test: For any given metric, ask “So what?”. If you can’t articulate a clear and meaningful business decision that can be made based on the metric, it’s likely a vanity metric.
3. Track Cohort-Based Retention: Instead of just tracking the total number of users, track user retention on a cohort basis. This involves grouping users by the week or month they signed up and then measuring what percentage of each cohort remains active over time.
4. Measure Engagement, Not Just Pageviews: Instead of focusing on pageviews, measure user engagement. This could include metrics like time on site, pages per session, bounce rate, or the completion of key actions within an application.
5. Focus on Per-User Metrics: Instead of tracking aggregate revenue, track metrics like Average Revenue Per User (ARPU) or Customer Lifetime Value (CLV). These per-user metrics provide a much clearer picture of the health and profitability of the business.
6. Implement A/B Testing: Use A/B testing to establish clear causal links between changes you make and their impact on key metrics. This is a powerful way to move from correlation to causation.
7. Create a Metrics Dashboard: Develop a dashboard that displays a small number of key, actionable metrics. This helps to focus the team’s attention on what’s most important and avoids the distraction of vanity metrics.
8. Regularly Review and Refine Metrics: The metrics that are important today may not be the ones that are important tomorrow. Regularly review your key metrics to ensure they are still relevant and aligned with your evolving business goals.

4. Implementation

Implementing a strategy to avoid vanity metrics and focus on actionable insights requires a conscious and deliberate shift in mindset and process. The first step is to establish a clear hierarchy of goals for the organization. This involves defining the overall mission, breaking it down into strategic objectives, and then identifying the key results that will indicate progress toward those objectives. Once these are in place, the team can begin to brainstorm potential metrics, always asking the critical question: “How does this metric help us understand our progress toward our key results?” This process should be a collaborative one, involving team members from different functions to ensure that the chosen metrics are relevant and meaningful across the organization. A useful framework for this is the AARRR (Acquisition, Activation, Retention, Referral, Revenue) model, which provides a structured way to think about the user lifecycle and to identify key metrics for each stage.

With a set of actionable metrics defined, the next step is to implement the necessary tools and processes to track them effectively. This may involve using analytics platforms like Google Analytics, Mixpanel, or Amplitude, and it will almost certainly require the creation of a centralized dashboard where the key metrics are displayed and regularly reviewed. It is crucial to foster a data-informed culture where team members are encouraged to ask critical questions about the data and to use it to inform their daily work. This means moving beyond simply reporting the numbers and instead focusing on the insights and actions that can be derived from them. For example, instead of just reporting the number of new users acquired, a team might analyze the conversion rate of different acquisition channels to determine where to focus their marketing efforts. Real-world examples of this in practice include SaaS companies that focus on Monthly Recurring Revenue (MRR) and churn rate rather than just the total number of registered users, or e-commerce companies that obsess over customer lifetime value and conversion rates rather than just website traffic.

5. 7 Pillars Assessment

6. When to Use

• Early-Stage PR and Fundraising: While ethically questionable, vanity metrics can sometimes be used in the very early stages of a startup to generate initial buzz, attract media attention, or secure seed funding from less sophisticated investors.
• Gauging Brand Awareness: High-level metrics like social media reach or website traffic can be used as a rough proxy for brand awareness, as long as they are not mistaken for indicators of deeper engagement or business success.
• Team Morale Boost: In some limited contexts, a sudden spike in a vanity metric (e.g., a viral video) can be used as a short-term morale boost for the team, as long as it is accompanied by a clear-eyed analysis of its actual business impact.
• Competitive Analysis: Vanity metrics can be used to get a quick, high-level sense of a competitor’s marketing efforts or perceived market presence, but this should be followed up with a more in-depth analysis.
• Identifying Potential Data Issues: A sudden, unexpected change in a vanity metric can sometimes be an indicator of a technical problem, such as a broken analytics implementation.
• As a Starting Point for Deeper Analysis: Vanity metrics can sometimes serve as a starting point for a more rigorous analysis. For example, a spike in website traffic could prompt a deeper investigation into the source of that traffic and the behavior of those users.

7. Anti-Patterns and Gotchas

• Confusing Correlation with Causation: The most common gotcha is to assume that because two metrics are moving in the same direction, one is causing the other. This can lead to flawed decision-making.
• Focusing on Cumulative Charts: Cumulative charts that only ever go up and to the right are a classic sign of vanity metrics. They look impressive but provide no insight into the current health of the business.
• Ignoring Segmentation: Looking only at aggregate data can hide serious problems. For example, a growing number of new users could be masking a high churn rate among existing users.
• Choosing Metrics Based on What’s Easy to Measure: It’s tempting to focus on the metrics that are easiest to track, but these are often not the most important ones. This can lead to a false sense of security.
• Using Metrics to Justify Past Decisions: Metrics should be used to inform future decisions, not to justify past ones. This is a form of confirmation bias and can be a major obstacle to learning and improvement.
• Failing to Connect Metrics to Business Goals: If you can’t explain how a particular metric connects to a specific business goal, it’s probably a vanity metric. This is a sign of a poorly defined measurement strategy.

8. References

1. Ries, E. (2011). The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business.
2. Tableau. (n.d.). Vanity Metrics: Definition, How To Identify Them, And Examples.

3. [ProductPlan. (n.d.). *Vanity Metrics

   Definition and Overview*.](https://www.productplan.com/glossary/vanity-metrics/)

4. Cutler, J. (2022). What Are Vanity Metrics and How to Stop Using Them. Amplitude Blog.
5. Yoskovitz, A., & Croll, A. (2014). Lean Analytics: Use Data to Build a Better Startup Faster. O’Reilly Media.