Lean Startup is a methodology designed to get companies to adopt a Build, Measure, Learn product development cycle. This emphasizes that teams quickly build an MVP (Minimum Viable Product) of a new product, measure how it performs, and learn from the experiment. The goal is to quickly go through this cycle to maximize learning and reduce costs in a short amount of time. The end result, ideally, is a more effective and agile company.
There are a few subcategories within Lean Startup. One is Lean Analytics, which covers the Measure and Learn part of the cycle. Because if you can’t measure, then you can’t learn. Lean companies need to know what’s important to track, why they should track, and how to track it.
Ben Yoskovitz and Alistair Croll wrote the Lean Analytics book. At Microconf 2013, Yoskovitz gave a talk about the measure part of the loop and some tips for measuring what matters. Below are the highlights from his talk.
What Makes a Good Metric?
Before we can know what a good metric is, we first need to define what analytics is.
Yoskovitz defines analytics as the measure of movement towards business goals. Once you know what your business goals are, you’ll then need measurements to know if you’re making progress towards your goals.
A good metric must also be comparative. A 2% conversion rate doesn’t tell you much if you have nothing to compare it to. How was the metric last month? Last year? Are your conversions increasing? A good way to track conversion rates overtime is with cohort analysis.
It must also be understandable. It shouldn’t be complicated, everyone should understand what the metric measures.
The metric should also be a ratio or a rate. Absolute numbers shouldn’t be used. Amount of users is useless, percentage of daily active users is better. Ratios and rates are comparative, which help you make better decisions.
Good metrics change how you behave. If the metric moves and you don’t know why it matters or what to do with it, then it’s a bad number. If it doesn’t change how you behave, it’s a bad metric.
This doesn’t mean you should only track one metric. If you find a metric you like but don’t know what to do with it, track it and put it in the back and worry about it later on.
Types of Metrics
There are two types of metrics: qualitative and quantitative.
Qualitative is talking to customers or prospects. In a Lean company this is typically customer interviews. This feedback is difficult to aggregate and score, but the insights it provides into the metric is valuable.
Quantitative is the numbers. They don’t give you answers, they help you ask better questions.
You discover things qualitatively, you prove them quantitatively.
Beneath qualitative and quantitative, there are vanity and actionable metrics. Vanity metrics make you feel good, but don’t change how you act. Actionable metrics are the ones that change your behavior.
Another dimension is the exploratory vs reporting metrics. Reporting metrics are the day-to-day managerial metrics that tell you how your business is performing.
Exploratory metrics are speculative; they’re used to find unknown insights. In its early days, Circle of Friends had about 10 million users, but terrible engagement. The team looked at their data and found the service was popular with moms, so they pivoted their product (and name) to target moms. They took a hit on total users (which is a vanity metric) but ended up getting a much more valuable and engaged user base.
Finally, there are lagging vs leading metrics/indicators. Lagging tells you something historical, it reports the news. A leading indicator helps you predict or tells you what’s going to happen. Churn is a lagging indicator, because it tells you the amount of customers who have canceled over a certain amount of time. Customer complaints could be the leading indicator for churn, because they are likely to predict churn. Buffer has discovered their leading indicator for customer retention:
Buffer's leading indicator for retention: Users who post 15+ times in the first week most likely to be active 3/6/9 months later
— Leo Widrich (@LeoWid) February 18, 2015
Because they have this data, Buffer can work to get new users to post 15+ times in their first week. This can be done in their onboarding process, adding some tweaks within the product that help users discover interesting things to share, as well as provide a great product that people want to use when they share on social media.
Most startups have to use lagging metrics because they only have historical information. As they grow and gather more data, they’ll want to discover the leading indicators for their business.
Analytical Superpowers
Yoskovitz begins this part of his talk by showing a chart showing ice cream consumption correlating with drownings. The more ice cream people eat, the more drownings there are. It is, of course, nonsense to believe that consuming ice cream causes drowning. It is more plausible that drownings occur with the seasons, which also correlates with ice cream consumption. Drownings happen in the warmer months (more people in the water), which is also when people eat ice cream. This is where we get into correlation and causality.
When something is correlated, it means that two variables change in similar ways (ice cream and drownings), likely because they are linked to something else. Causal is an independent factor that directly impacts the dependent one. In our example, it would be the summer months that are causal to ice cream consumption and drowning.
Here’s where this helps you:
Correlation helps you predict the future, because it gives you an indication of what’s going to happen. Causality lets you change the future. If you could drop the temperature during the summer months, that would cause fewer people to eat ice cream or go swim and play in the water. Look for correlation in your data, test for causality to try and figure out how the numbers work together, and if you can find causality (which is difficult) then you can optimize the causal factor.
Lean Analytics Framework
This framework looks at the business you’re in, and the stage you’re at:
It is possible for companies to combine business models, such as being a SaaS company on a mobile platform. When forming your business model, you have to know your customers and their buying process. How and why they buy, where they are in the ecosystem of their business, how they currently budget things, etc.
Don’t follow the leader. Test your business model out. Don’t assume a recurring revenue model (like SaaS) works best if your customers don’t like buying like that.
Lean Analytics Stages
Along with each stage are “gates” that are required to get through for the company to advance to the next stage:
In empathy, you want to make sure you’ve found a problem so painful that people are willing to pay you for a solution. This is more applicable for B2B products. If you’ve found this, you can move to stickiness.
In stickiness, you build an MVP product for (usually) an early adopter crowd. In this stage, you’re looking for user engagement and retention. You’ll know this if they use your product and stick around for a long enough time that it becomes clear you’ve provided value for them. If you get this, you can move to virality.
In virality, you’re looking to acquire customers in a cost efficient way. If you can scale it, then you can move to revenue.
If you’ve got all these, you can begin to focus on the math and ensuring the economics work. This doesn’t mean you wait to charge money until this point, it means that you focus on optimizing revenue and your LTV:CAC ratio. LTV means the amount of money you can expect to receive from a customer, and CAC is the cost to acquire a customer. You find the ratio by dividing your LTV by your CAC. In general, your LTV should be 3x your CAC, assuming your margins are healthy. Once you know you’ve got all the math figured out and you can run a sustainable company, you then move to scale.
Once in scale, you focus on growth. The approaches for growing a business is unique to each company.
Going through these steps is an important part of building the foundation. If you skip straight to scale without first making sure the math works or you have a market for your product, then the foundation will eventually crumble. Skip the steps at your own risk.
The One Metric That Matters
To find the metric that matters, you need to look at the stage you’re at and the business you’re in. At each stage in your business, you focus on the one metric that you work to improve. You pick the metric and set yourself a target or line in the sand. To find the target, you can do research and look at other similar companies in your space and the benchmarks they have.
In a SaaS company like OfficeDrop, their metric is paid churn. With this, their target is to keep it below 4%. Anything below that and they’re doing well, if it goes above they’re having problems and need to investigate.
Everybody in the company needs to know and focus on the metric that matters.
Lean Analytics Cycle
This is where you take all the concepts and put them together in a pragmatic approach to executing on what you do in your business. It’s ultimately all about business problems. You identify the problem, pick the one metric that matters, draw a line in the sand (or target), and get started. Here’s what the flowchart looks like:
Begin by picking a KPI (Key Performance Indicator), draw a line in the sand (or a target), and find a potential improvement. If you don’t have a lot of data on how to make an improvement, you make a guess. If you do have data, you look for commonalities that tell you what you should try to do.
Then you write your hypothesis, which basically says “if I do ____, I believe ____ will happen, which will get me ___ outcome.” Drill this hypothesis into your head by writing it down on paper and/or whiteboard so that you’ll look at it everyday. You need to stick to this and be reminded everyday what you’re trying to accomplish.
From there you make change in production or create a test where only 50% of your users see your experiment. You then measure your results and see if you moved the needle closer to your target/line in the sand. If you did, then you’ve reached the success box. You then begin again by finding the next KPI you need to focus on. If you didn’t, you can learn from the data and maybe pivot your business, business model, or target market. You can also draw a new line/target or try again.
Video & Slides
Video link – Q&A begins at 39:23
About the Author: Zach Bulygo (Twitter) likes marketing, finance, and learning about different businesses.
Comments (3)