The Limits of Being Data-Driven: Measuring Yesterday, Not Tomorrow

Content from Andrew Chen’s newsletter (with an AI-translated English–Chinese parallel version added) We are endlessly surrounded by data but also lack the ability to make sense of it all. It’s enormously frustrating but we commonly encounter discussions with statements like the following: We are constantly surrounded by data but lack the ability to understand it all. This is enormously frustrating, yet we often encounter discussions containing statements like these: “Oh the metrics went down? That’s just seasonality. Next quarter will be better” “Oh, the metrics went down? That is just seasonality. Next quarter will be better.” “Yes, the metrics are up! That’s because of this amazing new thing we just shipped” “Yes, the metrics are up! That is because of this amazing new thing we just shipped.” “Yes, the A/B test showed this metric went down but this other went up, so it’s actually okay” “Yes, the A/B test showed that this metric went down, but another went up, so it is actually fine.” “We have to wait on the A/B test to see how the feature is doing, the data is still collecting” “We have to wait for the A/B test to see how the feature is doing. The data is still being collected.” “I signed up for a 5% increase last quarter on this key metric, and we hit that goal. Why didn’t revenue go up? No idea” “Last quarter I committed to increasing this key metric by 5%, and we met that goal. Why did revenue not increase? No idea.” “People love this feature, just look at the data, we just need to onboard them better!” “People love this feature. Just look at the data. We only need to onboard them better!” “Our new product feature is totally working, but the current metrics don’t reflect it since there’s a time lag. Wait until next month” “Our new product feature is working perfectly, but the metrics do not yet reflect it because of a time lag. Wait until next month.” “The clickthrough rate on this email was 0.15% and the other was 0.14%. That’s why “This email had a clickthrough rate of 0.15%, while the other had 0.14%. That is why Etc etc etc, and so on and so on Many of you are now smiling because you’ve been there! Both on the receiving end, as well as having these kinds of words come out of your mouth :) I know I certainly have. Many of you are smiling now because you have been there! Both on the receiving end and having said these things yourself :) I know I certainly have. Why is it that we have so much data about how people use our products, and we are able to speculate as to what might make them stickier, or more valuable — yet truth-seeking is so difficult? We get all the aforementioned excuses (and more) because of the “fog of war” as we pick the product. This is doubly critical to figure out as many of our jobs now regularly involve reviewing data, then making decisions. This happens while reviewing dashboards and graphs (at product meetings, reviewing decks, board meetings, etc). Or writing about metrics you’re plan to increase as part of a quarterly review. We are in this fog constantly. Why do we have so much data about how people use our products, and the ability to speculate about what might make them stickier or more valuable, yet find the truth so hard to establish? The “fog of war” surrounding product choices gives rise to all the excuses above and more. Understanding this is critical because so many of our jobs now involve reviewing data and then making decisions. We do this when examining dashboards and charts—in product meetings, presentation reviews, board meetings, and so on—or writing about the metrics we plan to increase in a quarterly review. We are constantly in this fog. The root cause(s) for all our problems The root causes of all our problems In our quest to be data-driven, we hit many core problems based on how we collect/analyze/decision the data. Here are some of the variations: In trying to become data-driven, we encounter fundamental problems in how we collect and analyze data and make decisions with it. Here are some variations: Measuring yesterday when you should be thinking about tomorrow. Data comes from the market you have today, not who you want to have. Products often resonate with a small and loyal audience, but they are often niche, early adopter types. If you make decisions using data from this audience, you may not be able to make the right decisions that get you to a broader market. Your initial nerdy audience might ask for nerdy/configurable features, or expert features, whereas a broader market wants “easy to use.” Your data often doesn’t make this argument, and instead you end up chasing a local maximum. Measuring yesterday when you should be thinking about tomorrow. Data comes from the market you have today, not the audience you hope to have. Products often resonate with a small, loyal audience of niche early adopters. If you base decisions on that audience’s data, you may not make the decisions needed to reach a broader market. Your initial geeky audience may ask for highly configurable or expert features, while a broader market wants ease of use. Your data often does not make that case, so you end up pursuing a local maximum instead. Analysis paralysis. The cost of acquiring the data and its analysis can become cost prohibitive. Of course you’d like to have data for every decision. But if you need smart data analysts to pull the data, analyze it, check for corner cases, segment and analyze it, and then call a meeting to discuss (then analyze more, then discuss again), then it’ll just take too long. So it’s often better to just analyze the hell out of big things and otherwise ignore the small stuff, even though a truly data-driven team would bristle at this approach. Analysis paralysis. The cost of acquiring and analyzing data can become prohibitive. Of course you want data for every decision. But if you need skilled analysts to retrieve it, analyze it, check edge cases, segment and analyze it, and then call a meeting to discuss it—followed by more analysis and another discussion—it takes too long. Often it is better to analyze major issues thoroughly and leave small ones alone, even if a truly data-driven team resents that approach. Intuition is fast, A/B tests are slow. Related to the last point, sometimes when you become an expert about the customer, and have great intuition, you just know how to do the right thing. And now just the right thing that can move the needle in the short-term, but also the thing that will create better, long-term value. You can make a bunch of these decisions quickly, and just ship the product, and not try to A/B test every single one. Sometimes a startup’s market is moving so fast that this is actually the right approach. Intuition is fast; A/B testing is slow. Related to the previous point, once you know your customers well and have strong intuition, you sometimes simply know the right thing to do. That choice can move the needle in the short term and create greater long-term value. You can make a series of decisions quickly and ship the product without trying to A/B test every one. Sometimes a startup’s market moves so quickly that this is the right approach. The tyranny of the majority. When you deal with millions of datapoints of usage, you necessarily have to aggregate people together into segments. You look at averages, and compare segments. Yet sometimes these groupings are wrong — what about the VIPs? What about the high value users? If you make decisions that appeal to a broad audience, you might attract low-value international users at the expense of high-value US users — is more always better? It takes a while to unentangle which segments you actually care about, particularly if they are the minority of your users. The tyranny of the majority. When handling millions of usage data points, you have to aggregate people into segments, examine averages, and compare segments. But those groupings are sometimes wrong. What about VIPs and high-value users? Decisions that appeal to a broad audience might attract low-value international users at the expense of high-value US users. Is more always better? It takes time to untangle which segments actually matter, especially when they are a minority of your users. A/B tests are good at measuring big things quickly, but not other things. The biggest tool that a data-driven team can use is to use A/B tests. Yet to reach statistical significance quickly, and to iterate on a regular basis (weeks not months), you end up comparing big metrics movements that happen quickly. Signup rate is easy to measure, as is clickthrough rate. But what about 90 day retention rate for the minority of your users that are the most high-value? That’s hard. And so often people just ignore them. A/B tests are good at measuring large changes quickly, but not everything else. A/B testing is a data-driven team’s most powerful tool. Yet reaching statistical significance quickly and iterating regularly—in weeks rather than months—means comparing large metric movements that happen quickly. Signup and clickthrough rates are easy to measure. But what about 90-day retention among the small minority of your most valuable users? That is difficult, so people often ignore it. There’s a lot of noise. Factors like seasonality adds a lot of noise. So do big ad campaigns, or when competitors launch new products, or even if the month you’re in has an extra weekend versus another. It’s often hard to compare metrics cleanly, so unless there’s a big effect, you often have to just grit your teeth and make a guesstimate. There is a lot of noise. Seasonality creates considerable noise, as do large advertising campaigns, competitors launching products, or even a month having an extra weekend compared with another. Clean metric comparisons are often difficult. Unless the effect is large, you may simply have to grit your teeth and make an educated guess. Historical data may not predict future trends. This is particularly true with emerging technologies. If you take a product idea, and add AI to it, will that be better or worse? Will that increase conversion or retention or put other key metrics into decline? What if you take an idea and add a web3-first design? Sometimes, technology is just discontinuous and it’s impossible to know what will happen next. Sometimes you just have to go for it. I live with this problem every day in my venture capital job, where you’d like to simplify investing as a “you should always invest when a product has X attribute.” But the problem is, the successes are so few (a few dozen a year at most, out of tens of thousands) that you can’t really build a high-conviction predictive model Historical data may not predict future trends. This is particularly true of emerging technologies. If you take a product idea and add AI, will it be better or worse? Will conversion or retention increase, or will other key metrics fall? What if you add a web3-first design? Technology is sometimes discontinuous, making it impossible to know what comes next. Sometimes you simply have to try. I face this problem every day in venture capital: you want to reduce investing to “always invest when a product has attribute X.” But successes are so rare—at most a few dozen a year out of tens of thousands—that you cannot really build a highly reliable predictive model Correlation and causation. There’s a common path of analysis where a product team tries to figure out how their most engaged users became that in the first place. They quickly find that they do some setup step more often than others — like setting their profile photo or inviting a friend or something. The effect looks strong, and then an A/B test is run forcing everyone to take this step. The result is… nothing. It turns out high-intent users do X and then become high-intent. It’s correlation not causation. Shrug. This appears all the time, in many places, and just wastes everyone’s time. Correlation and causation. Product teams often try to understand how their most engaged users became that way. They soon discover that these users perform some setup step more often than others, such as adding a profile photo or inviting a friend. The effect looks strong, so they run an A/B test forcing everyone to take that step. The result is… nothing. High-intent users do X and then become high-intent. That is correlation, not causation. Shrug. This happens all the time, in many places, and just wastes everyone’s time. Some things are important, but hard to measure. How do you measure the effect of a new product feature on long-term brand? You probably can’t. What about how psyched your employees are about a new feature set? Or what investors think? Or maybe you have an n=1 super strategic and important partner you want to impress with some killer new tech. How does that show in an A/B test? Some of these factors are very important in the success of a new product, but it’s hard to get them to show up properly in a graph. Sometimes you just have to make the right call, if these things are important to you. Some things matter but are hard to measure. How do you measure a feature’s effect on the long-term brand? You probably cannot. What about employees’ enthusiasm for a new feature set, or investors’ opinions? Perhaps you have one especially important strategic partner—n=1—whom you want to impress with powerful new technology. How does that appear in an A/B test? Some of these factors are crucial to a product’s success, yet difficult to capture properly in a chart. If they matter to you, sometimes you simply have to make the right call. I could come up with dozens more of these. The ultimate point I want to make is that there are a lot of gotchas, and as the webcomics mention, a lot of these data-driven problems will crop up, there are too many dashboards, and too many conflicting opinions. And eventually the “highest paid person” (the HIPPO) just makes the decision, which is often where we started. I could think of dozens more examples. The point is that there are many traps. As the webcomics suggest, data-driven problems keep arising, with too many dashboards and conflicting opinions. Eventually, the “highest-paid person”—the HIPPO—makes the decision, which is often where we began. For the more data-driven among you, this might seem horrible. But let me also make the argument that this is great, and actually works! For those of you more committed to being data-driven, this may sound terrible. But I also want to argue that it is great and actually works! Competing against tech’s dominant culture Competing with technology’s dominant culture In the past two decades, because of the success of the Google/Meta product management cultures spreading into many new companies, we have now implemented OKRs across the industry. As a result, we have tens of thousands of product managers signing up for +5% increases in their metrics and aiming their hundreds of thousands of engineers at this problem. This dominant culture is one of data, of optimization, of taking small concrete steps to move metrics that are big and easy to understand. And you could argue that the success of many of the biggest tech products — the ones that don’t seem to evolve much at their core, but just inch along — are a success based on this strategy. Over the past two decades, the successful spread of Google’s and Meta’s product-management cultures to new companies has brought OKRs across the industry. Tens of thousands of product managers now commit to increasing their metrics by 5% and direct hundreds of thousands of engineers toward that problem. This dominant culture emphasizes data, optimization, and small, concrete steps that move large, readily understood metrics. You could argue that many of the biggest technology products—whose cores seem to change little as they progress slowly—owe their success to this strategy. What do you do if you are their competitor? For example if you are a startup competing with one of the FAANG companies? What do you do if you are their competitor—for example, a startup competing with one of the FAANG companies? The reality is, you can’t race their race. If you are data-driven in the same way they are, have the same dominant PM culture, and make decisions as slow as them, you will lose. They have more resources, can move the ball inches forward each time, and it’s hard to win as a result. The reality is that you cannot compete on their terms. If you are data-driven in the same way, have the same dominant product-management culture, and decide just as slowly, you will lose. They have more resources and can advance the ball a few inches at a time, making them hard to beat. Instead, you may have to do the opposite to compete: Instead, you may need to do the opposite to compete: mostly intuition rather than waiting for data Rely mainly on intuition rather than waiting for data super fast decision-making rather than incremental decisions Make decisions extremely quickly rather than incrementally big moves rather than small ones Make big moves rather than small ones Of course, I will always advocate to optimize the things worth optimizing for — like signup flows and key notifications. But a startup’s core product strategy might need something more drastic, and intuitive. Of course, I will always advocate optimizing what is worth optimizing, such as signup flows and key notifications. But a startup’s core product strategy may require something more radical and intuitive. Data-driven vs data-informed vs data-ignorant Data-driven, data-informed, and data-ignorant I write all of this with the deep pains and scars of using data in almost every product decision I’ve been apart of over the past decades. Of course I will always use the data. Yet in my twenties as a feral young founder, I erred towards relying on data too much — a desire to be data-driven, at all costs. But I found that often took me to local maxima. I write this with the deep pain and scars of having used data in almost every product decision I have participated in over the past few decades. Of course I will continue to use it. But as a wild young founder in my twenties, I erred by relying too heavily on data—wanting to be data-driven at any cost. I found that this often took me to local maxima. Later on, I learned to be data-informed, still having intuition but overlaying data on top. There’s a happy medium here. But I think there is also an argument — particularly for zero-to-one situations — to just be ignorant of the quantitative data, and instead, just train on intuition. Just make decisions based on your expert qualitative opinion about the market, your customers, and your competition. And only optimize based on data later on, when it’s more appropriate. Later, I learned to be data-informed: retaining intuition while layering data on top. There is a middle ground. But there is also an argument, especially in zero-to-one situations, for ignoring quantitative data and training your intuition instead. Make decisions based on your expert qualitative judgment of the market, customers, and competitors. Optimize using data later, when it is more appropriate. @andrewchen