Note 20 | Totals Keep Growing. Why Might the Product Not Be Getting Better?
These are Robin Shen’s original reading notes and commentary, written in his own words after reading Eric Ries’s The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. They are not the original book, excerpts from it, or an official publication. Methods and concepts from the book are attributed to Eric Ries; other sources are identified where stated. For reference only; not investment advice.
English translation of Robin’s Chinese reading notes and commentary. Passages are paraphrased, not verified verbatim quotations from the original English edition.

Cumulative registrations hit a new high, and revenue keeps rising. Has the team necessarily done the right thing? In Chapter 7, Eric Ries uses IMVU’s experience to remind us that an engine of growth is running, and that this month’s product work has made the engine more effective, are two different propositions. If you look only at totals, a company may count the results of past decisions as a contribution from every current feature.
As Ries recalls, IMVU in its early days kept fixing bugs and improving the interface and the experience, yet at the month-end board meeting it became clear that users’ important behaviors had not changed in any obvious way. The team had believed the problem was quality, and therefore worked harder and changed more often. When the numbers still did not move, frustration easily turned into blame for people’s ability and efficiency. Ries is both a participant and a person with an interest in IMVU, so the boundary of this as a first-person account should be kept.
To obtain feedback, the team used small amounts of advertising to keep reaching new users. The purpose was not to inflate promotional results, but to observe behavior after a change. The point of the practice is to give a new version a chance to meet new people. The advertising costs in Ries’s recollection should not be treated as a budget standard that can be copied today, still less should the amount spent be taken as proof that the data are sufficient to support a judgment.
Cohort analysis changed the view. It does not add everyone together, but looks separately at a group of people who arrived in a particular period, and at how many later completed login, repeat use, payment, or other behaviors. You can then ask whether a new group, using the revised product, truly obtained value more easily than the previous group. The purpose of cohorts is not to make reports look more professional. It is to give new users facing different versions a consistent standard of behavior that can be compared.
This comparison brought a previously easy-to-hide problem to the surface: even if new users keep arriving, product adjustments may not improve each group’s key behaviors. Totals can rise because people keep coming in, yet they cannot by themselves tell us whether the current improvements in design or marketing are actually effective. The vanity metrics Ries warns against are not a claim that totals are never useful. They are a warning not to let them replace a test of progress and of cause and effect.
The numbers also changed the use of user interviews. Ries admits that in the past it was easy to explain negative comments as having found the wrong people, and to feel reassured after meeting more enthusiastic ones. Only when behavioral data showed that the effort was ineffective did the team truly ask why. Simplifying the flow of using the product with existing friends does not mean people want to invite friends to join. The problem may not be that the process is too troublesome, but that the team is optimizing for a goal users do not care about.
When the direction changed to helping people meet new friends, product work began to correspond more closely to a real need. This is not data automatically producing a new idea. It is quantitative results forcing the team to admit that the original explanation was insufficient, and then, through conversation, producing a new hypothesis. Quantitative and qualitative work therefore complement each other: the former shows whether behavior has changed; the latter helps you understand possible reasons. Interviews alone, or a dashboard alone, can leave people stuck in the account they prefer.
I think the reminder for a board is especially direct: when the numbers stall, do not first demand that engineering go a little faster and that requirements be written in a little more detail. If the direction is wrong, more precise execution will only postpone facing the problem. The conclusion is to keep totals in order to understand scale, use grouped behavior to see improvement, and leave a place for unfriendly feedback. Do not let a handsome curve automatically award a prize to every current decision.
Three suggestions
Put a behavioral funnel grouped by period of arrival next to the cumulative user report, and keep tracking the same key steps, rather than changing to a better-looking metric in each round.
Arrange limited contact with new users for the sake of learning, say in advance which change you will observe, do not apply the historical advertising prices in the book, and do not treat the volume of acquisition itself as product improvement.
When a metric does not move, first interview people who actually ran into difficulty or left, keep the negative responses, check whether a premise about the need is wrong, and only then decide whether to change the execution or the direction.
#TheLeanStartup | Source: Chapter 7, “Measure”
These are reading notes and reflections based on my personal understanding. I recommend reading them alongside the original book for the full context.
Comments
What do you think? Share your experience or questions.
Sign in to comment, reply or like. Sign in / Sign up
Loading comments…