Note 8 | “We Learned a Lot” Is Not an Account. Customer Behavior Is the Evidence

These are Robin Shen’s original reading notes and commentary, written in his own words after reading Eric Ries’s The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. They are not the original book, excerpts from it, or an official publication. Methods and concepts from the book are attributed to Eric Ries; other sources are identified where stated. For reference only; not investment advice.

English translation of Robin’s Chinese reading notes and commentary. Passages are paraphrased, not verified verbatim quotations from the original English edition.

Illustration for the reading notes on The Lean Startup: “We Learned a Lot” Is Not an Account. Customer Behavior Is the Evidence

“We have no results, but we learned a lot.” The sentence may be a genuine reflection, or it may be the most convenient explanation of failure. In Chapter 3, Eric Ries does not ask anyone to accept the value of learning unconditionally. He first explains why it is hard to trust: anyone can tell a story after the fact, without proving that the effort produced progress.

What he calls validated learning is not how much material was read or how many meetings were held. It is using empirical evidence from real customers to identify which parts of the strategy help build a sustainable business. It has to be able to resist self-consolation. It cannot rest only on a founder’s claim that they now understand more.

This also changes the test of value and waste. In mature manufacturing, the features customers care about are usually clearer, so you can ask which steps in production are superfluous. A startup may not yet know who the customer is or what is valuable to them. Work that deepens the team’s understanding of the customer’s need therefore becomes important progress; effort that cannot produce that understanding has to be examined again.

As Ries recalls, IMVU kept revising the product early on without driving the key behaviors. Only after the strategic direction changed did new product work begin to correspond more closely to a real need. The contrast matters: the team did not suddenly become more diligent; the effort began to go into what users actually cared about. The volume of output can stay the same and the productivity can be entirely different.

He describes a split test that changed the product description from “avatar chat” to “3D instant messaging,” so that new users were randomly shown different versions. As he recounts it, the latter version performed better on registration and later payment. The point is not that a particular phrase is inherently effective, but that the team compared the wording against actual behavior, rather than choosing a preferred line of copy by a vote in a meeting.

By contrast, the team once offered one-to-one virtual tours, hoping to solve users’ lack of understanding of the product, and the effort did not make people more likely to use it or pay. Attentive service itself cannot prove that an assumption is correct. If you only record how many tours were given, it is easy to mistake busyness for value; only when you see that behavior has not changed can you tell whether the path is worth continuing.

Qualitative understanding and quantitative observation therefore cannot replace each other. Behavioral data show whether a change had an effect; interaction with users helps you understand why. In Ries’s experience at IMVU, quantitative goals forced deeper contact, and what the contact revealed was then brought back into product tests. That is closer to the learning he describes than a survey that yields “we would probably use this,” or a gaze fixed only on a string of numbers.

A positive change, however, does not mean you can announce at once that the business model is fully established. In the case, the team was still exploring, and kept using new findings to design later tests. The value of validated learning is that it narrows what is unknown and improves the quality of judgment. It is not a way to package every favorable result as a final proof. You should claim only as much as the evidence answers.

The chapter also offers a more economical reflection: whether supporting fewer compatible networks, or even beginning with a feature description and a download link, would have been enough to expose the problem in the original assumption. This is not a call to develop nothing. It is a question about how much you need to do in order to obtain a particular understanding. The work of a test should follow the question, not the habits of engineering.

I think a good learning report should be able to say what you originally believed, what behavior you saw, and what decision you therefore changed. This is a way of putting it I have drawn from the chapter, not a standard form Ries provides. It gives supporters a chance to examine the reasoning, and it saves the team from having to choose between “look only at revenue” and “listen only to the story.”

Three suggestions

First name the customer understanding this round of work is meant to change, and state the observable behavior, so you do not attach a learning rationale to the results only after the fact.

Keep both useful and useless tests, and compare the changes with customer responses. Do not show only the successful version, and do not treat hours of service as evidence.

Look for a leaner way to test the next question, do only the work needed to support a judgment, and keep qualitative interaction so you can understand the reasons behind the numbers.

Robin’s note | Taiwan startups and venture capital

If an early-stage team in Taiwan needs to explain progress to investors, it can replace a feature list with a comparison of hypotheses, evidence of behavior, and changes in decision. That does not guarantee support, but it lets both sides discuss the same question, rather than each imagining a different meaning of “learning.”

#TheLeanStartup | Source: Chapter 3, “Learn”

These are reading notes and reflections based on my personal understanding. I recommend reading them alongside the original book for the full context.

Comments

What do you think? Share your experience or questions.

Loading comments…