Note 21 | Shipping a Feature Is Not the End. Make Validation a Shared Responsibility
These are Robin Shen’s original reading notes and commentary, written in his own words after reading Eric Ries’s The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. They are not the original book, excerpts from it, or an official publication. Methods and concepts from the book are attributed to Eric Ries; other sources are identified where stated. For reference only; not investment advice.
English translation of Robin’s Chinese reading notes and commentary. Passages are paraphrased, not verified verbatim quotations from the original English edition.

Agile iteration is running smoothly, and features are delivered on time. Why might the company still not be learning enough to move the business? In Chapter 7, Eric Ries discusses Grockit and points out that the gap may not be in discipline, but in a team that is only responsible for completing decisions, and does not jointly test whether those decisions work. He discloses an interest in Grockit, so the case should be read as his observation and account, not as an independent comparative study.
As Ries recounts, this online learning company had clear user stories and a development cadence, and the team could also find people who liked new features. But metrics such as total users and total questions answered could be used at different times to prove progress, and it was hard to tell what a particular change had actually altered. If you wait until after the fact to reconstruct a release, users, and other contemporaneous factors, it often takes a great deal of time, and the team is already busy with the next round of work.
One change was to treat a new feature directly as a split test: at the same time, different groups encounter different versions, and differences in behavior are observed, rather than looking only at totals before and after a release. Ries admits that this requires extra work in tracking and statistics; but if you can discover early which features have no effect, you may in the long run save even more unrelated development. A test is not an extra decorative approval. It is a basis for changing the order of development.
The “lazy registration” example in the book is representative. The team had first let visitors experience the learning and register later, and maintained a complex flow for that purpose. After a comparison with a version that registered after seeing marketing materials, Ries recounts, the two groups showed no difference in key behaviors. This is limited to Grockit’s situation at the time, and is not a declaration that experience-before-registration is always ineffective. What it truly shook was the team’s belief that a particular best practice was a necessary condition.
Another change was the definition of done on the kanban board. After the technical work is finished, the item still has to enter validation, so it is clear whether the feature is worth keeping. Limiting work in progress at each stage forces people to deal with items that have been built but have no answer. What Ries criticizes is not agile tools themselves, but a process that lacks validation. If new tasks keep arriving, results without learning can stay hidden forever behind delivery numbers.
Good metrics also have to meet what Ries calls the “three A’s.” Actionable means people can see the relationship between an action and a result, so that departments do not claim credit when numbers rise and blame one another when they fall. Accessible means the people involved can understand and obtain them: compared with a vague click count, describing how many people took what action makes it easier to connect a report back to a real problem, rather than leaving it as jargon for the analytics department.
Auditable addresses trust. When a beloved feature is judged ineffective, the team may first doubt the data, so a report has to be able to go back to original records and real users for a check, and the way it is generated should avoid unnecessary complexity. This is not a license to expose personal data at will in order to inspect numbers. I take the management meaning to be that, within an appropriate scope, evidence should be traceable, rather than asking everyone to believe a report that cannot be checked.
I think the most valuable part of this approach is not finding a permanently correct person to rank features, but letting anyone’s proposal be tested. When a manager’s idea also has to answer the same questions, learning is no longer only a subordinate’s homework. The conclusion is to extend the definition of done from delivery to validation, and then use results that can be understood and checked to discuss the next step. Only then is there a chance to turn local busyness into overall progress.
Three suggestions
Before work starts on a new feature, write a testable behavioral hypothesis and a method of comparison, and put post-completion validation onto the board, so that a technical delivery is not treated as business success.
Present reports in plain sentences about how many people completed what behavior, keep them open to the teams involved, and write clearly the question and result of each experiment, rather than piling up totals alone.
Establish a path back to the data source and a sample check, reduce unnecessary transformations, and for important disputes return to actual records and to users for confirmation, rather than letting rank decide the answer.
#TheLeanStartup | Source: Chapter 7, “Measure”
These are reading notes and reflections based on my personal understanding. I recommend reading them alongside the original book for the full context.
Comments
What do you think? Share your experience or questions.
Sign in to comment, reply or like. Sign in / Sign up
Loading comments…