Note 38 | Success Theater and Stealth Research Are Only Science in Costume
These are Robin Shen’s original reading notes and commentary, written in his own words after reading Eric Ries’s The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. They are not the original book, excerpts from it, or an official publication. Methods and concepts from the book are attributed to Eric Ries; other sources are identified where stated. For reference only; not investment advice.
English translation of Robin’s Chinese reading notes and commentary. Passages are paraphrased, not verified verbatim quotations from the original English edition.

People at the start of the twentieth century, seeing today’s scientists and engineers, would probably be astonished. In Chapter 13, however, Eric Ries says that if Taylor were alive, he might sneer at the way we manage entrepreneurship and innovation. The management practices that organize these smart people often lack scientific rigor. Ries even calls them, outright, a pseudoscience.
He lists several forms of it, which I put in my own words. First, approving a new project by intuition rather than by facing the facts. Ries also says, however, that this is not yet the root, because all innovation begins with a vision; the key is what you do afterward. Second, innovation teams put on a success theater, selectively looking for data that support them, rather than running a genuine experiment. Third, they hide and do stealth research, ignoring customer feedback and outside scrutiny, and running experiments without limits in a vacuum with no data. Fourth, they mark certain numbers on a chart of vanity metrics and use them to prove cause and effect, without being able to judge whether the causation truly exists. Fifth, they use learning as an excuse for failure.
The fifth is the easiest to overlook. Ries does not oppose learning from failure, but he requires that the learning be capable of validation: if something has truly been learned, it should in the next cycle be turned into a result that can be tested. A model of customer behavior has to be built, and you have to show how the product is changing that behavior step by step, before the vision can be said to have been truly validated. In other words, “we learned a lot” cannot be only a feeling. It should change the next round of hypotheses and practice.
Ries also warns that the Lean Startup itself may become a new pseudoscience. If people only make a show around nouns such as pivot and minimum viable product, history will repeat. He looks back at how scientific management once went off course: an excessive emphasis on standardization and procedure, neglecting creative work; an emphasis on efficiency and mechanization, neglecting the human factor; an emphasis on planning and control, neglecting flexible adaptation. In Ries’s view, this set the development of management back several decades, and later movements in management had to come out and correct it.
More interesting is a passage of self-reflection. Ries points out that many of the views Taylor regarded as scientific are, by today’s lights, prejudices: for example, the belief that people of certain origins or genders are naturally superior, and that those of lower status should be strictly supervised by their superiors. He then asks: when later people look back at us, which prejudices will they find? Which forces do we believe in blindly? After the Lean Startup’s first success, what may it ignore? Ries says that what we now know is only the tip of the iceberg. If we can stop wasting everyone’s time, we have no idea what wonders people can create.
My reading is that what these forms of pseudoscience share is not a lack of data, but that the data have no chance to overturn the original idea. Success theater picks only the good news; stealth research avoids the test; charts of totals confuse cause and effect; the excuse of learning lets failure not have to turn into a next step. They all make the team look well founded, while in fact it has never taken the risk of being proved wrong. A genuine experiment is the opposite: before it starts, you have to say clearly what kind of result would make us change our minds.
This also explains why Ries discusses the problem of prejudice in the same place. If even someone like Taylor, who valued experiment, could treat prejudice as science, we today also cannot assume that we have no blind spots. The best defense is to let the method itself accept the same test, rather than treating it as a dogma that cannot be questioned.
I think that when investors listen to a presentation, they can notice whether what is being shown is a handsome curve of totals, or a comparison whose expectation was stated in advance. The conclusion is that the core of a scientific spirit is not how many charts there are, but a willingness to let evidence change a decision. Even the method you believe in should keep room to be questioned.
Three suggestions
Change every experiment report into three columns: the hypothesis, the expected change, and the actual change, and fill in the first two columns before the experiment begins.
Whenever a presentation uses a chart of totals to claim cause and effect, require a grouped or controlled comparison to be attached; otherwise treat it only as an observation, not as a conclusion.
A review of failure must write the specific proposition to be validated in the next cycle and the criterion of judgment. Only if it can be written has learning truly been completed.
#TheLeanStartup | Source: Epilogue, “Waste Not, Want Not”
These are reading notes and reflections based on my personal understanding. I recommend reading them alongside the original book for the full context.
Comments
What do you think? Share your experience or questions.
Sign in to comment, reply or like. Sign in / Sign up
Loading comments…