Randomness In, Randomness Out

Excerpted from A Random Search for Excellence:

Many managers have found the prescriptions in one or more of these studies helpful, perhaps even enormously so.  And we’re sympathetic to the notion that if it works, don’t knock it.  But we’ve come to the disturbing conclusion that every one of the studies that we’ve investigated in detail is subject to a fundamental, irremediable flaw that leaves us with no good scientific reason to have confidence in their findings.

It is this:  success studies aren’t studying unexpectedly successful companies.  Rather, they are, by an overwhelming majority, studying a collection of firms that have taken fortunate random walks.  In other words, they are not studying great companies, they are studying lucky companies.  Consequently, the inputs to every success study we can lay our hands on are the wrong inputs.  This has material consequences for the confidence we can have in the advice offered, for no matter how rigorous the data collection, no matter how Aristotelian the logic, to deviate a bit from the old aphorism, “randomness in, randomness out”.

 

I’ve long been troubled by the noncumulative nature of the prescriptions associated with success that one finds in various success studies. Is M&A critical or not?  Does leadership matter, and, if so, what kind?  Is success strategy-led or is execution what it’s really all about?  It does not good to conclude that “everything’s important” since that provides no guidance in allocating resources – especially management time.

 

In addition, many prescriptions for success feel “platitudinous”:  make sure you have the “right” people.  How helpful; I always thought having the wrong people was the way to go.  More generally, if the opposite of a given prescription isn’t defensible, then the prescription itself has no real substance.

 

These two characteristics – contradictory prescriptions and empty advice – are unlikely to be a consequence of any deficiencies on the part of the researchers involved.  Our hypothesis is that it is an inevitable result of the fact (if we may be so bold) that each study examines a different sample from a random distribution.  Since the resulting patterns in behavior – the foundation of the conclusions drawn – are therefore more imposed than inferred, it’s no surprise that everyone sees a different pattern.  In addition, studying randomness makes it very difficult to come up with falsifiable claims since every company in the sample is essentially “the same” – or at least, indistinguishable from one another, which is a weaker, but from a method perspective, equivalent claim.

 

 

In short, if one hopes to reach nonrandom conclusions, one needs to begin with a nonrandom sample.

Comments

[David Jones]

I applaud your efforts on this project. If you're looking for a place to start, I suggest you look at the Baldrige Criteria and Baldrige winning companies www.baldrige.nist.gov .

To briefly answer two of your questions: the best companies have the best leadership, without exception; and strategy setting and deployment are both necessary for success.

[David Zorn]

Thank you for bringing a more rigorous approach to the study of management and organizations. Anecdotes are entertaining, but they're hard to employ.

[Rick Payne]

Your research to date raises some very important questions. I look forward to seeing the work develop. Clearly the scope is already huge but it would be great if you could engage some of your international colleagues in extending this rigorous approach to countries outside the USA.

[Gerald Nanninga]

After reading through the data in your paper (A Random Search for Excellence), I came to a completely different conclusion than the one voiced in the document. The underlying current written between the lines in the article seemed to be that “style does not determine substance.” In other words, the relationship between the way a company is managed (management “style”) and the way it performs (its “substance” as defined by ROA or TSR) seems to be predominantly random. Therefore, don’t put a lot of hope into recommendations of how to improve your substance by changing your style.

This line of reasoning leaves the reader in a rather hopeless state--Just try to be lucky and see if you can find a fable with questionable relevancy.

While all of this may have some truth in it, there is also a second truth in the data which I find very uplifting and practical. This other truth is that while style may not determine substance, positioning does. Getting positioning right (something you have control over) can help you be successful.

Consider the two data conclusions from the paper:

1) The most likely outcome for a firm in any decile is to repeat that decile in the following year.
2) This stickiness in performance is especially pronounced at the high and low ends of the spectrum.

What these two conclusions say to me is this: Once you have established your position in the market, it tends to stick. If you pick a winning position in the marketplace, you tend to remain a winner. If you pick a losing position, you tend to remain a loser. Therefore, rather than worrying about “style” we should focus on one’s market position. If you get positioning right, it can overcome a lot of other randomness and get you “sticky” in the right outcome place regardless of style.

Winning positions are the sweet spot between three forces: consumer desires, internal capabilities and marketplace vulnerabilities. In other words, if you are able to profitably meet a desire better than anyone else and the market allows you to own that position in the mind of the consumer, then you have a winning position. (I go into the process of finding a winning position in a lot more detail in one of my blogs:
http:// planninga-from-nanninga.blogspot.com/2008/09/analogy-207-eight-questions.html)

To learn more, also read what Al Reis and Jack Trout have written on positioning, or better yet, read all of my blogs that are tagged with positioning (http: //planninga-from-nanninga.blogspot.com)

[Barry Johnson]

David Jones post on 03/27/2009 may help us identify some of the ”drivers” (independent variables) that potentially produce the distinguishing differences in performance outcomes (dependent variables) in any sample taken from an ever changing, non-homogeneous population.
David advises us “If you're looking for a place to start, I suggest you look at the Baldrige Criteria and Baldrige winning companies.” He has pointed in a promising direction.

I suggest we need to look at the factors that these organizations have in common to discern how they were recognized for “performance excellence.” That information would be found in the Criteria for Performance Excellence (downloaded at http:/ /www.quality.nist.gov/Criteria.htm. It is popular to refer the organizations recognized as “excellent” by the Baldrige National Quality Program, other similar programs (there are approximately 35 in the United States and may be 80 or so in other countries), and researchers.

Yes you can learn a lot about how to become “excellent” by studying the two dozen or so Baldrige-winning application summaries since 2000 http:/ /www.quality.nist.gov/Contacts_Profiles.htm but you can learn more if you study – an better yet – apply the Criteria they followed to earn the recognition they achieved.

Collins makes the same argument. It is not the company; it’s the principles. The corollaries are: #1 Any organization can become “great” if it follows the principles identified by “Good to Great” research. #3 Organizations can lose their “excellent performance If the stop following the principles.

The table below compares the six “Good to Great” principles to the six Baldrige Categories. I see many similarities and a high degree of correaltion.
Good To Great Principles Baldrige Categories
Level 5 Leadership Category 1 Leadership
First Who … Then What Category 5 Workforce Focus
Confront the Brutal Facts Category 4 Measurement, Analysis and Knowledge Management
Hedgehog Concept Category 2 Strategic Planning
Culture of Discipline Category 3 Customer Focus
Technology Accelerator Category 6 Process Management

I also see a significant difference. In Good to Great, Collins does not provide many specifics on how to implement each of the six principles that distinguished the G2G companies. For example, what steps does someone take to become a “Level 5 Leader,” or to “get the right people on the bus and the wrong people off the bus,” or to “confront the brutal facts,” etc? Collins’ major findings were that Level 5 Leaders displayed the paradoxical personal qualities of humility and indomitable will. Great (pun intended). How do I become humble? How do I learn to have indomitable will?

If this proposed relationship makes sense, then we have found a way to bridge the “how to” gaps in Good To Great. We can turn to the Baldrige Core Values and Criteria requirements to identify the specific things we might consider if we want to implement each of the six steps that lead from “good” to “great.”
There is a more in depth analysis of these relationships in the Winter 2009 American Society for Quality (ASQ) Quality Management Forum pages 10-14.

[Michael Raynor]

It’s been great to see the conversation around this work begin to develop. We’re genuine when we say that this research is a work in progress, and we’re hopeful that the large community of informed and insightful practitioners, commentators, and consultants to management will see fit to help us improve our research program. As the saying goes, none of us is as smart as all of us…

David Jones observes that the Baldrige criteria might be helpful in identifying the sources of long-term success. As an erstwhile TQM specialist (it’s where I began my career almost twenty years ago) I have a pretty good sense of what Baldrige was. A quick look at the current criteria suggests a useful evolution away from “merely” quality to a more comprehensive set of criteria that are asserted to be important success factors. And it’s difficult to disagree: leadership, strategic planning, customer focus, process management… no one could claim with a straight face that any of these doesn’t matter.

There are, however, a great number of competing frameworks for thinking about the determinants of organizational success – not least the frameworks developed in the “success studies” that we examine in the monograph. Were we to take a purely “theory testing” approach to our own research, we could spend from now until eternity examining our focal companies against these different frameworks, seeking to “adjudicate” among them and pick as a “winner” the one with the best explanatory power.

Such an approach would have much to recommend it, but only if the frameworks themselves were built in scientifically rigorous ways. And it is that premise that our research to date has led us to question in the strongest possible terms. In other words, it’s not clear (enough) to us that existing frameworks are much more than solid common sense frameworks justified through a post-hoc analysis of (essentially) a randomly-chosen sample of fortunate random walkers. As a result, we’re not willing to devote the significant energy and resources required to do this kind of work to merely testing theories that have not (in our minds) risen the level of testable theories.

That’s not to say that we will be willfully ignorant of existing explanations of what drives success; at minimum, with so many out there, some of them have to be right just by chance alone. So as we work through our clinical research program, we’ll be looking for evidence that prior work has “gotten it right.” In that instance, our contribution, we hope, will be to provide a more nearly solid scientific foundation for those conclusions.

At the same time, we are open to generating new explanatory and ultimately predictive frameworks, and that requires a more nearly tabula rasa approach to the research, since we’ll need to be open to generalizing across very different industries and companies with respect to behaviors that might not be captured in existing ways of looking at the problem.

Consequently, although we hope to guided by prior work such as Baldrige, we think we’d be potentially shortchanging what the research might reveal if we were beholden to it.

Rick Payne makes a good point about expanding the research beyond companies listed on US exchanges. The challenge here is simply the availability and comparability of data. We’re working through how best to cope with the survivor and other selection biases that will be introduced by making these comparisons, but we are definitely moving toward the development and analysis of a global database of companies analyzed using our method.

Gerald Nanninga (who mentions our study on his blog – thanks for the shout out, Gerald!) observes that the stickiness of performance illustrates the importance of positioning. I’m not sure that’s true. What our research (so far) implies, at most, is that if you want good performance next year, you maximize the odds of achieving that by delivering good performance this year.

I’m the first to admit that this is hardly helpful advice… It’s why our research project is just getting started, not coming to a close, with the publication of this monograph: we can say nothing about the causes of good performance based on our observation that good performance persists. It will take a lot of hard work to figure out what those causes are. And in that work, Gerald’s claim that “position is what matters” is precisely the sort of claim that we hope to be able to test empirically. Do companies that deliver exceptionally good performance achieve it via better position (which I’ll take to mean “strategic position” in a market), or better operations (as some have suggested) or some other set of structural or behavioral traits? We’re entirely open to the possibility that “position is what matters”…but our findings to date offer no support for that belief, and the support we’ve seen for most advice tends to be post hoc and anecdotal. Our take on it is that most of the “conventional wisdom” in business is reasonable and defensible, but has shaky empirical foundations. If we’re lucky (!) we’ll be able to make some contribution to both substantiating what is already felt to be true and, perhaps, generating some new insights as well.