Setting the Standards of Excellence

Excerpted from A Random Search of Excellence:

 

There’s one more step before we can determine what constitutes statistically remarkable performance:  we have to define a benchmark. For the purposes of our success study, we have defined two categories of exceptional performance, tentatively labeled “Miracle Workers” and “Long Runners.”  The former deliver whatever number of 9th decile years is statistically unlikely given their lifespans, while the latter deliver whatever number of years in the 6th-8th decile band that is similarly improbable.  Our definition of excellence is motivated by a desire to contrast the behaviors of firms that do exceptionally well for a long period of time with those who do merely well (rather than exceptionally well) over long periods.

 

Finally, we have identified “Average Joes” – companies with statistically unremarkable life span, performance level, and performance variability.  Our intent, in the time honored tradition of the success study, is to compare and contrast the behaviors of these firms and thereby tease out the necessary and sufficient conditions for exceptional performance.

 

 

 

Every success study needs a definition of successful companies and predefined categories of firms to compare.  (Unless you’re In Search of Excellence, in which case no comparisons are required J.)  There doesn’t seem to be a strong theoretical reason to pick any particular set of performance profiles to compare.  Looking at “climbers” vs. “tumblers”, “winners” vs. “losers”, “good” vs. “great” – all make good sense.

 

We were unable to find clear evidence of firms that had clear – that is, statistically significant – performance trends over time, so comparing rising with falling firms wasn’t an option for us.  As a result, we settled on looking at “Miracle Workers”, “Long Runners” and “Average Joes”.  Think of it as “great” vs. “good” vs. “okay”.

Finding the Right Measure of Success

Finding the Right Measure of Success

 

 

Excerpt from A Random Search of Excellence:

 

When looking for outperforming companies, the first question is always how do you measure performance?

 

A common, but hardly unanimous, practice is to look at total returns to shareholders (TSR):  Blueprint to a billion used sales, Built to Last used a composite measure of general esteem, The Breakthrough Company used primarily sales growth.  Some studies wanted to examine a specific phenomenon (how to get to a billion in sales), while others had to cope with constraints on data availability (if you’re looking at private companies, you can’t use TSR).

 

We took the view that at first principles, the object of all of these studies, ours included, is to identify noteworthy management practices.  This has some important implications, of which perhaps the most surprising is that TSR is arguably the worst measure to use.

 

Shareholder returns are a function of the capital market’s estimate of future performance.  A good fraction of TSR tells the story of changing hopes for the future rather than delivering on past promises.  Consequently, strong returns over time are often largely the result of consistent upside surprises that serve to ratchet up expectations, which is then made manifest in a rising stock price.

 

This is perhaps most evident in companies that deliver great performance but lackluster TSR:   a company can deliver fabulous profitability and eye-popping growth and have a share price that goes nowhere because at some point in the past the markets “figured them out” and priced that performance into the shares.

 

Now, if one is interested in what behaviors surprise investors, TSR might be just the ticket.  But if great management is what you want to understand, operating measures of performance are much better.  Of the operating measures available, we like return on assets as an overall measure of profitability.

 

 

 

The cult of shareholder value has its place. After all, shareholders are people, too.  But a firm’s ability to generate shareholder value is as much a function of how shareholders view it as how well it is managed.  If anything, shareholder returns either positive or negative reflect changes in expectations for the future.

 

So, shareholder value is certainly a valid measure, but it’s likely not the best measure of corporate performance if you want to understand the contribution of managers to that performance.  And so we’re focused on ROA.  It’s not perfect, either, though, and so we’re working hard to qualify our findings in keeping with the limitations of this measure.

Leading “Gurus” Respond to Their Critics – Including Us – in Boston Globe “Luck Inc.” Article

There was a great story in The Boston Globe on the role of luck in business success, and I’m happy to note that this research was prominently mentioned. (http://www.boston.com/bostonglobe/ideas/articles/2009/04/12/luck_inc/?page=full).

What got my attention in “Luck Inc.: The 7 secrets of really, really lucky companies” was that, for the first time I’m aware of, Tom Peters and Jim Collins engaged their critics (sort of), including us (sort of). I’d like to comment on three points in particular.

First, Peters seemed to take the view that some of the criticisms of the field research reported in success studies are missing the point. Peters was quoted in the article: “‘It's not the same kind of data that you would use to refute or confirm Einstein's theory of relativity,’ says Peters. ‘It's exploratory social science research.’”

It’s been a while since I’ve read Search cover to cover, but that level of circumspection seems to be a recent addition to the conversation surrounding this type of work. Seems to me you can’t have it both ways: making bold claims about deep causal structures, and when the models don’t appear quite as predictive as you might have hoped, invoking post hoc disclaimers.

Second, in a magnanimous defense of the underlying utility of the most popular success studies, an outspoken critic of the genre, Dr. Robert Sutton, Professor, Management Science and Engineering Faculty at the Stanford Technology Ventures Program, notes that “There's value in mastering the obvious…If Jim Collins' impact is to get people to do stuff that they know they should do already - facing the hard truths or being selfless or whatever - I certainly don't think that's a bad thing.”

It’s difficult to disagree with that observation, though to me it seems very different from what people think they are buying, and it’s certainly not what they’re told they’re getting. It’s de rigueur in this kind of work to claim that one’s findings are enormously counter-intuitive – in other words, that the advice does NOT amount to what everyone already knows they are doing (see page 10 of the 2001 edition of Good to Great, for example).

Third, in what is framed in the article as a direct response to our observations about “luck”, Collins points to unpublished work he is doing that shows that luck does not, in fact, explain the difference between the winners and losers in his model. “There is, however, a significant difference in how the winners and losers view the role of luck - and therein will lie an absolutely fascinating chapter!” he writes in an e-mail.”

It is a perfectly reasonable hypothesis that highly successful companies view luck differently, or take better advantage of their lucky breaks, or create more lucky breaks, than less successful companies. If one were able to explain how that all works, so much the better. But, as Drake Bennett of The Boston Globe rightly points out, our objection is not that luck creates success, but that it masks it. We do not claim that “successful companies were just lucky.” Rather, our view is that until you sort out the role of luck, you can’t know whether you’re looking at successful companies or merely lucky ones.

In other words, unless Collins’ new sample of successful companies can be shown to have achieved performance beyond what would have been expected due to levels of luck (good or bad) that are attributable to the system itself (our “common cause variation”), then the findings will remain highly suspect because there will be no good reason to believe that there are any differences in the underlying attributes of the two firms. We’ll remain stuck in a Kipling-esque just-so world building generalizations on the post hoc identification of proximate causes. This sort of research necessarily starts with sampling on the dependent variable, and that imposes some very exigent demands on what constitutes a proper sample.

(As an aside, Collins, in a comment that seems of a kind with Peters’ “exploratory social science” back-peddling, objects to the characterization of his work as “success studies”. According to The Boston Globe article, “it is more concerned, [Collins] argues, with discovering why companies do not become great than with why they do.” Really?!? Good to Great’s subtitle is “Why Some Companies Make the Leap…and Others Don’t.” At a minimum, that seems to seek to explain both why some companies become great and why others don’t. In addition, much of the advice seems to be “positive” (“do this to be successful”):have a hedgehog concept, first who then what, and so on. That’s the sort of thing that amounts to an explanation of why some companies do “make the leap.” It seems to me that Collins’ major works are legitimately characterized as studies of the causes of success, not its impediments.)

I don’t claim to understand Collins’ views on these issues based a couple of sentences in one article. But even so, I’m increasingly of a mind that the point we hope to make in the monograph might actually be part of a bone fide paradigm shift. That is, a change in the “rules of the game” when it comes to understanding the managerial determinants of corporate success. To date, the archetype has been Search and Good to Great, and much of what has followed in that tradition feels very much like Kuhnian “normal science”. Subsequent efforts have sought to extend the explanatory power of this basic approach and to tidy up various puzzles that remain in the wake of the defining studies. These projects buy into the basic assumptions of what constitutes a phenomenon worthy of study, the schema of concepts required to understand that phenomenon, and the rules of evidence and reasoning to be applied when drawing conclusions. For example, The Breakthrough Company is billed as Good to Great for small companies.”

Paradigms break when the anomalies pile up to the point that a new set of frameworks is needed to understand what’s going on. The approach we’re taking just might be exploring the constituents of a “new set of frameworks” for research into corporate performance. We’re working on a paper for the Ivey Business Journal (www.iveybusinessjournal.com) that will be coming out in the summer that will try to make this point. Watch for it.

Michael E. Raynor, of Deloitte Consulting LLP, is a Distinguished Fellow with Deloitte Research. In his client projects and research, he explores the challenges of corporate strategy, innovation and growth.

Randomness In, Randomness Out

Excerpted from A Random Search for Excellence:

Many managers have found the prescriptions in one or more of these studies helpful, perhaps even enormously so.  And we’re sympathetic to the notion that if it works, don’t knock it.  But we’ve come to the disturbing conclusion that every one of the studies that we’ve investigated in detail is subject to a fundamental, irremediable flaw that leaves us with no good scientific reason to have confidence in their findings.

It is this:  success studies aren’t studying unexpectedly successful companies.  Rather, they are, by an overwhelming majority, studying a collection of firms that have taken fortunate random walks.  In other words, they are not studying great companies, they are studying lucky companies.  Consequently, the inputs to every success study we can lay our hands on are the wrong inputs.  This has material consequences for the confidence we can have in the advice offered, for no matter how rigorous the data collection, no matter how Aristotelian the logic, to deviate a bit from the old aphorism, “randomness in, randomness out”.

 

I’ve long been troubled by the noncumulative nature of the prescriptions associated with success that one finds in various success studies. Is M&A critical or not?  Does leadership matter, and, if so, what kind?  Is success strategy-led or is execution what it’s really all about?  It does not good to conclude that “everything’s important” since that provides no guidance in allocating resources – especially management time.

 

In addition, many prescriptions for success feel “platitudinous”:  make sure you have the “right” people.  How helpful; I always thought having the wrong people was the way to go.  More generally, if the opposite of a given prescription isn’t defensible, then the prescription itself has no real substance.

 

These two characteristics – contradictory prescriptions and empty advice – are unlikely to be a consequence of any deficiencies on the part of the researchers involved.  Our hypothesis is that it is an inevitable result of the fact (if we may be so bold) that each study examines a different sample from a random distribution.  Since the resulting patterns in behavior – the foundation of the conclusions drawn – are therefore more imposed than inferred, it’s no surprise that everyone sees a different pattern.  In addition, studying randomness makes it very difficult to come up with falsifiable claims since every company in the sample is essentially “the same” – or at least, indistinguishable from one another, which is a weaker, but from a method perspective, equivalent claim.

 

 

In short, if one hopes to reach nonrandom conclusions, one needs to begin with a nonrandom sample.