A Simple Way to Evaluate the Claims in a Business Book

In Scotland, courts have three possible verdicts available to them: guilty, not guilty, or not proven. The latter two are good enough to acquit the defendant, but “not proven” is definitely not as strong as not guilty. “Not proven” is the verdict that should be rendered on so many business books written today.

In yesterday’s book review of Sales Growth, I made the comment that their research methods fell short of the gold standard for research methods, and I would like to elaborate on that comment a bit. They typical business “how-to” book follows the same underlying formula:

1. We studied X number of companies and found the same common characteristics. The top companies all did _____.
2. Therefore, if you do ____, you too will succeed.
3. Here’s how to do it.

There are plenty of possible holes in #1: Just for starters,

• was the sample large enough to allow for statistical significance,
• were those companies were chosen at random,
• can “did X” be defined precisely enough to differentiate between those who “did X“and those who “did not do X”?

But let’s assume for now that the answer to those three question is yes; it still does not prove anything. Here’s a simple demonstration of why. In a very simple claim, where there is only one factor X that companies did or did not do, there are four possible combinations, best illustrated by this simple chart:

The formula in most business books focuses only on the top right quadrant: these guys did X, and they succeeded. What they leave out are these other important questions:

• Did anyone do X and fail?
• Did anyone succeed even though they did not do X?

(You could also look at the bottom-left box, but that’s assumed in the claims of the book.)

Without considering these two questions, the book is, at best, incomplete.

And that’s just one factor. When the book says something like: “the seven things that successful companies do”, the complexity expands exponentially and the likelihood of their claims goes down accordingly.

This is not to say that the claims in the book are wrong. Business moves too fast and the stakes are too high to allow for rigorous experimentation in most cases, but you should always keep these questions in mind. A little healthy skepticism is always in order; let the reader beware.