Book Review: When Can You Trust the Experts?

Daniel Willingham wrote When Can You Trust the Experts: How to Tell Good Science from Bad in Education to help educators sift out good science from bad in education, but it is an excellent primer for anyone in business who is contemplating making a change based on recommendations from experts making scientific claims. In business, as in education, the demand for advice is as high as it has ever been, and there no shortage of willing suppliers. That makes it more important than ever to know how to separate good advice from bad.

The ability to discern good science from bad is important if you are making important corporate decisions. If you’re a buyer, you like to complete as much research as possible before talking to salespeople. We’ve been told that buyers are much better informed than in the past, but are you better informed, or simply more informed?

If you’re reading this, you’re one of the elite who likes to keep up with the latest ideas, but how do you know which ones are likely to be true? When top experts offer conflicting advice, how do you evaluate their respective claims? It’s even critical in our personal lives. What diet should we follow? How do we know?

My own field of sales and communications training is full of “experts” peddling myths, sometimes unknowingly, such as learning styles and the often-quoted statistic that 93% of communication is non-verbal. (Full disclosure: I once had both of those ideas in my training material.)

Willingham tells us that we can learn to think more like scientists. He does not claim that we can be as good as a trained scientist with a background in the particular field, because their practices and habits of mind take years to develop, and background knowledge is essential in evaluating many claims. But, we can do much better at it than we are now.

Four steps for more informed decisions:

Step 1: Strip it and Flip It

Strip it means taking out all the hyped-up, vague or emotional language that expert persuaders wrap around it to make it go down smoothly. Simply try to write the following sentence:

If I do X, there is a Y percent chance that Z will happen.

By boiling it down to those stark terms you gain clarity on what the person is trying to say, so that you can test it. If the claim can’t be expressed clearly, then that’s a red flag right there.

Flipping it means expressing its reverse corollary. For example, when hamburger is advertised as 85% lean, it also means it’s 15% fat, which doesn’t sound near as appetizing. You can also flip the action: what happens if you don’t do what they recommend?

Step 2: Trace It

If they have the credentials, many experts will simply say, “Trust me, I’m an expert.” If they don’t, they will say, “Trust me, because I have expert sources.” There’s nothing wrong with that, of course, because life would be impossible if we had to research everything ourselves. But for important decisions, a little digging under the surface can make a big difference, and it’s so easy to do on the internet. One of the problems is that the claim may be based on a kernel of truth from a valid study, but the expert may overstate the claim (as in the 93% myth mentioned above). Go to the original source when you can.

Step 3: Analyze It

Willingham takes the reader on a tour of some basic analysis and critical thinking skills, including sample size and statistical significance. Moderately sophisticated readers won’t learn much here. I do like one important point he makes, though: “If a claim sounds like a breakthrough, it’s probably a sham, because unheralded breakthroughs in science are exceedingly rare.”

Step 4: Should I Do it?

Finally, you have to decide whether to implement the change, buy the product, etc. This chapter takes you through two comprehensive checklists. If you follow these, you may not be guaranteed a good decision, but you’ll certainly have a defensible one if it doesn’t pan out!

Will this book help you?

Before I make my closing observation, let me first say that I think When Can You Trust the Experts? is a good book, and explains a useful (albeit basic) approach that can help make you a better consumer of information and improve your decision making. BUT…there is a lot of sound advice that is worth taking that does not meet the stringent standards outlined in the book. The irony is that the book itself offers a prime example. Let’s just apply Step 1 to the book. What is Willingham’s claim?

The cover flap states:

“When Can You Trust the Experts? offers parents, teachers, administrators, and policymakers the tools they need to ask tougher questions, think more logically about why an intervention might or might not work, and ultimately make more informed decisions.”

If we strip this claim (including evidence cited in the book itself), we get,

“If you follow the prescriptions in this book…

…there is a ____% chance…

…that you will make more informed decisions.”

This brings up a few questions:

• Do I have to follow all the prescriptions? (Let’s use the 16 questions on pages 208-209 as a template?)
• How will I know if I have followed them? (Some of the questions are hard to answer in a binary fashion.)
• What is the % chance? (I did not see any specific quantified claims in the book.)
• How will you know your decisions are more informed?
• How much more?
• Will more informed mean better outcomes from the decisions?

So, as you can see, it’s pretty tough to get good decision-making down to a specific methodology, but I do believe this book makes a strong case that some methodology is better than none at all.