In my books and blogs, I try hard to ensure that what I write is backed up by evidence, and so I rely heavily on scientific papers, particularly those reporting the results of psychological studies. But, what if much of what I read is wrong?
That may be the case, according to Stuart Ritchie, in his new book, Science Fictions: How Fraud, Bias, Negligence and Hype Undermine the Search for Truth.
But Ritchie is not anti-science. Far from it. Think of his book as an intervention by a concerned family member, who only wants the best for science.
Much of what you read in the popular press about scientific “advances” is incorrect. That’s partly the fault of the media, but a large part of the blame rests on scientists themselves, and the way they react to a system that incentivizes poor scientific behavior and practice. It reminds me of what W. Edwards Deming said: “A bad system will beat a good person every time.”
“Every time” is surely an exaggeration when it comes to science, but the situation is bad enough. Researchers have failed to replicate about half of all studies in psychology. It makes depressing reading to learn that papers that I have cited in my books and blog posts are among these, including power posing, fixed v. growth mindset, marshmallow test. Other fields, while not as dire, show similar patterns, including economics, neuroscience, and biology.
How does it happen? Ritchie cites several reasons, but the most important is one that most of my readers will easily recognize. Scientists, like salespeople, are generally subject to a quota, except for them it’s about publishing papers rather than revenue. The number of papers published, the frequency with which they get cited in other papers, and the prestige of the journals they are in are measures that are used in tenure and grant decisions.
But as Goodhart’s Law states, “when a measure becomes the target it ceases to be a good measure.” Just as salespeople quickly figure out ways to game their system, it should not be a surprise that scientists do so as well. A few resort to outright fraud; some inadvertently or purposely let their bias affect the results they find and/or report; some make mistakes in their haste to publish; and the journals themselves and especially the popular press hype even modest results beyond what the results show.
What to do about it? Ritchie lays out a lot of ideas to reform the system. They seem sound, albeit perhaps a bit idealistic. I’m not a scientist myself, so I’m not sure I can credibly comment on his ideas. But as a consumer of science, I can focus on what I can do about it—and what you can do about it. It behooves me to be more critical and careful. For me, the most important section is the appendix, which has ten suggestions for how to read a scientific journal article. A lot of this I’ve known, but having a clearer picture of the scope of the problem will make me much more careful and skeptical in the future. I will take more care in reading individual studies and look to see if the results have been replicated elsewhere.
I can’t guarantee that I won’t get fooled again, but the authors are going to have to work a lot harder for that to happen!
Because what choice is there? In the end, even though it can be discouraging to read about the shortcomings of science, it’s important to paraphrase what Churchill said about democracy: “It’s the worst from of pursuing knowledge…except for all the others that have been tried.”
There will be plenty of post-mortem analysis of the US intelligence community’s failure to anticipate the precipitous collapse of the Afghan military over the past few weeks. It will take a lot of explaining (or rationalization) to show why, after spending more than $80 billion over 20 years to train the Afghan government forces, such a massive effort yielded such pitiful results.
Whatever transpires from the analysis, it will almost certainly contain multiple factors and reasons, and it would be irresponsible to speculate this early on all of these. When disasters happen, the dots line up and connect perfectly in hindsight, but of course the view is never that clear before it happens.
But one of the factors is worth discussing here, because it applies so well to the business world as well. An article in today’s New York Times states: “Part of the problem, according to former officials, is that the can-do attitude of the military frequently got in the way of candid, accurate assessments of how the Afghan security forces were doing.”
It sounds obvious, but it’s hard to prevent. Imagine being a junior officer being asked to report on how your training efforts are going. What would be the effect on your career prospects if you candidly reported that despite your best efforts, you were pessimistic about their impact?
Now imagine yourself being an account executive or a product manager in a forecasting meeting. Would you do it any differently?
Probably not. America has long been a country where the power of positive thinking is enshrined in our culture. And overall, I believe that has been a good thing. It has enabled us to accomplish incredible feats and helped build the most powerful economy in the world.
But it can go too far. Can-do is almost a religion, where pessimism is a sin and realism is suspected to be. So, when someone reports or forecasts to their managers, they’ll naturally tend to shade toward the bright side, even when they think they’re being absolutely candid. Those managers will in turn take those reports and shade them slightly when they report up to the next level, so you can imagine the possibilities for overconfidence when the “ground truth” gets to the top.
If you’re a leader, here are three things you can do to reconcile can-do and candor:
- Take this saying out of your vocabulary: “Don’t bring me a problem unless you bring me a solution.” If your subordinate finds out about a problem that they’re unequipped to handle, do you really want them to keep quiet about it?
- Go to the gemba. That’s lean-speak for going to the scene of the action to see for yourself.
- Break the path into detail. In his book, Perfectly Confident, business professor Don Moore cites the well-known example that 93% of people rate themselves in the top half of all drivers. But when they ask them to rate themselves on individual skills such as signaling, using mirrors, or backing up, their “overplacement” reduced substantially. Detail helps the dots line up into a truer picture.
I’ve stressed enough that value is determined by the listener, so you may be thinking by now, “What about me, don’t I get to get something out of my communication?”
Of course you do. It’s the same as selling a product: the buyer gets to decide if they get value, but naturally the producer is entitled to a fair profit if they do.
Lean communication is not a recipe for just rolling over and giving your listeners anything they want to hear; ultimately you communicate to accomplish your own outcomes and benefits. In fact, you do your listeners a favor if you have a clear idea of your own purpose, and if you make your ask explicit at the beginning. It reduces the tension of waiting for the other shoe to drop, and allows them to listen for what they need to make a decision.
A lot of people are reluctant to ask directly for what they want because they’re afraid of rejection, but research has shown that people are about twice as likely to say yes as they estimate. Don’t be afraid to ask. My neighbor Rocky once needed to get some carpet replaced. The installer quoted him a price, and Rocky asked, can you do it for less? The guy knocked $1,000 off without blinking an eye.
One question—a thousand dollars. That’s pretty lean, in my book!
If you want to set yourself an ambitious standard to improve your communication, ask yourself before an important discussion or presentation: “Would they pay to hear what you have to say?”
You may not get an unqualified yes, but the exercise of trying to achieve it will force you to think carefully about the practical value your listeners receive, in the form of useful information that they can use to decide or act in a way to improve their personal or business outcomes.
Of course, in real life, no one actually pays to hear what you have to say, right?
Actually, they do.
They pay in the scarcest resource they have: their attention. Economist Herbert Simon wrote:
“What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”
If Simon was correct when he wrote those words in 1971, imagine how much more important it is 50 years later to repay your recipients’ precious attention.
Attention is difficult and costly. So, do your best to make it worth your listeners’ time and effort. Attention is not just time; it’s time and effort. Focus on RoTE: Return on Time and Effort. Or, in lean communication terms: give them value, briefly and clearly.
See also: The Economics of Lean Communication
