Superforecasting: The Art and Science of Prediction
There is a huge market for forecasting in our country, from political talking heads on TV, to investment advisors, to the approximately 20,000 intelligence analysts working in our intelligence establishment. But, while the stakes for getting it wrong can be huge (see Iraq’s WMD), there is no formal reliable way of measuring or analyzing the track records of those doing the predicting. Pick your favorite commentator or columnist, what’s their hit rate on their predictions? That’s impossible to answer, first because no one has compared what they said would happen to what did happen, and even if they did, so many of their predictions are so vaguely worded that most of them can easily claim they meant something else and wiggle off the hook.
Philip Tetlock is trying to change that. Beginning in the 1980s, he has been studying how good experts are at prediction (answer, just slightly better than a drunk monkey throwing darts). One of his findings was that pundits who were the most confident tended to be wrong more often, but they also got on TV more often. They are hired more for their ability to tell a compelling story with confidence than for their track record in forecasting.
This latest book details his findings from a four-year project funded by IARPA, the Intelligence Advanced Research Projects Activity, to test the forecasting performance of several different teams of experts. It was a large test which asked over 500 questions to more than 20,000 participants between 2011 and 2015. It was also rigorous, with questions designed to eliminate the wiggle room problem. For example, they asked, “Will any country withdraw from the Eurozone in the next three months? How many additional countries will report cases of the Ebola virus in the next eight months?”
The study found that about 2% of participants, which he calls superforecasters, are consistently more accurate in their predictions. By identifying the superforecasters, and then testing different combinations and variables, he was able to tease out what makes them successful, and the bulk of the book explains the traits, techniques and habits of thought that make for superior judgment.[1]
The basic theme is that it’s not superior intellect that distinguishes the SFs, but how they think. Here are just a few of his recommendations:
- Break down tough problems into their components, and make estimates or judgments about those.
- Pay attention to base rates first, and then adjust. For example, I may think that my friend is very likely to strike it rich in a very difficult venture, because I start with knowing how smart he is. But if I begin by considering that the odds are 50 to 1 against success, I could double his chances and still think it’s very unlikely.
- Be actively open-minded, not only being open to new information but looking for it. Once you have formed a judgment, pay attention to new information, especially anything that would call your initial judgment into question.
- Write down your forecasts and your reasoning, because the mere fact of writing it will help distance you emotionally from your first prediction. If it’s important that you get it right, take the further step of writing down all the reasons you might be wrong, and then synthesize the two.
- Expand your range of alternatives. Most people have a three-position dial about predictions: yes, no, and even odds. You can force yourself to become more systematic about your own thinking by adopting a 7-point scale as recommended by the National Intelligence Council as you see here:
Remote Very unlikely Unlikely Even Probably, likely Very likely Almost Certain
Even better, use percentages. It won’t guarantee[2] you’re right, but it will force you to examine your own thinking and help you adopt a more nuanced viewpoint.
There’s far more good advice than I can summarize, but frankly I’m struggling a little in deciding whether to recommend that you read Superforecasting. On the plus side, I predict that it is very likely that if you read and apply its lessons, you will become a better thinker. On the other hand, it’s an even chance that you will become a worse persuasive communicator. That’s because an effective salesperson radiates confidence about the future they recommend, while effective forecasters are far more cautious and humble about their predictions.
My personal choice would be to begin with better thinking. First, for the obvious point that you owe it to yourself and to others to give them your best thinking. Second, sustainable influence depends on credibility, which will in the long run correlate strongly with your predictions. It’s true that TV pundits who are the most confident in their predictions tend to be wrong most often, and they don’t suffer for it. But when people are putting their own reputations or money at stake based on what you predict, they tend to have longer memories.
[1] I use judgment in the sense that they are better predictors, not that they necessarily make better decisions.
[2] In fact, you may have noticed that the seven-point scale does not include certainty on either side.
To know that we know what we know and that we do not know what we do not know, that is true knowledge.[1]
Confucius
If you could know – and prove – beyond a reasonable doubt that everything you say or write is true, you would quickly become immensely credible. You would also probably live on another planet.
Credibility is nothing but the probability estimate that others form when deciding whether to rely on what you tell them. You’re credible when they assume a reasonably high probability that what you say is correct.
But even though credibility is something that others assign to you, it has to begin with your own probability estimate. Any time you utter something controversial, you put your personal credibility at risk. It may be a slight risk, as when you tell someone they would probably like that new restaurant, or a huge risk, as when you passionately advocate a major investment for your company. So, you weigh the evidence in your mind, maybe carefully and analytically, or maybe intuitively, to figure out how certain you are before you decide whether to take the risk.
Since you can’t be sure of everything, the next best thing is to be able to accurately measure how sure you should be. For example, you may be 100% sure that the sun will rise tomorrow, but how sure are you that it will rain tomorrow, or that the project you’re proposing will cut costs in half? If you think it’s a high probability, you might estimate the chances at 80%. If you have no clue, your estimate would be 50%, if you think it’s possible but not probable, it might be 20%.
But here’s the rub. How accurate is your estimation of certainty? Calibration is a measure of the accuracy of your own probability estimate about what you believe to be true. It’s a measure of how closely your level of certainty accords to the true facts. If you are generally accurate, you’re said to be well-calibrated. If you’re over- or under-confident in your certainty, you are poorly calibrated.
Just as some people know more than others, some people are better calibrated than others. So, for example, in one of the simplest tests you may answer ten questions and if you’re 70% certain about each of your answers, you will get seven right if you’re well-calibrated, fewer than seven if you’re overconfident, and more than seven if you’re underconfident. Most people are overconfident; one study that gave a quiz to over 2000 people found that fewer than 1% were not overconfident.[2]
Overconfidence is not all bad – it encourages difficult efforts and can help you sell your ideas. It will tend to increase your credibility in a single situation, because listeners will take cues from your perceived confidence. Your level of certainty about what you’re saying will affect the confidence with which you express it, which will in turn affect how much listeners believe you.
But excessive overconfidence can definitely hurt your credibility by increasing the odds that you will be shown to be wrong. We all know people who are often wrong, but never in doubt – just watch any of the early stages of American Idol to see this overestimation displayed to a painful degree. In fact, studies have shown that the people with the least competence are the most likely to overestimate their actual knowledge. It’s called the Dunning-Kruger Effect[3]. Justin Kruger and David Dunning of Cornell University ran a study that measured subjects’ objective performance in tests of humor, grammar and logic, and found that those scoring in the bottom quartile were the most overconfident of their abilities; scoring on average in the 12th percentile, they rated themselves on average in the 62nd.
But there’s also an external aspect, which is others’ perceptions of how well-calibrated you are. If you’re well-calibrated, you are less likely to run ahead of your facts and get yourself into trouble, which is a good thing for long-term credibility.
Some very few people are underconfident in their estimate of certainty.[4] They are less sure of their knowledge, which certainly lowers the risk of being proven wrong, but also limits their influence. Their uncertainty may show through in their expression, or they may be less apt to speak up on behalf of their position or interests.
So, being well-calibrated will improve your credibility in two ways. First, it will help you avoid the extremes of over- and underconfidence. Second, by being perceived to be well-calibrated, or self-aware, you can be more credible to your listeners.
Because most people are overconfident, improved calibration will most likely cause you to dial back your confidence a little when you speak. Although it would seem that being tentative would lower your credibility, it depends on the situation. One area where perceptions of credibility have immediate and important consequences is in criminal trials, and researchers have found – in mock trials – that jury members are affected by how well-calibrated they perceive witnesses to be. Jury members were initially more likely to believe witnesses who expressed certainty about what they had seen than those who were less sure. But when their testimony was later shown to be wrong in a minor detail, the effects were reversed. The confident ones were seen as less credible, while the unsure ones were seen as more credible.
If you’re already seen as an expert, being a little less sure may help. Experts who express some uncertainty were found in one study to be seen as more credible than when they expressed certainty[5]. The author of the study ascribes this to the surprise factor that makes people pay closer attention to their message and hence be more influenced. But I think there may be a different explanation. Showing that you know you could be wrong makes you seem more self-aware (better calibrated) and open-minded, which plays better with educated audiences.
What’s the lesson we can draw from this? Don’t get ahead of your facts. Be transparent about your levels of confidence. When you’re unsure of something, say so. It will make you more credible when you say you’re sure.
How to improve your calibration
Calibration can be improved through training and experience. It begins with awareness of the problem and acceptance of the fact that you are probably susceptible to it. Here are a half-dozen ways to get better.
Test your calibration. ProjectionPoint has a test on their website that allows you to test your calibration. Simply seeing the results, if they are bad, will make you aware of the need to improve your calibration.
Separate fact from opinion. As Richard Feynman said, “The most important thing is not to fool yourself. And you’re the easiest person to fool.”
Keep track. Experience tends toreduce overconfidence and improve calibration, as long as you learn from that experience. It’s no accident that two of the best-calibrated professions are bookies and meteorologists. This is because they get rapid feedback on their decisions, and are held accountable for being wrong.
Be more foxy. As we saw previously, hedgehogs, who know one thing very well, tend to be less calibrated than foxes, who have more breadth of knowledge. He found that hedgehogs were not only wrong more often than foxes, but that they were less likely to recognize or admit that they were wrong when events did not match their predictions
Try not to make up your mind too quickly. Early judgments can serve as anchors, so that if you adjust your position in light of new information, you will probably not adjust as far. If you do, be on the lookout for confirmation bias, which is the general tendency to notice evidence that supports your view and be less apt to seek out or even notice contrary evidence. Follow Charles Darwin’s example:
“I had also, during many years, followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favourable ones. Owing to this habit, very few objections were raised against my views which I had not at least noticed and attempted to answer.”
Practice productive paranoia. When you’re very confident and it’s important, try extra hard to find holes in your idea. Individually, you can take the time to list reasons why you might be wrong. With colleagues, you can conduct a PreMortem: imagine that it is some future time and your idea has failed, and try to figure out all the ways it could have happened.[6]
If you follow these six practices, I’m 90% confident that your calibration will improve, and 75% confident that your personal credibility will also.
[1] Quoted in “Managing Overconfidence, by J. Edward Russo and Paul J.H. Schoemaker, Sloan Management Review, Winter 1992.
[2] Russo and Schoemaker 1992.
[3] Kruger, Justin; David Dunning (1999). “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”.
[4] Russo and Schoemaker say that public accountants are slightly underconfident.
[5] Experts Are More Persuasive When They’re Less Certain, Zakary Tormala, Harvard Business Review, March 2011.
[6] The term was coined (I believe) by Gary Klein, in his book, The Power of Intuition.
Knows one thing very well
If you want to develop maximum credibility, is it better to be a hedgehog or a fox?
I’m referring to the terms that Isaiah Berlin used in an essay classifying thinkers as either hedgehogs or foxes. According to parable the hedgehog knows one thing very well, and the fox knows a lot of things.
Is there a clear advantage of one style over the other? A hedgehog thinker would answer yes; he would come down squarely on one side or the other and support his position to the hilt. And he would be very credible in doing so.
For developing credibility, there is a lot to be said for being a hedgehog. By concentrating on one area, you can focus your learning and develop deep expertise which can differentiate you from others and establish your reputation. According to Jim Collins, in his book Good to Great, the “hedgehog concept” was one of the factors that helped that companies make the leap to greatness. They focused on one thing and did it really well. Because they figured out what they could be good at, they had the advantage of clarity, focus, and confidence.
When applied to individual success, the hedgehog concept makes perfect sense because it ties neatly into the popular idea that when you can find something you are passionate about, you will succeed because you will pour your heart and soul into it. That level of concentration can get you to the point where you know more than anyone else, and you will have rock-solid credibility in your field[1]. And, with knowledge accumulating so rapidly in so many fields, that concentration will keep you abreast of the state of the art. You would probably not want to go to a doctor who stops reading medical journals so he can broaden his mind in other areas.
Hedgehogs also have an advantage when it comes to persuasion: they are listened to more and are more believable when they speak, because of their confidence and conviction. In fact, I would propose that the most successful bloggers are definitely hedgehogs, and most best-selling business books are written by hedgehogs. When readers and listeners are suffering from ever-shrinking attention spans, no one has time to explore the shades of grey between black and white. Very few people nowadays have patience for nuanced arguments; we prefer to be like Harry Truman, who complained that he wanted to meet a one-armed economist, so that he would never have to hear him say, “on the other hand…”
But could there be a downside to single-minded concentration on one big thing? Philip Tetlock thinks so. Tetlock studied the track records of political commentators over 15 years, those folks you see on the Sunday talk shows who get paid to make predictions about what will happen during political crises and trends. He found that hedgehogs were not only wrong more often than foxes, but that they were less likely to recognize or admit that they were wrong when events did not match their predictions[2].
The advantage that foxes have is that they are more likely to seek out new information from a broader range of sources, and are comfortable with uncertainty and new information. When something happens that contradicts their view of the world, they treat it as new information, not as an aberration or an exception. They try to incorporate it into a more nuanced viewpoint rather than finding reasons to exclude it from their thinking. They are also better calibrated, meaning that they have a clearer estimation of what they know and don’t know.
Knows a lot of things
The irony is that although foxes are more often right, they get less air time. In fact, judging from seeing the same faces all the time on television despite being woefully off on their predictions, it pays to be a hedgehog. Maybe people just have short memories, so being wrong is rarely fatal.
So, which is better?
I’m going to try to answer the question in a foxy hedgehog style. In other words, there are clear advantages for each style, depending on the situation. We’ll look at three different activities – learning, thinking, and persuading – each considered over the short term and the long term, for a total of six different scenarios.
Learning. In focusing on what to study and learn, there is no question that you should be a hedgehog early in your career, so that you can establish solid credentials and expertise in one particular area. It could be an academic discipline such as electrical engineering or finance, or it could just be becoming a deep expert on your craft, or even your specific accounts. You want to get so knowledgeable and so good at what you do that you become recognized as the “go-to” person. So many fields require years of focused learning and improvement to master that it behooves you not to scatter your attention early.
Even when you think you’ve mastered your field, you can’t stop learning. Because we are learning so much about so many things, there is a half-life to knowledge in any field, and a lot of “facts” that you take for granted are probably wrong, or will eventually be proven wrong.
Yet, as you get promoted to positions of higher responsibility, you’re going to find yourself dealing with more and more situations that your expertise has not prepared you for. In order to avoid Peter Principle, you have to become more of a fox; get curious about the wider world, broaden your knowledge, and learn to blend in different perspectives. In general, learn like a hedgehog early to establish a secure base, and then evolve into a fox.
Thinking style. I come down solidly on the foxy side on this one, both for the short and the long term. We all crave the comfort of simplicity, the stability of “timeless” ideas, and the seductive clarity of apparent patterns, but that’s dangerous in a complex and dynamic environment. Especially as you rise in an organization, more and more of the problems you will have to solve are people problems, and people are too diverse and complicated to fit into the neat little certainties that hedgehogs love. Plus, the world changes too quickly for a single big idea to hold its power for very long. Check out the subsequent track records of the “great” companies that Collins wrote about for all the confirmation you need about that.
You should almost always think like a fox. (I would say always, but that would turn it into an oxymoron.)
Persuasive expression. In the short run, it’s clearly more persuasive to speak like a hedgehog, because listeners respond to clarity and confidence. If your only goal is to win others over to your point of view – right now – then go full hedgehog on them. That’s what TV pundits and transactional salespeople do, because it works.
But if you care about long term credibility, keep in mind that most people have excellent memories about things that affect them directly. They will pay attention and they will keep score, and they will be especially more likely to remember your inevitable mistakes unless you learn to speak with a certain amount of fox-like humility. It’s good to have conviction and clarity, but sustainable agreements are more likely when you remain open to others’ reactions and inputs. They will feel greater ownership of the decision taken and you may actually learn something that will improve your original idea. To encapsulate this approach, it’s hard to improve on Bob Johansen’s phrase, “strong opinions, which are weakly held.”
In summary, the choice between being a hedgehog or a fox is a false trade-off. The most effective way to go through life is to strive to be that rare hybrid known as foxy hedgehog!
Knowledge has to be improved, challenged and increased constantly, or it vanishes.
Peter Drucker
Spring cleaning is an optimistic time, a time to refresh and revitalize your surroundings. Closets get cluttered, stuff gets lost, and things break down, wear out, or go out of style. The same thing can happen to our minds.
One of the side effects of the research I’ve been doing recently for my book on personal credibility is that I get to revisit books and articles that I may have read several years ago, and rediscover useful information that I had either forgotten or overlooked when I first read them. It’s always funny to see what I’ve highlighted or starred, then promptly ignored. Or maybe something I read then makes different sense now in light of different life experiences I’ve had or new things I’ve learned.
For example, I re-read some sections in Bruce Gabrielle’s Speaking PowerPoint: The New Language of Business
Whenever you read a book or take a course, you only retain and use a small part of what it contains. Besides the natural limitations of your memory, you will gravitate to one or two ideas or techniques, and as you use them more often they will become more firmly implanted into your routine while everything else will wither away.
For example, you might learn several different approaches to handling a price objection. In the days and weeks after the training, you try two or three approaches; maybe one works well, one bombs, and one is somewhere in the middle. You’ll focus on the successful experiences and drop the ones that didn’t work. But there’s a case to be made for revisiting the ones that didn’t work. Maybe they would actually work better in a different situation, or maybe times have changed. But you may never know because you’ve forgotten them—unless you do the occasional spring cleaning.
Here’s a few things you can do:
Toss out broken old ideas. Have you unlearned anything recently? Example: I “learned” once that we can hold 7 items at one time in working memory. I’ve since found out that this is wrong; the actual number is 3-4, which is very important to know if you want to communicate effectively.
Rediscover stuff you’ve forgotten. Re-reading a favorite book that taught you some valuable lessons is like getting back in touch with an old friend; you’ll wish you hadn’t waited so long. While I personally try to keep up with the latest books in fields that interest me, I still find that old books are the best. One of my favorites is Moving Mountains by Henry Boettinger.
Freshen up stale stuff. I apply this one to my training. Some of the examples and anecdotes need to be changed once in a while even though my new students are hearing them for the first time, because it keeps me excited about telling them.
Recharge. Sure, motivational books tell you stuff you already know, but it doesn’t hurt to give yourself a boost once in a while. If you want to try something new, I recommend EDGY Conversations: How Ordinary People Can Achieve Outrageous Success
Learn something new. When you’ve cleared out all the clutter, you’ll have room in your mind for something new. Maybe it’s a practical skill that might take you to the next level in your career, or something less immediately practical but maybe more fulfilling, such as learning a language or taking an online course.
