Calibration: How Well Do You Know What You Know?

To know that we know what we know and that we do not know what we do not know, that is true knowledge.[1]

                Confucius

If you could know – and prove – beyond a reasonable doubt that everything you say or write is true, you would quickly become immensely credible. You would also probably live on another planet.

Credibility is nothing but the probability estimate that others form when deciding whether to rely on what you tell them. You’re credible when they assume a reasonably high probability that what you say is correct.

But even though credibility is something that others assign to you, it has to begin with your own probability estimate. Any time you utter something controversial, you put your personal credibility at risk. It may be a slight risk, as when you tell someone they would probably like that new restaurant, or a huge risk, as when you passionately advocate a major investment for your company. So, you weigh the evidence in your mind, maybe carefully and analytically, or maybe intuitively, to figure out how certain you are before you decide whether to take the risk.

Since you can’t be sure of everything, the next best thing is to be able to accurately measure how sure you should be. For example, you may be 100% sure that the sun will rise tomorrow, but how sure are you that it will rain tomorrow, or that the project you’re proposing will cut costs in half? If you think it’s a high probability, you might estimate the chances at 80%. If you have no clue, your estimate would be 50%, if you think it’s possible but not probable, it might be 20%.

But here’s the rub. How accurate is your estimation of certainty? Calibration is a measure of the accuracy of your own probability estimate about what you believe to be true. It’s a measure of how closely your level of certainty accords to the true facts. If you are generally accurate, you’re said to be well-calibrated. If you’re over- or under-confident in your certainty, you are poorly calibrated.

Just as some people know more than others, some people are better calibrated than others. So, for example, in one of the simplest tests you may answer ten questions and if you’re 70% certain about each of your answers, you will get seven right if you’re well-calibrated, fewer than seven if you’re overconfident, and more than seven if you’re underconfident. Most people are overconfident; one study that gave a quiz to over 2000 people found that fewer than 1% were not overconfident.[2]

Overconfidence is not all bad – it encourages difficult efforts and can help you sell your ideas. It will tend to increase your credibility in a single situation, because listeners will take cues from your perceived confidence. Your level of certainty about what you’re saying will affect the confidence with which you express it, which will in turn affect how much listeners believe you.

But excessive overconfidence can definitely hurt your credibility by increasing the odds that you will be shown to be wrong. We all know people who are often wrong, but never in doubt – just watch any of the early stages of American Idol to see this overestimation displayed to a painful degree. In fact, studies have shown that the people with the least competence are the most likely to overestimate their actual knowledge. It’s called the Dunning-Kruger Effect[3]. Justin Kruger and David Dunning of Cornell University ran a study that measured subjects’ objective performance in tests of humor, grammar and logic, and found that those scoring in the bottom quartile were the most overconfident of their abilities; scoring on average in the 12^th percentile, they rated themselves on average in the 62^nd.

But there’s also an external aspect, which is others’ perceptions of how well-calibrated you are. If you’re well-calibrated, you are less likely to run ahead of your facts and get yourself into trouble, which is a good thing for long-term credibility.

Some very few people are underconfident in their estimate of certainty.[4] They are less sure of their knowledge, which certainly lowers the risk of being proven wrong, but also limits their influence. Their uncertainty may show through in their expression, or they may be less apt to speak up on behalf of their position or interests.

So, being well-calibrated will improve your credibility in two ways. First, it will help you avoid the extremes of over- and underconfidence. Second, by being perceived to be well-calibrated, or self-aware, you can be more credible to your listeners.

Because most people are overconfident, improved calibration will most likely cause you to dial back your confidence a little when you speak. Although it would seem that being tentative would lower your credibility, it depends on the situation. One area where perceptions of credibility have immediate and important consequences is in criminal trials, and researchers have found – in mock trials – that jury members are affected by how well-calibrated they perceive witnesses to be. Jury members were initially more likely to believe witnesses who expressed certainty about what they had seen than those who were less sure. But when their testimony was later shown to be wrong in a minor detail, the effects were reversed. The confident ones were seen as less credible, while the unsure ones were seen as more credible.

If you’re already seen as an expert, being a little less sure may help. Experts who express some uncertainty were found in one study to be seen as more credible than when they expressed certainty[5]. The author of the study ascribes this to the surprise factor that makes people pay closer attention to their message and hence be more influenced. But I think there may be a different explanation. Showing that you know you could be wrong makes you seem more self-aware (better calibrated) and open-minded, which plays better with educated audiences.

What’s the lesson we can draw from this? Don’t get ahead of your facts. Be transparent about your levels of confidence. When you’re unsure of something, say so. It will make you more credible when you say you’re sure.

How to improve your calibration

Calibration can be improved through training and experience. It begins with awareness of the problem and acceptance of the fact that you are probably susceptible to it. Here are a half-dozen ways to get better.

Test your calibration. ProjectionPoint has a test on their website that allows you to test your calibration. Simply seeing the results, if they are bad, will make you aware of the need to improve your calibration.

Separate fact from opinion. As Richard Feynman said, “The most important thing is not to fool yourself. And you’re the easiest person to fool.”

Keep track. Experience tends toreduce overconfidence and improve calibration, as long as you learn from that experience. It’s no accident that two of the best-calibrated professions are bookies and meteorologists. This is because they get rapid feedback on their decisions, and are held accountable for being wrong.

Be more foxy. As we saw previously, hedgehogs, who know one thing very well, tend to be less calibrated than foxes, who have more breadth of knowledge. He found that hedgehogs were not only wrong more often than foxes, but that they were less likely to recognize or admit that they were wrong when events did not match their predictions

Try not to make up your mind too quickly. Early judgments can serve as anchors, so that if you adjust your position in light of new information, you will probably not adjust as far. If you do, be on the lookout for confirmation bias, which is the general tendency to notice evidence that supports your view and be less apt to seek out or even notice contrary evidence. Follow Charles Darwin’s example:

“I had also, during many years, followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favourable ones.  Owing to this habit, very few objections were raised against my views which I had not at least noticed and attempted to answer.”

Practice productive paranoia. When you’re very confident and it’s important, try extra hard to find holes in your idea. Individually, you can take the time to list reasons why you might be wrong. With colleagues, you can conduct a PreMortem: imagine that it is some future time and your idea has failed, and try to figure out all the ways it could have happened.[6]

If you follow these six practices, I’m 90% confident that your calibration will improve, and 75% confident that your personal credibility will also.

[1] Quoted in “Managing Overconfidence, by J. Edward Russo and Paul J.H. Schoemaker, Sloan Management Review, Winter 1992.

[2] Russo and Schoemaker 1992.

[3] Kruger, Justin; David Dunning (1999). “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”.

[4] Russo and Schoemaker say that public accountants are slightly underconfident.

[5] Experts Are More Persuasive When They’re Less Certain, Zakary Tormala, Harvard Business Review, March 2011.

[6] The term was coined (I believe) by Gary Klein, in his book, The Power of Intuition.