Calibration: How Well Do You Know What You Know?

August 18th, 2014
Measure twice, speak once

Measure twice, speak once

To know that we know what we know and that we do not know what we do not know, that is true knowledge.[1]


If you could know – and prove – beyond a reasonable doubt that everything you say or write is true, you would quickly become immensely credible. You would also probably live on another planet.

Credibility is nothing but the probability estimate that others form when deciding whether to rely on what you tell them. You’re credible when they assume a reasonably high probability that what you say is correct.

But even though credibility is something that others assign to you, it has to begin with your own probability estimate. Any time you utter something controversial, you put your personal credibility at risk. It may be a slight risk, as when you tell someone they would probably like that new restaurant, or a huge risk, as when you passionately advocate a major investment for your company. So, you weigh the evidence in your mind, maybe carefully and analytically, or maybe intuitively, to figure out how certain you are before you decide whether to take the risk.

Since you can’t be sure of everything, the next best thing is to be able to accurately measure how sure you should be. For example, you may be 100% sure that the sun will rise tomorrow, but how sure are you that it will rain tomorrow, or that the project you’re proposing will cut costs in half? If you think it’s a high probability, you might estimate the chances at 80%. If you have no clue, your estimate would be 50%, if you think it’s possible but not probable, it might be 20%.

But here’s the rub. How accurate is your estimation of certainty? Calibration is a measure of the accuracy of your own probability estimate about what you believe to be true. It’s a measure of how closely your level of certainty accords to the true facts. If you are generally accurate, you’re said to be well-calibrated. If you’re over- or under-confident in your certainty, you are poorly calibrated.

Just as some people know more than others, some people are better calibrated than others. So, for example, in one of the simplest tests you may answer ten questions and if you’re 70% certain about each of your answers, you will get seven right if you’re well-calibrated, fewer than seven if you’re overconfident, and more than seven if you’re underconfident. Most people are overconfident; one study that gave a quiz to over 2000 people found that fewer than 1% were not overconfident.[2]

Overconfidence is not all bad – it encourages difficult efforts and can help you sell your ideas. It will tend to increase your credibility in a single situation, because listeners will take cues from your perceived confidence. Your level of certainty about what you’re saying will affect the confidence with which you express it, which will in turn affect how much listeners believe you.

But excessive overconfidence can definitely hurt your credibility by increasing the odds that you will be shown to be wrong. We all know people who are often wrong, but never in doubt – just watch any of the early stages of American Idol to see this overestimation displayed to a painful degree. In fact, studies have shown that the people with the least competence are the most likely to overestimate their actual knowledge. It’s called the Dunning-Kruger Effect[3]. Justin Kruger and David Dunning of Cornell University ran a study that measured subjects’ objective performance in tests of humor, grammar and logic, and found that those scoring in the bottom quartile were the most overconfident of their abilities; scoring on average in the 12th percentile, they rated themselves on average in the 62nd.

But there’s also an external aspect, which is others’ perceptions of how well-calibrated you are. If you’re well-calibrated, you are less likely to run ahead of your facts and get yourself into trouble, which is a good thing for long-term credibility.

Some very few people are underconfident in their estimate of certainty.[4] They are less sure of their knowledge, which certainly lowers the risk of being proven wrong, but also limits their influence. Their uncertainty may show through in their expression, or they may be less apt to speak up on behalf of their position or interests.

So, being well-calibrated will improve your credibility in two ways. First, it will help you avoid the extremes of over- and underconfidence. Second, by being perceived to be well-calibrated, or self-aware, you can be more credible to your listeners.

Because most people are overconfident, improved calibration will most likely cause you to dial back your confidence a little when you speak. Although it would seem that being tentative would lower your credibility, it depends on the situation. One area where perceptions of credibility have immediate and important consequences is in criminal trials, and researchers have found – in mock trials – that jury members are affected by how well-calibrated they perceive witnesses to be. Jury members were initially more likely to believe witnesses who expressed certainty about what they had seen than those who were less sure. But when their testimony was later shown to be wrong in a minor detail, the effects were reversed. The confident ones were seen as less credible, while the unsure ones were seen as more credible.

If you’re already seen as an expert, being a little less sure may help. Experts who express some uncertainty were found in one study to be seen as more credible than when they expressed certainty[5]. The author of the study ascribes this to the surprise factor that makes people pay closer attention to their message and hence be more influenced. But I think there may be a different explanation. Showing that you know you could be wrong makes you seem more self-aware (better calibrated) and open-minded, which plays better with educated audiences.

What’s the lesson we can draw from this? Don’t get ahead of your facts. Be transparent about your levels of confidence. When you’re unsure of something, say so. It will make you more credible when you say you’re sure.

How to improve your calibration

Calibration can be improved through training and experience. It begins with awareness of the problem and acceptance of the fact that you are probably susceptible to it. Here are a half-dozen ways to get better.

Test your calibration. ProjectionPoint has a test on their website that allows you to test your calibration. Simply seeing the results, if they are bad, will make you aware of the need to improve your calibration.

Separate fact from opinion. As Richard Feynman said, “The most important thing is not to fool yourself. And you’re the easiest person to fool.”

Keep track. Experience tends toreduce overconfidence and improve calibration, as long as you learn from that experience. It’s no accident that two of the best-calibrated professions are bookies and meteorologists. This is because they get rapid feedback on their decisions, and are held accountable for being wrong.

Be more foxy. As we saw previously, hedgehogs, who know one thing very well, tend to be less calibrated than foxes, who have more breadth of knowledge. He found that hedgehogs were not only wrong more often than foxes, but that they were less likely to recognize or admit that they were wrong when events did not match their predictions

Try not to make up your mind too quickly. Early judgments can serve as anchors, so that if you adjust your position in light of new information, you will probably not adjust as far. If you do, be on the lookout for confirmation bias, which is the general tendency to notice evidence that supports your view and be less apt to seek out or even notice contrary evidence. Follow Charles Darwin’s example:

“I had also, during many years, followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed to my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from the memory than favourable ones.  Owing to this habit, very few objections were raised against my views which I had not at least noticed and attempted to answer.”

Practice productive paranoia. When you’re very confident and it’s important, try extra hard to find holes in your idea. Individually, you can take the time to list reasons why you might be wrong. With colleagues, you can conduct a PreMortem: imagine that it is some future time and your idea has failed, and try to figure out all the ways it could have happened.[6]

If you follow these six practices, I’m 90% confident that your calibration will improve, and 75% confident that your personal credibility will also.


[1] Quoted in “Managing Overconfidence, by J. Edward Russo and Paul J.H. Schoemaker, Sloan Management Review, Winter 1992.

[2] Russo and Schoemaker 1992.

[3] Kruger, Justin; David Dunning (1999). “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”.

[4] Russo and Schoemaker say that public accountants are slightly underconfident.

[5] Experts Are More Persuasive When They’re Less Certain, Zakary Tormala, Harvard Business Review, March 2011.

[6] The term was coined (I believe) by Gary Klein, in his book, The Power of Intuition.

Listening Math

August 14th, 2014

Have you noticed that most of the training we need is about what we already know, but don’t do enough of? That’s certainly the case with listening skills. We all know how important listening is, but we all fall short of the level we’re capable of.

The interesting thing about listening skill is that we all can perform at top levels when we’re really, really motivated, so it’s not a question of knowing what to do; it’s about executing at the necessary level consistently.

If you think you’re a good listener, here’s some listening math to ponder:

50/25/10 A study showed that the average person remembers only 50% of what was said immediately after a 10 minute oral presentation. After 24 hours, the figure drops to 25%, and to 10% after a week. The first 50% is not a memory problem, it’s a listening problem, as anyone knows who hears a person’s name for the first time and “forgets” it ten seconds later.


500/125 In standard American spoken English, we speak at about 125 words per minute, but process words mentally at about 500 words per minute. I’m not sure how scientific the thinking speed measurement is, but it’s obvious that we can think much faster than others can speak. That’s why it’s so easy to get distracted while listening to someone else. We think we can listen and think about something else at the same time, but we’re actually rapidly switching back and forth – except when we forget to switch back.


583 The number of people killed when two 747s collided at Tenerife airport in 1977, caused primarily by a chain of listening errors and misunderstandings between pilots and air traffic control, and between pilots and copilots.


80/45 Those of us in business spend up to 80% of our waking hours in communicating almost half (45%) of our communication efforts consist of listening. There’s a lot of effectiveness left on the table if we’re not listening to our full potential.


43,8 The average length of a political sound bite on national news in 1968, and the average length in 1988. CBS News tried to counter this trend by mandating a minimum 30-second sound bite, but had to abandon the effort when people would not listen.


18 “That’s the average time it takes a doctor to interrupt you as you’re describing your symptoms. By that point, he/she has in mind what the answer is, and that answer is probably right about 80% of the time.”  Jerome Groopman, How Doctors Think


N = 1 One of the reasons that doctors (and possibly ourselves) tune out is that we think we’ve heard it all before. Maybe the other person needs advice with a problem, and it’s something we’re familiar with. Every conversation has a sample size of 1, because every person feels themselves unique, and maybe if we listen a little longer we may learn something new ourselves.


51+ This is a number I made up. 51+ represents the minimum level of responsibility you should take for your side of the conversation. When listening, don’t just passively take in the other’s words; meet them more than halfway and make sure you get their meaning. When talking, don’t assume that they got it just because you said it, make sure.



Festina Lente: Strategic Patience for Persuasive Communication

August 12th, 2014

Tortoise and rabbitThe first two articles in this series dealt with ways to benefit from exercising patience during persuasive conversations, on the time scale of seconds and minutes. Those time frames require tactical patience, and requires developing new habits, so that you can practice the skills without having to think about them in the moment. In this article, we turn our attention to the times between persuasive conversations, or the power of patience over days, weeks and months.  This is strategic patience, and it may require a change in your attitude and in your thinking processes, as you navigate the intricacies of relationships, decision processes, preparations and negotiations over time.

Relationships take time. Last week, I experienced an example of strategic impatience which will probably be familiar to you. I accepted a LinkedIn connection request from someone I did not know; I was flattered because she said she enjoyed reading my blog. Then, not 24 hours after connecting, she sent me an email aggressively trying to sell her company’s services. In this case, her haste was not only ineffective, it was counterproductive. (It wasn’t a total loss for me; I learned how easy it is to remove a connection on LinkedIn)

My aggressive new friend probably knows that trusting relationships take time, but she let her hunger for immediate results override her common sense. Maybe it’s not her fault: I’m sure she’s under pressure from a sales manager who wants sales now.

We all want a lot of strong relationships at the top of the relationship pyramid, where people who are able to help you take your phone calls. But the problem for most people is that they only pay attention to others in their network when they need something. If they haven’t patiently nurtured their network by staying in touch and by giving instead of taking, that will be too late.

Acceptance time. One of the reasons that you need patience is that persuasion implies change, and change usually takes time. You may think your idea is brilliant, but you’ve forgotten that it took you time to arrive at that conclusion; you’ve forgotten what it’s like not to know what you know, and not to believe what you believe. If it took you time to get to that point, why do you think you can short-cut that process for others?

They will need time to absorb the new information, to think about it, discuss it with others, and quite frankly just to get used to the idea. It’s hard to get someone to change their mind quickly or too far in one jump. You might need patience to nudge the needle, or otherwise your efforts may fail or even backfire. Pushing too far, too fast, will activate their inner two-year-old, and they will assert their independence by shoving back.

Patience is especially critical when you are involved in complex sales. It’s natural to want to rush your customers through their decision process as quickly as possible, especially when your solution will have such an impact on their business that they leave money on the table every day that they don’t implement it. So you step over potential allies to try to get right to the C-Level where decisions are made quicker, you give incentives (i.e. drop your price) to close this quarter, and you save time on your presentations by throwing in stock slides and stale information.

Nemawashi is the Japanese name for the patient preparation of an idea by talking to all the relevant people behind the scenes, long in advance of when you need the decision to be made. It may seem like it takes extra time, but besides improving your chances of gaining agreement, it actually saves time by shaping the conditions for the agreement you want and by enabling much more rapid and focused execution once the decision is taken.

Measure twice, cut once. It’s hard to avoid a cliché on this one, but if you haven’t got time to it right the first time, when will you have time to do it over? The patience to prepare and plan is probably the most obvious – and ignored – application of the concept there is.  Whether you’re preparing for a presentation, meeting, sales call or conversation, a few hours of preparation can save you weeks or months of work and worry. Whether it’s one more tough question you can anticipate, or double-checking your facts, or researching additional attendees, this can be some of the most profitable time you can spend.

Negotiation strength. In negotiations, the side in a hurry will lose, because there is no reason to accept a poor bargain until you have to. When time is on your side, impatience to get a deal closed is a form of unilateral disarmament. Unfortunately, this seems to be the dynamic at work in Afghanistan, as the US has announced a strict timetable for withdrawal by the end of 2014. As our adversaries like to say, “You have the watches, but we have the time.”

It takes patience to build patience. If this series on patience in persuasion has inspired you to work on strengthening your patience, please realize that it will take time. Patience at the tactical level is a skill, and proficiency takes time.

The only impatience you should have is to get started immediately.

Festina Lente: The Power of Patience Part 2

August 7th, 2014
Put your money on the guy in green

Put your money on the guy in green

In part 1 of this series, we began with an easy challenge: develop your patience muscle less than a second at a time. If you found that hard, stop reading right now; you’re not ready for this yet. Go back and work on your split-second patience – this will still be here when you’re ready.

But if you found the split seconds easy to handle, you’re ready to step up to the next challenge: developing patience on the minutes time scale. You’ll need it during conversations, where the challenge remains the same: how to improve effectiveness by curbing your impatience for immediate results.

At the seconds time scale, your weaknesses may work against you: caring more about your own needs and wanting to dominate the conversation. At the minutes level, ironically, it’s your strengths you have to guard against. Being passionate about your idea, adding value, solving problems, and planning ahead for the conversation are certainly good things, but they can also try your patience.

Let’s take one common example. Doctors would certainly seem to be models of wanting to help people, but a 1984 study found that the average time they listened to patients before interrupting was 18 seconds, and in 2001 another study found that the average was down to 12 seconds. What are the consequences? According to Jerome Groopman, author of How Doctors Think, by the time the doctor interrupts, he or she has already homed in on a diagnosis. About 80% of the time, the diagnosis is right, but the other 20% becomes less likely to be corrected either because the doctor has anchored on a diagnosis, or because the patient is unlikely to volunteer additional information.

In this example, the cost of misdiagnosis is clear and severe, but in daily persuasive communication the costs of impatience may be less clear yet still important. People may ignore good advice, customers may not buy the right product, and band-aids will get slapped over real problems.

The solution is not to abandon your strengths, but to recognize how they can make you impatient and then carefully guard against them.

  • If you like to solve problems, keep in mind that problems are often better solved at deeper levels than they are first described. The other person may not fully describe their problem, or may even need to work it out for themselves in the actual conversation. When you hear about something you know you can solve, hold off solving and ask a few more questions.
  • If you’re passionate about your idea, recognize that passion may be contagious, but it does require at least some incubation period in the other person. Don’t try to “close” them too fast.
  • If you want your advice to be heard, you can deliver it right away, but if you want your advice to be acted on, you have to find a way to make it the other person’s idea. That requires the patience to ask the right questions and let them work it out for themselves.
  • If you like to plan ahead for the conversation by preparing an agenda and list of questions, keep doing so, but allow for the conversation to take unexpected turns; use your prepared plan as a safety net, not a straitjacket.

If you can develop the habits of withholding judgment until you’ve heard the full story, of asking a few more questions before jumping to solution mode, and of keeping your attention on the process without looking ahead to results, the investment of patience at the minutes level will return you hours, days or months, in the form of more effective persuasion and more sustainable agreements. As the old saying goes, “you can speed up buying by slowing down selling”.

First the seconds, and then the minutes. The next section will address strategic patience at the level of days, weeks and months.