ROBERT SIEGEL, HOST:
In making a decision, does it help to hear that there's a 40 percent likelihood that a job you are thinking about will lead to a promotion? Or it's a 70 percent likelihood you'll be able to park within a mile of the movie theater? Well, this week we're considering probabilities and how useful or, perhaps, misleading they can be. Yesterday, we heard about weather forecasts and it turned out that a 20 percent chance of rain is a more mysterious concept than we had thought. Well, today, probability at an agency even more mysterious than the National Weather Service - the Central Intelligence Agency.
KENNETH POLLACK: There was a real injunction that no one should ever use numbers to explain probability.
SIEGEL: That's Kenneth Pollack who used to be a military analyst at the CIA, where he now teaches intelligence analysis. Pollack says CIA analysts are told if you are asked what the chances of something happening, use words.
POLLACK: Almost certainly or highly likely or likely or very unlikely.
SIEGEL: What's the problem with numbers?
POLLACK: Assigning numerical probability suggests a much greater degree of certainty than you ever want to convey to a policymaker. What we are doing is inherently difficult. Some might even say it's impossible. We're trying to protect the future. And, you know, saying to someone that there's a 67 percent chance that this is going to happen, that sounds really precise. And that makes it seem like we really know what's going to happen. And the truth is that we really don't.
SIEGEL: So Ken Pollack was surprised by the accounts of one especially high-profile event, in which CIA analysts and others in the intelligence agencies used numbers, very specific numbers, to express probabilities. Let's go back to May 1, 2011.
(SOUNDBITE OF ARCHIVED RECORDING)
PRESIDENT BARACK OBAMA: Good evening. Tonight, I can report to the American people and to the world that the United States has conducted an operation that killed Osama bin Laden, the leader of al-Qaida.
SIEGEL: According to writers who investigated the decision to send Navy SEALs to the compound in Abad Abad, Pakistan, the estimates of certainty that bin Laden was the man they'd spotted in the compound covered a range. Both Peter Bergen and Mark Bowden wrote separately that the lead analyst at the CIA put his confidence level at 90 percent or 95 percent. The deputy director of the CIA was at 60 percent. Other analysts, they say, settled on a much lower number - 40 percent. A week after the raid, President Obama went on "60 Minutes" and acknowledged that it'd been a tough decision, because the evidence was circumstantial.
(SOUNDBITE OF TV SHOW, "60 MINUTES")
OBAMA: At the end of the day, this was still a 55-45 situation. I mean, we could not say definitively that bin Laden was there.
SIEGEL: Other accounts have Obama concluding it was basically 50-50 - a coin flip. Reading these accounts, Jeff Friedman wondered about how the intelligence analysts had presented their views and whether they could've done it better. Friedman researches national security decision-making as a postdoctoral fellow at Dartmouth. He's the co-author of a paper examining the decision to stage the raid on Abad Abad. And he says this - people's beliefs about how probable something is are subjective and how much we trust those people is also subjective. But Jeff Friedman argues you can still picture the subjective judgments effectively, especially if you remember where people making those judgments are coming from, as in the bin Laden case.
JEFF FRIEDMAN: The low estimate of 30 or 40 percent likelihood that bin Laden was at Abad Abad was issued by a CIA red team. And the red team, which is a common institutional practice, is to be skeptical on purpose, right? They were meant to poke holes in the intelligence. So they, of course, came out the lowest estimate - 30 or 40 percent. The deputy director of Intelligence, Michael Morell, says that he assessed that there was a 60 percent chance that bin Laden would be at Abad Abad. And he tells President Obama explicitly that he's lowballing that assessment a bit because he remembers this assessment of Iraq's weapons of mass destruction. And he knows how easy it is to sort of connect these dots and to be overoptimistic in Intelligence. What does President Obama do, in the end? He says it's a coin flip. We're going with 50-50. So he ends up implicitly giving the most weight to the estimates at the bottom that are the least credible.
SIEGEL: So what would've been, to your way of thinking, the clearest way in which analysts could have put, to President Obama, in that case, their estimate of the likelihood that Osama bin Laden was in that compound in Abad Abad?
FRIEDMAN: Here's what I think would've been the clearest way to present that information. To present a clear likelihood, which is some kind of weighted average of people's views - to also say clearly how much confidence people had in this discussion. So for instance you could say, we believe the chances bin Laden's in Abad Abad is 65 percent. But people disagree about this and personal estimates range from 40 to 95 percent. And then last, to say how much this is about to change. So for instance, the United States has been tracking this particular compound for about nine months. There have been something on the order of 40 intelligence reviews. This is probably the best we're going to do. Now, I think if you say those three things - likelihood, confidence and responsiveness - I think that would still be subjective and you could still debate that. But I think that would have been, at least, a much more structured way to present that situation to the president than what he received.
SIEGEL: But wouldn't my next question as president be, who says 45 percent?
FRIEDMAN: It would and it should, by the way. When you present information to the president, you want to present information that allows him to ask the right questions. Who thinks what and why? Why do people disagree? That's all important. But the point is, in this case, they did that. I mean, the story is that the president met with his advisers and kept going around the table and asking people to state their views and to resolve them. And even once these views were stated, even once everybody had had a chance to talk about why they disagreed, the question was then, OK, what do we do now? And I think that stumbling block is something that I think is a lot more tractable than many people believe.
SIEGEL: But for all of Jeff Friedman's confidence in quantified expressions of probability, former CIA analyst, Ken Pollack, adds this caution. Sure, you can say something is 95 percent probable and the opposite is just 5 percent, but...
POLLACK: You wait long enough and you will get the 5 percent probability happening.
SIEGEL: What's a 5 percent event that's happened - an event of consequence that you can think of?
POLLACK: The United States winning the Battle of Midway. It is stunning to me that we won that battle and won it as decisively as we did. And yet, it was arguably the most important victory of the Pacific War.
SIEGEL: Sometimes long shots win and slam dunks bounce off the rim. Most of us aren't called upon to forecast military or political events. But deciding to take a prescription drug or to have surgery is common. We'll hear about probability at the doctor's office tomorrow and Friday. Transcript provided by NPR, Copyright NPR.
