Of the many reasons I have a soft spot for the intelligence community, one is this: imagine how hard it is to produce a document that is both precise and readable.
That’s the same challenge I face on the blog, but the IC can’t include gratuituous jokes of a certain kind just to liven things up. (It also helps that you, dear readership, tend to be a couple of standard deviations above the mean when it comes to technical facility.)
I was amused by a chart that our intern — the talented Alex Kahan, who New America shares with Brookings — found in an old China DIE (PRC Strategic Forces: How Much is Enough? DIE FE 7-74, 3 December 1974 . Thanks to Bill Burr, who actually provided us with the copy.)
Look familiar? Here is a chart from the 2007 NIE, Iran: Nuclear Intentions and Capabilities
It is fascinating to see that earlier generations faced the same challenges and how little the basic structure of the language has changed. You could use the 1974 chart to decipher the 2007 NIE without too much confusion.
The one change, of course, has been the post-politicization effort to make analysts express confidence levels, which I don’t think are independent of the estimative language.
What does it mean, for example, to have “low confidence that Iran probably has imported at least some weapons-usable fissile material …” How can evidence in which you have little confidence or no confidence, that lead you to conclude that something is not merely plausible, but probable?
Maybe its just my pet peeve, but such language seems mostly to serve as a permission slip for the Doug Feith’s of the world to disregard the estimate.
Having written a fair amount of NIE and similar language myself in the mid ’70s through mid ’80s, these definitional matters are all too familiar. And, though it happened after my days, the recent trend to ever more obscure formulations does seem likely to cause readers to set the intelligence estimates aside.
Ah, a posting after my own heart.
Here’s some value added.
Words of Estimative Probability
Sherman Kent is the father of the national estimates process. Reading an estimate is not like reading a novel or technical journal; it is its own type of work that requires a different sort of attention to detail.
As an IC analyst, the flaw you point out at the end of your post really bugs me: it should not be possible to say “we have low confidence XX probably will happen.” We should say either “XX probably will happen” or “We have low confidence XX will occur.” We should never mix the two.
Otherwise, you open yourself up to crazy situations where I can credibly say: “I assess with high confidence North Korea might have nuclear weapons.” Clearly this means nothing.
I’ve pointed this out several times to the folks that right these boxes, but to no avail. It took so long to negotiate the “blue box” published in the 2007 NIE (it actually was developed a few years earlier) that no one wants to re-do it for the whole community, no matter what the flaws. Oh well.
Jeffrey, try a little harder to be less irritating. And burn an effigy of Doug if you really have to.
Estimative language (likely, unlikely, probably, etc.) conveys the analyst’s probabilistic sense that the event under consideration will or will not happen (e.g. “It is likely that Iran halted their nuclear program in 2003”).
Confidence conveys the analysts’ sense of how likely it is that they are wrong in that estimate.
The easiest way to imagine this is to think about two different analysts who both say, in response to a policymaker’s question, “It is likely that X will occur.” The analysts do not know one another and in fact, for purposes of this example, work for entirely different countries.
The first analyst is an expert in the field, had lots of time, excellent sources and used structured analytic methods. The second analyst was new on the job, had poor sources and was asked for an off the cuff estimate.
Both analysts made the same estimate but the first, arguably, has reason for a much higher level of analytic confidence.
Used properly (and admittedly, this is rare), this type of language leads to an estimate that is more, rather than less, difficult to ignore. Imagine if the famous Aug 6 2001 PDB titled “Bin Ladin Determined To Strike US” had read “Bin Ladin Highly Likely To Strike US. Confidence: High” (or even better “Bin Ladin Highly Likely To Strike US In Next 6 Months. Confidence: High”). I would suggest that either of the last two formulations would not be easy to dismiss.
This general kind of thing was mentioned in last April’s briefing on the al Kibar reactor. See the section starting with
SENIOR INTELLIGENCE OFFICIAL 1: I might add something that would helpful here. These are lessons learned that are – that came out of previous experience about how to put more rigor into our process. So there’s a difference between evidence and an assessment.
“burn an effigy of Doug [Feith] if you really have to.”
yeah, poor old Doug. All he did was cherrypick, distort and manipulate intelligence to help stampede the nation into an unnecessary war that cost thousands of American lives. Why pick on him?
You asked another interesting question: “How can evidence in which you have little confidence or no confidence, lead you to conclude that something is not merely plausible, but probable?”
The answer is, in my opinion, that the intel analyst often doesn’t have a choice.
Imagine that the analyst has been asked a question and has come to the conclusion that X is likely albeit with low confidence. Lets also imagine that time is somewhat of the essence. There is a summit or a crisis or some other cause of time pressure (in other words, business as usual…).
The analyst in this situation has three options. First, the analyst can say that X is likely without reference to confidence. This is clearly misleading. Policymakers will almost certainly assume at least moderate confidence and would likely be upset, when things go south, to find out that the analyst had omitted the tiny detail of their lack of confidence in their own estimate.
The second option is to refuse to make an estimate — to beg off because the analyst’s confidence is so low. This has the unwelcome effect of making the policymaker the analyst (particularly when time is of the essence). Policymakers are almost always tasked with doing something (even if it is only making sure that nothing happens). If the analyst refuses to provide an estimate, then the policymaker still has to do something but has been left to figure it out on their own. As a result, the policymaker’s impression of intelligence, once again, is unlikely to be very favorable.
The third option and the one we teach as the “best practice” at Mercyhurst is to provide an estimate but to indicate the analyst’s level of confidence in that estimate as well. It is far from a perfect solution, of course, but at least the policymaker has the benefit of the analyst’s properly caveated best estimate.
This sort of thing is very similar — at least metaphorically — to the way data is presented in scientific publications: You present the value of a quantity (here, likelihood) produced by measurement, calculation, or whatever. Then you give the uncertainty (often presented graphically as the somewhat misnamed “error bars”), which correspond to what we’re calling “confidence.”