labAnalysis of competing hypotheses. Contradictions decide
Richards Heuer's method weighs every explanation against the data at once and counts the evidence against each. The lab uses it for the step of the cycle at which no fewer than two hypotheses are weighed.
Richards Heuer described the analysis of competing hypotheses in his book "Psychology of Intelligence Analysis", published by the CIA in 1999. The lab uses it most often at the "Analysis and alternative hypotheses" step of its analytical cycle: no fewer than two explanations are weighed against the data.
Why this method
The usual path goes like this: the analyst picks the most plausible explanation and looks for what supports it. Support is almost always found, because most evidence agrees with several explanations at once. That is how confirming the hypothesis happens, the first of the logical errors we wrote about.
Heuer's method turns the order around. All the hypotheses are considered at the same time, and for each one the analyst looks for what contradicts it.
Eight steps
- Hypotheses. Gather every explanation worth considering. It helps when they come from people with different views.
- Evidence. List the facts, arguments and assumptions, and also the absence of what should have been there under one hypothesis or another.
- Matrix. Hypotheses across, evidence down. Each cell records whether the item agrees with the hypothesis.
- Refining. Reword the hypotheses and remove the evidence that does not tell them apart.
- Tentative conclusions. Assess the hypotheses by trying to refute them, not to support them.
- Sensitivity. Check what happens to the conclusion if one or two key items prove false.
- Report. State the relative likelihood of every hypothesis, not only of the leading one.
- Milestones. Name the signs that will show events taking another course.
How to read the matrix
The cover of this post has three hypotheses and five items of evidence. The first item agrees with all three hypotheses. To ordinary reasoning it looks weighty; to the method it is empty, because it does not help choose between the hypotheses.
Contradictions decide. The first hypothesis has two, the second three, the third one. The third hypothesis holds up better than the others, and that is still not a conclusion: it is the candidate worth checking further.
What the method does not do
The method does not compute a probability. The count of contradictions shows where to look and does not measure likelihood: items of evidence are not equal, and one strong contradiction can remove a hypothesis entirely.
The method is only as good as its set of hypotheses. If the true explanation is not in the matrix, the method picks the best of the wrong ones. That is why the first step matters most, and why the hypotheses are gathered by people with different views.
The method also depends on the quality of the evidence. Every item carries the grade of its source, and a contradiction from a source graded E5 weighs less than one from a source graded B2.
How the result reaches the note
The matrix stays with the working materials, and the note receives the conclusion drawn from it. The Alternatives block names no fewer than two explanations and says why they are weaker than the main one. In the Forecast block the probability is written as a word of the scale. The milestones from the eighth step go into the Indicators block.
A reviewer from outside the team checks that the alternatives in the note are real. A hypothesis invented only to be refuted does not count as an alternative.