labThe source and the message. Two grades instead of one
Every message in the lab's work gets two marks. A letter grades the reliability of the source, a digit the credibility of the message itself, and neither is derived from the other.
A message in the lab's work gets two marks. A letter from A to F says how reliable the source is. A digit from 1 to 6 says how credible the message itself is. The scale is old: it is known as the Admiralty system, and intelligence services and police forces in many countries use it.
The full scale is in the lab's methodology. This post explains why there are two grades and how to set them on open sources.
Why two grades
The reliability of a source comes from its past: how often it has been right and whether it has access to what it reports on. The credibility of a message comes from the present: whether what it says agrees with what is already established, and whether anyone else confirms it.
A single grade carries the reputation of the source over to every sentence it produces. A reliable register can be wrong in a single entry. An anonymous channel sometimes passes on something true. Two marks make it possible to say so plainly.
The letter: reliability of the source
- A — completely reliable. A long history of correct reports and direct access to the subject.
- B — usually reliable. Mistakes have happened, but rarely.
- C — fairly reliable. More correct reports than wrong ones.
- D — usually unreliable. More wrong reports than correct ones.
- E — unreliable. The source's history speaks against it.
- F — reliability cannot be judged. There is no history.
On open sources the letter depends on what the source records itself. A state register is reliable for the date a company was registered: that entry is made by the register. For information it records from an applicant's words, it is as reliable as the applicant.
A new source with no history gets F. Giving it C "to be safe" passes off ignorance as a grade.
The digit: credibility of the message
- 1 — confirmed by independent sources.
- 2 — probably true. It agrees with what is established, but there is no independent confirmation.
- 3 — possibly true. It agrees in part.
- 4 — doubtful. It leans against what is established.
- 5 — improbable. It contradicts what is established.
- 6 — credibility cannot be judged. There is nothing to check it against.
A 1 needs independent sources. Five publications reprinting one report give one source. Counting them as five is one of the eight logical errors we wrote about earlier.
How a mark reads
A mark is the letter and the digit together. B2 means the source is usually reliable and the message is probably true. C3 is the most common mark, and there is nothing wrong with that: open sources are seldom better.
F6 means that grading is impossible. It says nothing about the message being false, and a text does not render such a mark as "unreliable information".
E1 also happens: an unreliable source passed on a message that was confirmed independently. The message is confirmed, and the source stays unreliable for its next message.
How a mark reaches the note
Every fact in a note carries the mark of its source. The analyst derives the confidence of a conclusion (high, medium or low) from the marks taken together. No average mark is computed: letters and digits set an order, and an average of them measures nothing.
A reviewer from outside the team checks that the confidence of a conclusion is no higher than the marks under it allow. This is part of the second gate of the analytical cycle.
Common mistakes
- Grading by fame. A large outlet gets an A even when it retells other people's words about the subject.
- Grading by default. Every source gets C3, and the scale stops telling anything apart.
- Promotion after one match. A source that was right once gets a B.
- A mark without a date. Reliability changes over time, so a mark is set as of a date and reviewed.