labThe forecast register. How a forecast is checked by its outcome
A lab forecast is recorded before its date arrives, and after the date the outcome is recorded, whatever it is. The accumulated outcomes show how well the words of the scale match what happens.
A forecast is checked by what happens afterwards. So every forecast the lab makes goes into the register before its date arrives, as the lab's methodology requires.
What is recorded
- Date. When the forecast was made.
- Wording. What exactly will or will not happen.
- Probability on the scale. One of the seven words of probability.
- Horizon. Near, strategic or far.
- Author. Who made the forecast and answers for it.
- Date of the check. The date after which the outcome is known.
- Outcome. Recorded after the date: it came true or it did not.
A wording that can be checked
The main difficulty of the register is the wording. "The situation in the industry will deteriorate" cannot be checked: after the date any outcome can be read in the forecast's favour. A wording that can be checked names the event, the party it happens to and the date: "by 31 December of such-and-such a year the regulator will take such-and-such a decision".
If a forecast cannot be written that way, it is not recorded as a forecast. It stays as reasoning in the Significance block of the note.
The outcome is recorded whatever it is
Memory keeps the hits and forgets the misses. A register into which outcomes are entered selectively gives the same picture as memory, only on paper. So after the date the outcome is always recorded, and a forecast that did not come true stays in the register next to one that did.
A disputed outcome is settled by a reviewer from outside the team, and the decision is recorded too.
Calibration
Once enough outcomes have accumulated, it will be possible to see for each word of the scale what share of the forecasts using it came true. If the share is below the word's range, the analysts overrate their grounds. If it is above, they underrate them. This comparison is called calibration.
Measuring the accuracy of probabilistic forecasts is an old skill. Meteorologists have used the score proposed by Glenn Brier since 1950. Philip Tetlock, in his book "Expert Political Judgment" (2005), and the Good Judgment Project he led with Barbara Mellers showed that the accuracy of political forecasts can be measured too, and that it improves when forecasters see their outcomes. The project won the forecasting tournament that IARPA, the research agency of the US intelligence community, ran from 2011 to 2015.
With few forecasts, calibration says nothing. While outcomes are few, the lab says so and draws no conclusions about calibration.
The debrief after the event
The register answers the question "did it come true". The debrief after the event answers the question "why". It records what the forecast got right, what it missed and which assumptions proved wrong. Its findings feed back into the setting of new tasks: this is the last step of the analytical cycle, "Knowledge archive and calibration".