| Summary: | This paper summarizes
an empirical comparison of the accuracy of forecasts included in analysis
reports developed by professional intelligence analysts to comparable forecasts
in a prediction market that has broad participation from across an intelligence
community. To compare forecast accuracy, 99 event forecasts were extracted
from qualitative descriptions found in 41 analysis reports and posted on the
prediction market. Quantitative probabilities were imputed from the
qualitative forecasts by asking seasoned professional analysts, who did not
participate in the prediction market, to read the reports and to infer a
quantitative probability based on what was written. These readers were also
asked to provide their personal probabilities before and after reading the
reports. There were two statistically significant results of particular
interest. First, the primary result is that the prediction market forecasts
were more accurate than the analysis reports. On average prediction market
probabilities were 0.114 closer to ground truth than the analysis report
probabilities. Second, in cases where analysts (readers) updated their
personal probabilities in a direction opposite to what the reports implied,
analysts tended to update their probabilities in the correct direction. This
occurred even though, on average, reading the reports did not make readers more
accurate.
|
|---|
