Title: Evaluation and Testing of Rare Event Predictors

Dragos Margineantu, The Boeing Company (Boeing Research & Technology) 

Abstract

The last decade has witnessed several successes of deployed systems that
have learning and inference components. One of the hardest tasks lying
ahead for learning systems is the prediction and detection of high-risk
low-probability events. Small counts (or zero counts) are notoriously
difficult to address especially because those rare observations or
events have high costs associated with missing them. An even more
challenging task is the testing and evaluation of predictors and
detectors that claim rare event and anomaly detection capabilities, and
no statistical, inference, or learning magic will solve the problem in
the absence of any knowledge on how the observed values in the data were
generated.
Our recent research has focused on a set of statistical tests for
estimating confidence intervals for the risk of high-cost classification
decisions and a Bayesian extension to these test, that allows the
incorporation of prior knowledge on characteristics of the task and of
the detection model - for estimating the distribution of the risk of
decisions. Several experimental results on the tests and on employing
several structural models for the probability of observations show the
benefits of a Bayesian approach to the testing of these rare event
predictors, but the challenges lying ahead are many.