Monitoring Operational Selection Systems Through Frequency Counts: An Application of Bayesian Predictive Inference

摘要

Military selection systems are put into place in an attempt to reduce high attrition rates in training. Typically, though, after the selection system has experienced some operational use, the rejection rates and attrition rates are observed to fluctuate over time. The question then naturally arises, Do these fluctuations represent some substantial change in the real world that should be investigated, or do they merely reflect the statistical vagaries seen in any small sample size. In this technical memorandum the question just posed is addressed quantitatively by standard Bayesian statistical techniques. A predictive inference can be made about future frequency counts based on the empirical data of the past frequency counts. This kind of analysis is helpful whenever there is concern that something fundamental might have caused the rejection rates and/or attrition rates of the selection system to change. For example, the rejection rate of a given selection system is suspected of having dramatically changed over the past few years. Before we can attempt to track down the cause of this alleged rate increase, we must first establish that the increase in frequency counts is not simply due to statistical fluctuations inherent in small sample sizes. It is quite easy to be misled into thinking that a 'trend' based on relatively small numbers portends a significant change in the underlying rate parameter. The techniques detailed in this report will help researchers disentangle sample size fluctuations from external perturbations to the rate parameter. In the case of any selection system, these techniques can be employed to determine whether there is a justification for investigating such fundamental changes as a shift in the hardware configuration, a change in the ability levels of the candidate population, changes to the training regime, or changes in the validity of the current prediction algorithm.