Estimating the Probability of Being the Best System: A Generalized Method and Nonparametric Hypothesis Test.

摘要

This thesis provides two new approaches for comparing competing systems. Instead of making comparisons based on long run averages or mean performance, the first paper presents a generalized method for calculating the probability that a single system is the best among all systems in a single trial. Unlike current empirical methods, the generalized method calculates the exact multinomial probability that a single system is best among competing systems. The ability to avoid time consuming empirical estimate techniques could potentially result in significant savings in both time and money when comparing alternate systems. A Monte Carlo simulation is conducted comparing the empirical probability estimates of the generalized integral method, calculated using a bootstrapping procedure and density estimation technique, with those of two related estimation techniques, Procedure BEM (Bechhofer, Elmaghraby, and Morse) and Procedure AVC (All Vector Comparisons). All test cases show comparable performance in empirical estimation accuracy of the generalized integral method with that of the current methods analyzed. The second paper proposes the use of a distribution-free ordered comparisons procedure to test whether one population is stochastically larger (or smaller) than the other. This procedure is suggested as a useful alternative to the wellknown Wilcoxon rank sum and Mann-Whitney tests. A Monte Carlo study is conducted, comparing the methods based on simulated power and type I error. All test cases show a marked improvement in performance of the ordered comparisons test versus the Wilcoxon rank sum test, when testing for stochastic dominance. A table of critical values is presented and a large sample approximation is given.