Using the conditional grade-of-membership model to assess judgment accuracy bruce cooil and sajeev varki

摘要

Consider the case where J instruments are used to clssify each of I objects relative to K nominal categories. The conditional grade-of-membership (GoM) model provides a method of estimating the classification probabilities of each instrument (or "judge") when the objects being classified consist of both pure types that lie exclusively in one of K nominal categories, and mixtures that lie in more than one category. Classification probabilities are identifiable whenever the sample of GoM vectors includes pure types from each category. When additional, relatively mild, assumptions are made about judgment accuracy, the identifiable correct classification probabilities are the greatest lower bounds among all solutions that might correspond to the observed multinomial process, even when the unobserved GoM vectors do not include pure types from each category. Estimation using the conditional GoM model is illustrated on a simulated data set. Further simulations show that the estimates of the classification probabilities are relatively accurate, even when the sample contains only a small percentage of approximately pure objects.