Since the early 1900's, the intraclass correlation coefficient (ICC) has been used to quantify the level of agreement among different assessments on the same object. By comparing the level of variability that exists within subjects to the overall error, a measure of the agreement among the different assessments can be calculated. Historically, this has been performed using subject as the only random effect. However, there are many cases where other nested effects, such as site, should be controlled for when calculating the ICC to determine the chance corrected agreement adjusted for other nested factors. We will present a unified framework to estimate both the two-level and three-level ICC for both binomial and multinomial outcomes. In addition, the corresponding standard errors and confidence intervals for both ICC measurements will be displayed. Finally, an example of the effect that controlling for site can have on ICC measures will be presented for subjects nested within genotyping plates comparing genetically determined race to patient reported race.;In addition, when determining agreement on a multinomial response, the question of homogeneity of agreement of individual categories within the multinomial response is raised. One such scenario is the GO project at the University of Pennsylvania where subjects ages 8-21 were asked to rate a series of actors' faces as happy, sad, angry, fearful or neutral. Methods exist to quantify overall agreement among the five responses, but only if the ICCs for each item-wise response are homogeneous. We will present a method to determine homogeneity of ICCs of the item-wise responses across a multinomial outcome and provide simulation results that demonstrate strong control of the type I error rate. This method will subsequently be extended to verify the assumptions of homogeneity of ICCs in the multinomial nested-level model to determine if the overall nested-level ICC is sufficient to describe the nested-level agreement.
展开▼
