Estimators of different delta coefficients based on the unbiased estimator of the expected proportions of agreements

Abstract

To measure the degree of agreement between two observers that independently classify $n$ subjects within $K$ categories, it is common to use different kappa type coefficients, the most common of which is the $\kappa_C$ coefficient (Cohen's kappa). As $\kappa_C$ has some weaknesses -such as its poor performance with highly unbalanced marginal distributions-, the $\Delta$ coefficient is sometimes used, based on the $delta$ response model. This model allows us to obtain other parameters like: (a) the $\alpha_i$ contribution of each $i$ category to the value of the global agreement $\Delta=\sum \alpha_i$; and (b) the consistency $\mathcal{S}_i$ in the category $i$ (degree of agreement in the category $i$), a more appropriate parameter than the kappa value obtained by collapsing the data into the category $i$. It has recently been shown that the classic estimator $\hat{\kappa}_C$ underestimates $\kappa_C$, having obtained a new estimator $\hat{\kappa}_{CU}$ which is less biased. This article demonstrates that something similar happens to the known estimators $\hat{\Delta}$, $\hat{\alpha}_i$, and $\hat{\mathcal{S}}_i$ of $\Delta$, $\alpha_i$ and $\mathcal{S}_i$ (respectively), proposes new and less biased estimators $\hat{\Delta}_U$, $\hat{\alpha}_{iU}$, and $\hat{\mathcal{S}}_{iU}$, determines their variances, analyses the behaviour of all estimators, and concludes that the new estimators should be used when $n$ or $K$ are small (at least when $n\leq 50$ or $K\leq 3$). Additionally, the case where one of the raters is a gold standard is contemplated, in which situation two new parameters arise: the $conformity$ (the rater's capability to recognize a subject in the category $i$) and the $predictivity$ (the reliability of a response $i$ by the rater).