Estimation of r and k from ANOVA tables

When we come to analyse the data it is usually appropriate to carry out an analysis of variance ( ANOVA). For the first design we carry out a one-way ANOVA (X by subject) and for the second we perform a two-way ANOVA (X by rater and subject). In the latter case we assume that there is no subject by rater interaction and accordingly constrain the corresponding sum of squares to be zero. We assume that readers are reasonably familiar with an analysis of variance table. Each subject has been assessed by, say, k raters. The one-way ANOVA yields a mean square for between-subjects variation (BMS) and a mean square for within-subjects variation (WMS). WMS is an estimate of Var(E) in eqn (1). Therefore the square root of WMS provides an estimate of the instrument's standard error of measurement. The corresponding estimate of r is given by where rX is used to represent the estimate of rX rather than the true, but unknown, value. In the case of k = 2, r becomes

In the slightly more complex two-way ANOVA, the ANOVA table provides values of mean squares for subjects or patients (PMS), raters (RMS), and error (EMS). We shall not concentrate on the details of estimation of the components of eqn (3) (see Fleiss(2) or Streiner and Norman(3)) but simply note that rXa is estimated by where n is the number of subjects (patients) in the study.

In reporting the results of a reliability study, it is important that investigators give some idea of the precision of their estimates of reliability, for example by giving an appropriate standard error or, even better, an appropriate confidence interval. The subject is beyond the scope of this chapter, however, and the interested reader is referred to Fleiss(2) or Dunn(!) for further illumination.

Finally, what about qualitative measures? We shall not discuss the estimation and interpretation of k in any detail here but simply point out that for a binary (yes/no) measure one can also carry out a two-way analysis of variance (but ignore any significance tests since they are not valid for binary data) and estimate rXa as above. In large samples rXa is equivalent to k3. A corollary of this is that k is another form of reliability coefficient and, like any of the reliability coefficients described above, will vary from one population to another (i.e. it is dependent on the prevalence of the symptom or characteristic being assessed).

Hearing Aids Inside Out

Hearing Aids Inside Out

Have you recently experienced hearing loss? Most probably you need hearing aids, but don't know much about them. To learn everything you need to know about hearing aids, read the eBook, Hearing Aids Inside Out. The book comprises 113 pages of excellent content utterly free of technical jargon, written in simple language, and in a flowing style that can easily be read and understood by all.

Get My Free Ebook


Post a comment