Transient Error and Coefficient Alpha: A Call for Cautious Practice when Applying and Interpreting Alpha in Personnel Selection Settings

Abstract

Reliability is an integral component in determining the worth of results from any measure. There are a number of estimates used to represent reliability, which vary in terms of the sources of error addressed, underlying assumptions about the data, statistical theory, and formulae applied; but in the areas of personnel selection research and practice coefficient alpha (also known as Cronbach’s alpha and simply referred to as alpha below) is by far the most widely reported. Alpha’s popularity is mostly due to two commonly accepted properties of the statistic. First, it is a measure of internal consistency so, unlike test-retest or inter-rater estimates of reliability, data used to generate the coefficient can be gathered from a single test administration. Second, alpha is generally considered a conservative statistic, or more specifically the coefficient is thought to estimate reliability’s lower boundary. While the convenience of calculating alpha is inarguable, the assumption that it is an underestimate of reliability is not always warranted. It has recently been demonstrated that transient errors, a source of variability not often assessed, can actually inflate the alpha coefficient and cause reliability to be overestimated. The current study investigates the effect of transient error and echoes the call to present additional diagnostic information that has recently been introduced to the professional literature, such as Alpha’s Standard Error (ASE; Duhacheck & Iacobucci, 2004). The benefits of calculating confidence intervals surrounding the alpha coefficient and substituting the upper and lower boundaries in place of the point estimate when performing a variety of calculations used in personnel selection practice and research are demonstrated. The present research calls for greater caution interpreting and applying reliability estimates in this high stakes setting.