SME Differences and Their Importance on the Content Validity of Job Analysis Ratings

Abstract

Subject matter expert ratings differences in job analyses were studied to see how these differences may impact the content validity of selection processes. Due to the increased use of content validation strategies in employment testing, it is important to examine any potential rating differences that may arise from a job analysis due to using different types of raters and raters with varying levels of experience. This study examined rating differences among incumbents and supervisors and incumbents with low versus high levels of experience for both the Entry Firefighter and Fire Captain ranks. The findings showed that there were statistically significant differences found in the Fire Captain sample for incumbents and supervisors in the areas of Communication/Documentation, Management/Supervision, Emergency Services Delivery, and Inspections and Investigations and for different levels of experience in the areas of Fire Station Administration and Maintenance, Communication/Documentation, and Management/Supervision. There were no significant differences found in the Entry Firefighter sample.