Training an AI-Based Chatbot to Infer Workplace Competencies

Abstract

Competency modeling is used within a wide range of contexts within organizations such as selection, development, and promotion decisions to name a few. Assessing relevant competencies comes with its challenges; designing and implementing these models is time consuming and expensive. The purpose of the present study is to design and train AI models to infer competencies through a text-based chat interaction with an AI chatbot. The models were trained using text responses to behaviorally based competency interview questions as predictors and raters used behaviorally anchored rating scales (BARS) to score the responses and these scores were used as the criteria (ground truth). The training sample consisted of 297 full-time employees and the test sample included 210 college students. The psychometric properties of machine-inferred competency scores were examined including reliability (split-half), convergent and discriminant validity, criterion-related validity, and generalizability. The results showed promising evidence for split-half reliability, convergent validity, and generalization of the model, but less promising results for discriminant validity and criterion validity.