Predicting Vocational Interests through an AI-based Chatbot

Abstract

Emerging research has provided compelling evidence that vocational interests serve as robust predictors of various life and work-related outcomes. However, vocational interests assessment did not see much innovation since its inception and still relies heavily on the traditional self-report method. The current study investigated the feasibility of assessing vocational interests using an AI chatbot, where user-generated text inputs were gathered and analyzed utilizing machine learning algorithms. I examined the psychometric properties of machine-inferred interests scores, namely their (1) internal consistency, (2) split-half reliability, (3) factorial validity, (4) convergent validity and discriminant validity, and (5) criterion-related and incremental validity. Participants in the training sample (n = 229) were recruited from the SONA system at a large public university in Florida, while participants in the independent test sample (n = 88) were recruited from the Business School of a large public Southeastern university. Participants first interacted with an AI chatbot for approximately 45 minutes and then completed an online vocational interests measure on Qualtrics. They were then asked to report their college GPA, ACT or/and SAT scores, and high school GPA, and completed surveys that measured major satisfaction, withdrawal intentions, absenteeism, health complaints, and subjective physical health. The results provided encouraging evidence for machine-inferred vocational interests scores in terms of internal consistency, split-half reliability, factorial validity, and convergent and discriminant validity, but failed to support the criterion-related and incremental validity.