| dc.description.abstract | A new mobile application (app) was developed by the Military REACH team. It delivered research-backed professional development content to practicing human and family service providers over the course of eight weeks. The app aimed to enhance their ability to provide services that meet the unique needs of Service member, Veteran, and military family clients in family life education and clinical settings. To this end, the app delivered summaries of recent military family research which highlight findings, methods, and suggestions of how to practically apply findings.
To examine the efficacy of this app, evaluation was conducted in two phases, guided by a real-world evaluation approach (Bamberger, 2006; 2012; 2019). In Phase I, human and family service providers were randomly assigned to the initial intervention group or delayed intervention group and both groups completed online pretest surveys (Time 1). The delayed group functioned as a control comparison during Phase I as they were kept on a waitlist for eight weeks while the initial intervention group utilized the app. At the end of the first eight weeks (Time 2), the initial group completed a posttest, and the delayed group repeated their pretest (retest). In Phase II, delayed intervention participants were also given access to the app for eight weeks before completing a posttest, and during this time frame, the initial group completed a “cool down” period without app use before completing a follow up survey. This phase was used to further support the efficacy of the app by examining initial effects within the delayed intervention group and sustained effects within the initial group.
This dissertation examined the efficacy of the app focusing on the first two levels of Kirkpatrick’s (2016) multilevel assessment approach for program evaluation: 1) reaction and satisfaction, and 2) knowledge and learning. Participants were 258 human and family service professionals. The majority were women (82.2%) and identified as racially White (76.0%). Participants were, on average, 36.47 years old (SD = 10.59). Roughly half were students (45.3%). Most held a master’s degree (63.6%); 27.9% had a 4-year degree. Using 2:1 allocation ratio, 162 participants were randomly assigned to the initial intervention protocol and 96 were randomly assigned to the waitlist-delayed intervention protocol. There were no significant differences between the groups in terms of sex, race, ethnicity makeup, or age distribution.
Regarding Kirkpatrick’s first level of program evaluation (reaction and satisfaction), participants rated the app as engaging and helpful for their professional development. Regarding Kirkpatrick’s second level of program evaluation (knowledge and learning), the initial and delayed intervention groups showed significant differences in their rate of change across all learning outcomes over time, including military cultural competence (i.e., military cultural knowledge and skills, military cultural awareness, and military client intervention confidence) and research self-efficacy (i.e., research comprehension self-efficacy and research evaluation self-efficacy). In Phase I, the initial intervention group demonstrated significant increases across all military cultural competence and research self-efficacy variables; those in the waitlist group did not report any changes. In Phase II, delayed intervention participants were given access to the app and subsequently reported significant increases across all military cultural competence and research self-efficacy variables; the initial group reported no changes. Findings suggest that the app is effective in bolstering military cultural competence and research self-efficacy with some initial evidence of sustain impact.
Changes in outcomes of interest were examined in the context of several potentially moderating variables including age, student status, military connection, perfectionism (i.e., striving for high standards and fearing failure), and ambiguity tolerance (i.e., being able to withstand stimuli that are not straightforward). Few moderation effects emerged. Implications of continuing education delivered in a mobile app format are discussed, specific to promoting military cultural competence among human and family service providers. | en_US |
