Evaluating Dissemination and Implementation Efforts in Parent-Child Interaction Therapy by Carisa Carol Wilsie A dissertation submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 4, 2012 Keywords: Parent-Child Interaction Therapy, training, dissemination, implementation, observational coding, Dyadic Parent-Child Interaction Coding System Copyright 2012 by Carisa Carol Wilsie Approved by Elizabeth Brestan Knight, Chair, Associate Professor of Psychology Christopher Correia, Associate Professor of Psychology Alejandro Lazarte, Assistant Professor of Psychology Frank Weathers, Professor of Psychology ii Abstract With a push for evidence-based practice in psychology (EBPP), dissemination to community mental health care providers has been occurring. Parent-Child Interaction Therapy (PCIT) is one type of EBPP that is currently being trained across the United States and internationally; however, there is little research reporting on these efforts and no research has reported any implementation outcomes for trainees. This study evaluated dissemination efforts to 31 trainees across 4 different training times. Self-report and objective data revealed that trainees gained knowledge of the core principles of PCIT during a 40-hour workshop. Furthermore, they felt competent and comfortable to use PCIT effectively after completing the 40-hour workshop. Of the 31 trainees, 16 have completed, coded, and successfully video-recorded a behavioral observation between a caregiver-child dyad. No variables significantly predicted percent reliability scores in coding the behavioral observations; however, several variables together predicted a large amount of variance in the reliability scores. Limitations of this study and future directions for research are discussed in relation to PCIT dissemination and implementation training. iii Acknowledgments A project of this size required a lot of work from numerous people committed to PCIT dissemination research. A big thanks to the Parent-Child Research Lab members that worked on this project ? Shalonda Brooks, Tim Thornberry, Jamie Travis, and a host of undergraduates. Of course, our fearless leader, Elizabeth Brestan Knight, spent a countless number of hours preparing for trainings and providing her vast knowledge of PCIT through consultation. I have learned more from her than I could have ever imagined before coming to Auburn and will forever be indebted for the wonderful opportunities she has given me. My committee, Chris Correia, Alejandro Lazarte, and Frank Weathers, along with my outside reader, Bill Sauser, were a great help with this project as well. The clinical psychology program at Auburn is full of great supervisors and colleagues that were in full support of me throughout my years and for that I am humbled and blessed. Of course, I would not have been able to survive this whole process without the love and support of my family and friends. Many hours have been spent listening to me lament over the woes of trying to graduate with a doctoral degree. The most of which had to be endured by my loving husband, Chad Wilsie, who stood by me every step of the way, never complaining about having to do more cleaning than me. This dissertation represents a huge accomplishment in my life that I am proud to have completed. iv Table of Contents Abstract ........................................................................................................................................... ii Acknowledgments.......................................................................................................................... iii List of Tables ................................................................................................................................. vi List of Abbreviations .................................................................................................................... vii Introduction ......................................................................................................................................1 Dissemination of EBPP........................................................................................................1 Dissemination Examples ......................................................................................................2 Parent-Child Interaction Therapy ........................................................................................8 Dissemination of PCIT ........................................................................................................9 Hypotheses .........................................................................................................................13 Method ...........................................................................................................................................14 Participants .........................................................................................................................14 Trainers ..............................................................................................................................15 Measures ............................................................................................................................15 Procedure ...........................................................................................................................18 Results ............................................................................................................................................22 Discussion ......................................................................................................................................29 Limitations .........................................................................................................................32 Future Directions ...............................................................................................................39 v Conclusion .........................................................................................................................44 References ......................................................................................................................................46 Appendix ........................................................................................................................................53 vi List of Tables Training Characteristics across Training Times ............................................................................54 List of Trainee Numbers and Attrition Reasons ............................................................................55 Totals for Completed, Drop-Outs, and In-Process Trainees ..........................................................57 Measure Characteristics .................................................................................................................58 Listing of DPICS-III Codes ...........................................................................................................59 List of Trainee Numbers and How Many Practice Sessions to Reach Reliability .........................61 Comparison of Mean Ratings from Pre- to Post-Training .............................................................63 List of Trainee Number, Percent Reliable, and Inter-Rater Reliability on DPICS Coding ...........64 Potential Predictors of Coding Reliability with Actual Clients .....................................................65 Best Sets of Predictors with Respective R Squared Values ...........................................................66 vii List of Abbreviations APA American Psychological Association ASD Autism Spectrum Disorders CAC Child Advocacy Center CDI Child Directed Interaction DPICS Dyadic Parent-Child Interaction Coding System EBPAS Evidence-Based Practice Attitude Scale EBPP Evidence-Based Practice in Psychology I/O Industrial/Organizational IRR Inter-Rater Reliability MST Multi-Systemic Therapy NCTSN National Child Traumatic Stress Network PCIT Parent-Child Interaction Coding System PDI Parent Directed Interaction RRT Remote Real-Time TF-CBT Trauma-Focused Cognitive-Behavioral Therapy Triple P Positive Parenting Program 1 Introduction The American Psychological Association (APA) released a policy statement in 2005 encouraging the use of evidence-based practice in psychology (EBPP; APA, 2005). Specifically stated, EBPP is ?the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences? (APA, p. 1). The report also states that ?researchers and practitioners should join together to ensure that the research available on psychological practice is both clinically relevant and internally valid? (APA, p. 1). Researchers have been investigating the efficacy of different treatment approaches in an effort to improve the quality of treatment options for therapists (e.g., Spielmans, Gatlin, & McFall, 2010; Weisz, Jensen-Doss, & Hawley, 2006). However, the challenge with efficacy work is that treatments are examined in university-based clinics, leaving open the possibility of restricting the treatment?s ecological validity and clinical relevance for clients within the community. Important steps following efficacy studies in the research laboratories are transportability studies in which evidence-based treatment options are utilized by community-based mental health providers. In order to bring EBPP treatments to clients in the community, therapists must be trained on how to implement such treatments with fidelity. Ultimately, therapists in the community will require training efforts outside of research laboratories in order to implement EBPP on a widespread basis. Dissemination of EBPP Clinical researchers are currently investigating the best methods to transport empirically- based treatments to the agencies in the community that can utilize them. Herschell, Kolko, 2 Baumann, and Davis (2010) reviewed 55 studies evaluating training outcomes and methods of adult or child psychotherapy techniques. They concluded that trainings with multiple components provided better training outcomes, whereas trainings involving readings, self- directed trainings, or didactic workshops alone do not routinely result in positive outcomes. Readings pertaining to the trainings were reported to be necessary but not sufficient for skill acquisition. Self-managed training techniques demonstrated improvements in trainee knowledge or skills, but are more effective when paired with expert consultation. Didactic workshops increased trainee knowledge but not the application of the knowledge or demonstration of clinical skills. The researchers also reported that follow-up trainings can help to improve outcomes after workshops. As described by Herschell and colleagues, the body of literature describing training outcomes is limited by poor methodology, small sample sizes, and limited power. Current dissemination research studies are also hampered by a lack of control groups, random assignment, standardized assessment, and follow-up assessment. However, important first steps are being made to assess what methods should be used to disseminate EBPP. Dissemination Examples Many different family- or child-treatment approaches have been in the forefront of the dissemination literature. Some of these treatment approaches include Multi-Systemic Therapy (MST; e.g., Edwards, Schoenwald, Henggeler, & Strother, 2001; Henggeler, Schoenwald, & Pickrel, 1995), Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT; e.g., Cohen & Mannarino, 2008; Cohen, Mannarino, & Deblinger, 2006), Positive Parenting Program (Triple P; e.g., Sanders, 1999; Seng, Prinz, & Sanders, 2006), and treatment approaches for children with Autism Spectrum Disorders (ASD; e.g., Vismara, Young, Stahmer, Griffith, & Rogers, 2009). Models of dissemination used by these treatment approaches include distance or web-based 3 learning, live training with ongoing consultation, and learning collaborative models. Each of these dissemination techniques will be discussed below within the framework of the aforementioned treatment models. Multi-Systemic Therapy (MST). MST is an intensive treatment approach used to help children and adolescents with considerable behavioral and emotional problems. Within the theoretical framework is the concept that the youth and family?s school, work, peer, community, and cultural institutions are all interconnected, influencing behavior. MST is family and community based with the goal of keeping the child or adolescent placed within the home. Guiding treatment are the strengths and weaknesses of the family members. Therefore, services are provided within the appropriate setting, such as home, school, or the community, to address specific needs (Edwards et al., 2001; Henggeler et al., 1995). The MST model began in university-affiliated settings but has been used in dissemination efforts within the community across many randomized trials with promising results in reducing juvenile delinquency and improving family cohesion (Edwards et al., 2001; Henggeler et al., 1995). MST uses a training model which includes a five-day workshop for training and orientation, one-and-one-half days of quarterly trainings, weekly on-site group supervision, and weekly telephone consultation with an MST expert. The five-day training for MST includes both didactic and experiential components. Weekly on-site supervision meetings focus not only on case recommendations, but also to monitor adherence to the treatment model. Dissemination efforts of MST require that therapists be full-time employees assigned to the MST program, teams have two to four therapists on each team, caseloads not exceed six families, and teams hold weekly group supervision meetings and weekly telephone consultation with an MST expert. In addition, supervisors are required to dedicate at least 50 percent of their time to the MST 4 program. In order for these guidelines to be met, stakeholders must be identified before implementing MST treatment (Edwards et al.). To assess adherence to the MST model, monthly surveys of a standardized Therapist Adherence Measure (Henggeler & Borduin, 1992; Heggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002) are administered to families. This internet-based measure is administered so that supervisors and therapists can receive immediate feedback (Edwards et al., 2001). Supervisors are expected to be more versed in MST treatment than other team members (Edwards et al.). Supervisors are monitored for adherence as well with the Supervisor Adherence Measure (Henggeler et al.; Schoenwald, Henggeler, & Edwards, 1998) given to the therapists on the treatment team. Henggeler and colleagues provided evidence for correlations between these two measures, indicating that the more adherence to the protocol that supervisors kept, the more adherence the therapists kept to the protocol. Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT). TF-CBT can be utilized for the treatment of children and adolescents following a traumatic event with outcomes such as Post-Traumatic Stress Disorder, depression, anxiety, and other trauma-related symptoms. Using an acronym for CRAFTS, TF-CBT is comprised of the following characteristics: components- based, respectful of cultural values, adaptable and flexible, family-focused, therapeutic relationship is central, and self-efficacy is emphasized (Cohen et al., 2006). Cohen and Mannarino (2008) discussed the different forms of TF-CBT training that have been administered. Each of these models is discussed below. Distance or web-based learning. As discussed by Cohen and Mannarino (2008), distance or web-based learning is convenient, affordable, and efficient. There are options to allow participants to work at their own pace, link to other resources, view examples, and print out 5 materials as needed. TF-CBTWeb (accessible at www.musc.edu/tfcbt) is a web-based learning system designed to allow clinicians to learn the model of TF-CBT using videos, scripts, and handouts alongside discussions of clinical decision-making skills, what to expect, and cultural issues. The training is free of charge and is set up so that trainees move through modules one at a time, without skipping any material. Reports from trainees indicate an increase in knowledge of the core components of TF-CBT and high satisfaction with the course (Cohen & Mannarino, 2008). The TF-CBTWeb system has advantages and disadvantages as highlighted by Cohen and Mannarino (2008). Advantages include convenient, free training for the trainee without having to travel, which is also cost-effective for the trainer. However, disadvantages include not having discourse about questions that the trainees may have and the requirement of internet access. The intent of the web-based training is for more people to be exposed to the material, encouraging them to seek further training or information within the treatment manual (Cohen & Mannarino, 2008). Organizational readiness/live training/ongoing consultation. The first step for a live training, as suggested by Cohen and Mannarino (2008), is for an organization to be prepared for implementing the treatment before the training. Once the organization is ready, an initial training occurs, followed by continued supervision and consultation. The continued consultation offers the chance for trainers to monitor fidelity to the treatment while also providing recommendations for how to adapt the treatment to families within the organization. What can also occur, when needed, are follow-up and/or booster trainings because of staff turnover. Projects in which TF- CBT trainers used this model of dissemination have had favorable outcomes, with therapists able 6 to implement the model with fidelity and having improved outcomes with families (Cohen & Mannarino, 2008). The organizational readiness, live training, and continued consultation option for TF- CBT has advantages and disadvantages. Advantages include help with case conceptualization and difficult clinical decisions during consultation meetings and an agency having the ability to make an informed decision about the potential of completing the training and having the space and necessary means with the organizational readiness process. Disadvantages include the increased cost and time associated with the continued supervision and organizational readiness process (Cohen & Mannarino, 2008). Agencies often have very limited funds for training and continuing education, making the extra cost of an organizational readiness visit prohibitive. Learning collaboratives. Learning collaboratives involve setting up learning communities where therapists can communicate with one another, facilitating the spread of the treatment model. Monthly metrics are maintained that focus on the number of clients seen rather than client outcomes. Teams work together within an agency to make informed goals and clinical decisions by using PDSA cycles: plan-go-study-act. Therapists will decide to test a strategy for some small change in the protocol or therapy process and report to the team how this worked in a therapy session. Team members can get ideas for their own clients based on the other team members? success (Cohen & Mannarino, 2008). In one example of a learning collaborative, TF-CBT was delivered through a learning collaborative style after being sponsored by the National Child Traumatic Stress Network (NCTSN) to engage entire organizations in training. Therapists were able to hold high fidelity to the treatment model, sites were able to demonstrate competence in using TF-CBT, and family engagement was improved (Cohen & Mannarino, 2008). 7 Positive Parenting Program (Triple P). Triple P is a parenting skills intervention for parents with children with behavior problems, with the goal of reducing future behavior problems (Sanders, 1999). There are both group and individual protocols, with different levels of treatment, depending on the severity of the child?s behavior problems (Sanders, 1999; Seng et al., 2006). Seng and colleagues (2006) described a Triple P training program, typically involving 20 participants. There are different treatment levels that Triple P can be delivered to parents, so participants were trained at different levels as well. The first part of the training involved a two- or three-day didactic, depending on the level of treatment, which entailed acquisition of knowledge and skills. Another training occurred six to eight weeks later with role-played situations where therapists were required to demonstrate at least 80 percent of the skills they were taught to utilize with the families. If therapists did not reach the 80 percent mark, trainers coached the trainees in using the skills. All trainees who completed the training program met this criterion by the completion of training (Seng et al.). Other researchers have also reported positive results with a Triple P training program, with trainees demonstrating increased confidence and proficiency in using the skills learned (e.g., Sanders, Murphy-Brennam, & McAuliffe, 2003; Sanders, Tully, Turner, Maher, & McAuliffe, 2003). Early Start Denver Model. Vismara and colleagues (2009) discussed training community-based therapists on a treatment model for infants and toddlers with ASD called the Early Start Denver Model. This intervention occurs during the context of play where the therapist works first with a child on acquiring social communicative skills and then coaches the parent to facilitate skills for his or her child (Vismara et al.). 8 Vismara and colleagues (2009) compared two types of training for the Early Start Denver Model ? distance learning (telehealth) and live instruction. Training consisted of self-instruction with the training materials, one 10-hour didactic seminar and one 3-hour didactic seminar, and four hours of team supervision. Those therapists in the live instruction group were on-site at the trainings, while those in the distance learning group observed the didactic seminars remotely from a television screen with screen shots of the slides, videos, and lecture. Outcomes were assessed by reviewing video-taped sessions with clients along with a self- rated scoring sheet from the session completed by the trainee and data sheets tracking child progress. Results were promising for both training methods as therapists were able to use the program to teach parents, and child outcomes improved, including number of functional verbal utterances, attention to the adult, and number of social initiations to the adult. However, the live instruction method with didactic workshops and team supervision was better at helping therapists improve in using the skills needed for the model (Vismara et al., 2009). Parent-Child Interaction Therapy Another type of treatment within EBPP that is the focus of this proposed study is Parent- Child Interaction Therapy (PCIT; Eyberg, 1988; McNeil & Hembree-Kigin, 2010). PCIT is an empirically supported treatment program for behaviorally disruptive children (Brestan & Eyberg, 1998; Chambless & Ollendick, 2001) and for reducing future rates of child physical abuse (Chaffin et al., 2004). PCIT has also been adapted and used with additional populations such as Mexican American families, children with Developmental Disabilities, and children with Separation Anxiety Disorder (e.g., McCabe & Yeh, 2009; McDiarmid & Bagner, 2005; Pincus, Santucci, Ehrenreich, & Eyberg, 2008). Caregivers are taught specific skills to facilitate interaction with their children and are coached by therapists to use these skills during a 9 concentrated time period with other opportunities to practice in the home. PCIT is comprised of two distinct phases: Child-Directed Interaction (CDI) and Parent-Directed Interaction (PDI). During CDI, the child leads the activity and the parent uses specific skills to facilitate the child?s direction. These skills include praising, reflecting, imitating, describing, and showing enjoyment while ignoring any minor negative behavior and avoiding any questions, commands, or negative talk. During PDI, the parent learns to give effective commands and how to implement time-out, thereby improving his or her discipline strategies with the child. PCIT concludes with parents learning to generalize skills to other siblings and settings outside of therapy. PCIT is characterized by didactics, modeling, role-playing, and coaching by the therapist with time for the parent to practice at home. Dissemination of PCIT Historically, PCIT has been disseminated within graduate programs, pre-doctoral internships, and post-doctoral fellowships, using a co-therapy model with an experienced PCIT therapist as a mentor. PCIT is currently being disseminated in new ways, outside of university- affiliated training. Clinicians in the community, especially in child service systems, are receiving training from experienced PCIT trainers across the United States and internationally. However, there is not much research on the best way to train new PCIT therapists. PCIT Training Guidelines were created by a PCIT Advisory Board training committee which included the key components that should be included by trainers. These guidelines were distributed through an established PCIT listserv, referred to during recent international PCIT conferences, and are available on the PCIT website (www.pcit.org). Currently, PCIT is in a transition period with the establishment of PCIT International, Inc., a corporation that will provide oversight to the PCIT dissemination process. A certification process is currently being established with a certification 10 exam that will follow participation in PCIT training. In addition, there is a prospect of standardized training procedures and materials. The new training committee for PCIT International, is currently working on the best way or ways to disseminate PCIT. Without empirical research on the training process, this will be a difficult task, especially with the many possible avenues for dissemination. Currently, there are 13 trainers recognized as ?master trainers? by PCIT International, Inc., and listed on the PCIT website (www.pcit.org). Becoming a master trainer involves submitting a set of video-recordings of the trainer administering PCIT or supervising others administering PCIT and hosting a site visit for two members of the Board of Directors who review training and consultation materials. The PCIT Training Guidelines address many of the factors that are intended to address the dissemination process in training; however, factors of training related to implementation of PCIT are not covered as well as the dissemination factors. Herschell (2010) discussed how passive implementation strategies, such as a workshop alone, do not adequately prepare trainees to implement an EBPP with their clients. Rather, Herschell encourages trainers to use active implementation training strategies, such as ?staff selection, training, coaching, staff performance assessment, data systems, facilitative administration, and systems intervention? (p. 254). The PCIT Guidelines prescribe fidelity checks and consultation calls, involving active implementation training, but leave some liberties to the trainers to navigate for themselves, such as fidelity checks structure and expectations (i.e., percentage of fidelity), structure of the consultation calls, and expectations of coding behavioral observations integral to PCIT. In the first study involving PCIT dissemination, Herschell, McNeil, Urquiza, McGrath, Zebell, Timmer, and Porter (2009) evaluated two forms of conducting PCIT training. One group of masters or doctoral level therapists read the manual while another group of trainees received a 11 two-day intensive training with didactic and experiential components. Both groups were assessed for knowledge, skill, and satisfaction at four time points. Herschell and colleagues found that reading a treatment manual was not sufficient for reaching mastery of skills or gaining knowledge. For the experiential group, few trainees reached mastery of the skills after two days of intensive training; however, trainee degree type predicted successful acquisition of skills, but not knowledge. While this study provided initial evidence for the transportability of PCIT, the length of the workshop is no longer consistent with current PCIT training practice (40 hours of face-to-face contact). In addition, the researchers did not evaluate the factors predicting the acquisition of coding skills needed for PCIT, but only acquisition of the skills that are taught to parents and the skills involved in coaching parents. The ability to accurately code behavior observations of parent-child interactions is an integral part of helping clients to learn the skills needed for positive treatment outcomes. Further, a weakness of this paper is that the researchers did not evaluate the implementation of the skills learned, but only the training process during the workshop. Other studies have investigated implementation for other treatment approaches and indicated that higher fidelity to a treatment protocol can produce better clinical treatment outcomes (Henggeler, Pickrel, & Brondino, 1999; Huey, Henggeler, Brondino, & Pickrel, 2000; Schoenwald, Henggeler, Brondino, & Rowland, 2000). Funderburk, Ware, Altshuler, and Chaffin (2008) discussed the use of telemedicine to address some of the challenges of the dissemination of PCIT. They discussed the use of direct practitioner coaching during sessions for avoiding drift, defined as ?a misapplication of the model, often involving either technical errors or abandonment of core and requisite components? (Funderburk et al., p. 379). In university-affiliated training, live practitioner coaching is convenient with on-site trainers. However, within PCIT dissemination, trainers are not on-site 12 during therapy sessions. Drift may be more likely to happen without live coaching of the therapists in training. It is difficult to replicate the same PCIT training that occurs within university-affiliated training programs without the live coaching of the therapists. The common practice of telephone consultation meetings and video recording sessions for PCIT training is intended to mimic the coaching of practitioners when the trainer is not on-site. However, these methods are very different than live practitioner coaching and leave ample room for incorrect implementation and differing case conceptualization. Therefore, Funderburk and colleagues discussed the use of telemedicine techniques for live practitioner coaching. The telemedicine technology utilized for PCIT dissemination by Funderburk and colleagues (2008) is Remote Real-Time (RRT) coaching. Teleconferencing equipment with a television-quality image and dual-channel voice communication is used in an observation room behind a one-way mirror where the trainer can observe the trainee?s session and talk with the trainee. In addition, if needed, the equipment can be moved into the therapy room and the trainer can talk with the family alongside the trainee. This set-up allows for live practitioner coaching and opportunities for case conceptualization for the trainer. The RRT system can also be combined with phone consultation meetings. Initial surveys of therapists involved with training using RRT coaching and phone consultation reported that RRT coaching was more helpful but phone consultation was more comfortable. Overall, trainees preferred RRT coaching to phone consultation for their follow-up supervision (Funderburk et al.). The difficulty with the telemedicine systems that are current options for RRT coaching is the cost for the agencies that are receiving training. With estimates at $5,000 for each system, it is not feasible for agencies to purchase what is needed for RRT coaching. Some trainers or their universities have purchased a telemedicine system that is portable and transported between 13 agencies where trainees are located. However, the costs are doubled when this occurs as the trainer must have the same system to communicate with the agencies. The previous literature review suggests that treatments that are considered to be EBPP can be disseminated successfully to community-based mental health providers through various methods of trainings. PCIT training has occurred but there are currently only two empirical studies with research findings to determine the best way to disseminate this evidence-based treatment approach; one of the studies (Herschell, McNeil, Urquiza, McGrath, Zebell, Timmer, & Porter, 2009) presents a model that is no longer consistent with PCIT training guidelines and the other (Funderburk et al., 2008) is not outcome focused. The goals of this study were to add to the body of literature concerning dissemination, extend the limited literature base of PCIT training, and to evaluate one method of disseminating PCIT across four different training periods. Further, this paper will investigate the implementation of coding skills used during PCIT, which, to date, no study has assessed. Hypotheses Archival data from four PCIT training workshops were used to evaluate workshop outcomes. It was hypothesized that trainees would increase their objective knowledge of the core components of PCIT, self-reported knowledge of the learning objectives, and self-reported competency to use PCIT in practice after the training when compared to before the training. It was also hypothesized that trainees? acquisition and experience of coding observations during practice sessions with other trainees would predict their performance of coding with real clinical cases. Finally, it was hypothesized that acquisition of coding skills during a training workshop would predict future coding performance over and above what was predicted by trainee characteristics. 14 Method Participants Trainees were mental health professionals from various agencies. Across four different training times, there were 31 total trainees. See Table 1 for a list of training times, how many trainees attended each training, and the type of agency where trainees worked. All trainees completed a Demographic Questionnaire to gather information about their professional characteristics. The mean age of participants was 39.1 (SD = 9.4). Eighty-seven percent of the trainees were female (n = 27). Race or ethnicity was acknowledged as White or Caucasian for 67.7 percent of the trainees (n = 21), Asian for 25.8 percent (n = 8), and Black or African American for 6.5 percent (n = 2). Thirty-six percent of the trainees worked in a hospital or medical center setting (n = 11), 29 percent in a child advocacy center (n = 9), 26 percent in a counseling center (n = 8), 3 percent in private practice (n = 1), and 7 percent in another setting (n = 2). For highest degree obtained, 36 percent had a masters? degree (n = 11), 32 percent had an MSW (Masters of Social Work and Medical Social Worker) degree (n = 10), 19 percent had a Ph.D. (n = 6), and 10 percent had bachelors? degrees (n = 3). On average, trainees worked 40.5 hours per week (SD = 7.9), with an average of 17.7 hours per week with direct client contact (SD = 10.3). On average, the trainees had worked with children and families for 12.3 years (SD = 7.7), with a range from 1 to 20 years. The average amount of parent training courses taken before the PCIT workshop was 1.3 (SD = 1.9), with a range from 0 to 8. This sample of trainees was unique in that 8 of the 31 trainees, or 26 percent, were from Singapore. The Ministry of Health for Singapore funded the training and invited the trainers to 15 travel to Singapore to administer the PCIT training. Therefore, the lead trainer and author traveled to Singapore to administer the PCIT training within their agency in August of 2010. Funding for the other trainings was also provided by the State of Georgia, Governor?s Office for Children and Families; a Federal Program Improvement Grant from the National Children?s Alliance; the State of Maryland, Governor?s Office for Children; and Insuring the Children of Southern Ohio/Northern Kentucky. Although 31 trainees completed a 40 hour PCIT training workshop, they are currently at different stages of the training process. Forty-two percent of our initial roster of trainees has dropped out of the consultation phase of PCIT Training. Table 2 provides a list of trainees and where they are in the training process, including reasons why they dropped out of the process. Totals for the number of trainees who have completed the training process, those who are in the middle of the process, and those who dropped out of the training process are in Table 3. Trainers The lead trainer, Elizabeth Brestan-Knight, Ph.D., is an expert PCIT therapist who was trained by the originator of PCIT, Sheila Eyberg, Ph.D., during graduate school. She is recognized as a Master Trainer by PCIT International. Other trainers included four graduate students in Dr. Brestan-Knight?s research laboratory, the Parent-Child Research Lab, at Auburn University with research and clinical experience using PCIT. The graduate students ranged in their years of graduate training and their experience using PCIT (from 1 to 6 years). Measures Prior to the start of the training and immediately following the training, participants completed several study measures. Measures included in this study are listed in Table 4 with the assessment strategy and type of measurement used. Participants also completed a measure five to 16 seven months following the training during an on-site advanced PCIT training. Throughout the entire training and supervision process, trainees used a behavioral observation coding system (specified below) to code either parent-child or role-played interactions. PCIT Quiz. The PCIT Quiz had 28 objective multiple-choice questions about PCIT, covering both CDI and PDI topics, to assess trainee knowledge of PCIT core principles. It was developed for another research study (Lee, Wilsie, & Brestan-Knight, in press) and used with students who learned about PCIT in an undergraduate Psychology class. Lee and colleagues reported an internal consistency (Cronbach?s alpha) of .73 and a test-retest reliability with control participants of .63 (p = .001; Pearson Correlation). The PCIT Quiz was administered both before and after a 40-hour face-to-face workshop. Pre-Training Learning Objectives Survey and Post-Training Learning Objectives Survey. The Pre-Training Learning Objectives Survey and Post-Training Learning Objectives Survey were developed especially for the 40-hour workshop by using the learning objectives from each didactic presentation, which corresponded to the learning objectives created for CE documentation. These questionnaires had 33 items each, which addressed each learning objective from the entire 40-hour workshop by giving the trainee the option to choose from the following choices on a Likert-type scale: ?not very knowledgeable,? ?somewhat knowledgeable,? and ?very knowledgeable.? The Pre-Training measure was administered immediately before beginning a 40-hour face-to-face workshop and the internal consistency (Cronbach?s Alpha) was .96. The Post-Training measure was administered immediately following a 40-hour face-to-face workshop and the internal consistency (Cronbach?s Alpha) was .93. Pre-Training Competency Survey and Post-Training Competency Survey. The Pre- Training Competency Survey and the Post-Training Competency Survey were also developed 17 especially for the 40-hour workshop. The Pre-Training Competency Survey was made up of 17 items, while the Post-Training Competency Survey had the same 17 items with an additional two items. These questionnaires assessed the trainees? self-reported competency and comfort in using PCIT. Items were placed on a Likert-type scale with the options ?strongly disagree,? ?disagree,? ?neutral,? ?agree,? and ?strongly agree? or the options ?very uncomfortable,? ?somewhat uncomfortable,? ?neutral,? ?somewhat comfortable,? and ?very comfortable.? The Pre-Training measure was administered immediately before beginning the 40-hour face-to-face workshop and the internal consistency (Cronbach?s Alpha) was .95. The Post-Training measure was administered immediately following the 40-hour face-to-face workshop and the internal consistency (Cronbach?s Alpha) was .89. Dyadic Parent-Child Interaction Coding System ? 3rd Edition (DPICS-III). The DPICS-III (Eyberg, Nelson, Duke, & Boggs, 2004) is a useful measure to observe the parent- child interactions between a dyad within a child-led play situation (low demand), parent-led play situation (moderate demand), and clean-up situation (high demand). The DPICS-III can be used to assess the interactions of parents and children using both verbal and physical categories to comprise a composite score for positive and negative behavior. A description of the DPICS-III codes used in this project can be found in Table 5. To evaluate reliability for the DPICS-III, standardization studies on the DPICS-II categories can be referred to. These studies point to adequate to strong inter-observer reliabilities (Cohen?s kappa) for all DPICS categories except one uncommon code, Child Negative Touch (Eyberg et al., 2004). The validity of the original DPICS categories has been demonstrated with most of those categories still intact (Eyberg et al.). Discriminative validity and treatment sensitivity for the DPICS have also been demonstrated across several studies (Bessmer, 1998; 18 Eyberg & Matarazzo, 1980; Eyberg & Robinson, 1982; Foote, 2000; Schuhmann, Foote, Eyberg, Boggs, & Algina, 1998; Webster-Stratton & Hammond, 1990). For this study, parent-child interactions were recorded on DVD prior to the start of treatment and then observed and coded by trainees, graduate students, and undergraduate research assistants using the abridged version of the DPICS-III manual. Reliability coders were undergraduate and graduate students from Auburn University who were trained by Dr. Brestan- Knight. When used for research purposes, DPICS-III training typically involves weekly meetings with a coding team to discuss coding questions and practice coding video-taped observations. For this study, research assistant coders were required to meet 80 percent reliability with an advanced, reliable coder on at least two video-recorded observations. In addition, all research assistant coders completed a coding workbook with explanations of all the codes and quizzes to allow for practice of coding. The DPICS-III coding training typically takes a semester to complete. The reliability coders? codes were compared with codes sent from trainees for percent agreement. Procedure Trainees or their agency requested PCIT training from Auburn University. The trainings occurred at four different time periods at four different locations (see Table 1). These live, face- to-face trainings lasted for 40 hours over five days. Trainings included didactic presentations, modeling and role-playing of skills, and coding practice. The most recent version of the PCIT Training Guidelines (2009) was followed as closely as possible for all training groups. A 40-hour curriculum was created by the Parent-Child Research team at Auburn University based on original materials posted by Dr. Sheila Eyberg on the University of Florida?s PCIT website (http://pcit.phhp.ufl.edu). Didactic presentations were intended to give trainees an 19 overview of the core components of PCIT. Trainees were exposed to the theory and history of PCIT, including its origins and the theoretical frameworks that it draws upon. Other didactic presentations covered the specifics of assessment during PCIT, including what outcome measures to use and how to administer, score, and interpret each measure. Trainees were also presented the material that is typically taught to parents during each phase of PCIT. Another important area of PCIT that was emphasized during the didactic talks was the process of the generalization of skills beyond the therapy session, including in the home and in public places. Other research didactic presentations focused on the empirical research surrounding PCIT, including how PCIT has been tailored to fit different populations, how PCIT has been used in a group format, and how the skills taught in PCIT can apply to non-parent populations working with children. Trainees were provided with research articles that were covered during the didactic presentations and encouraged to refer to empirical materials when making clinical decisions about applying the protocol to their clients. Trainees were required to code five-minute DPICS interactions throughout the training with an advanced, reliable coder. The codes that were used are described in Table 5. Percent agreement between the trainee?s codes and the codes of the advanced, reliable coder was calculated for each interaction. Each trainee was required to meet 80 percent reliability for their DPICS codes at least once. See Table 6 for a list of trainees by participant number and how many practice sessions it took each trainee to reach 80 percent reliability. The trainees also had to meet mastery criteria for both the CDI and PDI skills, which were the same mastery criteria that parents must meet before completing treatment. The mastery criteria for CDI required the trainees to role-play a five-minute observation during which they used 10 labeled praises, 10 reflections, and 10 behavior descriptions, with fewer than three questions, commands, and 20 criticisms. The PDI mastery criteria required trainees to role-play a five-minute observation during which they gave at least four commands, 75 percent of which were effective commands as taught through PCIT, and followed through correctly using the time-out procedure. The role- plays were conducted with another trainee acting like a child within the PCIT age range. The objective of role-playing observations was two-fold as trainees rotated ?roles:? some of them practiced their CDI mastery skills (as ?the parent?) while others practiced their DPICS coding skills (as ?the therapist?) at the same time. After each role-play, trainers discussed the DPICS codes with the trainees to facilitate discussion and learning. Once trainees reached reliability on DPICS coding towards the end of the workshop, they also practiced coaching during role-play situations. Therefore, role-plays during the workshop allowed for practicing multiple skills at the same time, allotting ample time to learn all skills, especially coding skills, which typically take undergraduates a semester to learn. Following the 40-hour face-to-face live training, trainees are required to have follow-up consultation until they see two families through completion of the PCIT protocol (in order to qualify for the PCIT certification exam). This consultation included group supervision calls and video reviews. The supervision calls ranged from weekly to monthly, depending on agency funding and availability and the PCIT caseload of the clinicians. The PCIT Training Guidelines specified that contact could be no less than once per month. The format of the supervision calls varied, but generally followed a structure created by the lead trainer. Video reviews included a total of seven sessions during PCIT, the initial DPICS observation session, the two Teach sessions from CDI and PDI, the first Coach session from both CDI and PDI, and sessions which covered generalization to the home and public places. Integrity checks, which are included in the PCIT Manual (Eyberg, 1999), were adapted for the video reviews, with space for narrative 21 comments about the session and sent to trainees for each of the seven reviewed sessions. Graduate students first reviewed the therapy session tapes using the integrity checks and offered comments. Undergraduate students watched the initial DPICS observations, completed integrity checks, and coded for reliability. These reviews and the tapes were observed by the lead trainer and then sent to the trainees as feedback. An advanced training occurred five to seven months following the initial 40-hour training. This part of the training focused on refining coaching skills and providing opportunities for further DPICS coding practice. If available, families who were in treatment during the time of the on-site advanced training presented for their session and were observed by the lead trainer and trainees behind a one-way mirror or through a live video feed. Whenever possible, the lead trainer assisted the trainee using a co-therapy model to offer clinical advice. These sessions were encouraged to be coaching sessions, to help refine the coaching skills. Trainees also watched live and video-recorded DPICS sessions for coding practice. Their codes were compared to an advanced, reliable coder to check for percent agreement. Trainees completed follow-up measures at the time of the advanced training as well. 22 Results Several t-tests were conducted to evaluate the change in the trainees? responses on objective and self-report rating scales from pre- to post-training. It was hypothesized that trainees would increase in objective and self-reported knowledge of core PCIT principles and competency in using PCIT. The average score on the PCIT Quiz before the workshop was 74.5 (SD = 10.9, Range 54 to 93) and the average score after the workshop was 95.9 (SD = 3.5, Range 89 to 100). See Table 7 for a comparison of pre-training and post-training mean scores and deviations. Given that this is an objective test, the average scores can be converted to a letter grade using the same grading scales common for schools. Notably, the trainees in this study improved from a C average to an A+ average from pre- to post- training. Based on the measured differences between on the PCIT Quiz before the 40-hour workshop and after the workshop, a statistically significant difference emerged in trainees? objective performance (t (30) = -11.70, p < .001). The calculated effect size was large (d = -2.64), indicating a clinically significant difference between the quiz scores before and after the workshop. To determine future areas to improve during the workshop, item analyses were conducted on the post-training quiz. All 31 trainees answered correctly on 18 of the 28 items. The three lowest item scores were identified. On the question, ?Which is not a benefit of CDI play sessions?? the correct answer was ?enhances children?s creativity.? Forty-two percent (n = 13) of the trainees endorsed an incorrect answer. Twelve trainees chose ?has a calming effect on the child? and one trainee chose ?improves children?s self-esteem.? On the question, ?If the child does not comply with what the parent said in an appropriate amount of time, what should the 23 parent do?? the correct answer was ?give them a warning.? Thirteen percent (n = 4) of the trainees endorsed an incorrect answer, all choosing ?put them directly in time-out.? On the question, ?How many phases does PCIT have?? the correct answer was ?2.? Thirteen percent (n = 4) of the trainees endorsed an incorrect answer. Three chose ?3? and one person chose ?5.? The Pre-Training Learning Objectives Survey was administered immediately before beginning a 40-hour face-to-face workshop. The average score of the Likert-scale items was converted to an overall percentage on the Learning Objectives Survey, and the pre-training average percentage score was 43.3 (SD = 10.2, Range 33 to 73). The Post-Training measure was administered immediately following a 40-hour face-to-face workshop. The average score of the Likert-scale items was converted to an overall percentage on the Learning Objectives Survey, and the post-training average percentage score was 83.7 (SD = 10.8, Range 58 to 99). See Table 7 for a comparison of pre-training and post-training mean scores and deviations. On the self- reported Pre- and Post-Training Learning Objectives Survey, trainees reported a statistically significant improvement in their knowledge of the learning objectives outlined during training (t (29) = -18.25, p < .001). The calculated effect size was large (d = -3.85), indicating a clinically significant difference between the scores before and after the workshop. To determine future areas to improve during the workshop, item analyses were conducted on the Post-Training Learning Objectives Survey. Based on a 3-point Likert Scale (?not very knowledgeable? = 1, ?somewhat knowledgeable? = 2, and ?very knowledgeable? = 3), the highest three items were ?Regarding my ability to demonstrate and/or model the PRIDE skills to caregivers, I feel?? ( M = 2.87, SD = .35), ?Regarding my ability to differentiate between positive reinforcement and negative reinforcement and how to implement each technique, I feel?? (M = 2.87, SD = .35), and ?Regarding the Child Directed Interaction phase skills, I 24 feel?? ( M = 2.83, SD = .38). The three lowest scores were ?Regarding my ability to summarize and critique the literature pertaining to the efficacy and dissemination of PCIT in a group format, I feel?? ( M = 1.90, SD = .66), ?Regarding how PCIT can be used in a group format, I feel?? (M = 2.00, SD = .64), and ?Regarding the use of motivational techniques to enhance completion of PCIT, I feel?? ( M = 2.13, SD = .68). The Pre-Training Competency Survey was administered immediately before beginning the 40-hour face-to-face workshop. The average score of the Likert-scale items was converted to an overall percentage on the Competency Survey, and the pre-training average percentage score was 47.6 (SD = 13.8, Range 22 to 80). The Post-Training measure was administered immediately following the 40-hour face-to-face workshop. The average score of the Likert-scale items were converted to an overall percentage on the Competency Survey, and the post-training average percentage score was 82.4 (SD = 5.7, Range 72 to 99). See Table 7 for a comparison of pre- training and post-training mean scores and deviations. Using the self-reported Pre- and Post- Training Competency Survey, trainees reported a statistically significant improvement in their PCIT skill competency following training (t (30) = -16.22, p < .001). The calculated effect size was large (d = -3.30), indicating a clinically significant difference between the scores before and after the workshop. To determine future areas to improve during the workshop, item analyses were conducted on the Post-Training Learning Objectives Survey. Based on a 5-point Likert Scale, the three highest rated items were ?Overall, I am satisfied with the training I received? (M = 4.77, SD = .44; scale ?strongly disagree? = 1, ?disagree? = 2, ?neutral? = 3, ?agree? = 4 and ?strongly agree? = 5), ?I am satisfied with PCIT as a therapeutic approach? (M = 4.71, SD = .53; scale ?strongly disagree? = 1, ?disagree? = 2, ?neutral? = 3, ?agree? = 4 and ?strongly agree? = 5), and 25 ?Regarding my competency in understanding/conveying the CDI skills of PCIT, I feel?? (M = 4.58, SD = .50; scale ?very uncomfortable? = 1, ?somewhat comfortable? = 2, ?neutral? = 3, ?somewhat comfortable? = 4, and ?very comfortable? = 5). The three lowest rated items were ?My use of PCIT in the future will be influenced by the use of the treatment by other clinicians where I work? (M = 2.90, SD = 1.90; scale ?strongly disagree? = 1, ?disagree? = 2, ?neutral? = 3, ?agree? = 4 and ?strongly agree? = 5), ?My use of PCIT in the future will be influenced by my patients? similarity to the patients I observed during training? (M = 3.13, SD = .86; scale ?strongly disagree? = 1, ?disagree? = 2, ?neutral? = 3, ?agree? = 4 and ?strongly agree? = 5), and ?I feel I am effective as a PCIT therapist? (M = 3.57, SD = .73; scale ?strongly disagree? = 1, ?disagree? = 2, ?neutral? = 3, ?agree? = 4 and ?strongly agree? = 5). Several questions included on the Competency Survey are of particular interest as they assess broad beliefs about competency, comfort, and effectiveness, rather than narrowed constructs. On the statement, ?I feel competent to effectively conduct PCIT treatment,? trainees reported significantly improved Likert-scale scores from pre-training (M = 1.65, SD = .91) to post-training (M = 4.13, SD = .50; t (30) -16.25, p < .001) and effect size indicated a large clinically significant difference (d = -3.38). On the statement, ?I feel comfortable implementing PCIT,? trainees reported significantly improved Likert-scale scores from pre-training (M = 1.84, SD = .97) to post-training (M = 4.29, SD = .59; t (30) -16.06, p < .001) and effect size indicated a large clinically significant difference (d = -3.05). On the statement, ?I feel I am effective as a PCIT therapist,? trainees reported significantly improved Likert-scale scores from pre-training (M = 1.67, SD = .88) to post-training (M = 3.57, SD = .73; t (29) -12.96, p < .001) and effect size indicated a large clinically significant difference (d = -2.35). In other words, trainees reported 26 feeling more competent, comfortable, and effective as a PCIT therapist after the 40-hour workshop when compared to ratings before the workshop. In short, self-report and objective data revealed that trainees gained knowledge or believed they gained knowledge of the core principles of PCIT during the 40-hour workshop. Furthermore, they felt competent and comfortable to use PCIT effectively after completing the 40-hour workshop. Also of interest in this study was identifying predictors for trainee coding accuracy on the DPICS behavior observation coding system with actual clients following the workshop. Of the 31 trainees who had completed the 40-hour training workshop, 52 percent (n = 16) of those trainees have completed, coded, and successfully video-recorded a DPICS observation. Nineteen percent (n = 6) of the trainees are currently attempting to video-record and code a DPICS observation with a client. The remaining trainees dropped out of the consultation process before sending a DPICS observation. The average percent reliable score for the 16 trainees was 73 percent (SD = 12.9) and the median was 73 percent. The range was large, spanning from 39 to 94 percent. Sixty-nine percent (n =11) did not meet 80 percent, which was the DPICS coding criteria during the training, and suggests a marked reduction in coding skill since their workshop performance. For 30 percent (n = 5) of the tapes, the reliability coder?s (not the trainees?) codes were compared to another advanced, reliable graduate student coder to assess Inter-Rater Reliability (IRR). The average IRR for the two advanced, reliable coders was 83 percent (SD = 4.3), with a median of 83 percent. A list of the trainees by number with reliability scores and IRR (when applicable) can be found in Table 8. Specifically, it was hypothesized that trainees? performance in coding during the workshop using video-recorded and role-played situations would predict how accurate the 27 trainees would be when they later coded sessions with their own clients. Further, it was hypothesized that these variables would predict reliability with actual clients over and above other trainee characteristics. To determine predictors of coding performance with actual clients following the 40-hour workshop, multiple regression was utilized. Variables from the trainee workshop performance were number of practice sessions it took for trainees to reach 80 percent reliability with an advanced, reliable coder and the average DPICS reliability across all practice sessions. Also entered as possible predictors were training site, agency where the trainee worked, education level, hours worked per week with direct client contact, years of experience working with children and families, and number of courses taken to learn about parent training. Training site and agency where the trainee worked were significantly correlated so agency where the trainee worked was left out of the subsequent analyses. However, none of these potential predictors were significant predictors of how reliable the trainees were when they coded sessions with actual clients. See Table 9 for the amount of variance for which each predictor accounted, and the significance value. To characterize this dataset without respect to generalizability, the SAS statistical program identified combinations of predictors and respective variance accounted for by the combinations. Table 10 lists the best model for each number of variables with the respective R square value. It appears the best combination of predictors is education level, training site, hours of direct client contact per week, and the average DPICS reliability during the workshop, as the variance accounted for does not increase substantially in the models with 5, 6, or 7 predictors. Fifty-four percent of the variance in the reliability during the first actual DPICS session with a client was accounted for by the combination of education level, training site, hours of direct client contact per week, and the average DPICS reliability during the workshop. Training site 28 was a continuous variable which characterized the trainings in the order of the trainings administered by the training team. So, essentially, training site can be considered more of a trainer variable rather than a trainee variable in that the trainers improved their methods for teaching the DPICS through subsequent trainings. Two of the other three variables, education level and hours of direct client contact per week, were variables that characterized trainees before they started the training workshop. The last variable in the set is the average DPICS reliability during the workshop, which was a variable determined during the workshop. Although these results cannot be generalized beyond this small sample size, it is a depiction of the current sample. 29 Discussion Using measures from all 31 trainees, these data suggest that the described 40-hour face- to-face workshop was adequate in facilitating an increase in trainees? knowledge of the basic aspects of PCIT. Trainees also reported feeling competent and comfortable to provide PCIT effectively after the workshop. Findings in this study were not only statistically significant, but also clinically significant, which is an important finding given that other dissemination efforts have found statistical but not clinical improvements (Schumacher, Madson, & Norquist, 2011). These results are good indicators that the PCIT Training Guidelines (which require a 40-hour face-to-face workshop for trainees) are sufficient in terms of trainee acquisition of basic PCIT knowledge. While workshop content most likely varies across trainers and training sites, this study provides a good base to start formulating ideas of how to structure the training process and what should be included in a workshop. Based on the item analyses, trainees were satisfied with PCIT as a therapeutic approach and the PCIT workshop. The workshop covered the PRIDE skills and how to teach caregivers the PRIDE skills well, but may not convey the theorized benefits of the CDI play sessions. Trainees indicated that, compared to other information conveyed during the training, they felt the least knowledgeable about the information concerning PCIT in a group format, including the use of motivational techniques. It is important to note that, due to time constraints during some of the workshops, not all trainees were able to spend adequate time discussing PCIT use in a group format. Trainees also reported not feeling effective as a PCIT therapist (compared to other item responses) but ratings on the same item showed clinically significant differences between the 30 item when asked before and after the workshop. It is quite possible that trainees would never feel effective as a PCIT therapist after a workshop alone, despite having improved feelings of effectiveness following the workshop when compared to before the workshop. Rather, supervised experience with actual clients within their own agency may be the ultimate factor to improve feelings of effectiveness. Follow-up measures may help to examine the influence of having PCIT clients and receiving consultation about PCIT clients. Given that the sample size was small in this study, the results from the predictors of DPICS coding performance with actual clients cannot be generalized to other samples. However, the results of this study do have some implications for PCIT trainers to consider given the scarcity of data available pertaining to PCIT dissemination. The best model accounting for the variance in the DPICS coding performance included education level, training site, hours of direct client contact per week, and the average DPICS reliability during the workshop. Within these variables are both trainee and training characteristics. This could imply that multiple factors are involved in successful use of skills learned in a PCIT workshop with clients and that these factors are not all acquired during the workshop. Rather, some of the characteristics that predict accurate DPICS implementation may relate to trainee past experience and even experience of the training team. In effect, for administrators deciding who to send to trainings, determining which clinicians would demonstrate better PCIT skills after the workshop may not be an easy task. DPICS training may be an important part of being a successful PCIT therapist, as better coders may make better therapists, although this has yet to be studied. Studies suggest that higher fidelity to a treatment protocol produces better treatment outcomes (Henggeler, Pickrel, & Brondino, 1999; Huey, Henggeler, Brondino, & Pickrel, 2000; Schoenwald, Henggeler, Brondino, & Rowland, 2000). PCIT studies may yield similar results in the area of trainee 31 outcomes. Trainees may have difficulties teaching parents the skills if they do not know the skills well enough to code them reliably. Such potential results would provide evidence that higher fidelity to (or reliability with) a coding protocol produces better trainee outcomes (as measured by the ability to convey didactic information to parents). If this were the case, DPICS training would be integral to training successful PCIT therapists. Trainers can utilize the information gathered in the current study in training DPICS skills to other therapists. One of the variables that was important in this sample in determining use of skills after the workshop was the trainees? coding performance during the workshop. Therefore, training during the workshop is imperative. If the average DPICS reliability during the workshop helps to determine better use of skills following the workshop, trainers should utilize time during the workshop to strengthen this average. During the workshops described in the current study the schedule had to stay somewhat flexible with trainee and agency needs; however, the amount of workshop time planned to spend practicing DPICS coding ranged from around 7 hours (approximately 18 percent of workshop time) to around 13 hours (approximately 32 percent of workshop time), with multiple practice sessions (ranging from 6 to 11 during the workshop time). Given that DPICS coding can be difficult for undergraduate coders (who are required to have at least a 3.5 grade point average and a weekly meeting with an expert coder over a semester), our training team dedicated large amounts of time to teaching DPICS skills during the workshop. Other trainers may consider dedicating more time to learning DPICS coding during the workshop. Even with the same training team, there was quite a bit of variation in the amount of time practicing DPICS coding. This variation occurred for various reasons including the size of the trainee groups during the different workshops, the trainers learning through experience if more or less time was needed for 32 DPICS practice, and that some agencies had difficulty learning the skills. It is assured that comparing training teams across different sites would also result in variability. Some direction from PCIT International, as the agency overseeing the PCIT dissemination process, might better standardize the DPICS training process during the workshop. Regardless, more work needs to be conducted on the best way to train coders. Limitations While the PCIT Quiz was used by Lee and colleagues (in press), citing good pilot data for the use of this instrument, this study is limited by the use of other measures that have not yet been validated. These measures were developed by the Auburn University Parent-Child Research Lab specifically for the training of PCIT principles in various settings, including the 40-hour workshop, and will eventually be validated with more opportunities in the future to assess trainees? PCIT knowledge and competence. It would be helpful if other training sites used similar measures as well to determine if trainees learn similarly across training sites. Soon PCIT therapists will have the ability to become certified to use PCIT with clients by taking an online continuing education quiz through the PCIT International website. These data can be used by PCIT International to determine gaps in knowledge across training sites (much like the licensing exam for clinical psychologists), which could also inform future training approaches. One recommendation is for PCIT International and master trainers to use an agreed upon set of measures as a shared database that could be developed for the evaluation of PCIT training as a whole. Another limitation was this study?s small sample size, which limits the generalizability of these results. With the amount of time the training process takes, only a limited number of therapists can be trained by one training team at any given time. The period of time for PCIT 33 consultation calls can range from one to two years. Fixsen, Blas?, Naoom, and Wallace (2009) reported that successful implementation of an EBPP will likely take two to four years to complete, and PCIT is no exception. There are considerable time demands for trainers for both the workshop and consultation phases of training (approximately 100 hours total per training cohort). With the time commitment that is required to follow trainees through two completed PCIT cases, it is a difficult task to obtain a large sample of trainees for research. These analyses were conducted using the pilot data from the first four trainings administered by Auburn University Parent-Child Research Lab. The results of this study can only characterize the current sample but do provide a good start for identifying potential predictors of accurate PCIT skill implementation. Unfortunately, this sample?s performance in using DPICS coding skills with actual clients was not entirely determined by the examined factors from the workshop. Rather, both training and individual factors prior to the workshop were identified as a part of the model that predicted DPICS coding skills. Therefore, it may be beneficial for trainers to think about such factors prior to committing to training with certain agencies. It may be beneficial for trainees to have spent a majority of their time in direct client contact on a weekly basis, rather than administrative work. Clearly, it takes longer to recruit training families if individuals do not carry a large clinical caseload, extending the amount of time it takes for a trainee to complete the consultation phase of training and increasing trainee attrition. Future studies could assess whether skill acquisition is faster - or applied more accurately - for those who spend more time in direct client contact. One potential difficulty in analyzing the DPICS practice coding sessions used during the trainings is that these practice sessions were conducted with role-played scenarios rather than with standardized video-recorded observations of real parent-child dyads. Therefore, situations 34 varied between trainees, making it more difficult to determine which factors led to learning the DPICS coding system. While it is difficult to measure, these coding situations are, in a sense, randomly determined by the trainees in each training, ultimately resulting in a random mixture of situations for each trainee. Friedberg, Gorman, and Beidel (2009) caution that role-played situations may not approximate real-world situations, making these role-plays futile activities. This note is especially applicable to the current study as the trainees in the role-played situations during the training workshop were attempting to meet the mastery criteria for their CDI skills (10 labeled praises, 10 reflections, and 10 behavior descriptions with less than three total questions, commands, and criticisms); however, the observations with actual clients that trainees coded for skill review were conducted at pre-treatment, before the parents learned any of the PCIT skills. Therefore, the role-played observations from the training workshop included strong use of the target skills, essentially making them not realistic when compared to pre-treatment DPICS sessions with actual clients. Anecdotally, the trainees in Singapore spoke very quickly during the first few practice sessions; however, once they had an opportunity to code they began to talk slower, perhaps in an attempt to help out their colleagues who were coding the interactions. Friedberg and colleagues described situations in which beginning clinicians role-played being in a therapy session. Training to use the DPICS is different, and perhaps more complicated, in that coding is a matter of classification rather than production of verbal behavior. Coding also involves clinical decision making in real time, which occurs very quickly. In addition, the role- played practice sessions during our PCIT trainings may not be accurate portrayals of how parent- child dyads will react in sessions and may, therefore, not be preparing clinicians for coding actual parent-child interactions. All trainees in this study met 80 percent reliability at least once during the role-played or video-recorded observations; however, only 31 percent (n = 5, see 35 Table 8) of the trainees that sent DPICS work samples met our target of 80 percent reliability. Schmacher and colleagues (2011) reported that similar loss of skills occurred during a Motivational Interviewing training, hindering the implementation of the EBPP; further, they reported that intensive training did not promote skill acquisition that was necessary for real- world implementation of Motivational Interviewing. What could strengthen the research process in PCIT dissemination is the development of DPICS training videos to allow for standardized coding situations across all trainees. Such a video would also help trainees to learn the procedure of conducting a DPICS observation, which was also a challenge for this cohort of trainees. Fidelity to the DPICS procedure is a less complicated, but separate skills set, from coding accuracy. While this study assessed coding accuracy, one study for the future would be to evaluate the trainees? fidelity to the DPICS set-up and procedure as outlined in the DPICS manual. It would also be helpful to investigate the extent to which potential PCIT trainees have experience with empirically based assessments, such as the DPICS. The attrition of trainees was another challenging aspect of this study. Woltmann and colleagues (2008) reported that 24-month turnover rates, including jobs within agencies, negatively predicted fidelity scores when implementing evidence-based treatments for mental health. If agencies want to be trained on and implement evidence-based treatments, it is important to consider turnover rates when determining how appropriate training may be for employees. For this study, all trainees that started the 40-hour PCIT workshop completed it. However, many trainees were not able to continue the consultation phase of the training following the workshop. Table 2 lists the trainees with their progress in the training, along with reasons, if any, why they had to discontinue the consultation phase before seeing two PCIT clients to completion. Trainees had to cease consultation services for multiple reasons including 36 not having access to the client population best-suited for PCIT (n = 2), leaving the agency (n = 3), changing job responsibilities within the agency before the end of the consultation process (n = 6), or a difference in theoretical orientation between the trainees and PCIT?s strong behavioral base (n = 2). Another important finding from Woltmann and colleagues to consider was that not all turnover was negative, meaning that some less qualified staff left their positions and were replaced with more qualified staff; in addition, sometimes a realignment within the team occurred under the support of the consultants. Therefore, attrition may not always be considered negative. However, it does pose a challenge for building an agency-supported practice with PCIT-trained therapists. The PCIT Training Guidelines suggest that two trainees per site should complete the PCIT training process together. Trainees have qualitatively reported that the support of a co-therapist is valuable during sessions with the first couple of PCIT clients, especially a co-therapist who completed the same training workshop. Further, agencies may be more likely to fund consultation calls and the technology used with PCIT if there are multiple therapists delivering PCIT within the agency. Some of the attrition from PCIT training is expected with the dynamic nature of funding and jobs. However, some of the trainees from this sample might have benefitted from knowing more about PCIT and the requirements of the training prior to initiating the process. Cohen and Mannarino (2008) recommend that trainers conduct an organizational readiness process prior to conducting a live workshop. The lead trainer in the current study communicated what was expected of the trainees to a contact person within each agency in charge of the funding for the trainings; however, many trainees were asked or required by their agency to participate in the training without receiving much information about the training process prior to the PCIT workshop. Also, at times supervisors often did not understand the requirements of the training. 37 One recommendation is that agencies should sign a contract prior to the training workshop, outlining the training requirements and work products expected from the trainees. Furthermore, agency administrators could fill out a type of readiness scale assessing if the agency would be a good fit for PCIT therapists. Agencies must first determine if PCIT is a good match for their clientele. Further, agencies must also establish which therapists would be best suited for learning PCIT as a therapeutic approach. Schumacher and colleagues (2011) reported that baseline skills and trainee motivation were important to consider in Motivational Interviewing training. Herschell, Kogan, Celedonia, Gavin, and Stein (2009) reported various considerations by administrators when deciding which therapists to include in training, including whether they volunteered, clinician seniority, credentials, familiarity with the therapy being disseminated (specifically Dialectical Behavior Therapy), staffing needed, and diversity. Administrators may need more direction on how to select clinicians and what factors to weight more heavily than others. One possibility for preselecting PCIT trainees is to obtain a work sample from potential trainees prior to their enrollment in a workshop. The DPICS Manual (Eyberg et al., 2004) has an accompanying workbook containing quizzes that could be completed prior to the workshop. Agencies could select trainees by the highest scores achieved on the workbook quizzes. Future studies could assess if workbook scores prior to the workshop were good predictors of trainee performance with the PCIT skills. Finally, therapists who are going to participate in the training should be involved before the start of the workshop to clarify the needs of the trainers as well as the agencies. However, it may be difficult for an organization to engage in a readiness process prior to the start of a workshop because of limited funding. Such a visit would require more funds for the trainer to 38 make an extra trip to the agency and supervisors would need to schedule extra time to spend with the trainer. Perhaps a portion of the first day of a PCIT training workshop could be used to talk with both administrators and therapists to gain agency buy-in and clarify what is expected from the start. Established measures would be useful tools to identify therapists most suited for the training. Aarons (2004) described the use of Evidence-Based Practice Attitude Scale (EBPAS) in evaluating how to tailor dissemination efforts. The EBPAS consists of 15 items with Likert-scale response options. Four dimensions of attitudes toward adoption of EBPP were identified including intuitive appeal, likelihood of adopting given the requirements, openness to new practices, and perceived divergence of usual practice with research-based/academically developed interventions. A scale similar to the one described by Aarons may have been useful in reducing attrition during our PCIT training. Another major limitation of the present study is the lack of a control group for comparison. Trainees were evaluated at pre- and post-training but not compared to a group of therapists who did not complete any training in PCIT. In addition, 42 percent of the trainees who completed the workshop dropped out before the end of the consultation phase (see Tables 2 and 3). It is currently unclear whether those trainees who continued past the workshop phase to the consultation phase would demonstrate statistically different skill demonstration compared to those who only completed the workshop. Without gathering DPICS observations with codes from trainees who only completed the 40-hour workshop, there is no way to determine if their skills differ from the trainees who continued through the consultation phase of training. A future investigation could solicit work samples from ?drop-out? trainees to determine whether there is added benefit to skill acquisition during the consultation phase. 39 Future Directions There are multiple possible areas of interest for PCIT dissemination literature to consider. Future research in this area could compare a live workshop to other forms of training, such as distance learning or self-paced methods. While the live workshop evaluated in this study produced significant gains in knowledge and competence for the trainees, other modes of training may produce equally significant gains with less investment of time and money by the trainers or trainees. Of particular interest would be a better way to communicate with international trainees, as the time-difference as well as technology constraints can hinder the evaluation process following the 40-hour face-to-face workshop. Currently, the Auburn University Parent-Child Research Lab is using video conferencing equipment to continue the consultation calls; however, the technology for these calls is quite expensive, limiting contact to less than monthly calls. In addition, the team has to wait for international postage to deliver the videos to be reviewed, creating a longer time delay between the provision of PCIT services by the trainees and feedback from the trainers. More advanced avenues for distance learning education that are affordable and available for agencies should be explored to determine more efficient ways to deliver training to international audiences. While the PCIT Training Guidelines (found at www.pcit.org) specify that a training must contain 40 hours of face-to-face time during the workshop, there is no direction about if the 40 hours need to occur over 5 consecutive days or can be broken into several phases. Of the four trainings in this study, three different formats were used. One of the trainings occurred across five consecutive days, two occurred with three days and then two days about four weeks later, and one occurred over two days followed by a three day break before the last three days of training. Seng and colleagues (2006) reported that delivery of Triple P is typically a two- or 40 three-day didactic followed by another training six to eight weeks later. Future studies could assess the best method of delivery for a PCIT 40-hour workshop that is beneficial for training needs, trainers, and trainees. Given that this was a pilot study to determine if trainees used the skills taught during the workshop, future studies could follow trainees past the 40-hour workshop and the pre-treatment DPICS sessions to assess their fidelity in using the PCIT protocol with clients in later sessions. While trainees can learn about PCIT in the workshop, it is imperative to know how well these trainees adhere to the PCIT manual on the core components. In addition, there are skills used in PCIT other than those skills used in DPICS coding. The area with perhaps the most variability between trainees, and established PCIT therapists, is the coaching skills set. PCIT therapists coach caregivers to use the skills that the therapists have taught them while the caregiver-child dyads are playing. Coaching occurs across both the CDI and PDI phases of treatment, both requiring a different approach to coaching. DPICS is a part of coaching at its most basic level; however, coaching must also progress in difficulty to include process statements and attributions of parent success to be clinically effective. Of interest would be how trainees are taught their coaching skills, how these skills develop with actual clients, and what coaching skills produce better client outcomes. Future studies could evaluate the development and use of coaching skills through the development of a coaching skills coding system and systematic analysis of both experienced and novice PCIT therapist verbal behavior during the coaching portions of PCIT. This coding system would also need to include a sequential analysis of parent verbal behavior and therapist verbal behavior to investigate the therapist statements that lead to better in-session parenting behavior. 41 Dissemination research needs to be refined to ultimately result in more positive treatment outcomes for families. Future studies in the area of PCIT dissemination could assess client outcomes as well as trainee outcomes. Vismara and colleagues (2009) reported that trainees in the Early Start Denver Model for infants and toddlers with ASD track child progress with data sheets. PCIT trainers could also track client progress intentionally, rather than only trainee progress. Tracking client outcomes is a benefit to the face-to-face workshop and consultation model, as other training models, such as learning collaboratives, do not do this. However, the face-to-face workshop and consultation model could track client outcomes better as, often, more of the focus is on trainee outcomes. Client outcomes could be tracked by asking trainees to send their trainers copies of outcome measures (e.g., the Eyberg Child Behavior Inventory (Eyberg & Pincus, 1999), child socio-emotional measures, and post-treatment DPICS observations). Given that this study focused on DPICS coding, one aspect of PCIT assessment for measuring outcome, another variable that was not considered was previous trainee assessment experience. However, education may have been somewhat of an indicator as higher degrees in the mental health field can indicate more specialized training in assessment; perhaps those trainees in this study with higher degree types had more experience with assessment. One problem with the way that education was measured in this study was that several trainees were classified as the degree type they reported (bachelors, masters, etc.), but were currently enrolled in classes to obtain a higher degree than the one indicated. These trainees were at various points in obtaining the higher degree as well. Therefore, better categorization of education and previous experience with empirically-based assessment methods should be added to the demographics questionnaire and would be positive contributions in future studies. 42 Another area that is not prescribed by the PCIT Training Guidelines is how to structure consultation calls. Different agencies involved in the current study seemed to approach the consultation calls differently, even when they were given a structure by the trainers to follow. When trainees discussed aspects of PCIT, talk almost exclusively focused on treatment discussion, rather than DPICS coding discussion. However, DPICS coding is a fundamental skill in PCIT and some discussion needs to be dedicated to discussing coding questions and comments. Thus far, the feedback on coding is discontinuous in the PCIT training process in that trainees do not receive any feedback on DPICS coding beyond their first DPICS observation, per the PCIT Training Guidelines. Perhaps feedback on coding needs to continue beyond the first DPICS observation sent to the trainers. Continuing education opportunities in DPICS coding would also be beneficial to trainees. Mazzucchelli and Sanders (2010) reported that the Triple P training program offers web-based post-training support in the form of electronic newsletters, online question and answer forums via webinar, and access to a web-based ?Triple P Provider Network? which contains web-based resources tailored to each level of training that trainees have received. In addition, the PCIT Training Guidelines also specify that consultation should occur no less than once per month. However, in the current study, the agencies ranged in frequency of calls from weekly to biweekly to monthly. Fewer calls per month likely cause coder (and therapist) drift but, again, this is an empirical question. More research in the future could focus on how much feedback is necessary pertaining to coding and how often calls should occur and how much time the calls should dedicate to coding issues. Even within agencies, attendance rates to the calls varied. Future studies could evaluate if attendance to the consultation calls related to the DPICS coding performance or other fidelity checks. 43 A difficult aspect for agencies can also be obtaining clients quickly after the time of the workshop. Some therapists may not have openings in their schedule for new clients and will have to wait to take a PCIT client; other times therapists may have restrictions on referrals into their agency and may have to wait some time before an appropriate referral is made for PCIT (i.e., trauma-specific needs that have to be addressed quickly). Therefore, the time between the workshop and the first DPICS session varies across trainees. In the current study, the range of days between the workshop and the DPICS observation ranged from 14 to 296. The variability can mostly be accounted for by technological difficulties (e.g., recording not useable and time lapsed before the next appropriate referral), agency delay in referrals, or job requirements needing to be shifted before trainees could see PCIT clients. Future studies could assess the differences in lag time between the end of the workshop and when a therapist begins treatment with a family and how this could affect trainees? DPICS coding. There is one area of untapped research that could be valuable in the area of evidence- based treatment dissemination. Industrial/Organizational (I/O) Psychology researchers have investigated strategies for training for over 20 years and often refer to the concept of the transfer of training (Baldwin & Ford, 1988). Transfer of training has been broken down into two dimensions: generalization, or how knowledge and skills are applied, and maintenance, or how knowledge and skills learned persist over time (Blume, Ford, Baldwin, & Huang, 2010). Baldwin and Ford report that trainee characteristics, training design, and the work environment all contribute to learning and retention which leads to the transfer of training on the job. These same principles can be applied well to the dissemination efforts of PCIT and appear to be important factors for the current study as well. As demonstrated in the current study, trainee and training characteristics helped to explain a portion of the variance in using skills with actual clients. 44 Perhaps the I/O Psychology literature related to the transfer of training can serve as a theoretical grounding for the principles left under-investigated in the area of PCIT dissemination and implementation and other EBPP. Conclusion The traditional format for PCIT training is rooted in graduate programs within a mentor- mentee relationship and can develop for years. As there is more focus in the field of psychology on undergraduate and graduate education, lifelong learning efforts are ignored (Clay, 2010). PCIT trainers can develop successful models of training for professionals working in the community, fostering this lifelong learning. However, it is a challenge to replicate the apprenticeship model used in graduate programs, maintain EBPP training, and provide this service in an affordable format. It is clear that trainings need to extend beyond a workshop to active implementation strategies. The literature base pertaining to PCIT dissemination is very young, with only one published study with outcome data, which was published before the establishment of the PCIT Training Guidelines (Herschell, McNeil, Urquiza, McGrath, Zebell, Timmer, & Porter, 2009). There are no studies, to date, that evaluate PCIT implementation. The current study has some preliminary findings that could help the progress of the field as trainers try to establish the most effective methods of PCIT dissemination and implementation. Trainees feel confident and gain knowledge related to the core components of PCIT. However, the dissemination efforts are relatively easy compared to the more difficult implementation efforts, the latter of which there is less direction for trainers with regard to the most effective methods. The results of the current study suggest that trainers cannot rely on clinician self-report alone. Rather, the trainees? skill levels need to be evaluated through an objective measure, such as DPICS coding reliability based on work samples submitted by trainees. As demonstrated by the 45 discussion generated by this pilot study, there is a tremendous amount of work ahead of the PCIT community in the area of dissemination and implementation. 46 References Aarons, G. A. (2004). Mental health provider attitudes toward adoption of Evidence-Based Practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research,6(2), 61-74. American Psychological Association. (2005). American Psychological Association statement: Policy statement on evidence-based practice in psychology. Retrieved from http://www.apa.org/practice/guidelines/evidence-based.pdf Baldwin, T. T. & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41(1), 63-105. Bessmer, J. L. (1998). The Dyadic Parent-Child Interaction Coding System II (DPICS II): Reliability and validity (Unpublished doctoral dissertation). University of Florida, Gainsville, FL. Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta- analytic review. Journal of Management, 36, 1065-1105. doi: 10.1177/0149206309352880 Brestan, E. V., & Eyberg, S. M. (1998). Effective psychosocial treatments of conduct-disordered children and adolescents: 29 years, 82 studies, and 5,272 kids. Journal of Clinical Child Psychology, 27(2), 180-189. Chaffin, M., Silovsky, J. F., Funderburk, B., Valle, L. A., Brestan, E. V., Balachova, T., ...Bonner, B. L. (2004). Parent-Child Interaction Therapy with physically abusive 47 parents: Efficacy for reducing future abuse reports. Journal of Consulting and Clinical Psychology, 72(3), 500-510. doi: 10.1037/0022-006X.72.3.500 Chambless, D. L. & Ollendick, T. H. (2001). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685-716. Clay, R. A. (2010, December). Promoting lifelong learning in psychology. Monitor on Psychology, 64-65. Cohen, J., & Mannarino, A. P. (2008). Disseminating and implementing Trauma-Focused CBT in community settings. Trauma, Violence, & Abuse, 9(4), 214-226. Cohen, J. A., Mannarino, A. P., & Deblinger, E. (2006). Treating trauma and traumatic grief in children and adolescents. New York, NY: Guilford. Edwards, D. L, Schoenwald, S. K., Henggeler, S. W., & Strother, K. B. (2001). A multi-level perspective on the implementation of multisystemic therapy (MST): Attempting dissemination with fidelity. In G. A. Bernfeld, D. P. Farrington, & A. W. Leschied (eds.), Offender rehabilitation in practice: Implementing and evaluating effective programs (pp. 97-120). New York: J. Wiley & Sons. Eyberg, S. (1988). Parent-child interaction therapy: Integration of traditional and behavioral concerns. Child and Family Behavior Therapy, 10, 33-46. Eyberg, S. (1999). Parent-Child Interaction Therapy: Integrity checklists and session materials (Version 2.10). Unpublished manual. University of Florida, Gainesville, FL. Eyberg, S. M. & Matarazzo, R. G. (1980). Training parents as therapists: A comparison individual parent-child interaction training and parent group didactic training. Journal of Clinical Psychology, 36(2), 492-499. Eyberg, S. M., Nelson, M. M., Duke, M., & Boggs, S. R. (2004). Manual for the dyadic 48 parent-child interaction coding system: Third Edition. Unpublished manual. University of Florida, Gainsville, FL Eyberg, S. M. & Pincus, D. (1999). ECBI Eyberg Child Behavior Inventory and SESBI-R Sutter- Eyberg Student Behavior Inventory ? Revised, Professional manual. Odessa, FL: Psychological Assessment Resources, Inc. Eyberg, S. & Robinson, E. (1982). Parent-child interaction therapy: Effects on family functioning. Journal of Clinical Child Psychology, 11, 130-137. Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531-540. doi: 10.1177/1049731509335549 Foote, R. C. (2000). The Dyadic Parent-Child Interaction Coding System II (DPICS II): Reliability and validity with father-child dyads (Unpublished doctoral dissertation). University of Florida, Gainesville, FL. Friedberg, R. D., Gorman, A. A., & Beidel, D. C. (2009). Training psychologists for Cognitive- Behavioral Therapy in the raw world: A rubric for supervisors. Behavior Modification, 33(1), 104-123. doi: 10.1177/0145445508322609 Funderburk, B. W., Ware, L. M., Altshuler, E., & Chaffin, M. (2008). Use and feasibility of telemedicine technology in the dissemination of Parent-Child Interaction Therapy. Child Maltreatment, 13(4), 377-382. doi: 10.1177/1077559508321483 Henggeler, S. W. & Borduin, C. M. (1992). Multisystemic Therapy Adherence Scales. Unpublished manuscript, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, Charleston, SC. 49 Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and ?dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1(3), 171-184. Henggeler, S. W., Schoenwald, S. K., Liao, J. G., Letourneau, E. J., & Edwards, D. L. (2002). Transporting efficacious treatments to field settings: The link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child Psychology, 31(2), 155-167. Henggeler, S. W., Schoenwald, S. K., & Pickrel, S. G. (1995). Multisystemic Therapy: Bridging the gap between university- and community-based treatment. Journal of Consulting and Clinical Psychology, 63(5), 709-717. Herschell, A. D. (2010). Fidelity in the field: Developing infrastructure and fine-tuning measurement. Clinical Psychology: Science and Practice, 17(3), 253-257. doi: 10.1111/j.1468-2850.2010.01216.x Herschell, A. D., Kogan, J. N., Celedonia, K. L., Gavin, J. G., & Stein, B. D. (2009). Understanding community mental health administrators? perspectives on Dialectical Behavior Therapy implementation. Psychiatric Services, 60(7), 989-992. Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448-466. doi: 10.1016/j.cpr.2010.02.005 Herschell, A. D., McNeil, C. B., Urquiza, A. J., McGrath, J. M., Zebell, N. M., Timmer, S. G., & Porter, A. (2009). Evaluation of a treatment manual and workshops for disseminating, 50 Parent-Child Interaction Therapy. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 63-81. doi: 10.1007/s10488-008-0194-7 Huey, S. J., Henggeler, S. W., Brondino, M. J., & Pickrel, S. G. (2000). Mechanisms of change in Multisystemic Therapy: Reducing delinquent behavior through therapist adherence and improved family and peer functioning. Journal of Consulting and Clinical Psychology, 68(3), 451-467. doi: 10.1037//0022-006X.68.3.451 Lee, E. L., Wilsie, C. C., & Brestan-Knight, E. (in press). Using Parent-Child Interaction Therapy to develop a pre-parent education module. Children and Youth Services Review. doi: 10.1016/j.childyouth.2011.02.024 Mazzucchelli, T. G. & Sanders, M. R. (2010). Facilitating practitioner flexibility within an empirically supported intervention: Lessons from a system of parenting support. Clinical Psychology: Science and Practice, 17(3), 238-252. doi: 10.1111/j.1468- 2850.2010.01215.x McCabe, K. M. & Yeh, M. (2009). Parent-Child Interaction Therapy for Mexican Americans: A randomized clinical trial. Journal of Clinical Child and Adolescent Psychology, 38(5), 753-759. doi: 10.1080/15374410903103544 McDiarmid, M. D. & Bagner, D. M. (2005). Parent-Child Interaction Therapy for children with disruptive behavior and developmental disabilities. Education and Treatment of Children, 28(2), 130-141. McNeil, C. B. & Hembree-Kigin, T. L. (2010). Parent-Child Interaction Therapy (2nd ed.). New York: Springer. 51 Pincus, D. B., Santucci, L. C., Ehrenreich, J. T., & Eyberg, S. M. (2008). The implementation of modified Parent-Child Interaction Therapy for youth with Separation Anxiety Disorder. Cognitive and Behavioral Practice, 15(2), 118-125. doi: 10.1016/j.cbpra.2007.08.002 Sanders, M. R. (1999). Triple P-Positive Parenting Program: Towards an empirically validated multilevel parenting and family support strategy for the prevention of behavior and emotional problems in children. Clinical Child and Family Psychology Review, 2(2), 71- 90. Sanders, M. R., Murphy-Brennan, M., & McAuliffe, C. (2003). The development, evaluation, and dissemination of a training programme for general practitioners in evidence-based parent consultation skills. International Journal of Mental Health Promotion, 5, 13-20. Sanders, M. R., Tully, L. A., Turner, K. M. T., Maher, C., & McAuliffe, C. (2003). Training GPs in parent consultation skills: An evaluation of training for Triple P ? Positive Parenting Program. Australian Family Physician, 32, 1-6. Schumacher, J. A., Madson, M. B., & Norquist, G. S. (2011). Using telehealth technology to enhance Motivational Interviewing training rural substance abuse treatment providers: A services improvement project. The Behavior Therapist, 34(4), 64-70. Seng, A. C., Prinz, R. J., & Sanders, M. R. (2006). The role of training variables in effective dissemination of evidence-based parenting interventions. International Journal of Mental Health Promotion, 8(4), 20-28. Schoenwald, S. K., Henggeler, S. W., Brondino, M. J., & Rowland, M. D. (2000). Multisystemic Therapy: Monitoring treatment fidelity. Family Process, 39, 83-103. Schoenwald, S. K., Henggeler, S. W., & Edwards, D. L. (1998). MST Supervisor Adherence Measure. Unpublished manuscript, MST Institute, Charleston, SC. 52 Schuhmann, E. M., Foote, R. C., Eyberg, S. M., Boggs, S. R., & Algina, J. (1998). Efficacy of Parent-Child Interaction Therapy: Interim report of a randomized trial with short-term maintenance. Journal of Clinical Child Psychology, 27(1), 34-45. doi: 10.1207/s15374424jccp2701_4 Spielmans, G. I., Gatlin, E. T., & McFall, J. P. (2010). The efficacy of evidence-based psychotherapies versus usual care for youths: Controlling confounds in a meta-reanalysis. Psychotherapy Research, 20(2), 234-246. doi: 10.1080/10503300903311293 Vismara, L. A., Young, G. S., Stahmar, A. C., Griffith, E. M., & Rogers, S. J. (2009). Dissemination of evidence-based practice: Can we train therapists from a distance? Journal of Autism and Developmental Disorders,39, 1636-1651. doi: 10.1007/s10803- 009-0796-2 Webster-Stratton, C. & Hammond, M. (1990). Predictors of treatment outcome in parent training for families with conduct problem children. Behavior Therapy, 21(3), 319-337. Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61(7), 671-689. doi: 10.1037/0003-066X.61.7.671 Woltmann, E. M., Whitley, R., McHugo, G. J., Burnette, M., Torrey, W. C., Coots, L?.Drake, R. E. (2008). The role of staff turnover in the implementation of evidence-based practices in mental health care. Psychiatric Services, 59(7), 732-737. 53 Appendix 54 Table 1 Training Characteristics Across Training Times Training Number of Trainees Type of Agency Where Trainees Worked Time 1 12 Child Advocacy Center Time 2 5 Medical Center/Child Advocacy Center Time 3 6 Community Mental Health Center Time 4 8 Medical Center Total Number of Trainees = 31 55 Table 2 List of Trainee Numbers and Attrition Reasons Trainee Number Progress of Training If Dropped, Reason Why 1 Dropped Change in job responsibilities 2 In process 3 Dropped Change in job responsibilities 4 Dropped Left agency 5 Dropped Change in job responsibilities 6 In process 7 Dropped No appropriate referral base 8 Dropped No appropriate referral base 9 Dropped Change in job responsibilities 10 Dropped Change in job responsibilities 11 In process 12 Dropped Left agency 13 Dropped Difference in theoretical orientation 14 Dropped Difference in theoretical orientation 15 In process 16 In process 17 Completed 18 In process 19 Completed 20 In process 56 21 Completed 22 In process 23 In process 24 In process 25 In process 26 Dropped Left agency 27 In process 28 In process 29 In process 30 In process 31 Dropped Change in job responsibilities 57 Table 3 Totals for Completed, Drop-Outs, and In-Process Trainees Progress Number of Trainees Completed 3 Drop-Outs 13 In-Process 15 58 Table 4 Measure Characteristics Assessment Measure Strategy Scoring System PCIT Quiz Objective Multiple choice with four choices Learning Objectives Survey Self-Report 3-point Likert scale Competency Survey Self-Report 5-point Likert scale DPICS-III Objective Observational Coding Note. PCIT = Parent-Child Interaction Therapy, DPICS = Dyadic Parent-Child Interaction Coding System 59 Table 5 Listing of DPICS-III Codes Code Description Neutral Talk (TA) Statements that introduce information about people, objects, events, or activities, or indicate attention to the child, but do not clearly describe or evaluate the child?s current or immediately completed behavior. Behavior Description (BD) Non-evaluative, declarative sentences or phrases in which the subject of the sentence is the child and a verb describes the child?s ongoing or immediately completed (< 5sec.) observable verbal or nonverbal behavior. Reflective Statement (RF) A declarative statement that has the same meaning as a preceding child verbalization. The reflection may paraphrase or elaborate on the child?s verbalization but may not change the meaning of the child?s statement or interpret unstated ideas. Labeled Praise (LP) A verbalization expressing a positive evaluation of a specific behavior, activity, or product of the child. Unlabeled Praise (UP) A verbalization expressing a positive evaluation of the child, an attribute of the child, or a nonspecific activity, behavior, or product of the child. Question (QU) Verbal inquiries that are distinguishable from declarative statements by having a rising inflection at the end and/or by having 60 the sentence structure of a question. Questions request an answer but do not suggest that a behavior is to be performed by the child. Negative Talk (NTA) A verbal expression of disapproval of the child or the child?s attributes, activities, products, or choices. Also includes sassy, sarcastic, rude, or impudent speech. Command (CM) Statements in which the parent directs the vocal or motor behavior of the child. Commands may be direct or indirect in form. Note. DPICS = Dyadic Parent-Child Interaction Coding System 61 Table 6 List of Trainee Numbers and How Many Practice Sessions to Reach Reliability Trainee Number Number of Practice Sessions 1 1 2 7 3 1 4 3 5 3 6 10 7 3 8 7 9 4 10 4 11 2 12 2 13 7 14 3 15 2 16 3 17 1 18 4 19 4 20 6 62 21 4 22 1 23 6 24 1 25 6 26 2 27 5 28 4 29 3 30 5 31 1 M 3.7 SD 2.2 Note: M = Mean, SD = Standard Deviation 63 Table 7 Comparison of Mean Ratings from Pre- to Post-Training Measure Pre-Training Post-Training Statistics M SD M SD t p d PCIT Quiz 73.9 10.5 95.8 3.5 -11.70 <.001 -2.64 Learning Objectives Survey 42.7 9.8 83.4 10.8 -18.25 <.001 -3.85 Competency Survey 46.6 13.1 82.1 5.6 -16.22 <.001 -3.30 Note. PCIT = Parent-Child Interaction Therapy 64 Table 8 List of Trainee Number, Percent Reliable, and Inter-Rater Reliability on DPICS Coding Trainee Number Reliability IRR (if applicable) 2 73 % 77 % 8 74 % 11 39 % 84 % 12 65 % 13 69 % 14 70 % 15 78 % 16 85 % 17 94 % 83 % 18 58 % 82 % 19 65 % 20 86 % 21 83 % 22 76 % 89 % 29 80 % 30 68 % Note. IRR = Inter-Rater Reliability, DPICS = Dyadic Parent-Child Interaction Coding System 65 Table 9 Potential Predictors of Coding Reliability with Actual Clients Variable R square value p value Education Level .078 .296 Training Site .064 .345 Average DPICS Reliability During the Workshop .034 .494 Number of Courses Taken to Learn About Parent Training .011 .696 Hours Per Week of Direct Client Contact .005 .799 Years of Experience Working with Children and Families .002 .873 Number of Practice Session Taken to Reach 80% Reliability .000 .979 Note. DPICS = Dyadic Parent-Child Interaction Coding System 66 Table 10 Best Sets of Predictors with Respective R Squared Values Number in Model Variables R square value 2 Education, Training Site .286 3 Education, Training Site, Years of Experience .451 4 Education, Training Site, Hours of Direct Client Contact, Average DPICS Reliability During the Workshop .536 5 Education, Training Site, Hours of Direct Client Contact, Years of Experience, Average DPICS Reliability During the Workshop .567 6 Education, Training Site, Hours of Direct Client Contact, Years of Experience, Number of Courses Taken to Learn About Parent Training, Average DPICS Reliability During the Workshop .583 7 Education, Training Site, Hours of Direct Client Contact, Years of Experience, Number of Courses Taken to Learn About Parent Training, Average DPICS Reliability During the Workshop, Number of Practice Sessions Taken To Reach 80% Reliability .591 Note. DPICS = Dyadic Parent-Child Interaction Coding System