THE ROLE OF EMOTION IN COMPUTER SKILL ACQUISITION Except where reference is made to the work of others, the work described in this dissertation is my own or was done in collaboration with my advisory committee. This dissertation does not include any proprietary or classified information. ____________________________________ John Kenneth Corley II Certificate of Approval: ________________________ ________________________ Thomas E. Marshall R. Kelly Rainer, Jr., Chair Associate Professor Professor Management Management ________________________ ________________________ F. Nelson Ford George T. Flowers Associate Professor Interim Dean Management Graduate School THE ROLE OF EMOTION IN COMPUTER SKILL ACQUISITION John K Corley II A Dissertation Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 9, 2008 iii THE ROLE OF EMOTION IN COMPUTER SKILL ACQUISITION John Kenneth Corley II Permission is granted to Auburn University to make copies of this dissertation at its discretion, upon request of individuals or institutions and at their expense. The author reserves all publication rights. ________________________ Signature of Author ________________________ Date of Graduation iv VITA John Kenneth Corley II was born in Colorado Springs, Colorado on June 9, 1968, the son of John Kenneth Corley, Sr. and Janice Lillian Corley. He graduated from the University of Georgia in 1996 with a B.F.A. in interdisciplinary studies. In 2001, he graduated from Augusta State University with a Master?s of Business Administration. After working as an art director at Augusta Sportswear, a member of an in-house marketing group at Kennametal IPG, and general manager of Beaed Corporation, he entered graduate school at Auburn University. In 1998, he married the former Tammy Michelle Townsend of Augusta, Georgia. They have a son, John Kenneth Corley III (9), and a daughter, Hannah Leanne Corley (7). v DISSERTATION ABSTRACT THE ROLE OF EMOTION IN COMPUTER SKILL ACQUISITION John K. Corley II Doctor of Philosophy, August 9, 2008 (B.F.A., University of Georgia, 1996) (M.B.A., Augusta State University, 2001) 148 Typed Pages Directed by Rex Kelly Rainer, Jr. In response to recent calls for researchers to identify the underlying causes of training effectiveness, this study investigated the role of emotion in computer skill acquisition (CSA). The current study introduced empathetic emotion as a behavior model training (BMT) intervention within the observational learning model (OLM) of CSA developed and validated by Yi and Davis (2003). Recent research studies indicate that emotion is an integral part of cognitive processing and decision making in humans because emotion focuses attention on stimuli that are relevant to needs and goals. Research also indicates that emotional arousal tends to increase human memory. Attention and memory are very similar to the attention and retention process dimensions in Yi and Davis? (2003) OLM. Therefore, this study posited vi that an emotion-based training intervention would increase attention and memory thereby improving training outcomes. A training workshop, including a custom software simulation, was developed for the current research study. The software simulation recorded data related to training effectiveness and randomly assigned trainees to either a treatment group (receiving emotion-based performance feedback) or control group (receiving performance feedback void of emotion). Data related to individual differences, perceptions, and attitudes were collected using paper-based survey questions. Empirical testing of the proposed theoretical model and hypotheses was conducted using PLS-Graph, and multi-group comparisons were conducted using t-tests. The test results of the proposed model were favorable, but no significant differences were found between the training outcomes for the control and treatment groups. The findings of the empirical model indicate that emotion has a significant affect on learning processes, and highlights the need to better understand the role of emotion within the context of computer skill training. vii ACKNOWLEDGEMENTS This dissertation would not have been possible without the support of several individuals. To my wife, Tammy, and my children, John and Hannah, thank you for your constant love and support throughout this process. Without the sacrifices you have endured over the past few years this dissertation would not have been possible. It is to the three of you that I dedicate this dissertation. I would also like to thank my parents, Ken and Janice Corley, and my in-laws, Paul and Debra West, for your emotional and financial support. I would like to acknowledge the patience, encouragement, and intellectual stimulation generously provided by the chair of this dissertation, Dr. Kelly Rainer. It was a pleasure to work with you, and I am honored that you have served as Chair of my committee. To the members of the dissertation committee, I owe each of you a sincere thank you. The countless hours of consultation were instrumental to the quality of this research. To the other three musketeers, Jennifer Pitts, Frank Morris, and Evelyn Thrasher (my classmates), I cannot imagine trying to navigate the rigors of the PhD program without such an outstanding group of friends. To Dean Marc Miller, I owe you a sincere debt of gratitude for your constant encouragement and support over the years. You are a tremendous mentor and friend. viii Style manual used: Publication Manual of the American Psychological Association, Fifth Edition Computer software used: Microsoft Word 2003, EndNote 9.0 for Windows, Microsoft Excel 2003, Microsoft PowerPoint 2003, SPSS 13.0 for Windows, Amos 5.0, PLS-Graph 3.0. ix TABLE OF CONTENTS LIST OF TABLES ........................................................................................................... xii LIST OF FIGURES ......................................................................................................... xiv CHAPTER I INTRODUCTION......................................................................................... 1 Theoretical Foundation and Definitions ................................................................. 2 Behavior Modeling-Based Training ........................................................... 2 Contribution to the Field......................................................................................... 5 Extending the Conceptual Framework........................................................ 5 Research Objectives and Plan..................................................................... 7 Organization of the Dissertation ............................................................................. 9 CHAPTER II LITERATURE REVIEW .......................................................................... 10 Research Model and Hypotheses.......................................................................... 12 Behavior Modeling In Computer Skill Training....................................... 12 Training Outcomes.................................................................................... 14 The OLM of CSA ..................................................................................... 16 Pre-Training Individual Differences......................................................... 18 Modeling-Based Training Interventions................................................... 20 Conclusion ............................................................................................................ 27 CHAPTER III METHOD ................................................................................................. 30 Participants............................................................................................................ 31 Procedure .............................................................................................................. 32 x Design ................................................................................................................... 34 The Embodied Computer Agent ............................................................... 34 Measures ............................................................................................................... 35 Training Outcomes.................................................................................... 35 Observational Learning Processes ............................................................ 39 Pre-Training Individual Differences......................................................... 39 Emotional Response: Perceptions of Agent Caring.................................. 40 Statistical Analysis.................................................................................... 45 CHAPTER IV RESULTS................................................................................................. 47 Data Preparation.................................................................................................... 47 Participant Response................................................................................. 47 Psychometric Properties of Measures................................................................... 50 Confirmatory Factor Analysis................................................................... 50 Test of Model and Hypotheses ............................................................................. 55 Submodel One........................................................................................... 56 Submodel Two.......................................................................................... 59 Hypothesis Tests ....................................................................................... 62 Multiple Group Analysis........................................................................... 66 CHAPTER V DISCUSSION............................................................................................ 72 Findings for Full Sample ...................................................................................... 73 Findings for the Control and Treatment Groups................................................... 73 Training Outcomes.................................................................................... 73 Pre-training Individual Differences .......................................................... 74 Training Intervention ................................................................................ 76 xi Implications........................................................................................................... 77 Limitations ............................................................................................................ 78 Future Research .................................................................................................... 79 Conclusion ............................................................................................................ 80 REFERENCES ................................................................................................................. 82 APPENDICES .................................................................................................................. 91 APPENDIX A INSTRUCTIONAL DESIGN PROCESS................................................ 92 APPENDIX B PARTICIPANT RECRUITMENT........................................................... 99 APPENDIX C SCRIPT FOR TRAINING WORKSHOP FACILITATOR................... 101 APPENDIX D SOFTWARE SIMULATION DEVELOPMENT.................................. 108 APPENDIX E PRE-TRAINING SURVEY INSTRUMENTS ...................................... 120 Pre-Training Motivation to Learn Survey Instrument ........................................ 121 Pre-Training Software Self-Efficacy Survey Instrument.................................... 122 APPENDIX F POST TRAINING SURVEY INSTRUMENTS .................................... 123 Post-Training Perceptions of Computer Agent Survey Instrument .................... 124 Post-Training Observational Learning Processes Survey Instrument ................ 125 Post-Training Software Self-Efficacy Survey Instrument .................................. 126 APPENDIX G SCRIPT OF IN-CLASS RECRUITMENT ANNOUNCEMENT......... 127 APPENDIX H FOLLOW-UP EMAIL ANNOUNCEMENT FOR PARTICIPANT RECRUITMENT ............................................................................................................ 129 APPENDIX I INFORMATION LETTER FOR PARTICIPANTS................................ 131 xii LIST OF TABLES TABLE 1........................................................................................................................... 29 Summary of Study Hypotheses............................................................................. 29 TABLE 2........................................................................................................................... 35 Summary of Performance-based Feedback in the Software Simulation .............. 35 TABLE 3........................................................................................................................... 38 Adopted and Modified Measures of Task-Specific Software Self-Efficacy ........ 38 TABLE 4........................................................................................................................... 41 Summary of Study Measures................................................................................ 41 TABLE 5........................................................................................................................... 48 Descriptive Statistics............................................................................................. 48 TABLE 6........................................................................................................................... 53 Reliabilities, Convergent and Discriminant Validities, and Correlations of Latent Constructs - Measurement Model......................................................................... 53 TABLE 7........................................................................................................................... 54 Factor Structure Matrix of Loadings and Cross-Loadings - Measurement Model54 TABLE 8........................................................................................................................... 57 Construct Correlations for All Data, Control Group, and Treatment Group ? Submodel 1 ........................................................................................................... 57 TABLE 9........................................................................................................................... 58 Indicator Weights and Loadings for All Data, Control Group, and Treatment Group ? Submodel 1 ............................................................................................. 58 xiii TABLE 10......................................................................................................................... 59 Construct Correlations for All Data, Control Group, and Treatment Group ? Submodel 2 ........................................................................................................... 59 TABLE 11......................................................................................................................... 60 Indicator Weights and Loadings for All Data, Control Group, and Treatment Group ? Submodel 2 ............................................................................................. 60 TABLE 12......................................................................................................................... 67 Results of Hypothesis Tests for All Data, Control Group, and Treatment Group 67 TABLE 13......................................................................................................................... 68 Significant Tests of Group Differences ? Path Coefficients................................. 68 TABLE 14....................................................................................................................... 116 xiv LIST OF FIGURES Figure 1. Yi and Davis? Conceptual Framework............................................................... 2 Figure 2. Current Contribution to Yi and Davis? Conceptual Framework ........................ 8 Figure 3. Training Outcomes: Declarative Knowledge, Post-Training Software Self-Efficacy, and Immediate Task Performance are the Constructs Representing Training Outcomes within the Research Framework ................ 16 Figure 4. Formative Model Representing the Observational Learning Processes within the Research Framework...................................................... 17 Figure 5. Pre-Training Motivation to Learn and Pre-Training Software Self- Efficacy Represent Individual Differences within the Research Framework ....................................................................................................... 19 Figure 6. LeDoux?s Model of the Neurological Structure of Emotion............................ 21 Figure 7. Perceptions of Caring are Antecedents of Empathetic Emotional Response and Represent an Emotion-Based Training Intervention within the Research Framework....................................................................... 27 Figure 8. Complete Research Model ................................................................................ 28 xv Figure 9. Experimental Procedure ................................................................................... 33 Figure 10. Test of Proposed Model Using the Full Data Sample Data Sample................ 69 Figure 11. Test of Proposed Model for Treatment Group ................................................ 70 Figure 12. Test of Proposed Model for Control Group .................................................... 71 Figure 13. Initial Login Screen of the Custom Software Simulation Program............... 109 Figure 14. Second Screen of the Custom Software Simulation Program....................... 110 Figure 15. Introduction Screen of the Custom Software Simulation Program............... 111 Figure 16. ?Happy? Emotion Settings for People Putty Character................................. 113 Figure 17. "Sad" Emotion Settings for People Putty Character...................................... 113 Figure 18. "Neutral" Emotion Settings for People Putty Character................................ 114 Figure 19. Sample Presentation Window of the Custom Software Simulation Program.......................................................................................................... 115 1 CHAPTER I INTRODUCTION Organizations dedicate a significant portion of their annual budgets to maintaining and upgrading their information technology (IT). In fact, organizations in the United States spend over 50 billion dollars annually on training (Galvin 2001; Dolezalek 2005), and computer skill training represents the most frequent type of training provided to employees (Galvin 2001; Yi and Davis 2003). There are many approaches to facilitating computer skill acquisition (CSA), but the underlying causes of training effectiveness are not well understood (Martocchio and Webster 1992; Davis and Bostrom 1993; Olfman and Mandviwalla 1994; Santhanam and Sein 1994; Compeau and Higgins 1995; Lim, Ward et al. 1997; Yi and Davis 2003). Yi and Davis (2001) surveyed relevant research studies, noting that behavior modeling consistently yields better CSA when compared to other computer software training methods. In model-based training an instructor or trainer demonstrates the target behavior for individual learners, who then immediately attempt to repeat the observed target behavior. Yi and Davis (2003) explored this phenomenon by developing and validating an observational learning model of CSA. In brief, Yi and Davis (2003) presented their conceptual framework to ?provide an opportunity for IS researchers to identify, prioritize, and verify the effectiveness of specific techniques for improving 2 model-based training for computer skills? and thereby overcome the constraints of ?limited existing knowledge about the psychological mechanisms underlying model- based training? for CSA (see Figure 1). Figure 1. Yi and Davis? Conceptual Framework Theoretical Foundation and Definitions Behavior Modeling-Based Training Learning Outcomes Colquitt, LePine, et al.?s (2000) meta-analysis of 106 training studies found that the three most commonly examined outcomes of training research are: (a) declarative knowledge; (b) task performance; and (c) post-training efficacy. Yi and Davis (2003) include these three training outcomes in their Observational Learning Processes Model (OLM) of CSA positing that ?declarative knowledge and post-training self-efficacy function as two distinct causal mechanisms by which training interventions may influence performance.? Modeling-Based Training Interventions Observational Learning Processes Training Outcomes Pre-training Individual Differences 3 Post-training Software Self-Efficacy and Task Performance. ?Self-efficacy refers to one?s perceived performance capabilities for a specific activity (Kraiger et al., 1993, p.320).? Bandura?s (1986, 1997) social cognitive theory posits that self-efficacy is a major determinant of an individual?s performance. Researchers have posited that self- efficacy performs a self-regulatory and motivational role in controlling behavioral performance of acquired skills (Ackerman et al., 1995; Kanfer and Ackerman, 1989; Mitchell et al., 1994). Furthermore, Yi and Davis (2003) note that research on computer training found post-training software self-efficacy to be a significant predictor of task performance (Compeau and Higgings, 1995a; Gist et al., 1989; Johnson and Marakas, 2000; Martocchio and Judge, 1997; Martocchio and Webster, 1992). Declarative Knowledge and Task Performance. Many researchers have theorized about the process of an individual?s CSA, suggesting that knowledge evolves from a declarative or propositional form, through knowledge compilation, toward an automatic or procedualized form (Anderson, 1982; Glaser, 1990; Kanfer and Ackerman, 1989; Kozlowski et al., 2001; Kraiger et al., 1993; Martocchio, 1994; May and Kahnweiler, 2000; Olfman and Mandviwalla, 1994). Anderson defines declarative knowledge (1985, p. 199) as ?knowledge about facts and things.? ?Information content is obtained in the declarative phase either by verbal specifications of task objectives and instructions, or trainees ?may observe demonstrations of the task, may encode and store task rules, and may derive strategies for the task (Kanfer & Ackerman, 1989, P. 660)?? (Yi and Davis, 2003). We expect the training process to have a positive effect on training outcomes including post-training task performance. 4 Observational Learning Processes Yi and Davis? (2003) observational learning processes model (OLM) of computer software training and skill acquisition (CSA), is based on Bandura?s (1986) social cognitive theory which posits that four observational learning processes are responsible for the effects of modeling-based training: ?(1) attention processes regulate exploration and perception of modeled activities; (2) through retention processes, transitory experiences are converted for memory representation into symbolic conceptions that serve as internal models for response production and standards for response correction; (3) production processes govern the organization of constituent subskills into new response patterns; and (4) motivation processes determine whether or not observationally acquired competencies will be put to use? (Bandura 1986). Pre-Training Individual Differences Researchers have suggested that it is important to control for pre-training individual difference variables (Bostrom, Olfman et al. 1990; Olfman and Bostrom 1991; Venkatesh and Morris 2000). Motivation and self-efficacy are two pre-training individual differences commonly used in training research. Findings in a number of research studies suggest that measuring pre-training motivation to learn is a useful method for predicting training effectiveness (Baldwin, Magjuka et al. 1991; Colquitt, LePine et al. 2000). Yi and Davis (2003) defined pre- training motivation to learn as ?a trainee?s desire to master the content of the training program? (Noe 1986; Noe and Schmitt 1986). Yi and Davis (2003) noted that Bandura?s (1986) social cognitive theory ?argues that motivation to learn modeled skills increases trainees? active engagement in all four observational learning processes.? 5 Self-efficacy has been widely used and accepted within the field of training research (Colquitt, LePine et al. 2000). Following Yi and Davis (Yi and Davis 2003), this study also adopts the definition of self-efficacy presented by Kraiger et al. (1993), which states ?self-efficacy refers to one?s perceived performance capabilities for a specific activity.? Within the field of information systems research, pre-training software self- efficacy is reported to be a significant predictor of post-training software self-efficacy (Martocchio and Webster 1992; Martocchio 1994; Martocchio and Dulebohn 1994; Marshall, Rainer et al. 2003; Yi and Davis 2003). Contribution to the Field Extending the Conceptual Framework Recent studies in the field of neurology strongly indicate that emotion is an essential part of cognitive processing (LeDoux 1996). Consequently, emotion is believed to play a key role in decision making and learning processes because it focuses our attention on objects and situations that are most important to our needs and goals (Brave and Nass 2003). This increased attention focus also impacts memory, because emotional stimuli are generally remembered better than unemotional events (Thorson and Friedsad 1985). Emotional stimuli tend to be remembered better than unemotional events, because emotion focuses thought and attention on the causal events and situations (Thorson and Friedsad 1985). That is, emotion directs and focuses attention on objects or events that are considered relevant to meeting needs and attaining goals so that we can deal with them appropriately (Brave and Nass 2003). Higher levels of relevance or importance lead to higher levels of emotional arousal which result in a more forceful focus of attention 6 (Clore and Gasper 2000). Therefore, it follows that memory could improve CSA when a learner?s emotions are activated in response to their performance on a given task. Affective Computing ?Whether emotional expressivity by computers is or is not beneficial in human- computer interaction (HCI)? has become an important debate within the HCI community (Brave, Nass et al. 2005). On one side of the debate, HCI researchers contend that emotional expressivity is unnecessary at best and potentially both irritating and distracting (Brave & Nass, 2005). Other researchers who have pioneered the concept of affective computing (Bates 1994; Picard 1997), contend that incorporating emotional expressivity into the HCI ?is necessary to best leverage users? life-time of experience with social interaction (Brave, Nass et al. 2005)?. Picard (1997, p.3) defines affective computing as ?computing that relates to, arises from, or deliberately influences emotion.? Computer Agents There can be advantages to an interface that empathizes with users while attempting to meet their needs and goals (Klein, Moon et al. 1999). Brave, Nass, & Hutchinson (2005) recently conducted a HCI study that incorporated empathy into an HCI using an embodied (virtual) computer agent. Participants in the study reported higher levels of caring, likeability, and trust when ?other-oriented? empathetic text messages from an embodied computer agent were presented to participants. Brave et al. (2005) suggested that future research should investigate the use of empathetic emotion exhibited by an embodied computer agent in a computer assisted learning environment. Brave et al. (2005) noted that ?caring [computer] agents have an advantage in persuading and motivating users for two reasons: (a) people tend to trust information 7 coming from sources that care about them, and (b) caring tends to be reciprocal and people are more willing to comply with those they care about.? They suggest that emotional behavior could be integrated into a HCI through an embodied computer agent presented as a caring tutor or teacher. If an embodied computer agent elicits positive emotional arousal through ?other-oriented? empathetic language, it could have a significant impact on the interaction and subsequent performance of learners in a computer mediated learning environment. Batson et al. (1997) defines empathy as ?an other-oriented emotional response congruent with another?s perceived welfare.? This definition generally assumes that empathy is based on altruistic motives and caring for another. We expect the expression of empathetic emotion by a computer agent to have a significant and positive effect on training outcomes. Research Objectives and Plan As previously noted, emotion can bias cognitive processing, thereby influencing individual attention and memory which are very similar to the attention and retention dimensions of observational learning processes as outlined in Bandura?s (1986) social cognitive theory. Therefore, social cognitive theory further supports our claim that emotion can influence training outcomes. We introduced empathetic emotion as a behavior model training (BMT) intervention and expect training outcomes to improve due to the affect of emotion on attention and memory. In brief, the purpose of this study is to investigate the potential enhancement of users? computer skill acquisition when emotion is incorporated into BMT as a training intervention. The current study contributes to the literature dedicated to studying BMT by investing this training 8 intervention within the framework developed and validated by Yi and Davis (2003) (see Figure 2). Methodology A training workshop was developed for the current research study. The workshop focused on CSA for a specific set of software skills. Trainees watched a pre-recorded video of a trainer modeling the targeted software skills. Immediately following the video demonstration trainees used a custom software simulation designed to assess declarative knowledge and task performance. An embodied computer agent within the software simulation instructed the trainees to perform tasks and to answer multiple choice questions. The trainees were randomly assigned to either a treatment or control group. There were no differences between the two groups with one exception. The embodied computer agent provided immediate feedback based on performance, and the treatment group received other-oriented empathetic feedback while the control group received emotionally neutral feedback. Pre- and post-training data related to individual differences, perceptions of agent caring, software self-efficacy, and attitude were collected using paper-based survey questions. Figure 2. Current Contribution to Yi and Davis? Conceptual Framework Emotion-Based Training Interventions Observational Learning Processes Training Outcomes Pre-training Individual Differences 9 Organization of the Dissertation This research is presented in five chapters; the content of each chapter is briefly outlined below: Chapter I introduced and described the problem under investigation in the current research study. In addition, the previous discussion addressed the importance of this study in regard to its contribution to the IT research literature and its implications for both practitioners and researchers. The conceptual model was presented, and the proposed methodology was briefly summarized. Chapter II provides a theoretical foundation and support for this research study through a review of relevant literature. Theoretical support for the proposed model is discussed, and past literature is used to further support the research framework, the proposed research model, and to develop testable hypotheses. Chapter III describes the methodology to be used within this research study. The sample of interest is described, the experimental procedure is presented, the instruments adopted and developed for the experimental treatment is described, and the process of data collection is discussed. Chapter IV presents the empirical results of the data analysis and hypothesis tests. Descriptive statistics and test results are described and summarized. Chapter V provides as discussion of the research results and offers insights regarding the implications for future research and practical application in the corporate sector. The major contributions and limitations of the study are discussed. The Chapter concludes with a summary of the study and its findings. 10 CHAPTER II LITERATURE REVIEW Organizations dedicate a significant portion of their annual budgets to maintaining and upgrading their information technology (IT). Unfortunately, managers often fail to realize the expected returns on IT investment when employees fail to effectively use IT. Motorola estimates a thirty dollar gain in productivity over a period of three years for each dollar spent on employee training (Kirkpatrick 1993; Yi and Davis 2003). Organizations in the United States spend over 50 billion dollars annually on training (Galvin 2001; Dolezalek 2005), and computer skill training represents the most frequent type of training provided to employees (Galvin 2001; Yi and Davis 2003). There are many approaches to facilitating computer skill acquisition (CSA), but the underlying causes of training effectiveness are not well understood (Martocchio and Webster 1992; Davis and Bostrom 1993; Olfman and Mandviwalla 1994; Santhanam and Sein 1994; Compeau and Higgins 1995; Lim, Ward et al. 1997; Yi and Davis 2003). Yi and Davis (2001) surveyed relevant research studies, noting that some consistencies existed among studies that investigated the effectiveness of the various methods used to facilitate CSA (Yi and Davis 2001; Yi and Davis 2003). One such consistency is that behavior modeling yields better CSA than (a) traditional lecture-based instruction (Compeau and Higgins 1995; Simon, Grover et al. 1996; Simon and Werner 11 1996; Johnson and Marakas 2000; Bolt, Killough et al. 2001), (b) computer-aided instruction (Gist, Rosen et al. 1988; Gist, Schwoerer et al. 1989), and (c) self-study from a manual (Gist, Rosen et al. 1988; Gist, Schwoerer et al. 1989; Simon, Grover et al. 1996; Simon and Werner 1996). In model-based training an instructor or trainer demonstrates the target behavior for individual learners, who then immediately attempt to repeat the observed target behavior. Behavior modeling training (BMT) intervention is one of the most widely used and highly regarded methods of psychologically-based training intervention (Taylor, Russ-Eft et al. 2005). Yi and Davis (2003) explored this phenomenon by developing and validating an observational learning model of CSA which is ?designed to trace the influence of modeling-based interventions on training outcomes through their effects on observational learning processes (OLP).? The conceptual framework for their model is presented in Figure 1. In brief, Yi and Davis (2003) presented their conceptual framework to ?provide an opportunity for IS researchers to identify, prioritize, and verify the effectiveness of specific techniques for improving model-based training for computer skills? and thereby overcome the constraints of ?limited existing knowledge about the psychological mechanisms underlying model-based training? for CSA. This study argues that introducing emotion as a BMT intervention will increase attention and memory during training which will lead to better training outcomes. Few research studies address the psychology of emotion (Gross 1999; Zhang and Li 2005). However, recent studies in the field of neurology strongly indicate that emotion is an essential part of cognitive processing (LeDoux 1996). Consequently, emotion is believed to play a key role in decision making and learning processes because it focuses 12 our attention on objects and situations that are most important to our needs and goals (Brave and Nass 2003). This increased attention focus also impacts memory, because emotional stimuli are generally remembered better than unemotional events (Thorson and Friedsad 1985). Furthermore, evidence suggests that computer users ?orient towards computers as social actors, much as we orient socially towards other humans? (Reeves and Nass 1998; Nass and Moon 2000). ?Whether emotional expressivity by computers is or is not beneficial in HCI? has become an important debate within the HCI community (Brave, Nass et al. 2005). On one side of the debate, HCI researchers contend that emotional expressivity is unnecessary at best and potentially both irritating and distracting (Brave & Nass, 2005). Other researchers who have pioneered the concept of affective computing (Bates 1994; Picard 1997), contend that incorporating emotional expressivity into the HCI ?is necessary to best leverage users? life-time of experience with social interaction (Brave, Nass et al. 2005)?. Picard (1997, p.3) defines affective computing as ?computing that relates to, arises from, or deliberately influences emotion.? The purpose of this study is to investigate the potential enhancement of users? computer skill acquisition when emotion is incorporated into BMT as a training intervention (see Figure 2). Research Model and Hypotheses Behavior Modeling In Computer Skill Training Figure 2.2 represents an adaptation of Yi and Davis? (2003) original theoretical framework for OLP, and posits that emotion-based behavior modeling interventions will improve training outcomes due to their effects on attention and memory. Many training 13 methods are currently used to facilitate the acquisition of computer skills (Gattiker 1992), but the underlying causes of training effectiveness are not well understood (Martocchio and Webster 1992; Davis and Bostrom 1993; Olfman and Mandviwalla 1994; Santhanam and Sein 1994; Compeau and Higgins 1995; Lim, Ward et al. 1997; Yi and Davis 2003). Yi and Davis (2003) point out that within the literature, behavior model training (BMT) is consistently reported to have a significant positive impact on training outcomes and specifically in computer skill training. Behavior modeling training (BMT) intervention is one of the most widely used and highly regarded methods of psychologically-based training intervention (Taylor, Russ-Eft et al. 2005). BMT is based on Bandura?s (1977) social learning theory, and emphasizes five training approaches, which include (a) describing to trainees a set of well-defined behaviors (skills) to be learned, (b) providing a model or models displaying the effective use of those behaviors, (c) providing opportunities for trainees to practice using those behaviors, (d) providing feedback and social reinforcement to trainees following practice, and (e) taking steps to maximize the transfer of those behaviors to the job (Goldstein and Sorcher 1973; Robinson 1982; Decker and Nathan 1985; Taylor, Russ-Eft et al. 2005). These five training approaches provide a very systematic approach to developing and executing training sessions designed to facilitate CSA. Consequently, adaptation of the validated BMT model offers a well founded vehicle for testing the effectiveness of emotion-based training interventions within the context of the current study. 14 Training Outcomes Yi and Davis (2003) note that Colquitt et al.?s (2000) meta-analysis of 106 training studies found that the three most commonly examined outcomes of training research are declarative knowledge, task performance, and post-training efficacy. Yi and Davis (2003) include these three training outcomes in their Observational Learning Processes Model (OLM) of CSA positing that ?declarative knowledge and post-training self-efficacy function as two distinct causal mechanisms by which training interventions may influence performance.? Post-training Software Self-Efficacy and Task Performance ?Self-efficacy refers to one?s perceived performance capabilities for a specific activity (Kraiger et al., 1993, p.320).? Bandura?s (1986, 1997) social cognitive theory posits that self-efficacy is a major determinant of an individual?s performance. Research has indicated that self-efficacy and declarative knowledge is separate and distinct constructs (Kraiger et al., 1993; Marcolin et al., 2000; Martocchio, 1994). Furthermore, researchers have posited that self-efficacy performs a self-regulatory and motivational role in controlling behavioral performance of acquired skills (Ackerman et al., 1995; Kanfer and Ackerman, 1989; Mitchell et al., 1994). Research studies have often reported significant relationships between self-efficacy and performance (Ackerman et al., 1995; Kraiger et al., 1993; Mitchell et al., 1994, Salas and Cannon-Bowers, 2001). Yi and Davis (2003) note that research on computer training found post-training software self-efficacy to be a significant predictor of task performance (Compeau and Higgings, 1995a; Gist et al., 1989; Johnson and Marakas, 2000; Martocchio and Judge, 1997; Martocchio and Webster, 1992). 15 Declarative Knowledge and Task Performance Many researchers have theorized about the process of an individual?s CSA. Such theories suggest that knowledge evolves from a declarative or propositional form, through knowledge compilation, toward an automatic or procedualized form (Anderson, 1982; Glaser, 1990; Kanfer and Ackerman, 1989; Kozlowski et al., 2001; Kraiger et al., 1993; Martocchio, 1994; May and Kahnweiler, 2000; Olfman and Mandviwalla, 1994). Anderson defines declarative knowledge (1985, p. 199) as ?knowledge about facts and things.? ?Information content is obtained in the declarative phase either by verbal specifications of task objectives and instructions, or trainees ?may observe demonstrations of the task, may encode and store task rules, and may derive strategies for the task (Kanfer & Ackerman, 1989, P. 660)?? (Yi and Davis, 2003). Therefore, in the declarative stage individuals develop an initial surface-level, cognitive representation of the task. During the knowledge compilation phase of the process ?persons integrate the sequences of cognitive and motor processes required to perform the task? (Kanfer & Ackerman, 1989, P. 660). Knowledge structures for organizing and processing knowledge are developed and refined as they progress toward procedural knowledge (Anderson, 1985; Kraiger et al., 1993). Post-Training Task Performance We expect the training process to have a positive effect on training outcomes including post-training task performance. However, it is rather unlikely that trainees will move beyond the early stages of skill acquisition to automaticity immediately following training (Ackerman 1987; Glaser 1990) (See Figure 3). Therefore, we hypothesize: 16 Hypothesis 1: Declarative Knowledge will positively influence immediate task performance. Hypothesis 2: Post-Training Software Self-Efficacy will positively influence immediate task performance. Figure 3. Training Outcomes: Declarative Knowledge, Post-Training Software Self- Efficacy, and Immediate Task Performance are the Constructs Representing Training Outcomes within the Research Framework The OLM of CSA Yi and Davis (2003) recently developed and validated a formative observational learning processes model (OLM) of computer software training and skill acquisition (CSA) which is presented in Figure 4. Their OLM was based on Bandura?s (1986) social cognitive theory which posits that four observational learning processes are responsible for the effects of modeling-based training: ?(1) attention processes regulate exploration and perception of modeled activities; (2) through retention processes, transitory experiences are converted for memory representation into symbolic conceptions that serve as internal models for response production and standards for response correction; Post-Training Software Self- Efficacy Declarative Knowledge Immediate Task Performance 17 (3) production processes govern the organization of constituent subskills into new response patterns; and (4) motivation processes determine whether or not observationally acquired competencies will be put to use? (Bandura 1986). As previously noted, emotion can bias cognitive processing, thereby influencing individual attention and memory which are very similar to the attention and retention dimensions of observational learning processes as outlined in Bandura?s (1986) social cognitive theory. Therefore, social cognitive theory further supports our claim that emotion can influence training outcomes. Therefore, we hypothesize: Hypothesis 3: Observational Learning Processes will positively influence declarative knowledge. Hypothesis 4: Observational Learning Processes will positively influence Post-Training Software Self-Efficacy. Figure 4. Formative Model Representing the Observational Learning Processes within the Research Framework Attention Processes Observational Learning Processes Retention Processes Production Processes Motivation Processes 18 Pre-Training Individual Differences Pre-Training Motivation to Learn Researchers have suggested that it is important to control for pre-training individual difference variables (Bostrom, Olfman et al. 1990; Olfman and Bostrom 1991; Venkatesh and Morris 2000). Findings in a number of research studies suggest that measuring pre-training motivation to learn is a useful method for predicting training effectiveness (Baldwin, Magjuka et al. 1991; Colquitt, LePine et al. 2000). Yi and Davis (2003) defined pre-training motivation to learn as ?a trainee?s desire to master the content of the training program? (Noe 1986; Noe and Schmitt 1986). Yi and Davis (2003) noted that Bandura?s (1986) social cognitive theory ?argues that motivation to learn modeled skills increases trainees? active engagement in all four observational learning processes.? The findings in Yi and Davis? (2003) study support the argument that motivation to learn modeled skills increases trainees? active engagement in all four observational learning processes (See Figure 5). Therefore, we hypothesize: Hypothesis 5a: Pre-training motivation to learn will positively influence the attention processes of observational learning. Hypothesis 5b: Pre-training motivation to learn will positively influence the retention processes of observational learning. Hypothesis 5c: Pre-training motivation to learn will positively influence the production processes of observational learning. Hypothesis 5d: Pre-training motivation to learn will positively influence the motivational processes of observational learning. 19 Pre-Training Software Self-Efficacy Self-efficacy has been widely used and accepted within the field of training research (Colquitt, LePine et al. 2000). Following Yi and Davis (Yi and Davis 2003), this study also adopts the definition of self-efficacy presented by Kraiger et al. (1993), which states ?self-efficacy refers to one?s perceived performance capabilities for a specific activity.? Self-efficacy has been modeled as a training outcome and as an antecedent to training (Gist 1987; Tannenbaum, Mathieu et al. 1991; Yi and Davis 2003). Colquitt et al. (2000) reported a meta-analytic correlation of 0.59 between pre- and post-training self- efficacy. Within the field of information systems research, pre-training software self- efficacy is reported to be a significant predictor of post-training software self-efficacy (Martocchio and Webster 1992; Martocchio 1994; Martocchio and Dulebohn 1994; Marshall, Rainer et al. 2003; Yi and Davis 2003). The current research posits that among trainees pre-training software self-efficacy is a determinant of post-training software self- efficacy, and helps to account for some of the variation among the task performance and declarative knowledge of trainees (See Figure 5). We hypothesize: Hypothesis 6: Pre-training software self-efficacy will positively influence post-training software self-efficacy. Figure 5. Pre-Training Motivation to Learn and Pre-Training Software Self- Efficacy Represent Individual Differences within the Research Framework Pre-training Motivation to Learn Pre-training Software Self- Efficacy 20 Modeling-Based Training Interventions Emotion ?Since, in actual human behavior motive and emotion are major influences on the course of cognitive behavior, a general theory of thinking and problem solving must incorporate such influences.? -- Herbert Simon, 1967 The quote above comes from a conceptual article written by Herbert Simon more than three decades ago. He is considered by many to be one of the founders of modern computing and artificial intelligence. Unfortunately, research interest related to the integration of emotion into the HCI basically became dormant for a long period of time. Recently there has been a resurgence of research focused on the psychology of emotion (Gross 1999; Zhang and Li 2005). The pioneering work of researchers such as Scott Brave, Clifford Nass, Rosalind Picard, and their colleagues have greatly impacted this resurgence of interest. To better understand how emotion relates to cognitive processing and learning, it is important to understand how the human mind processes information and sensory stimuli. LeDoux?s (1995) research in the field of neuropsychology produced a model of the neurological structure of emotion which helps explain the importance of the relationship between emotion and cognitive processing. LeDoux?s Model: the Neurological Structure of Emotion In LeDoux?s model, the thalamus, the limbic system, and the cortex are the three key regions of the brain involved in cognitive processing (see Figure 6) (LeDoux 1996). The thalamus functions as a signal processor and is first in line to receive all sensory 21 input. The thalamus passes information for higher level processing to the cortex, and simultaneously passes information to the limbic system (LeDoux 1996). The limbic system constantly evaluates inputs based on relevance to meeting needs and goals. ?The limbic system is the seat of emotion, memory, and attention. It helps determine valence (i.e., whether you feel positive or negative toward something) and salience (i.e., what gets your attention (Picard 1997).? When the limbic system identifies a relevant input, it responds by sending two sets of signals in tandem to prepare the body and mind for an immediate and appropriate response. The first set of signals is sent to the body to coordinate physiological response. The second set of signals is sent to the cortex to concentrate cognitive processes, such as attention focus, on meeting needs and attaining goals (LeDoux 1996). Figure 6. LeDoux?s Model of the Neurological Structure of Emotion The direct pathway between the thalamus and the limbic system is responsible for basic or more primitive emotions, such as the initial fear response that occurs when a person is suddenly startled (LeDoux 1996). Emotions that fall into this category have been categorized as primary emotions (Damasio 1994). Many HCI research studies have Cortex Physiological Responses Thalamus Limbic System Sensory Input 22 indicated that primary emotions can be directly triggered through audio and video stimuli within a computer interface (Detenber and Reeves 1996; Reeves and Nass 1998). For example, if a large window abruptly appears on screen accompanied by a loud beep or tone, it can potentially startle a computer user and activate a primary fear response. The ability to activate primary emotions can be an important consideration in the design of a human computer interface (e.g., an alert message notifies a laptop user that the battery is low). However, HCI research generally focuses on secondary emotions that involve higher levels of cognitive processing; i.e., secondary emotions such as frustration and satisfaction which occur when the limbic system is activated through cognitive processing in the cortex (Brave and Nass 2003). Therefore, the cortex is capable of activating emotions independently (i.e., with no external sensory stimuli) by generating its own internal stimuli. An example of this phenomenon is the anxiety that a novice computer user experiences when they contemplate the difficulties involved with purchasing and learning a new software application (Brave and Nass 2003). It is also possible for emotion to be generated by both the interactions of the thalamus-limbic system and the cortex-limbic system. This process might occur when sensory inputs indicate a large object is rapidly approaching, causing an immediate, primary fear response (via the thalamus-limbic system interaction), which is then promptly counteracted (via the cortex-limbic system interaction) as higher cognitive processing identifies the large object as a harmless beach ball (Brave and Nass 2003). To thoroughly define the concept of emotion within this study it is important to note how it differs from related constructs. 23 Emotion versus Related Constructs Emotion versus Mood. The key difference between mood and emotion is specificity and intention. That is, an emotion is directed toward a specific object, while mood is experienced in a more diffuse or global manner (Brave and Nass 2003). Emotions ?imply and involve relationships with a particular object,? (Frijda 1994) and therefore they are intentional. Brave and Nass (2003) note that ?we get scared of something, angry at someone, and excited about some event.? A second point of departure between the constructs of emotion and mood is temporal bias. Recalling that emotions are intentionally directed toward an object, they bias action in the short-term by preparing both body and mind for an appropriate and immediate response (Brave and Nass 2003). In contrast, mood tends to bias cognitive processing for a longer period of time (Davidson 1994). Moods can generally be thought of as an affective filter through which all incoming stimuli are judged. Consequently, moods tend to regulate the activation thresholds for emotions (e.g., a good mood lowers the activation threshold for positive emotions), and conversely emotions tend to cause or contribute to moods (Brave and Nass 2003). Emotion versus Sentiment. Another construct often confused with emotion is sentiment. Emotion and mood are related to an individual?s emotional state. In contrast, sentiments are properties or attributes assigned to an object or class of objects, which are generalizations formed through direct experience or through social learning (Frijda 1994). Sentiments are judged by bringing an object to mind and observing the affective reaction (Clore 1994). Sentiments can persist indefinitely, while emotions and moods are fleeting (i.e., lasting minutes, hours, or days), (Frijda 1994). Therefore, an individual?s 24 sentiment towards interacting with an object will motivate them to either seek or avoid opportunities to interact with that object (Brave and Nass 2003). For example, if a computer user says that using Microsoft Access is frustrating, they are essentially stating that through past experience or social learning they have come to expect that interactions with the Microsoft Access program will result in a negative emotional state. As a result of this sentiment, the computer user will likely choose to avoid opportunities to interact with the Microsoft Access program. Affect: The Impact of Emotion on Attention and Memory Emotion-relevant thoughts tend to dominate cognitive processing, and therefore emotion has the ability to focus attention and become completely absorbing (Brave and Nass 2003). That is, emotion directs and focuses attention on objects or events that are considered relevant to meeting needs and attaining goals so that we can deal with them appropriately (Brave and Nass 2003). Higher levels of relevance or importance lead to higher levels of emotional arousal which result in a more forceful focus of attention (Clore and Gasper 2000). Emotional stimuli tend to be remembered better than unemotional events, because emotion focuses thought and attention on the causal events and situations (Thorson and Friedsad 1985). Events associated with negative emotion tend to be highly arousing, and tend to be remembered better than events associated with positive emotions (Newhagen and Reeves 1991; Reeves, Newhagen et al. 1991; Newhagen and Reeves 1992; Reeves and Nass 1998). Emotionally charged events or situations improve the memory of central details at the expense of background details (Heuer and Reisberg 1992; Parrott and 25 Spackman 2000). Therefore, it follows that memory could improve CSA when a learner?s emotions are activated in response to their performance on a given task. Activation of Emotion within the HCI Emotions are an immediate and appropriate response to a stimulus that has been determined to be relevant to needs and goals. It follows that the degree to which attaining a goal is facilitated by an interface directly affects the emotional state of the user (Brave and Nass 2003). There can be advantages to an interface that empathizes with users while attempting to meet their needs and goals (Klein, Moon et al. 1999). In a study conducted by Reeves & Nass (1998), one group of participants were presented with a word processing interface which offered praise for their ability to spell. Participants gave higher ratings of intelligence and likeability to the word processing interface that offered praise during the interaction. This study provides evidence that the positive affective emotional state of the participants was elevated by appealing to the needs of individual self-esteem (Reeves and Nass 1998). Empathy ? Intrinsic Motivation for Affective Computing There are many definitions of empathy. Following the pioneering HCI research of Brave et al. (2005), the current research adopts Batson et al.?s (1997) definition of empathy as ?an other-oriented emotional response congruent with another?s perceived welfare.? This definition generally assumes that empathy is based on altruistic motives and caring for another. Once again, emotional response can increase attention and memory. Therefore, if an embodied computer agent can elicit positive emotional arousal through ?other- 26 oriented? empathetic language, it could have a significant impact on the interaction and subsequent performance of learners in a computer mediated learning environment. Brave, Nass, & Hutchinson (2005) recently conducted a HCI study that incorporated empathy into an HCI using an embodied (virtual) computer agent. Participants in the study reported higher levels of caring, likeability, and trust when ?other-oriented? empathetic text messages from an embodied computer agent were presented to participants. Brave et al. (2005) suggested that future research could investigate the use of empathetic emotion exhibited by an embodied computer agent in a computer assisted learning environment. Caring Agents Brave and Nass (2005) concluded that ?caring agents have an advantage in persuading and motivating users for two reasons: (1) people tend to trust information coming from sources that care about them, and (2) caring tends to be reciprocal and people are more willing to comply with those they care about.? They suggest that dynamic adaptation of emotional behavior in an HCI may be best, and that it could be implemented through an embodied computer agent that acts as a tutor or teacher. By introducing empathy as a modeling-based training intervention through an embodied computer agent serving as a tutor, we expect to observe a significant and positive influence on training outcomes (See Figures 7). Therefore we hypothesize: Hypothesis 7a: The empathetic emotion-based training intervention will positively influence the attention processes of observational learning. Hypothesis 7b: The empathetic emotion-based training intervention will positively influence the retention processes of observational learning. 27 Hypothesis 7c: The empathetic emotion-based training intervention will positively influence the production processes of observational learning. Hypothesis 7d: The empathetic emotion-based training intervention will positively influence the motivational processes of observational learning. Figure 7. Perceptions of Caring are Antecedents of Empathetic Emotional Response and Represent an Emotion-Based Training Intervention within the Research Framework Conclusion The literature review provides support for the research hypotheses presented in chapter two (See Figure 8). The emotion-based training intervention provides a method of potentially supporting the pro-emotional or affective computing side of the debate among researchers in the human-computer interaction research community. Furthermore, the results have greater implications for practicing managers who seek to realize returns on IT investments through improved methods of computer skill acquisition. The literature review and hypotheses presented in chapter two provides the foundation for empirically testing the theoretical impact of incorporating emotion into computer skill acquisition training as a modeling-based training intervention. Empathetic Response Caring 2 Caring 3 Caring 4 Caring 1 Caring 5 28 Figure 8. Complete Research Model Perce p t i ons of A g e n t Carin g Atten tio n Processes Ob serv atio n a l Learni ng Processes Post-t rainin g Soft ware Self-E fficac y Pre-t r aining Mot i vat i on To Lea r n Pr odu ction Processes Reten tio n Processes Pre-t r aining Soft ware Self-E fficac y Declarative K now ledg e Imme diate Task Performa nce Mot i vat i on Processes H7 H7b H7c H7d H5a H5b H5c H5d H6 Training Outcomes Observational Learning Processes H4 Pre-Trainin g Individual Differ e nces H3 H1 H2 Training Interventio n 29 Table 1 Summary of Study Hypotheses Hypotheses Hypothesis 1 Declarative Knowledge will positively influence immediate task performance. Hypothesis 2 Post-Training Software Self-Efficacy will positively influence immediate task performance. Hypothesis 3 Observational Learning Processes will positively influence declarative knowledge. Hypothesis 4 Observational Learning Processes will positively influence Post- Training Software Self-Efficacy. Hypothesis 5a Pre-training motivation to learn will positively influence the attention processes of observational learning. Hypothesis 5b Pre-training motivation to learn will positively influence the retention processes of observational learning. Hypothesis 5c Pre-training motivation to learn will positively influence the production processes of observational learning. Hypothesis 5d Pre-training motivation to learn will positively influence the motivational processes of observational learning. Hypothesis 6 Pre-training software self-efficacy will positively influence post- training software self-efficacy. Hypothesis 7a The empathetic emotion-based training intervention will positively influence the attention processes of observational learning. Hypothesis 7b The empathetic emotion-based training intervention will positively influence the retention processes of observational learning. Hypothesis 7c The empathetic emotion-based training intervention will positively influence the production processes of observational learning. Hypothesis 7d The empathetic emotion-based training intervention will positively influence the motivational processes of observational learning. 30 CHAPTER III METHOD The experimental treatment in this study was a training intervention designed to assess the impact of emotional response, specifically empathy, on learning outcomes during computer-based software training. Based on the research framework developed and validated by Yi and Davis (2003), a training intervention was incorporated into a computer software training workshop developed specifically for this study (see Appendix A). In keeping with prior IS research involving CSA (Marakas, Yi et al. 1998; Johnson and Marakas 2000; Yi and Davis 2003), Microsoft Excel 2003 was chosen as the subject matter for the training workshop (see Appendix A). Many undergraduate business courses require students to use specialized computer software including Microsoft Excel (Winston 2004, p. vii). Consequently, the process of CSA for a business professional often begins while completing the requirements of an undergraduate business degree and continues throughout their professional career (Winston 2004, p. vii). With this in mind, undergraduate business students were recruited as participants. The use of students as surrogates in social science research is a controversial issue and has often been debated within the IS research literature. Burnette and Dunne (1986) suggest that students should only be used as subjects when they represent the subject of 31 interest. Bass and Firestone (1980) note that research findings which are not widely generalizable beyond a specific population, can provide evidence of causal relationships and testable hypotheses that can be extended to other subject populations. Despite the controversy, previous social science research seems to indicate that it is suitable to use students as surrogates when the participants? skills and experiences are considered appropriate for an experimental task (Chi and Glaser 1985; Hughes and Gibson 1991). Therefore, it seems reasonable to use undergraduate business students as participants in this research study. Participants A total of 260 participants (36.2% female and 63.8% male) ranging in age from 19 to 26 completed the research study. Participants were automatically and randomly assigned to one of two treatment groups by a computer program during the experimental training session. The control group consisted of 129 participants (37.2% female and 62.8% male) who received emotionally neutral (non-empathetic) feedback during the experimental training session. The experimental treatment group consisted of 131 participants (35.1% female and 64.9% male) who received empathetic feedback during the experimental training session. Participants were recruited from upper level (junior or senior level) undergraduate business courses at a large university in the Southeast region of the United States. They were offered extra credit points toward their course grade as compensation for participating in the study. See Appendix B for a detailed account of the recruitment process. 32 Procedure The procedure for implementing behavior model training within this study closely followed the framework validated by Yi and Davis (2003) (see Figure 9). As participants arrived at the computer lab, they were welcomed by a facilitator and directed to a vacant computer station. The facilitator led the training session using scripted instructions (see Appendix C). After all participants were seated, the doors were closed and the facilitator began the training session. Following a brief introduction, the facilitator distributed pre-training questionnaires and a set of self-adhesive labels. A unique participation code was printed on each self-adhesive label and this code was used to identify all data collected from each participant. After all participants had completed the pre-training questionnaires they were colleted by the workshop facilitator. A software training video was then simultaneously displayed on a large screen at the front of the room and on the monitor at each computer station. Following the training video, participants were instructed to run a custom software program previously installed on their lab computer. This software program (a) collected demographic data, (b) randomly assigned participants to one of two training conditions, and (c) presented an interactive software simulation of Microsoft Excel 2003. The software simulation developed for this study performed two functions (see Appendix D). First, it provided an opportunity for participants to briefly practice the software skills demonstrated in the training video. Second, it served as a learning assessment tool capable of providing personalized performance feedback to participants while assessing learning outcomes. After all participants had completed the software simulation the 33 facilitator distributed and collected post-training questionnaires which concluded the training workshop and data collection. Figure 9. Experimental Procedure Pre-training Questionnaire Introduction Post-training Questionnaire Video Demonstration Practice with Immediate Feedback from Non-Empathetic Agent Declarative Knowledge Tested Immediate Task Performance Tested within Software Simulation Treatment B: Empathetic Agent Video Demonstration Practice with Immediate Feedback from Empathetic Agent Declarative Knowledge Tested Immediate Task Performance Tested within Software Simulation Treatment A: Empathetic Agent 34 Design The two training conditions, (a) behavior modeling and (b) behavior modeling with attention enhancement, were identical except for the expression of empathetic emotion by an embodied computer agent in the latter condition. Several key elements of the experimental design for this study were adapted from a recent study by Brave et al. (2005) including the embodiment of a computer agent and measures of empathetic response (see Appendix D). The Embodied Computer Agent The computer agent was visually represented by one of three still images within the simulation, and it communicated asynchronously with participants via text messages displayed within a speech bubble. The agent guided all participants through the software simulation while presenting multiple choice questions and instructions. The multiple choice questions were designed to measure declarative knowledge while the instructions were designed to assess immediate task performance (see Table 4). Immediately after a participant responded to a question or instruction, the agent presented personalized feedback based on their performance. In the control group the agent was always represented visually by a human face void of expression. In contrast, the empathetic treatment group received performance feedback that was emotionally varied. That is, when a participant in the treatment group responded correctly, the agent was visually represented by a smiling human along with empathetic performance feedback. Conversely, when a participant in the treatment group responded incorrectly, the agent was visually represented by a frowning human face along with empathetic performance feedback (see Table 2) (see Appendix D). 35 Table 2 Summary of Performance-based Feedback in the Software Simulation Participant Response Control Group Treatment Group Image Feedback Image Feedback Correct No Emotion No Emotion Happy Empathetic Incorrect No Emotion No Emotion Sad Empathetic Measures Training Outcomes As previously noted in Chapter 2, the three most commonly examined outcomes of software training are declarative knowledge, task performance, and post-training efficacy. Following the research framework of Yi and Davis (2003) measures were developed and adopted to assess training outcomes. Declarative Knowledge Anderson defines declarative knowledge (1985, p. 199) as ?knowledge about facts and things.? In keeping with similar research studies of CSA (Yi and Davis 2001; Yi and Davis 2003), post-training declarative knowledge was assessed through multiple choice questions. Eight multiple choice questions were developed and deemed adequate to measure declarative knowledge during the instructional design (ID) process (see Table 4). The questions were reviewed by a panel of subject matter experts (SME) and further tested in a pilot study (see Appendix A). A grade of one or zero was recorded in the research database based on whether or not questions were answered correctly (correct = 0; incorrect =1). Both declarative knowledge and task performance were assessed within the custom software simulation developed for this study (see Appendix D). 36 Task Performance Task performance is a measure of how well a trainee can reproduce behavior modeled by a trainer. Task performance is often assessed by using human raters who observe trainees while they attempt to reproduce modeled behavior. The task performance data recorded by human raters can result in measurement error due rater bias. To address this issue, a panel of subject matter experts collaboratively recommended automating the process of assessing task performance (see Appendices A). Therefore, task performance assessment was incorporated into the software simulation as an automated process, and was operationalized using the embodied computer agent (see Appendix D). The agent presented a total of eleven individual tasks to participants within the software simulation program (see Table 4). A grade of one or zero was recorded in the research database based on whether or not the participant completed the task correctly (correct = 0; incorrect =1). Pre- and Post-Training, Task Specific Software Self-Efficacy Johnson and Marakas (2000) presented a framework of guidelines for measuring self-efficacy within a specific domain in which the skills are directly related to a known task. These guidelines suggest that software self-efficacy measures must: (a) elicit estimations of ability within a task-specific rather than general context; (b) avoid ability assessments that include cross-domain or general-domain skills; (c) and that the level of analysis (LOA) of the requested estimation of perceived ability must agree with the level of analysis of the task and subsequent performance measure. Following these guidelines, the LOA of task specific software self-efficacy were closely matched to the Microsoft Excel skills demonstrated in the training video (see Appendix A). 37 A total of 10 items were used to measure task specific software self-efficacy. Four of these items were directly adopted from the instrument developed by Johnson and Marakas (2000), and the remaining six items were modified items from the same instrument (Johnson and Marakas 2000). These modifications included a LOA that better matched the LOA of the learning outcome measures of task performance and declarative knowledge (see Tables 3 & 4). Consistent with prior research the task specific software self-efficacy measures captured the magnitude and strength of each individual?s perceived abilities (Compeau and Higgins 1995; Marakas, Yi et al. 1998). The magnitude was self-reported by participants when they first answered ?YES? or ?NO? regarding their perceived ability to perform a specific behavior (coded respectively as a 1 or 0). The strength was captured when the participant rated their ability on a scale of 1 to 10 (where 1 = ?Not at All Confident? and 10=?Totally Confident?). Following Lee and Bobko (1994), magnitude was then multiplied by the strength to obtain the final self-reported score for each item. Colquitt et al. (2000) reported a meta-analytic correlation of 0.59 between pre- and post-training self-efficacy. Within the field of information systems research, pre- training software self-efficacy is reported to be a significant predictor of post-training software self-efficacy (Martocchio and Webster 1992; Martocchio 1994; Martocchio and Dulebohn 1994; Marshall, Rainer et al. 2003; Yi and Davis 2003). Therefore, the same 10-item scale was used to measure both pre- and post-training software self efficacy. 38 Table 3 Adopted and Modified Measures of Task-Specific Software Self-Efficacy Item ID Item Origin MS Skill # prSSE1 & poSSE1 I believe I have the ability to use an Excel spreadsheet to share numeric information with others. Johnson & Marakas (2000) XL03S-1-1 prSSE2 & poSSE2 I believe I have the ability to enter numbers into an Excel Spreadsheet. Johnson & Marakas (2000) XL03S-1-1 prSSE3 & poSSE3 I believe I have the ability to use and understand the cell references in an Excel Spreadsheet. Johnson & Marakas (2000) XL03S-2-3 prSSE4* & poSSE4* I believe I have the ability to use and understand both relative and absolute cell references in an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-3 prSSE5 & poSSE5 I believe I have the ability to write a simple formula in an Excel Spreadsheet. Johnson & Marakas (2000) XL03S-2-3 prSSE6* & poSSE6* I believe I have the ability to write a basic conditional formula in an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-4 prSSE7* & poSSE7* I believe I have the ability to use an ?IF? function within an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-4 prSSE8* & poSSE8* I believe I have the ability to write a complex conditional formula in an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-4 prSSE9* & poSSE9* I believe I have the ability to write an embedded ?IF? statement in an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-4 prSSE10* & poSSE10* I believe I have the ability to copy formulas within an Excel Spreadsheet. Johnson & Marakas (2000)* XL03S-2-4 Note. * Indicates that the item is a modified version of the original instrument developed by Johnson and Marakas. The ?MS Skill #? is the Microsoft Office Specialist Skills Standard identification number for professional certifications that match the task specific self-efficacy measure. See Appendix A for further details. 39 Observational Learning Processes The sixteen-item scale used to measure observational learning processes (OLP) was adopted directly from Yi and Davis? (2003). Attention processes, retention processes, production processes, and motivation processes are posited to be formative factors for the second order OLP construct. Four items are used to measure each of these individual observational learning processes (see Table 3). Participants were asked to indicate on a seven-point Likert-type scale (0 = completely disagree, 4 = neither agree or disagree, 7 = completely agree) the degree to which they agreed or disagreed with the statements presented (see Appendix F). Pre-Training Individual Differences Pre-Training Motivation to Learn Pre-training motivation to learn is defined as ?a trainee?s desire to master the content of the training program? (Noe 1986; Noe and Schmitt 1986; Yi and Davis 2003). Yi and Davis (2003) validated their research framework and model using a four-item scale to measure pre-training motivation to learn. This scale was adapted from prior research related to pre-training motivation (Noe and Schmitt 1986; Hicks and Klimoski 1987; Baldwin, Magjuka et al. 1991; Martocchio and Dulebohn 1994). This study adopts the same four-item scale used by Yi and Davis (2003) to assess participants? pre-training motivation to learn advanced spreadsheet skills (see Table 3). Participants were asked to indicate on a seven-point Likert-type scale (0 = completely disagree, 4 = neither agree or disagree, 7 = completely agree) the degree to which they agreed or disagreed with the statements presented (see Appendix F). 40 Emotional Response: Perceptions of Agent Caring There are numerous definitions of emotion within the research literature (Kleinginna and Kleinginna 1981), but two generally accepted aspects of emotion are evident. First, emotion is a reaction to events deemed relevant to the needs, goals, and concerns of an individual. Second, emotion encompasses physiological, affective, behavioral, and cognitive components (Brave and Nass 2003). Therefore, emotion can only be measured indirectly by monitoring changes in these components while a subject is engaged in a HCI. A wide range of methods have been identified for measuring emotional response within HCI studies including: (a) neurological changes via electroencephalogram (EEG); (b) autonomic nervous system activity (e.g., heart rate, blood pressure, blood pulse volume, respiration, temperature, pupil dilation, skin conductivity, and muscle tension) via electromyography; (c) facial expression via computer-based pattern recognition and image capture; (d) changes in voice (i.e., pitch range, rhythm, amplitude, and duration); and, (e) self-report measures via survey questionnaires (Brave and Nass 2003, p. 88-92). Post-interaction questionnaires are currently the primary method for measuring emotion within HCI research studies (Brave and Nass 2003, p. 90). Therefore, self-reported, post-interaction survey questions were used to measure empathetic response within the current study. As previously noted, an embodied computer agent was used to illicit an emotional response in participants via other-oriented, emotional ques incorporated into performance feedback (see Appendix D). Following the resent study by Brave et al. (2005), participants? opinions about the emoting computer agent were used to indirectly measure empathetic emotional response (i.e., affective state). Specifically, perceptions of caring 41 were used as a post-interaction measure of the affective state of participants. Five, 10- point semantic differentials (i.e., adjective pairs) were adopted from Brave et al. (2005) who reported the scale to be very reliable (Cronbach?s ? = 0.88) (see Table 4 & Appendix F). Table 4 Summary of Study Measures Item ID Scale/Measure Source Training Outcomes Declarative Knowledge Developed for this DK1 Next, we need to enter the function name (in this case "IF") immediately followed by a beginning... (Please click on the correct answer below): (A) parentheses '(' ; (B) bracket '[' ; (C) squiggly line '{' Study DK2 Which operator would we use to test whether or not the restaurant's score is below the minimum required to prevent a mandatory shutdown? (Please click the correct choice listed below): (A) <= (Less than or equal) ; (B) >= (Greater than or equal) ; (C) < (Only Less Than) ; (D) > (Only Greater Than) DK3 Recalling that the health inspector must shutdown, warn, or pass each restaurant...What should we enter as the FALSE criteria for the logical test? (Please click on the correct answer below.) ; (A) "Passed Inspection" ; (B) "Warning" ; (C) Another "IF" Statement ; (D) Cell "F6" DK4 Which operator would we use to test whether or not the restaurant's score is below the minimum required to prevent a mandatory shutdown? (Please click the correct choice listed below): (A) <= (Less than or equal) ; (B) >= (Greater than or equal) ; (C) < (Only Less Than) ; (D) > (Only Greater Than) DK5 Using a SINGLE CLICK of the mouse, select the best choice for "result if false" for the second or "embedded" conditional statement. (Please click the correct answer from the list below): (A) "Warning" ; (B) "Score Below" ; (C) "Passed" ; (D) Cell "G8" DK6 Parentheses are required to complete the formula. How many parentheses are actually needed? (Please click on the correct answer below): (A) One Closing Parentheses ; (B) Two Closing Parentheses ; (C) Three Closing Parentheses ; (D) Three Opening Parentheses 42 DK7 In order to accurately copy this formula to the cells D4 through D22, what modification needs to be made to this formula? (Please click the correct answer below): (A) Make All Cell References Absolute ; (B) Make Only Cell C3 an Absolute Reference ; (C) Make All Cell References EXCEPT Cell C3 Absolute ; (D) Make All Cell References Relative DK8 Which of the following is an absolute cell reference? (Please click the correct answer below): (A) Cell C3 ; (B) Cell $G$6 ; (C) Cell G7 ; (D) Cell F7 Task Performance Developed for this TP1 Using a single click of your mouse, select the cell that represents the ideal location to enter the first formula. Study TP2 Using your keyboard, type the symbol needed to begin a formula in Excel. TP3 Using a SINGLE CLICK of your mouse, select the cell that should be evaluated by our logical test. TP4 Now, using a SINGLE CLICK of your mouse, select the evaluation criteria for shutting down a restaurant. (Hint: It is the cell that holds the lowest score a restaurant can receive without a mandatory shutdown.) TP5 Using your keyboard, type the character required to separate the "logical test" from the "result if true." TP6 Using a SINGLE CLICK of your mouse, select the cell that holds the action that should be taken by the health inspector if the result of the logical test is TRUE. TP7 Using a SINGLE CLICK of your mouse, select the cell that will be evaluated in our second (embedded) "IF" statement. TP8 Now, using a SINGLE CLICK of your mouse, select the evaluation criteria for issuing a warning to a restaurant. (Hint: It is the cell that holds the lowest score a restaurant can receive without a warning.) TP9 Using your keyboard, type the character required to separate the "logical test" from the "result if true." TP10 Using a SINGLE CLICK of your mouse, select the cell that holds the action that should be taken by the health inspector if the result of the second logical test is TRUE. TP11 Using your keyboard, press the key that will finalize our formula and enter it into the Excel spreadsheet for processing. Post-Training Task-Specific Software Self-Efficacy Johnson & Marakas poSSE1 I believe I have the ability to use an Excel spreadsheet to share numeric information with others. (2000) poSSE2 I believe I have the ability to enter numbers into an Excel Spreadsheet. poSSE3 I believe I have the ability to use and understand the cell references in an Excel Spreadsheet. 43 poSSE4 I believe I have the ability to use and understand both relative and absolute cell references in an Excel Spreadsheet. poSSE5 I believe I have the ability to write a simple formula in an Excel Spreadsheet. poSSE6 I believe I have the ability to write a basic conditional formula in an Excel Spreadsheet. poSSE7 I believe I have the ability to use an ?IF? function within and Excel Spreadsheet. poSSE8 I believe I have the ability to write a complex conditional formula in an Excel Spreadsheet. poSSE9 I believe I have the ability to write an embedded ?IF? statement in an Excel Spreadsheet. poSSE10 I believe I have the ability to copy formulas within an Excel Spreadsheet. Observational Learning Processes Attention Processes Yi and Davis (2003) olp_A1 I paid close attention to the video demonstration. olp_A2 I was able to concentrate on the video demonstration. olp_A3 The video demonstration held my attention. olp_A4 During the video demonstration, I was absorbed by the demonstrated activities. Retention Processes Yi and Davis (2003) olp_R1 I had an opportunity to summarize the key aspects of demonstrated computer operations. olp_R2 I had the opportunity to symbolically process the presented information. olp_R3 I had the opportunity to mentally visualize the demonstrated computer operations. olp_R4 I had the opportunity to mentally practice the demonstrated computer operations. Production Processes Yi and Davis (2003) olp_P1 I had the opportunity to accurately reproduce the demonstrated computer operations. olp_P2 I had enough practice of the demonstrated computer skills. olp_P3 The training provided me with the opportunity to produce the procedural steps demonstrated through the video. olp_P4 The training helped me practice the key component skills required to produce the demonstrated computer operations. Motivation Processes Yi and Davis (2003) olp_M1 The training provided information that motivated me to use Excel. olp_M2 The training helped me see the usefulness of Excel. olp_M3 The training increased my intention to master Excel. olp_M4 The training showed me the value of using Excel in solving problems. 44 Pre-Training Individual Differences Pre-Training Motivation to Learn Yi and Davis (2003) PTM1 I am very interested in taking this training session. PTM2 I am excited about learning the Excel spreadsheet skills that will be covered in this training session. PTM3 I will try to learn as much as possible in this training session. PTM4 I am motivated to learn the training material in this session. Pre-Training Task-Specific Software Self-Efficacy Johnson & Marakas prSSE1 I believe I have the ability to use an Excel spreadsheet to share numeric information with others. (2000) prSSE2 I believe I have the ability to enter numbers into an Excel Spreadsheet. prSSE3 I believe I have the ability to use and understand the cell references in an Excel Spreadsheet. prSSE4 I believe I have the ability to use and understand both relative and absolute cell references in an Excel Spreadsheet. prSSE5 I believe I have the ability to write a simple formula in an Excel Spreadsheet. prSSE6 I believe I have the ability to write a basic conditional formula in an Excel Spreadsheet. prSSE7 I believe I have the ability to use an ?IF? function within and Excel Spreadsheet. prSSE8 I believe I have the ability to write a complex conditional formula in an Excel Spreadsheet. prSSE9 I believe I have the ability to write an embedded ?IF? statement in an Excel Spreadsheet. prSSE10 I believe I have the ability to copy formulas within an Excel Spreadsheet. Experimental Treatment: Training Intervention Emotional Response: Perceptions of Agent Caring Brave, Nass, and AC1 Not Compassionate <---> Compassionate Hutchinson AC2 Selfish <---> Unselfish (2005) AC3 Unfriendly <---> Friendly AC4 Competitive <---> Cooperative AC5 Cold <---> Warm 45 Statistical Analysis Measure Validation and Model Testing Measure validation and model testing were conducted using Partial Least Squares (PLS) Graph 3.0 (Chin 1998; Chin and Frye 2001; Gefen and Straub 2005). PLS is a component-based structural equation modeling (SEM) technique that offers several advantages over the better known covariance-based SEM methods which use a maximum likelihood function to obtain parameter estimates for latent structural modeling. In general, the advantages of using PLS include minimal demands on measurement scales, sample size, and residual distribution (Chin 1998; Chin 1998, pp. vii-vxi). More specifically, PLS-Graph provides the added advantage of representing both formative and reflective constructs in the proposed SEM model for the current study. The PLS structural model and hypotheses were evaluated by examining path coefficients and their respective significance levels. Sample Size Using the general rule of thumb for PLS SEM analysis the minimum required sample size for the current study was determined to be N=220 (N=110 required for both the control and treatment groups). According to the rule of thumb (Chin 1988), the sample size for PLS needs to be greater than at least ten times (1) the largest number of formative indicators (i.e., the largest measurement model) or (2) the largest number of independent variables impacting a dependent variable (i.e., the largest structural equation). The task performance latent variable represented the largest measurement model with its 11 formative indicators. Therefore, the largest number of formative 46 indicators multiplied by a factor of ten yielded an estimate of N=110. A detailed description of the statistical analyses and their results is presented in Chapter IV. 47 CHAPTER IV RESULTS This chapter presents the quantitative analysis of the research data in four sections. The first section describes the participant response rate and the data preparation process. The second section presents the evaluation of the psychometric properties of the latent constructs using a confirmatory factor analysis conducted with PLS-Graph. The third section presents the tests of the hypothesized structural model for both the full sample and for each individual group (control and treatment). The fourth and final section presents results of the tests for significant differences between the estimated path coefficients for the control and treatment groups. Data Preparation Participant Response There were 622 individuals invited to participate, and 387 accepted the invitation for a response rate of 62.2%. A total of 127 (or 32.8%) of these participants were removed from the sample due to (a) technical issues, (b) failure of participants to follow instructions, and (c) randomly missing data. Early in the data collection process network security permissions prevented the software simulation program from recording learning outcome data into the research database. Before the issue could be resolved, the data collected from 83 participants were affected and subsequently removed from the final sample. A small group of 32 participants failed to follow the instructions of the facilitator 48 during the training workshop. As a result, their responses to the paper-based surveys could not be matched to the learning outcome data recorded by the software simulation. The data of 12 additional participants were determined to be missing at random, and were also removed using the case-wise deletion method. Descriptive statistics for the final sample of 260 participants, the control, and the treatment group are presented in table 5. Table 5 Descriptive Statistics All a Control b Treatment c Scale M SD M SD M SD Learning Outcomes Declarative Knowledge DK1 0.97 0.16 0.99 0.09 0.95 0.21 DK2 0.84 0.37 0.84 0.37 0.84 0.37 DK3 0.75 0.43 0.79 0.41 0.72 0.45 DK4 0.54 0.50 0.40 0.49 0.68 0.47 DK5 0.73 0.44 0.73 0.45 0.73 0.44 DK6 0.94 0.23 0.93 0.26 0.95 0.21 DK7 0.70 0.46 0.71 0.46 0.69 0.46 DK8 0.88 0.32 0.88 0.32 0.88 0.33 Task Performance TP1 0.88 0.32 0.88 0.33 0.89 0.32 TP2 0.72 0.45 0.69 0.46 0.74 0.44 TP3 0.82 0.38 0.84 0.37 0.81 0.39 TP4 0.90 0.30 0.88 0.33 0.92 0.27 TP5 0.87 0.34 0.84 0.37 0.89 0.31 TP6 0.75 0.43 0.75 0.43 0.76 0.43 TP7 0.61 0.49 0.62 0.49 0.60 0.49 TP8 0.85 0.35 0.85 0.36 0.85 0.35 TP9 0.93 0.25 0.95 0.23 0.92 0.27 TP10 0.85 0.35 0.85 0.36 0.85 0.35 TP11 0.92 0.27 0.92 0.27 0.92 0.27 49 Post-Training Task-Specific Software Self-Efficacy poSSE1 8.31 1.89 8.20 2.00 8.42 1.78 poSSE2 8.96 1.68 8.90 1.73 9.02 1.64 poSSE3 8.31 1.93 8.22 1.96 8.40 1.90 poSSE4 8.01 2.23 7.88 2.26 8.15 2.19 poSSE5 8.52 1.88 8.53 1.82 8.50 1.95 poSSE6 7.80 2.22 7.66 2.18 7.93 2.25 poSSE7 7.87 2.14 7.83 2.08 7.92 2.21 poSSE8 6.66 2.85 6.58 2.75 6.74 2.95 poSSE9 7.27 2.57 7.30 2.45 7.24 2.69 poSSE10 8.51 2.18 8.48 2.16 8.53 2.20 Observational Learning Processes Attention Processes olp_A1 6.26 1.03 6.40 0.83 6.12 1.18 olp_A2 5.97 1.27 6.04 1.18 5.90 1.35 olp_A3 5.46 1.35 5.49 1.34 5.43 1.36 olp_A4 5.20 1.42 5.25 1.38 5.15 1.47 Retention Processes olp_R1 5.42 1.25 5.48 1.19 5.37 1.32 olp_R2 5.61 1.28 5.63 1.26 5.60 1.31 olp_R3 5.87 1.16 5.95 1.07 5.79 1.25 olp_R4 5.67 1.33 5.74 1.30 5.61 1.35 Production Processes olp_P1 5.94 1.27 5.95 1.15 5.92 1.37 olp_P2 5.32 1.43 5.29 1.35 5.35 1.51 olp_P3 5.94 1.13 5.98 1.04 5.89 1.22 olp_P4 5.93 1.14 5.98 1.06 5.89 1.22 Motivation Processes olp_M1 4.83 1.51 4.84 1.51 4.83 1.52 olp_M2 5.75 1.34 5.73 1.27 5.76 1.41 olp_M3 5.08 1.50 5.10 1.47 5.07 1.54 olp_M4 5.89 1.25 5.88 1.22 5.90 1.28 Pre-Training Individual Differences Pre-Training Motivation to Learn PTM1 4.54 1.30 4.51 1.41 4.57 1.20 PTM2 4.68 1.35 4.81 1.32 4.56 1.37 PTM3 5.95 1.19 5.99 1.20 5.90 1.18 PTM4 5.30 1.28 5.38 1.30 5.21 1.26 50 Pre-Training Task-Specific Software Self-Efficacy prSSE1 6.80 2.37 6.57 2.37 7.02 2.35 prSSE2 8.60 1.97 8.53 1.92 8.68 2.03 prSSE3 6.61 2.66 6.40 2.59 6.81 2.71 prSSE4 4.56 3.24 4.31 3.10 4.80 3.37 prSSE5 6.90 3.04 6.83 2.95 6.98 3.14 prSSE6 4.66 3.23 4.70 3.08 4.63 3.39 prSSE7 3.80 3.19 3.67 2.98 3.93 3.39 prSSE8 2.39 2.74 2.19 2.52 2.59 2.93 prSSE9 2.08 2.59 1.96 2.46 2.20 2.72 prSSE10 6.60 3.07 6.62 3.08 6.57 3.07 Experimental Treatment: Training Intervention Emotional Response: Perceptions of Agent Caring AC1 5.02 1.46 4.74 1.47 5.29 1.39 AC2 5.45 1.50 5.34 1.44 5.56 1.55 AC3 5.71 1.43 5.37 1.58 6.04 1.18 AC4 5.61 1.50 5.57 1.36 5.66 1.63 AC5 5.15 1.45 4.89 1.52 5.40 1.33 Note. N a =260; N b =129; N c =131 Psychometric Properties of Measures Confirmatory Factor Analysis Eight reflective latent factors were measured using self-report survey items. The psychometric properties of these eight constructs were evaluated through a confirmatory factor analysis (CFA). The CFA was performed using a measurement model in which the first-order constructs were specified as correlated variables with no causal paths (Yi and Davis 2003). The measurement model was assessed using PLS to examine internal consistency reliability, convergent reliability, and discriminant reliability (Barclay, Higgins et al. 1995; Chin 1998; Compeau, Higgins et al. 1999; Yi and Davis 2003). The criteria used for these assessments are only applicable for constructs with reflective 51 indicators and are not appropriate for the constructs with formative indicators within the research model (Chin 1998; Gefen, Straub et al. 2000; Gefen and Straub 2005). Internal consistency reliability Internal consistency reliability (ICR), synonymous with composite reliability and similar to Chronbach?s alpha (Yi and Davis 2003), was calculated within PLS-Graph. ICR is calculated using the following formula: ( ) ( ) ( )( ) ??? ?+?= 2 22 1 iii ICR ??? where i ? is the standardized component loading of a manifest indicator on a latent construct (Chin 1998). Latent constructs with ICR values of 0.70 or higher are considered acceptable (Barclay, Higgins et al. 1995; Compeau, Higgins et al. 1999; Agarwal and Karahanna 2000). All of the ICR values for each of the eight latent constructs were calculated to be 0.89 or higher, and therefore were determined to be acceptable (see Table 6). Convergent & Discriminant Validity Two criteria were used to verify convergent and discriminant validity. First, the square root of the average variance extracted (AVE) by a latent construct from its indicators should (a) be at least 0.707 (i.e., AVE > 0.50), and should also exceed the calculated values of its correlations with other constructs (Fornell and Larcker 1981; Barclay, Higgins et al. 1995; Chin 1998; Yi and Davis 2003). AVE was calculated within the PLS-Graph program using the following formula: ( )( ) ??? ?+= 222 1 iii AVE ??? 52 where i ? is the standardized component loading of a manifest indicator on a latent construct (Chin 1998). Second, the standardized item loadings should (a) be at least 0.707, and (b) should also load higher on the construct it is intended to measure than on any other construct (Compeau, Higgins et al. 1999; Agarwal and Karahanna 2000; Yi and Davis 2003). Cross-loadings represent the correlations between latent variable component scores and the manifest indicators of other latent constructs (Chin 1998). The square root of the AVE for each construct was greater than 0.707 (see Table 6). The correlations presented in table 6 were also calculated within PLS-Graph. The remaining criteria could not be calculated within PLS-Graph, and required additional steps using both Microsoft Excel and SPSS. Specifically, the rescaled data matrix and the the latent variable scores (i.e., eta matrix) were taken from the PLS-Graph output manipulated within Excel. The new matrix was 261 (i.e., 260 cases plus a header row) rows by 54 (i.e., the participant ID, 45 columns of rescaled item scores, and eight columns of latent factor scores). This matrix created in Excel was then imported into SPSS. SPSS was then used to generate bivariate Pearson correlations between the rescaled item scores and the latent factor scores (Gefen and Straub 2005). The resulting factor structure matrix of loadings and cross-loadings calculated in SPSS is presented in Table 7. The factor structure matrix indicates that all items had loadings greater than 0.707 on their respective constructs, except for two pre-training software self-efficacy items (i.e, prSSE2 and preSSE9). All items loaded higher on their respective construct than on any other construct (see Table 7). 53 Despite the fact that the loadings for two of the pre-training software self-efficacy items (i.e., items prSSE2 and prSSE9 highlighted in table 7) were less than 0.707, they were not removed from the sample. The pre- and post-training software self-efficacy were identical and item loadings of the corresponding two post-training software self- efficacy items (i.e., items poSSE2 and poSSE9) were greater than the minimum requirement of 0.70. Furthermore, as noted in chapter 2, Colquitt et al. (2000) reported a meta-analytic correlation of 0.59 between pre- and post-training self-efficacy. Within the field of information systems research, pre-training software self-efficacy is reported to be a significant predictor of post-training software self-efficacy (Martocchio and Webster 1992; Martocchio 1994; Martocchio and Dulebohn 1994; Marshall, Rainer et al. 2003; Yi and Davis 2003). Additionally, researchers have argued that measurements instruments should be treated more holistically when using structural equation (SEM) tools (MacCallum and Austin 2000; Straub, Boudreau et al. 2004; Gefen and Straub 2005). Table 6 Reliabilities, Convergent and Discriminant Validities, and Correlations of Latent Constructs - Measurement Model Latent Construct ICR AVE 1 2 3 4 5 6 7 8 1 Empathetic Response 0.89 0.62 0.79 2 Pre-training Motivation 0.89 0.66 0.25 0.81 3 OLP: Attention 0.93 0.76 0.27 0.34 0.87 4 OLP: Retention 0.93 0.78 0.26 0.28 0.58 0.88 5 OLP: Production 0.90 0.70 0.22 0.29 0.59 0.74 0.84 6 OLP: Motivation 0.91 0.71 0.35 0.25 0.56 0.54 0.51 0.84 7 Pretraining SSE 0.93 0.57 0.14 0.20 0.01 0.16 0.22 0.12 0.75 8 Post-training SSE 0.96 0.69 0.19 0.32 0.30 0.44 0.52 0.37 0.58 0.83 Note. OLP = Observational Learning Process Dimensions; SSE = Task Specific Software Self-Efficacy; ICR = Internal Consistency Reliability, and should be 0.70 or greater; AVE = Average Variance Extracted by a latent construct from 54 its reflective indicators, and should be greater than 0.50; Diagonal elements in bold are the square roots of Average Variance Extracted (AVE) by latent variable from their reflective indicators; Off-diagonal elements are the correlations between latent constructs; To establish convergent and discriminant validity, the diagonal elements in bold should be at least 0.707 (i.e., AVE > 0.50) and exceed the off-diagonal elements in the same row and column. Table 7 Factor Structure Matrix of Loadings and Cross-Loadings - Measurement Model Construct/Item 1 2 3 4 5 6 7 8 1 Empathetic Response: Perceptions of Computer Agent as Caring AC1 0.79 0.20 0.26 0.21 0.16 0.32 0.12 0.17 AC2 0.78 0.13 0.15 0.21 0.17 0.24 0.10 0.11 AC3 0.82 0.25 0.26 0.19 0.20 0.27 0.14 0.17 AC4 0.76 0.19 0.17 0.26 0.20 0.27 0.14 0.19 AC5 0.79 0.21 0.20 0.16 0.15 0.29 0.06 0.12 2 Pretraining Motivation Processes PTM1 0.21 0.78 0.24 0.19 0.18 0.14 0.13 0.18 PTM2 0.15 0.83 0.23 0.21 0.24 0.16 0.13 0.24 PTM3 0.24 0.77 0.35 0.25 0.26 0.23 0.13 0.28 PTM4 0.21 0.86 0.28 0.27 0.26 0.28 0.25 0.34 3 OLP: Attention Processes olp_A1 0.17 0.27 0.88 0.48 0.54 0.52 -0.02 0.29 olp_A2 0.17 0.23 0.88 0.44 0.50 0.43 0.06 0.29 olp_A3 0.32 0.32 0.90 0.55 0.51 0.48 0.02 0.25 olp_A4 0.28 0.36 0.83 0.56 0.49 0.53 -0.03 0.21 4 OLP: Retention Processes olp_R1 0.27 0.27 0.57 0.85 0.65 0.45 0.14 0.34 olp_R2 0.25 0.24 0.51 0.90 0.67 0.47 0.12 0.37 olp_R3 0.25 0.26 0.52 0.91 0.66 0.48 0.17 0.43 olp_R4 0.14 0.22 0.45 0.87 0.62 0.49 0.14 0.42 5 OLP: Production Processes olp_P1 0.19 0.30 0.54 0.68 0.83 0.40 0.14 0.37 olp_P2 0.17 0.17 0.38 0.57 0.75 0.32 0.23 0.49 olp_P3 0.18 0.25 0.51 0.61 0.89 0.43 0.20 0.45 olp_P4 0.21 0.25 0.52 0.60 0.88 0.54 0.16 0.44 55 6 OLP: Motivation Processes olp_M1 0.27 0.22 0.48 0.50 0.53 0.77 0.11 0.34 olp_M2 0.30 0.19 0.47 0.41 0.38 0.88 0.10 0.29 olp_M3 0.31 0.20 0.45 0.46 0.40 0.88 0.06 0.29 olp_M4 0.30 0.23 0.49 0.44 0.41 0.84 0.14 0.33 7 Pre-Training Task Specific Software Self-Efficacy (prSSE) prSSE1 0.06 0.13 0.05 0.10 0.20 0.12 0.73 0.55 prSSE2 0.12 0.14 0.03 0.09 0.18 0.02 0.61 0.50 prSSE3 0.18 0.17 0.05 0.16 0.19 0.10 0.79 0.54 prSSE4 0.15 0.18 0.03 0.18 0.20 0.14 0.80 0.49 prSSE5 0.09 0.19 0.03 0.14 0.19 0.11 0.82 0.47 prSSE6 0.12 0.16 -0.01 0.15 0.16 0.11 0.79 0.37 prSSE7 0.07 0.13 -0.04 0.11 0.12 0.07 0.79 0.39 prSSE8 0.14 0.16 -0.04 0.09 0.11 0.08 0.75 0.34 prSSE9 0.06 0.07 -0.15 -0.02 -0.01 -0.02 0.64 0.24 prSSE10 0.07 0.14 0.08 0.18 0.27 0.15 0.79 0.47 8 Post-Training Task Specific Software Self-Efficacy (poSSE) poSSE1 0.15 0.24 0.23 0.31 0.41 0.29 0.52 0.81 poSSE2 0.13 0.26 0.19 0.27 0.36 0.22 0.47 0.79 poSSE3 0.18 0.25 0.26 0.34 0.41 0.32 0.49 0.84 poSSE4 0.14 0.28 0.20 0.35 0.41 0.33 0.51 0.85 poSSE5 0.16 0.30 0.27 0.38 0.46 0.30 0.52 0.86 poSSE6 0.17 0.29 0.29 0.41 0.45 0.35 0.46 0.86 poSSE7 0.16 0.27 0.29 0.43 0.51 0.37 0.46 0.87 poSSE8 0.15 0.21 0.20 0.39 0.42 0.29 0.46 0.80 poSSE9 0.16 0.27 0.27 0.42 0.46 0.30 0.47 0.83 poSSE10 0.19 0.33 0.29 0.37 0.42 0.30 0.45 0.82 Note. OLP = Observational Learning Process Dimensions; In order to establish convergent and discriminant validity, items should load greater than 0.707 on their respective constructs (bold). Items should also load higher on the construct they are intended to measure than on any other construct. Test of Model and Hypotheses The proposed structural model and hypotheses were tested using partial least squares (PLS) analysis. Bootstrapping (with 1000 subsamples) was performed in the PLS 56 Graph program so that the significance of the proposed model and hypotheses could be assessed via t-tests. As noted in chapter two, the theoretical framework developed by Yi and Davis (2003) conceptualized the four first-order OLP dimensions as formative indicators of the second-order OLP construct. Second-order latent constructs (i.e., the OLP construct) cannot be directly represented in PLS Graph 3.0. Consequently, it is necessary to test such models indirectly by separately testing the first-order constructs comprising a second-order construct in a submodel, and then treat the first-order factor scores as manifest indicators of the second-order construct in a separate model (Agarwal and Karahanna 2000; Yi and Davis 2003). Two submodels were used to indirectly test the proposed model and hypotheses (see Figure 10). After testing the proposed model with the full sample of all data, the full data sample was separated into two groups: (a) the control group (neutral or non-empathetic treatment); (b) the treatment group (empathetic treatment). The original data for each group was standardized separately, and the proposed model was tested for each individual group. This provided the means to test for significant differences between the path coefficients of the proposed model for the groups via t-tests. Submodel One Submodel one was tested using PLS Graph and included empathetic response as a training intervention, pre-training motivation to learn, and the four first-order OLP dimensions (see Figure 10). The construct correlations for submodel one are presented in table 8, and the indicator weights and loadings of submodel two are presented in table 9. The factor scores calculated for each OLP dimension in submodel one were used as inputs for the second-order OLP construct in submodel two (Yi and Davis 2003). 57 Table 8 Construct Correlations for All Data, Control Group, and Treatment Group ? Submodel 1 Construct Correlations ? All Data Latent Construct 1 2 3 4 5 6 (1) Pretraining Motivation --- (2) OLP: Attention 0.354 --- (3) OLP: Retention 0.289 0.600 --- (4) OLP: Production 0.299 0.589 0.740 --- (5) OLP: Motivation 0.256 0.568 0.534 0.509 --- (6) Empathetic Response 0.253 0.282 0.271 0.225 0.355 --- Construct Correlations ? Control Group Data Latent Construct 1 2 3 4 5 6 (1) Pretraining Motivation --- (2) OLP: Attention 0.413 --- (3) OLP: Retention 0.396 0.569 --- (4) OLP: Production 0.398 0.597 0.705 --- (5) OLP: Motivation 0.347 0.517 0.486 0.530 --- (6) Empathetic Response 0.311 0.381 0.263 0.249 0.408 --- Construct Correlations - Treatment Group Data Latent Construct 1 2 3 4 5 6 (1) Pretraining Motivation --- (2) OLP: Attention 0.320 --- (3) OLP: Retention 0.210 0.617 --- (4) OLP: Production 0.230 0.577 0.766 --- (5) OLP: Motivation 0.184 0.607 0.575 0.478 --- (6) Empathetic Response 0.236 0.232 0.312 0.229 0.327 --- Note. ALL = the full sample including both control and treatment group data; Control = Data for the control (non- empathetic feedback) group; Treatment = Data for the treatment group (empathetic feedback); OLP = Observational Learning Processes Dimensions; Pre-training Motivation = Pre-training motivation to learn; Empathetic Response = Perceptions of Agent Caring. 58 Table 9 Indicator Weights and Loadings for All Data, Control Group, and Treatment Group ? Submodel 1 All Control Treatment Indicator Weights Loadings Weights Loadings Weights Loadings OLP: Attention A1 0.248 0.850 0.263 0.812 0.262 0.891 A2 0.222 0.843 0.250 0.871 0.191 0.823 A3 0.337 0.912 0.330 0.892 0.335 0.929 A4 0.342 0.863 0.337 0.815 0.333 0.893 OLP: Retention R1 0.317 0.865 0.331 0.835 0.284 0.881 R2 0.296 0.907 0.305 0.873 0.279 0.934 R3 0.305 0.909 0.315 0.894 0.296 0.922 R4 0.215 0.843 0.218 0.802 0.243 0.886 OLP: Production P1 0.347 0.848 0.333 0.838 0.369 0.860 P2 0.229 0.729 0.197 0.726 0.283 0.752 P3 0.303 0.884 0.330 0.901 0.279 0.862 P4 0.309 0.876 0.320 0.877 0.267 0.858 OLP: Motivation M1 0.279 0.772 0.268 0.805 0.287 0.743 M2 0.297 0.874 0.324 0.851 0.257 0.893 M3 0.301 0.875 0.285 0.856 0.336 0.894 M4 0.310 0.844 0.309 0.858 0.313 0.826 Perceptions of Agent Caring AC1 0.283 0.800 0.273 0.811 0.295 0.775 AC2 0.227 0.768 0.182 0.717 0.272 0.814 AC3 0.266 0.817 0.311 0.846 0.221 0.786 AC4 0.261 0.765 0.233 0.753 0.281 0.789 AC5 0.234 0.782 0.260 0.809 0.209 0.739 Pre-training Motivation to Learn PTM1 0.250 0.746 0.312 0.809 0.120 0.622 PTM2 0.276 0.803 0.288 0.830 0.254 0.739 PTM3 0.353 0.813 0.318 0.795 0.412 0.856 PTM4 0.347 0.882 0.294 0.869 0.425 0.908 Note. ALL = the full sample including both control and treatment group data; Control = Data for the control (non- empathetic feedback) group; Treatment = Data for the treatment group (empathetic feedback); OLP = Observational Learning Processes Dimensions; PTM = Pre-training Motivation to Learn; A = Attention Processes; R = Retention Processes; P = Production Processes; M = Motivation Processes; AC = Perceptions of Agent Caring which is a pseudo 59 measure of empathetic response. All the indicators in submodel 1 were specified as reflective. Therefore, the loading scores (bold) were applied in the PLS analysis to test the proposed model. Submodel Two As previously noted, OLP is modeled as a formative second-order construct in submodel two. Chin (1998) notes that estimates of weights in a formative second-order construct can become unstable due to multicollinearity among first-order factors. To avoid this potential problem, submodel two was tested in PLS Graph using the factor loadings for the four first-order OLP dimensions calculated in submodel one rather than weights (Fornell and Bookstein 1982; Yi and Davis 2003)(see Figure 10). The construct correlations for submodel two are presented in table 10, and the indicator weights and loadings of submodel two are presented in table 11. Table 10 Construct Correlations for All Data, Control Group, and Treatment Group ? Submodel 2 Construct Correlations ? All Data Latent Construct 1 2 3 4 5 (1) OLP --- (2) Pre-training SSE 0.179 --- (3) Post-training SSE 0.501 0.603 --- (4) Declarative Knowledge 0.303 0.241 0.413 --- (5) Task Performance 0.370 0.143 0.457 0.601 --- Construct Correlations ? Control Group Data Latent Construct 1 2 3 4 5 (1) OLP --- (2) Pre-training SSE 0.176 --- (3) Post-training SSE 0.589 0.572 --- (4) Declarative Knowledge 0.311 0.244 0.372 --- (5) Task Performance 0.430 0.077 0.403 0.577 --- 60 Construct Correlations - Treatment Group Data Latent Construct 1 2 3 4 5 (1) OLP --- (2) Pre-training SSE 0.197 --- (3) Post-training SSE 0.449 0.625 --- (4) Declarative Knowledge 0.326 0.200 0.454 --- (5) Task Performance 0.326 0.220 0.499 0.752 --- Note. ALL = the full sample including both control and treatment group data; Control = Data for the control (non- empathetic feedback) group; Treatment = Data for the treatment group (empathetic feedback): OLP = Observational Learning Processes Dimensions; SSE = Task Specific Software Self-efficacy. Table 11 Indicator Weights and Loadings for All Data, Control Group, and Treatment Group ? Submodel 2 All Control Treatment Indicator Weights Loadings Weights Loadings Weights Loadings OLP: Dimensions olp_A -0.146 0.541 -0.225 0.513 0.004 0.545 olp_R 0.107 0.778 0.225 0.785 -0.172 0.691 olp_P 0.928 0.989 0.791 0.957 1.125 0.994 olp_M 0.134 0.580 0.268 0.679 -0.003 0.438 Pre-training Software Self-Efficacy pr_SSE1 0.167 0.777 0.177 0.756 0.159 0.798 pr_SSE2 0.154 0.675 0.157 0.664 0.151 0.683 pr_SSE3 0.164 0.828 0.168 0.819 0.161 0.836 pr_SSE4 0.147 0.806 0.149 0.776 0.145 0.829 pr_SSE5 0.144 0.823 0.155 0.772 0.135 0.868 pr_SSE6 0.111 0.750 0.106 0.720 0.114 0.778 pr_SSE7 0.116 0.741 0.115 0.712 0.116 0.761 pr_SSE8 0.102 0.692 0.106 0.662 0.098 0.712 pr_SSE9 0.071 0.571 0.065 0.523 0.074 0.604 pr_SSE10 0.144 0.799 0.156 0.819 0.135 0.788 Post-training Software Self-Efficacy po_SSE1 0.123 0.810 0.121 0.799 0.122 0.822 po_SSE2 0.114 0.790 0.107 0.758 0.119 0.816 po_SSE3 0.118 0.839 0.114 0.813 0.120 0.862 po_SSE4 0.126 0.855 0.130 0.844 0.120 0.861 po_SSE5 0.132 0.862 0.124 0.830 0.136 0.894 61 po_SSE6 0.120 0.856 0.121 0.842 0.121 0.871 po_SSE7 0.128 0.866 0.134 0.871 0.121 0.863 po_SSE8 0.106 0.794 0.124 0.823 0.095 0.770 po_SSE9 0.115 0.830 0.123 0.831 0.109 0.835 po_SSE10 0.117 0.824 0.115 0.820 0.122 0.828 Task Performance TP1 -0.015 0.191 0.084 0.214 -0.105 0.166 TP2 0.208 0.439 0.303 0.486 0.158 0.451 TP3 0.108 0.464 0.113 0.603 0.074 0.355 TP4 -0.093 0.260 0.046 0.371 -0.107 0.222 TP5 0.162 0.438 0.091 0.280 0.101 0.568 TP6 0.467 0.764 0.606 0.758 0.339 0.661 TP7 0.284 0.592 0.287 0.516 0.112 0.503 TP8 0.193 0.559 0.409 0.497 0.235 0.547 TP9 0.124 0.494 -0.339 0.059 0.285 0.663 TP10 0.097 0.654 -0.084 0.471 0.184 0.770 TP11 0.131 0.441 -0.163 0.163 0.252 0.583 Declarative Knowledge DK1 0.345 0.439 0.014 -0.048 0.538 0.614 DK2 0.173 0.238 0.113 0.264 0.094 0.141 DK3 0.316 0.530 0.084 0.243 0.343 0.555 DK4 0.058 0.139 0.075 0.115 0.090 0.232 DK5 0.201 0.583 0.413 0.710 0.080 0.437 DK6 0.404 0.512 0.211 0.419 0.488 0.551 DK7 0.370 0.643 0.391 0.728 0.200 0.494 DK8 0.159 0.443 0.428 0.645 0.103 0.411 Note. ALL = the full sample including both control and treatment group data; Control = Data for the control (non- empathetic feedback) group; Treatment = Data for the treatment group (empathetic feedback); OLP = Observational Learning Processes Dimensions; A = Attention Processes; R = Retention Processes; P = Production Processes; M = Motivation Processes; pr_SSE = Pre-training Task Specific Software Self-efficacy; po_SSE = Post-training Task Specific Software Self-efficacy; DK = Declarative Knowledge; TP = Task Performance; OLP, pre-training software self-efficacy, and post-training self-efficacy used loading scores (bold); declarative knowledge, and task performance used weights (bold). 62 Hypothesis Tests Hypothesis Tests for the Full Sample A summary of the model-testing results for the full sample of all data is presented in figure 10. Supporting Hypothesis 1, declarative knowledge had a significant positive effect on immediate task performance (? = 0.50, p < 0.001). Supporting Hypothesis 2, post-training software self-efficacy had a significant positive effect on immediate task performance (? = 0.25, p < 0.01). Supporting Hypothesis 3, observational learning processes had a significant positive effect on declarative knowledge (? = 0.36, p < 0.001). Supporting Hypothesis 4, observational learning processes had a significant positive effect on declarative knowledge (? = 0.41, p < 0.001). Supporting Hypotheses 5a through 5d, pre-training motivation (PTM) to learn had a significant effect on all four dimensions of OLP: (a) PTM on Attention Processes (? = 0.30, p < 0.001); (b) PTM on Retention Processes (? = 0.24, p < 0.01); (c) PTM on Production Processes (? = 0.26, p < 0.001); (d) PTM on Motivation Processes (? = 0.18, p < 0.001). Supporting Hypothesis 6, pre-training software self-efficacy had a significant positive effect on post-training software self-efficacy (? = 0.53, p < 0.001). Supporting Hypotheses 7a through 7d, the empathetic emotion-based training intervention (represented by perceptions of agent caring (AC) as a measure of empathetic response in partcipants) had a significant positive effect on all four dimensions of OLP: (a) AC on Attention Processes (? = 0.21, p < 0.01); (b) AC on Retention Processes (? = 0.21, p < 0.01); (c) AC on Production Processes (? = 0.16, p < 0.05); (d) AC on Motivation Processes (? = 0.21, p < 0.001). 63 The model accounted for substantial variances in post-training software self- efficacy (R 2 = 0.50) and immediate task performance (R 2 = 0.41); and, modest variances in declarative knowledge (R 2 = 0.09), attention processes (R 2 = 0.17), retention processes (R 2 = 0.13), production processes (R 2 = 0.11), and motivation processes (R 2 = 0.16). All hypotheses were supported when the full sample of all data was used to test the proposed model. All hypothesis tests are summarized in table 12. Hypothesis Tests for the Treatment Group Data A summary of the model-testing results for the treatment group data is presented in figure 11. Supporting Hypothesis 1, declarative knowledge had a significant positive effect on immediate task performance (? = 0.66, p < 0.001). Failing to support Hypothesis 2, post-training software self-efficacy did not a significant positive effect on immediate task performance (? = 0.20, NS). Supporting Hypothesis 3, observational learning processes had a significant positive effect on declarative knowledge (? = 0.33, p < 0.01). Supporting Hypothesis 4, observational learning processes had a significant positive effect on declarative knowledge (? = 0.34, p < 0.001). Only two of the Hypotheses 5a through 5d were supported when the model was tested using the data for only the treatment group. Pre-training motivation (PTM) to learn only had a significant effect on the attention processes dimension of OLP: (a) PTM on Attention Processes (? = 0.28, p < 0.01); (b) PTM on Retention Processes (? = 0.24, NS); (c) PTM on Production Processes (? = 0.26, p < 0.05); (d) PTM on Motivation Processes (? = 0.18, NS). Supporting Hypothesis 6, pre-training software self-efficacy had a significant positive effect on post-training software self-efficacy (? = 0.50, p < 64 0.001). Only one of the Hypotheses 7a through 7d where note supported when the model was tested using the data for only the treatment group. The empathetic emotion-based training intervention (represented by perceptions of agent caring (AC) as a measure of empathetic response in participants) had a significant positive effect on only the retention process and motivation process dimensions of OLP: (a) AC on Attention Processes (? = 0.17, NS); (b) AC on Retention Processes (? = 0.28, p < 0.01); (c) AC on Production Processes (? = 0.19, p < 0.05); and, (d) AC on Motivation Processes (? = 0.30, p < 0.001). The model accounted for substantial variances in post-training software self- efficacy (R 2 = 0.49) and immediate task performance (R 2 = 0.59); and, modest variances in declarative knowledge (R 2 = 0.11), attention processes (R 2 = 0.13), retention processes (R 2 = 0.12), production processes (R 2 = 0.09), and motivation processes (R 2 = 0.12). All hypothesis tests for the treatment group data are summarized in table 12. Hypothesis Tests for the Control Group Data A summary of the model-testing results for the control group data is presented in figure 12. Supporting Hypothesis 1, declarative knowledge had a significant positive effect on immediate task performance (? = 0.50, p < 0.05). Failing to support Hypothesis 2, post-training software self-efficacy did not a significant positive effect on immediate task performance (? = 0.22, NS). Failing to support Hypothesis 3, observational learning processes did not have a significant positive effect on declarative knowledge (? = 0.31, p < 0.05). Supporting Hypothesis 4, observational learning processes had a significant positive effect on declarative knowledge (? = 0.50, p < 65 0.001). All four of the Hypotheses 5a through 5d were supported when the model was tested using the data for only the treatment group. Pre-training motivation (PTM) to learn had a significant effect on all four of the process dimensions of OLP: (a) PTM on Attention Processes (? = 0.33, p < 0.01); (b) PTM on Retention Processes (? = 0.35, p < 0.001); (c) PTM on Production Processes (? = 0.36, p < 0.001); (d) PTM on Motivation Processes (? = 0.24, p < 0.05). Supporting Hypothesis 6, pre-training software self-efficacy had a significant positive effect on post-training software self-efficacy (? = 0.40, p < 0.001). Only two of the Hypotheses 7a through 7d where supported when the model was tested using the data for only the control group. The empathetic emotion-based training intervention (represented by perceptions of agent caring (AC) as a measure of empathetic response in participants) had a significant positive effect on only the attention process and motivation process dimensions of OLP: (a) AC on Attention Processes (? = 0.28, p < 0.01); (b) AC on Retention Processes (? = 0.15, NS); (c) AC on Production Processes (? = 0.14, NS); (d) AC on Motivation Processes (? = 0.33, p < 0.001). The model accounted for substantial variance in post-training software self- efficacy (R 2 = 0.57), for moderate variance immediate task performance (R 2 = 0.38); and, modest variances in declarative knowledge (R 2 = 0.10), attention processes (R 2 = 0.24), retention processes (R 2 = 0.18), production processes (R 2 = 0.18), and motivation processes (R 2 = 0.22). All hypothesis tests for the treatment group data are summarized in table 12. 66 Multiple Group Analysis After running the submodel in PLS Graph (with 1000 bootstrap subsamples) individually for the control and treatment groups, the tests for significant differences between the path coefficients were conducted via t-tests. Following the recommendation of Wynne Chin (2004) regarding multiple group comparisons, the path coefficients and the corresponding standard error (S.E.) were taken from the bootstrapped mean of subsamples output and used to calculate t-values using the following formula: 2 2 2 1 2_1_ .... samplesample samplesample ESES PathPath t + ? = The degrees of freedom used to assess the calculated t-values were calculated (rounded to the nearest integer) using the following formula (Chin 2004): () ? ? ? ? ? ? ? ? ? ? ? ? ? + + + + = 2 1 .. 1 .. .... .. 2 2 2 1 2 2 2 2 1 n ES m ES ESES fd samplesample samplesample The t-tests did not provide evidence of significant (p = 0.05) differences between the path coefficients of the control and treatment groups (see Table 13). 67 Table 12 Results of Hypothesis Tests for All Data, Control Group, and Treatment Group Hypotheses All Control Treatment Hypothesis 1 Declarative Knowledge will positively influence immediate task performance. YES YES YES Hypothesis 2 Post-Training Software Self-Efficacy will positively influence immediate task performance. YES NO NO Hypothesis 3 Observational Learning Processes will positively influence declarative knowledge. YES YES YES Hypothesis 4 Observational Learning Processes will positively influence Post-Training Software Self-Efficacy. YES YES YES Hypothesis 5a Pre-training motivation to learn will positively influence the attention processes of observational learning. YES YES YES Hypothesis 5b Pre-training motivation to learn will positively influence the retention processes of observational learning. YES YES NO Hypothesis 5c Pre-training motivation to learn will positively influence the production processes of observational learning. YES YES YES Hypothesis 5d Pre-training motivation to learn will positively influence the motivational processes of observational learning. YES YES NO Hypothesis 6 Pre-training software self-efficacy will positively influence post-training software self-efficacy. YES YES YES Hypothesis 7a The empathetic emotion-based training intervention will positively influence the attention processes of observational learning. YES YES NO Hypothesis 7b The empathetic emotion-based training intervention will positively influence the retention processes of observational learning. YES NO YES Hypothesis 7c The empathetic emotion-based training intervention will positively influence the production processes of observational learning. YES NO YES Hypothesis 7d The empathetic emotion-based training intervention will positively influence the motivational processes of observational learning. YES YES YES Note. ALL = the full sample including both control and treatment group data; Control = Data for the control (non- empathetic feedback) group; Treatment = Data for the treatment group (empathetic feedback); YES = Hypothesis Supported; NO = Hypothesis Not Supported. 68 Table 13 Significant Tests of Group Differences ? Path Coefficients Control Group Treatment Group Path P.C. S.E. P.C. S.E. T-Stat d.f. P-Value AC --> olp_A 0.278 0.082 0.173 0.101 0.806 51 0.424 AC --> olp_R 0.158 0.091 0.294 0.095 1.030 61 0.307 AC --> olp_P 0.139 0.089 0.193 0.090 0.428 64 0.670 AC --> olp_M 0.337 0.079 0.315 0.088 0.186 57 0.853 PTM --> olp_A 0.339 0.088 0.297 0.096 0.319 59 0.751 PTM --> olp_R 0.369 0.096 0.159 0.099 1.514 62 0.135 PTM --> olp_P 0.384 0.077 0.207 0.090 1.492 54 0.141 PTM --> olp_M 0.265 0.110 0.126 0.107 0.904 66 0.369 olp_A --> OLP N/A N/A N/A N/A N/A N/A N/A olp_R --> OLP N/A N/A N/A N/A N/A N/A N/A olp_P --> OLP N/A N/A N/A N/A N/A N/A N/A olp_M --> OLP N/A N/A N/A N/A N/A N/A N/A OLP --> DK 0.318 0.152 0.337 0.112 0.098 83 0.922 OLP --> poSSE 0.501 0.062 0.338 0.338 0.473 17 0.642 prSSE --> poSSE 0.485 0.059 0.566 0.055 1.005 68 0.319 DK --> TP 0.607 0.195 0.711 0.126 0.449 91 0.654 poSSE --> TP 0.145 0.135 0.181 0.130 0.190 67 0.850 Note. olp_ = Observational Learning Process Dimension; A = Attention processes; R = Retention Processes; P = Production Processes; M = Motivation Processes; AC = Perceptions of Agent Caring ? a measure of empathetic response; PTM = Pre-training Motivation to Learn; prSSE = Pre-training Software Self-efficacy; poSSE = Post-training Software Self-efficacy; DK = Declarative Knowledge; TP = Task Performance; P.C. = Path Coefficient ? similar to the beta weights in regression; T-Stat = the calculated value of the Student?s T test statistic; d.f. = the calculated degrees of freedom; P-Value = the p-value corresponding to the t-statistic in a two tailed t-test. 69 Figure 10. Test of Proposed Model Using the Full Data Sample Data Sample Perce p t i ons o f Age n t C a ri ng Atten tio n Processes Ob serv atio n a l Learni ng Processes Post-t rainin g Soft ware Self-E fficac y Pre-t r aining Mot i vat i on To Lea r n Pr odu ction Processes Reten tio n Processes Pre-t r aining Soft ware Self-E fficac y Declarative K now ledg e Imme diate Task Performa nce Mot i vat i on Processes 0.21** 0.21** 0.16* 0.31*** 0.30*** 0.24** 0.26*** 0.18* 0.54*** 0.78*** 0.99*** 0.58*** 0.36*** 0.41*** 0.50*** 0.25** 0.53*** R 2 = 0 . 52 R 2 = 0 . 41 R 2 = 0 . 09 R 2 = 0 . 17 R 2 = 0 . 23 R 2 = 0 . 11 R 2 = 0 . 16 Submodel 1 Submodel 2 70 Figure 11. Test of Proposed Model for Treatment Group Perce p t i ons of A g e n t Carin g Atten tio n Processes Ob serv atio n a l Learni ng Processes Post-t rainin g Soft ware Self-E fficac y Pre-t r aining Mot i vat i on To Lea r n Pr odu ction Processes Reten tio n Processes Pre-t r aining Soft ware Self-E fficac y Declarative K now ledg e Imme diate Task Performa nce Mot i vat i on Processes 0.17 0.28** 0.19* 0.30*** 0.28** 0.15 0.19* 0.11 0.55*** 0.69*** 0.99*** 0.44*** 0.33** 0.34*** 0.66*** 0.20 0.50*** R 2 = 0 . 49 R 2 = 0 . 59 R 2 = 0 . 11 R 2 = 0 . 13 R 2 = 0 . 12 R 2 = 0 . 09 R 2 = 0 . 12 Submodel 1 Submodel 2 71 Figure 12. Test of Proposed Model for Control Group Perce p t i ons o f Age n t C a ri ng Atten tio n Processes Ob serv atio n a l Learni ng Processes Post-t rainin g Soft ware Self-E fficac y Pre-t r aining Mot i vat i on To Lea r n Pr odu ction Processes Reten tio n Processes Pre-t r aining Soft ware Self-E fficac y Declarative K now ledg e Imme diate Task Performa nce Mot i vat i on Processes 0.28** 0.15 0.14 0.33*** 0.33** 0.35*** 0.36*** 0.24* 0.51*** 0.79*** 0.96*** 0.68*** 0.31* 0.50*** 0.50* 0.22 0.40*** R 2 = 0 . 57 R 2 = 0 . 38 R 2 = 0 . 10 R 2 = 0 . 24 R 2 = 0 . 18 R 2 = 0 . 18 R 2 = 0 . 22 Submodel 1 Submodel 2 72 CHAPTER V DISCUSSION The objectives of the current research were to test the influence of an emotion- based training intervention on training outcomes within the framework for behavior model training developed and validated by Yi and Davis (2003). More specifically, to investigate the impact of an emotionally expressive, embodied computer agent on training outcomes as mediated through the four dimensions of the observational learning processes model. This investigation addresses (a) the call for research studies investigating the underlying factors that lead to training effectiveness and (b) the ongoing debate within the human-computer interaction (HCI) research community regarding whether or not it is beneficial to incorporate human emotion into the HCI. Similar to the original findings of Yi and Davis (2003), the theoretical model of observational learning processes of computer skill training and skill acquisition was successful in explaining the mechanisms through which a modeling-based training intervention influences the training outcomes. This chapter examines the results of the research analysis presented in Chapter 4, and presents a discussion of the findings. Implications for both researchers and practitioners are presented. This chapter concludes with a discussion of the limitations and suggestions for future research. 73 Findings for Full Sample The theoretical model was well supported when tested using the full sample of all data including both the control and experimental treatment groups. In fact, when testing the model with the full data sample, all hypotheses were supported and all causal pathways within the model were significant in a positive direction at the (? = 0.05) level or better (see Figure 10). However, when the research model was tested separately using the respective data samples for the control and experimental treatment groups, the results were less favorable. Findings for the Control and Treatment Groups Training Outcomes The path coefficients between post-training self-efficacy and task performance for the control and treatments groups were (? control = 0.22; ? treatment = 0.20). This is consistent with the original findings of Yi and Davis (2003) who noted a significant path coefficient of (? = 0.23; ? = 0.01). However, hypothesis 2 regarding the causal pathway between post-training self-efficacy and immediate task performance was not supported when the research model was tested separately using the data for the control and treatment groups. Post-hoc power analyses using the software program GPower version 3.0.10 indicated that the calculated power level for the treatment and control groups fell below the suggested power level ( ??1 = 0.80), while the post-hoc power analysis for the full sample was approaching 1 ( ??1 Full Sample = 0.99) (Cohen 1988; Chin and Newsted 1999). Therefore, the failure to attain significance in the causal pathway between post-training self-efficacy and task performance for the individual groups could be attributed to a small effect size (? control = 0.22; ? treatment = 0.20) and a smaller 74 sample size for the individual groups versus the full sample of all data (N Full Sample = 260; N control = 129; N treatment = 131). Research studies have often reported significant relationships between self- efficacy and performance (Ackerman et al., 1995; Kraiger et al., 1993; Mitchell et al., 1994, Salas and Cannon-Bowers, 2001). More specifically, research findings related to computer training found post-training software self-efficacy to be a significant predictor of task performance (Compeau and Higgings, 1995a; Gist et al., 1989; Johnson and Marakas, 2000; Martocchio and Judge, 1997; Martocchio and Webster, 1992). Pre-training Individual Differences The Possible Confounding Affects of Mood Several of the hypotheses were not supported when the research model was tested separately for the control and treatment groups. However, the pattern of causal pathways that fell from significance could be a manifestation of a missing construct. Specifically, a measure of mood was not included in the model. As noted in Chapter 2, the key difference between mood and emotion is specificity and intention. That is, an emotion is directed toward a specific object, while mood is experienced in a more diffuse or global manner (Brave and Nass 2003). Mood tends to bias cognitive processing for a longer period of time than emotion (Davidson 1994). Moods can generally be thought of as an affective filter through which all incoming stimuli are judged. Consequently, moods tend to regulate the activation thresholds for emotions (e.g., a good mood lowers the activation threshold for positive emotions), and conversely emotions tend to cause or contribute to moods (Brave and Nass 2003). 75 When testing the research model using data collected from the control group, pre- training motivation had a significant positive effect on all four observational process dimensions. In contrast, when testing the research model using only the data collected from the treatment group, only the causal paths between pre-training motivation to learn and the attention and production process dimensions were found to be significant. The measures of pre-training motivation to learn and the perceptions of agent caring might be indirectly capturing variation attributable to mood. If so, theoretically it would follow that if empathetic feedback increased the level of perceived agent caring, the amount of variance attributable to mood and captured by perceived agent caring could also increase thereby reducing the amount of variance explained by the relationship between pre- training motivation to learn and the four observational learning process dimensions. With this in mind a post-hoc analysis of submodel one was conducted using the full sample of all data with the addition of a dummy variable. The dummy variable was added as a manifest indicator of group membership (where, empathetic treatment group = 1; control group =0) with a causal link to perceived agent caring (see Figure 10). The results indicated a significant positive relationship between the dummy variable representing the treatment group and perceived agent caring (? = 0.19; ? = 0.01). Therefore, evidence suggests that the empathetic performance feedback presented via an embodied computer agent had a positive and significant effect on trainees? perceptions of agent caring. Furthermore, the amount of variance attributable to mood and captured by perceptions of agent caring would be lower when testing submodel one using the control group data. This could explain why the causal pathways between perceptions of agent 76 caring and the retention and productions process dimensions failed to be significant using the control group data and conversely why the causal pathway between pre-training motivation to learn and the retention and motivation processes failed to be significant using the treatment group data. Training Intervention When testing the research model with data collected from the treatment group, all but one of the casual pathways between perceptions of agent caring and the four observational learning process dimensions was found to be significant. The relationship between the attention process dimension and the training intervention, representing the key causal path in the hypothesized research model, was not found to be significant. While this finding contradicted the hypothesized relationship, it is not entirely unexpected given the ongoing debate among researchers in the field of human-computer interaction. On one side of the debate, HCI researchers contend that emotional expressivity is unnecessary at best and potentially both irritating and distracting (Brave & Nass, 2005). Other researchers who have pioneered the concept of affective computing (Bates 1994; Picard 1997), contend that incorporating emotional expressivity into the HCI ?is necessary to best leverage users? life-time of experience with social interaction (Brave, Nass et al. 2005)?. Therefore, the lack of significance in the causal pathway between perceptions of agent caring and the attention processes dimension for the treatment group supports the notion that emotional expressivity within the HCI is unnecessary at best and potentially both irritating and distracting. That is, the embodied computer agent could 77 have caused the trainees to become distracted rather than increasing attention focus as hypothesized. Implications The effort and cost of incorporating emotion and intelligent agents within a computer-based training environment may outweigh the benefits in a practical sense. If the results of the analysis in the current study could be generalized to encompass all attempts to incorporate emotional expressivity into the human-computer interaction, then perhaps the effort and costs involved outweigh the benefits for practitioners. However, the significant effect of empathetic treatment on perceptions of agent caring found in the post hoc analysis using the full sample could improve the human- computer interaction experience. This could potentially change an individual?s sentiment regarding the use of computers or a specific software program. That is, if an individual becomes more comfortable using a software program such as Microsoft Excel when an intelligent embodied computer agent is incorporated into the human-computer interaction to guide, encourage, and support users then sentiments tend to cause a user to avoid the use of Excel could change. This phenomenon would not be captured in the current study due to its focus on immediate effects of incorporating an embodied computer agent into the training simulation. The bottom line is that most practitioners do not have the time and resources to invest in a phenomenon that may or may not truly exist. Therefore, until research consistently finds a value-added advantage in incorporating emotional expressivity into human-computer interactions, practitioners are well advised to carefully weigh the 78 benefits and costs of incorporating emotional expressivity into computer-based training environments. Limitations As with any research investigation or study, the limitations of the current study should be considered when evaluating the results. First, the central measure of interest in the research model was an emotion-based training intervention. Putting aside the ongoing debate regarding emotion expressivity within human-computer interactions, there is the issue of measuring emotion. Emotions can only be measured indirectly, and the most common method used in human-computer interaction research is post-experience, self- report, questionnaires. This obviously has implications of adding bias into the measurement of emotion as it represents data collected based on an individual?s recollection of an emotional response. The workshop itself was comprised of several segments including pre- and post- questionnaires, a pre-recorded training video, and a simulation in which the embodied computer agent appeared (see Appendices A and D). Although instructed otherwise, it is feasible that participants might have confused the embodied agent with the unseen training modeling target behavior and software skills in the pre-recorded training video. Every effort was made to avoid such confusion, but it existed the resulting data would be extremely biased. Another issue related to the design of the workshop training materials and is central to the behavior modeling training method. Participants generally are given time to try to repeat the observed target behavior modeled by a trainer. Research has indicated that extended practice and note-taking greatly increase the measures of the four 79 observational learning process dimensions (Yi and Davis 2001; Yi and Davis 2003). For the purposes of this study, a guided practice session and measures of training outcome assessments were incorporated into a computer software simulation. This represents an attempt to control for possible confounding effects related to how long or how engaged participants would become during an open, unguided practice session. This was considered necessary given time constraints, the characteristics of participants, and the reward system for participating in the training workshop. These limitations should addressed in future research studies. Future Research Future research studies involving embodied computer agents within a computer- based training environment should include measures of mood within the individual pre- training differences. This could better identify the source of variance found within the pre-training individual difference measures of perceptions of agent caring and motivation to learn noted in the current study. Also, alternative methods of measuring emotion could be employed including, but not limited too: : (a) neurological changes via electroencephalogram (EEG); (b) autonomic nervous system activity (e.g., heart rate, blood pressure, blood pulse volume, respiration, temperature, pupil dilation, skin conductivity, and muscle tension) via electromyography; (c) facial expression via computer-based pattern recognition and image capture; and (d) changes in voice (i.e., pitch range, rhythm, amplitude, and duration). In addition, encapsulating both the training video and the software simulation into one continuous training module would be advisable. This could avoid possible confusion in the self-reported measures regarding perceptions of an embodied pedagogical 80 computer agent and the trainer modeling target behaviors and software skills. Additionally, if the pre- and post-training measures were presented electronically and encapsulated within the training module all data, including the training outcome assessments, could be collected within a self-contained program. The ongoing debate within the HCI research literature regarding the whether or not incorporating emotional expressivity will likely continue for many years to come. Both sides of the debate have merit, but one thing is certain. The youth of today are exposed to emoting interfaces and technologies such as speech recognition and alternative input devices beginning at a very young age. Future expectations of what constitutes and effective human-computer interaction will continue to be shaped and dramatically impacted by new and converging technologies found today in the products like the iPhone by Apple computer and the Wii, Playstation 3, and X-Box gaming and entertainment systems. Future research should investigate the changing nature of the modern computer user in tandem with emerging technologies that quickly become disseminated into to modern society. What may be deemed ?unnecessary and potentially distracting today? may very well become not only value-added but also expected in the world of tomorrow. Conclusion The findings of the current research suggest that an embodied computer agent can be perceived as caring by computer users engaged in a human-computer interaction. This is consistent with previous research within the field of human-computer interaction (Brave, Nass et al. 2005; Lee, Nass et al. 2007; McQuiggan and Lester 2007). Within the context of the current study, the effect of incorporating an embodied computer agent into 81 a computer-based training environment did not extend beyond trainee perceptions to significant improvements in training outcomes as hypothesized. However, these results are still quite disturbing. A computer, the quintessential inanimate object, caused an emotional response in humans. Furthermore, the embodied agent employed to cause this emotional response was represented by nothing more than three still images of a computer generated character and text displayed cartoon-like in a speech bubble. Although no significant differences were found between the treatment and control groups, all causal pathways in the model were found to be significant when using the full sample of all data. This further validates the research framework and the observational learning model of computer software and skill acquisition developed by Yi and Davis (2003). This provides a foundation for future research to (a) further investigate the potential of embodied computer agents to improve the human-computer interaction, and (b) develop a further understanding of the underlying causes of effective computer skill training using Yi and Davis? (2003) validated model of observational learning model of computer software training and skill acquisition. 82 REFERENCES Ackerman, P. L. (1987). "Individual-Differences in Skill Learning - an Integration of Psychometric and Information-Processing Perspectives." Psychological Bulletin 102(1): 3-27. Agarwal, R. and E. Karahanna (2000). "Time flies when you're having fun: Cognitive absorption and beliefs about information technology usage." Mis Quarterly 24(4): 665-694. Anderson, J. R. (1985). Cognitive Psychology and Its Implications. New York, W.H. Freeman. Baldwin, T. T., R. J. Magjuka, et al. (1991). "The Perils of Participation - Effects of Choice of Training on Trainee Motivation and Learning." Personnel Psychology 44(1): 51-65. Bandura, A. (1977). Social learning theory. Englewood Cliffs, N.J., Prentice Hall. Bandura, A. (1986). Social foundations of thought and action : a social cognitive theory. Englewood Cliffs, N.J., Prentice-Hall. Barclay, D., C. A. Higgins, et al. (1995). "The Partial Least Squares Approach to Causal Modeling: Personal Computer Adoption and Use as an Illustration." Technology Studies: Special Issue on Research Methodology 2(2): 285-324. Bass, A. R. and I. J. Firestone (1980). "Implications of Representativeness for Generalizability of Field and Laboratory Research Findings." American Psychologist 35(5): 141-150. Bates, J. (1994). "The Role of Emotion in Believable Agents." Communications of the Acm 37(7): 122-125. Batson, C. D., M. P. Polycarpou, et al. (1997). "Empathy and attitudes: Can feeling for a member of a stigmatized group improve feelings toward the group?" Journal Of Personality And Social Psychology 72(1): 105-118. 83 Bolt, M. A., L. N. Killough, et al. (2001). "Testing the interaction effects of task complexity in computer training using the social cognitive model." Decision Sciences 32(1): 1-20. Bostrom, R. P., L. Olfman, et al. (1990). "The Importance of Learning Style in End-User Training." Mis Quarterly 14(1): 101-119. Brave, S. and C. Nass (2003). Emotion in Human-Computer Interaction. The Human- Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. J. A. Jacko and A. Sears. Mahwah, N.J., Lawrence Erlbaum Associates: 81-96. Brave, S. and C. Nass (2003). Emotion in Human-Computer Interaction. The human- computer interaction handbook : fundamentals, evolving technologies, and emerging applications. J. A. Jacko and A. Sears. Mahwah, N.J., Lawrence Erlbaum Associates: 81-96. Brave, S., C. Nass, et al. (2005). "Computers that are care: investigating the effects of orientation of emotion exhibited by an embodied computer agent." International Journal Of Human-Computer Studies 62(2): 161-178. Burnette, J. J. and P. M. Dunne (1986). "An Appraisal of the Use of Student Subjects in Marketing Research." Journal of Business Research 14(4): 329-343. Chi, M. T. H. and R. Glaser (1985). Problem Solving Ability. New York, Freeman & Company. Chin, W. W. (1998). "Issues and Opinion on Structural Equation Modeling." MIS Quarterly 22(1): vii-vxi. Chin, W. W. (1998). The partial least squares approach to structural equation modeling. Modern Methods for Business Research. G. A. Marcoulides. MahWah, NJ, Lawrence Erlbaum Associates: 295-336. Chin, W. W. (2004, 12/24/04). "PLS: Frequently Asked Questions - Multiple Group Analysis." from http://disc-nt.cba.uh.edu/chin/plsfaq/multigroup.htm. Chin, W. W. and T. A. Frye (2001). PLS Graph User's Guide: Version 3.0, Soft Modeling, Inc. 84 Chin, W. W. and P. R. Newsted (1999). Structural Equation Modeling analysis with Small Samples Using Partial Least Squares. Statistical strategies for small sample research. R. H. Hoyle. Thousand Oaks, Calif., Sage Publications: pp. 307-341. Clore, G. C. (1994). Why Emotions are Felt. The nature of emotion : fundamental questions. P. Ekman and R. J. Davidson. New York, Oxford University Press: 103-111. Clore, G. L. and K. Gasper (2000). Feeling is Believing: Some Affective Influences on Belief. Emotions and Beliefs: How Feelings Influence Thoughts. N. H. Frijda, S. Bem, A. Manstead and K. Oatley. Cambridge, Cambridge University Press: 10- 44. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ, Lawrence Earlbaum Associates. Colquitt, J. A., J. A. LePine, et al. (2000). "Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research." Journal of Applied Psychology 85(5): 678-707. Compeau, D. R. and C. A. Higgins (1995). "Application of Social Cognitive Theory to Training for Computer Skills." Information Systems Research 6(2): 118-143. Compeau, D. R., C. A. Higgins, et al. (1999). "Social cognitive theory and individual reactions to computing technology: A longitudinal study." MIS Quarterly 23: 145- 158. Damasio, A. R. (1994). Descartes' error : emotion, reason, and the human brain. New York, G.P. Putnam. Davidson, R. J. (1994). On Emotion, Mood, and Related Affective Constructs. The nature of emotion : fundamental questions. P. Ekman and R. J. Davidson. New York, Oxford University Press: 51-55. Davis, S. A. and R. P. Bostrom (1993). "Training End Users - an Experimental Investigation of the Roles of the Computer-Interface and Training Methods." Mis Quarterly 17(1): 61-85. Decker, P. J. and B. R. Nathan (1985). Behavior modeling training : principles and applications. New York, Praeger. 85 Detenber, B. H. and B. Reeves (1996). "A bio-informational theory of emotion: Motion and image size effects on viewers." Journal Of Communication 46(3): 66-84. Dolezalek, H. (2005). "2005 Industry Report." Training 42(12): 14-25. Fornell, C. and F. Bookstein (1982). "Two Structural Equation Models: LISREL and PLS Applied to Consumer Exit - Voice Theory." Journal of Marketing Research 19: 440-452. Fornell, C. L. and D. F. Larcker (1981). "Evaluating structural equations models with unobervable variables and measurement error." Journal of Marketing Research 18(1): 39-50. Frijda, N. H. (1994). Varieties of Affect: Emotions, Episodes, Moods, and Sentiments. The nature of emotion : fundamental questions. P. Ekman and R. J. Davidson. New York, Oxford University Press: 59-67. Galvin, T. (2001). "INDUSTRY 2001 REPORT. (Cover story)." Training 38(10): 40. Gattiker, U. E. (1992). "Computer Skills Acquisition: A Review and Future Directions for Research." Journal of Management 18(3): 547. Gefen, D. and D. Straub (2005). "A Practical Guide to Factorial Validity Using PLS- Graph: Tutorial and Annotated Example." Communications of the AIS 16: 91- 109. Gefen, D., D. Straub, et al. (2000). "Structural modeling and regression: Guidelines for research practice." Communications of the AIS 4(7): 1-76. Gist, M., B. Rosen, et al. (1988). "The Influence of Training Method and Trainee Age on the Acquisition of Computer Skills." Personnel Psychology 41(2): 255-265. Gist, M. E. (1987). "Self-Efficacy - Implications for Organizational-Behavior and Human-Resource Management." Academy of Management Review 12(3): 472- 485. Gist, M. E., C. Schwoerer, et al. (1989). "Effects of Alternative Training Methods on Self-Efficacy and Performance in Computer Software Training." Journal of Applied Psychology 74(6): 884-891. 86 Glaser, R. (1990). "The Reemergence of Learning-Theory within Instructional- Research." American Psychologist 45(1): 29-39. Goldstein, A. P. and M. Sorcher (1973). Changing supervisor behavior. New York,, Pergamon Press. Gross, J. J., Ed. (1999). Emotion and Emotion Regulation. Handbook of Personality: Theory and Research. New York, Guilford. Heuer, F. and D. Reisberg (1992). Emotion, arousal, and memory for detail. Hillsdale, N.J., Erlbaum Associates. Hicks, W. D. and R. J. Klimoski (1987). "Entry into training programs and its effects on training outcomes: A field experiment." Academy Of Management Journal 30(3): 542-552. Hughes, C. and M. L. Gibson (1991). "Students as Surrogates for Managers in a Decision-Making Environment: An Experimental Study." Journal of Mangement Information Systems 29(1): 153-166. Johnson, R. D. and G. M. Marakas (2000). "Research report: The role of behavioral modeling in computer skills acquisition - Toward refinement of the model." Information Systems Research 11(4): 402-417. Kirkpatrick, D. (1993). "Making It All Worker-Friendly." Fortune 128(7): 44-&. Klein, J., Y. Moon, et al. (1999). This Computer Responds to User Frustration. CHI '99, human factors in computing systems the CHI is the limit : CHI 99 conference proceedings. M. G. Williams. New York, NY, Association for Computing Machinery: xv, 629 p. Kleinginna, P. R., Jr. and A. M. Kleinginna (1981). "A categorized list of emotion definitions, with suggestions for a consensual definition." Motivation and Emotion 5(4): 345-379. Kraiger, K., J. K. Ford, et al. (1993). "Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation." Journal of Applied Psychology 78(2): 311-328. LeDoux, J. E. (1995). "Emotion: Clues from the brain." Annual Review of Psychology(46): 209-235. 87 LeDoux, J. E. (1996). The Emotional Brain. New York, Simon & Schuster. Lee, C. and P. Bobko (1994). "Self-Efficacy Beliefs: Comparison of Five Measures." Journal of Applied Psychology 79(3): 364-369. Lee, J. E. R., C. Nass, et al. (2007). "The case for caring colearners: The effects of a computer-mediated colearner agent on trust and learning." Journal of Communication 57(2): 183-204. Lim, K. H., L. M. Ward, et al. (1997). "An empirical study of computer system learning: Comparison of co-discovery and self-discovery methods." Information Systems Research 8(3): 254-272. MacCallum, R. C. and J. T. Austin (2000). "Applications of Structural Equation Modeling in Psychological Research." Annual Review of Psychology 51: 201- 226. Marakas, G. M., M. Y. Yi, et al. (1998). "The multilevel and multifaceted character of computer self-efficacy: Toward clarification of the construct and an integrative framework for research." Information Systems Research 9(2): 126-163. Marshall, T. E., R. K. Rainer, et al. (2003). "Complexity and control a determinants of performance with information technology innovations." Journal of Computer Information Systems 43(3): 1-9. Martocchio, J. J. (1994). "Effects of Conceptions of Ability on Anxiety, Self-Efficacy, and Learning in-Training." Journal of Applied Psychology 79(6): 819-825. Martocchio, J. J. and J. Dulebohn (1994). "Performance Feedback Effects in-Training - the Role of Perceived Controllability." Personnel Psychology 47(2): 357-373. Martocchio, J. J. and J. Webster (1992). "Effects of Feedback and Cognitive Playfulness on Performance in Microcomputer Software Training." Personnel Psychology 45(3): 553-578. McQuiggan, S. W. and J. C. Lester (2007). "Modeling and evaluating empathy in embodied companion agents." International Journal of Human-Computer Studies 65(4): 348-360. Nass, C. and Y. Moon (2000). "Machines and mindlessness: Social responses to computers." Journal of Social Issues 56(1): 81-103. 88 Newhagen, J. E. and B. Reeves (1991). Emotion and memory responses to negative political advertising. Television and political advertising. F. Biocca. Hillsdale, N.J., L. Erlbaum Associates: 197-220. Newhagen, J. E. and B. Reeves (1992). "The Evenings Bad-News - Effects Of Compelling Negative Television-News Images On Memory." Journal Of Communication 42(2): 25-41. Noe, R. A. (1986). "Trainees Attributes And Attitudes - Neglected Influences On Training Effectiveness." Academy Of Management Review 11(4): 736-749. Noe, R. A. and N. Schmitt (1986). "The Influence of Trainee Attitudes on Training Effectiveness - Test of a Model." Personnel Psychology 39(3): 497-523. Olfman, L. and R. P. Bostrom (1991). "End-user software training: An experimental comparison of methods to enhance motivation." Journal of Information Systems 1(4): 249-266. Olfman, L. and M. Mandviwalla (1994). "Conceptual Versus Procedural Software Training For Graphical User Interfaces - A Longitudinal-Field Experiment." Mis Quarterly 18(4): 405-426. Parrott, G. W. and M. P. Spackman (2000). Emotion and memory. Handbook of Emotions. M. Lewis and J. M. Haviland-Jones. New York, Guilford Press: 476- 490. Picard, R. W. (1997). Affective computing. Cambridge, Mass., MIT Press. Reeves, B. and C. Nass (1998). The media equation : how people treat computers, television, and new media like real people and places. Stanford, Calif. New York, CSLI Publications ; Cambridge University Press. Reeves, B. and C. I. Nass (1998). The media equation : how people treat computers, television, and new media like real people and places. Stanford, Calif. New York, CSLI Publications ; Cambridge University Press. Reeves, B. R., J. Newhagen, et al. (1991). "Negative and Positive Television Messages - Effects of Message Type and Context on Attention and Memory." American Behavioral Scientist 34(6): 679-694. 89 Robinson, J. C. (1982). Developing managers through behavior modeling. Austin, Tex. San Diego, Calif., Learning Concepts ; Distributed by University Associates. Santhanam, R. and M. K. Sein (1994). "Improving End-User Proficiency - Effects of Conceptual Training and Nature of Interaction." Information Systems Research 5(4): 378-399. Shaw, C., R. Shaw, et al. (2008). People Putty, Haptek, Inc. Simon, S. J., V. Grover, et al. (1996). "The relationship of information system training methods and cognitive ability to end-user satisfaction, comprehension, and skill transfer: A longitudinal field study." Information Systems Research 7(4): 466- 490. Simon, S. J. and J. M. Werner (1996). "Computer training through behavior modeling, self-paced, and instructional approaches: A field experiment." Journal Of Applied Psychology 81(6): 648-659. Straub, D., M. Boudreau, et al. (2004). "Validation Guidelines for IS Positivist Research." Communications of the Acm 14: pp. 380-426. Tannenbaum, S. I., J. E. Mathieu, et al. (1991). "Meeting Trainees Expectations - the Influence of Training Fulfillment on the Development of Commitment, Self- Efficacy, and Motivation." Journal of Applied Psychology 76(6): 759-769. Taylor, P. J., D. F. Russ-Eft, et al. (2005). "A meta-analytic review of behavior modeling training." Journal of Applied Psychology 90(4): 692-709. Thorson, E. and M. Friedsad (1985). The Effects of Emotion on Episode Memory for Television Commercials. Advances in Consumer Psychology. P. Cafferata and A. Tybout. Lexington, MA, Lexington: 131-136. Venkatesh, V. and M. G. Morris (2000). "Why don't men ever stop to ask for directions? Gender, social influence, and their role in technology acceptance and usage behavior." Mis Quarterly 24(1): 115-139. Winston, W. L. (2004). Microsoft Excel Data Analysis and Business Modeling. Redmond, Washington, Microsoft Press. 90 Yi, M. Y. and F. D. Davis (2001). "Improving computer training effectiveness for decision technologies: Behavior modeling and retention enhancement." Decision Sciences 32(3): 521-544. Yi, M. Y. and F. D. Davis (2003). "Developing and validating an observational learning model of computer software training and skill acquisition." Information Systems Research 14(2): 146-169. Zhang, P. and N. Li (2005). "The Intellectual Development of Human-Computer Interaction Research: A Critical Assessment of the MIS Literature (1990-2002)." Journal of the Association of Information Systems 6(8): 227-292. 91 APPENDICES 92 APPENDIX A INSTRUCTIONAL DESIGN PROCESS 93 Several key objectives were identified leading to the decision to develop custom training materials including: (a) an experimental treatment operationalized through immediate performance feedback delivered by an intelligent computer agent (Brave, Nass et al. 2005); (b) presenting behavioral modeling based training via a prerecorded video (Yi and Davis 2003); (c) measuring task performance and declarative knowledge within a computer-based simulation of the software environment; (d) measuring task specific software self-efficacy that is highly integrated with the subject matter of the training session; and (e) a time limitation of approximately thirty minutes given the target participant pool and related constraints. Commercial training materials for Microsoft Excel 2003 were evaluated based on their ability to meet all objectives. Unfortunately, no third party training materials met all previously stated objectives. A group of three subject matter experts (SME), including the primary researcher, met to evaluate the situation. Each group member had eight or more years of experience in the development and deployment of computer-based training materials for the Microsoft Excel software program at the undergraduate level. The group collaboratively decided that developing custom training materials would be the only option capable of meeting all objectives. Furthermore, the primary researcher was advised to limit the focus of the custom training materials to a small set of task specific software skills. The experimental treatment in this study was a training intervention designed to assess the impact of emotional response, specifically empathy, on learning outcomes during computer-based software training. Based on the research framework developed and validated by Yi and Davis (2003), a training intervention was incorporated into a computer software training workshop developed specifically for this study (see Appendix 94 A). In keeping with prior IS research involving CSA (Marakas, Yi et al. 1998; Johnson and Marakas 2000; Yi and Davis 2003), Microsoft Excel 2003 was chosen as the subject matter for the training workshop (see Appendix A). Many undergraduate business courses require students to use specialized computer software including Microsoft Excel (Winston 2004, p. vii). Consequently, the process of CSA for a business professional often begins while completing the requirements of an undergraduate business degree and continues throughout their professional career (Winston 2004, p. vii). With this in mind, undergraduate business students were recruited as participants. Instructors in higher education who teach students to solve business problems in the classroom using Excel were asked to which skills students tend to struggle with during informal interviews. They were also asked what skills were required within the context of their course requirements. These interviews revealed a common Excel skill considered problematic across the courses (including information systems, accounting, finance, economics, and quantitative analysis) was related to using absolute and relative cell references in Excel formulas and functions. Another commonality was that students in general have a hard time grasping the concept of embedding functions inside of excel formulas, and the concept of logical tests to evaluate data. Given this information the standards for Microsoft Excel Expert certifications were reviewed and the concept of embedded IF functions was identified as a skill set that would address the common issues identified in the informal interviews. A business scenario was developed to demonstrate the use of absolute cell references, relative cell references and embedded IF functions in Excel. The three SME?s reviewed the proposed 95 business scenario and collaboratively agreed on a business problem that would incorporate the use of the target software skills. The primary researcher then audio taped a step-by-step ?talk through? while working through the process of using the target software skills to solve the business problem. The steps were broken down in to micro tasks and following the framework for developing task specific software self-efficacy (SSE) measures as outlined by Johnson and Marakas (2000), existing SSE items used within the research model developed and validated by Yi and Davis (2003) were adapted to closely match the micro tasks identified in the ?talk through? analysis. The measures of declarative knowledge and task performance were also taken directly from the step-by-step micro tasks required to solve the business problem in the custom scenario. Performance feedback messages to be delivered by the embodied computer agent were then derived from the declarative knowledge and task performance measures for both incorrect and correct responses. Relevant literature was reviewed to identify examples of empathetic language found to illicit emotional response in computer users (Reeves, Newhagen et al. 1991; Reeves and Nass 1998; Nass and Moon 2000; Brave and Nass 2003; Brave, Nass et al. 2005). Two sets of messages were developed for (a) neutral, non-empathetic of performance feedback and (b) empathetic performance feedback. Each response was printed on index cards and a card sorting technique was employed. Five instructors in higher education were given the definition of empathy and asked to separate the messages into two groups, empathetic and emotional neutral performance feedback. They were also asked to offer feedback. The performance feedback messages were revised through four iterations of the card sorting technique. 96 After the forth iteration all five instructors agreed 100% based on the card sorting method. A final set of nine graduate students were asked to sort the messages using the card sorting technique, and sorted the cards into the designated groups with 97% level of agreement. One final message was revised when one of the graduate students noted concern over the wording of one of the messages. The final set of declarative knowledge questions and task performance micro tasks that were incorporated into the software training simulation along with the corresponding performance feedback messages can be found in table 14 of appendix D. A training video that included introductory slides, a video capture of and excel spreadsheet demonstrating the target behavior also included voice over narration. The narrator in the pre-recorded training video introduced the workshop as noted in the following slides and scripts prior to demonstrating the target behavior in an Excel spreadsheet. Slide 1 Narration Slide 1: ?Hello and welcome to this training session on advanced Excel techniques.? 97 Slide 2 Narration Slide 2: ?Our learning objectives are to help you understand: ? How to use an IF function in Excel; ? How to create complex formulas using embedded IF functions; ? How to copy formulas to evaluate a list; and, ? How to use both relative and absolute cell references within an Excel formula? Slide 3 Narration Slide 3: ?First of all, let?s take a look at a typical IF function within Excel. The IF function in Excel is usually displayed as?? 98 Slide 4 Narration Slide 4: ?Within the IF function three things are specified: 1. A logical test that compares or evaluates the contents of a cell; 2. The value to be displayed if the logical test is true (For example if we compared two values ? is one less than two ? the result would be true); 3. The value to be displayed if the logical test is false (For example if we compared two values ? is two less than one ? the result would be false)? Slide 5 Narration Slide 5: Read the business scenario and then begin the excel software demonstration to solve the business problem. 99 APPENDIX B PARTICIPANT RECRUITMENT 100 Step 1 - Acquiring Access to a Pool of Potential Participants The several instructors within the college of business at a large university in the southeast region of the United States were contacted regarding the research study. They were asked to allow their students to participate in a software training workshop for research purposes. Several instructors agreed and offered extra credit in their courses to students who voluntarily chose to participate in the research study. Step 2 ? The First Contact with Potential Participants The recruitment process began with a brief announcement delivered at the beginning of regularly scheduled lecture classes. The script of this in-class announcement can be found in Appendix G. The students were informed that they would receive further information in an email message (see Appendix G). The email message included a URL link for a web site where participants could reserve a seat for a workshop training session at a date and time that fit conveniently within their schedules. This was necessary due to the limited number of computers available in the computer lab where the training workshops were conducted. Step 3 ? Attending the Training Workshop Participants who reserved a seat through the online reservation system arrived simply needed to arrive at the self-selected training session. A list of participants was pulled from the reservation web site and printed before each session. After completing the workshop participants were asked to sign the reservation list next to their name as confirmation of their participation. 101 APPENDIX C SCRIPT FOR TRAINING WORKSHOP FACILITATOR 102 Notes and Script to Facilitator for Training Workshop ? As Students Arrive: Note to Instructor: As students arrive in the lab the greet them, give them a copy of the information letter for participants, direct them to an unoccupied computer station, and let them know approximately how much time they have before the session begins. ? Close the Lab Door and Begin: Note to Instructor: At the time the current workshop training session is scheduled to begin close the computer lab door and begin the workshop. ? Introduction Announcement to Begin Workshop: Instructor Script: ?Welcome to this training seminar on advanced Excel spreadsheet skills. We expect this training session to last about 30 minutes, but we have scheduled one hour for this training session to make sure everyone has enough time to complete the workshop without feeling rushed. You should have received an information letter that informs you of your rights as a participant in this research study as you entered the computer lab. The information letter also has contact information listed if you have questions regarding your participation after completing the workshop. If you were not given a copy of the letter when you arrived please let me know and I will give you one now.? ? Explain the Method for Anonymous Data Collection: Instructor Script: ?I will now pass out a set of self adhesive labels to each of you. 103 These labels have a unique participation code printed on them. This code will be used to identify coordinate all of the research data that will be collected from you during this training workshop. The research data will be collection will be anonymous, but please be careful to follow the instructions regarding how and when to use the labels and your unique participation code.? ? Distribute Labels: Note to Instructor: Distribute a set of labels to each participant, and make certain that everyone has a received them. ? Explain How Participation and Extra Credit Will Be Confirmed: Instructor Script: ?Now I am sure all of you want to make sure that you receive the extra credit offered by your instructor for participating in this workshop. You have registered online to attend this specific time which was necessary to manage the limited seating available in the computer lab. So, we have a list of everyone who should be present at this training session, and the list has been printed out and placed on the table near the exit door. After completing the workshop please take a moment to sign the list in the space provided next to your name. This list will be used to confirm your participation in the study, and will be forwarded to your instructor to ensure that you receive extra credit for your participation. ? ? Distribute the Pre-training Surveys with Instructions: Instructor Script: ?I am now passing out the two pre-training questionnaires. Please adhere one of the self-adhesive labels to the front right corner of each sheet of paper 104 you receive. This label is printed with a code that will be used to coordinate the data collected from you while still keeping your identity anonymous.? ? Survey Instructions: Note to Instructor: Use the document camera at the instructor station to project each survey instrument on the screen at the front of the room. Read the instructions for completing the survey and ask if there are any questions. Ask them to keep the surveys at their computer station until the end of the workshop. ? Behavior Modeling Training Video: Note to Instructor: While the students are completing the surveys, queue up the pre- recorded training video on the instructor station computer so that it is ready to play as soon as everyone completes the survey. It should take approximately five minutes for everyone to complete the surveys. Confirm that everyone has finished the surveys, and then introduce the video as scripted. Instructor Script: ?I will know play a training video for you that will last approximately 12 minutes. The trainer in the video will list the objectives and skills that will be demonstrated, describe the advanced Excel technical skills on a conceptual level, and then demonstrate how to apply these skills to solve a business problem.? ? Introduce and Give Instructions for the Training Simulation: Note to Instructor: When the video ends give instructions on the use of the software simulation as scripted. Instructor Script: ?Now, please login to the computer at your lab station. There is a 105 CD located at each computer station. After logging into the lab computer insert the CD into the CD player. This will automatically start a custom software simulation that has the look and feel of Microsoft Excel, but you will not actually be using the Microsoft Excel software you will be interacting with a simulation. The simulation program will initially prompt you to enter a participation code. When you see this prompt locate the code printed on the self-adhesive labels you were given at the beginning of this training seminar and enter the code printed on the labels. This will allow the researchers to coordinate the data collected from the paper surveys with the data automatically collected by the software simulation. After entering your participation code the simulation program will begin. You will be presented with an interface that looks exactly like Microsoft Excel, and on a panel to the right you will see an image of computer based tutor. This computer tutor will prompt you to complete tasks, answer multiple-choice questions, as it guides you through the process of solving the same business problem presented in the training video. Your personal training tutor will also provide you immediate feedback regarding your performance.? ? Explicitly State that the Computer Tutor is Not a Real Person: Instructor Script: ?Please keep in mind that your personal computer tutor is not a real person and is completely computer generated. If you experience any computer problems or have questions please quietly raise your hand and I will come to your lab station and assist you in anyway that I can.? 106 ? Tell Them What to do After Completing the Simulation: Instructor Script: ?When you complete the simulation, please log out of your lab computer.? ? Instructions After Logging Out: Instructor Script: ?While you are working through the simulation I will place a set of post-training survey questions face down at your workstation. After you log out of your computer, you can begin completing the post-training survey questions. As previously instructed, place a self-adhesive label at the top right corner of each sheet of paper and then answer all survey questions.? ? Instructions for Exiting the Computer Lab: Instructor Script: ?After completing the surveys, your participation in this training workshop is complete and may leave the computer lab at your convenience. Other participants may still be working through the simulation or completing the surveys after you are finished. So, when you are ready to leave please quietly gather up your completed pre- and post-training surveys, double-check to make sure each sheet has a label, and place your paper surveys in the box located next to the extra-credit participation list. Please do not forget to sign the workshop participation list for this training session, before you exit the lab. This will ensure that you will receive the extra credit from your instructor for participating.? ? Thank Them for Participating and Tell Them to Begin the Simulation: Instructor Script: ?I?d like to thank you in advance for attending the training 107 workshop. Your participation is greatly appreciated. You may now begin the software training simulation.? 108 APPENDIX D SOFTWARE SIMULATION DEVELOPMENT 109 Software Simulation The software simulation was a custom computer program designed to (a) assess learning outcomes and (b) provide an opportunity for trainees to repeat the behavior modeled in the software training video. The simulation was created using Macromedia Authorware version 7, and deployed as an executable file on an autorun compact disk (CD). That is, the CD was equipped to run the software simulation program automatically as soon as it was inserted into the CD reader of a computer. Anonymous Simulation Login and Tracking When the software simulation program started it immediately displayed a screen that prompted the trainees to enter a participation code (see Figure 13). This option allowed the trainees to remain anonymous while the participation code was used to match the training outcome data collected by the simulation program with the data collected to via the paper-based surveys. Further information regarding participant anonymity and the use of participation codes can be found in Appendices B and C. Figure 13. Initial Login Screen of the Custom Software Simulation Program 110 Treatment Group Assignment Process After entering a participation code a second screen appeared prompting the trainees to indicate their gender and ethnicity (see Figure 14). This demographic information was used to assign trainees to a treatment group. The simulation program created three variables using data gathered from this screen and assigned a numerical value to each one: (a) Gender (Male = 10; Female = 20); (b) Ethnicity (African American = 1; Asian = 2; Caucasian = 3; and Hispanic = 4); and (c) demographic = (the sum of the numerical values stored in the variables Gender and Ethnicity). When the trainee clicked the submit button the simulation program inserted the demographic variable and the participation code into a remote database on the local area network. It then checked the database and returned a record count of the number of database entries that matched the demographic variable self-reported by the trainee. If the value of the record count was an even number the trainee was assigned to the experimental treatment group and if it was an odd number the trainee was assigned to the control group. Therefore, the computer automatically and randomly assigned trainees to groups while ensuring that the demographic characteristics of trainees were evenly distributed across the two groups. Figure 14. Second Screen of the Custom Software Simulation Program 111 Software Simulation Window Layout The third screen within the simulation presented the learning objectives of the training simulation and followed a basic three-section layout that remained constant throughout the simulation (see Figure 15). The three-section layout of the simulation presentation window included: (a) a narrow title bar across the top of the window; (b) a large area in the bottom right, taking up most of the window area, represented the active workspace where the interactive Excel task environment was displayed; and (c) a narrow side panel on the left of the screen where the embodied computer agent and guided instructions were presented. Figure 15. Introduction Screen of the Custom Software Simulation Program 112 The Embodied Computer Agent Following Brave et al. (2005), the computer agent was embodied, or visually represented, by one of three still images and communicated asynchronously with trainees via text messages displayed within a speech bubble (see Figure 16). In the interest of the scientific method, lifelike still images of a computer-generated, female character were used rather than photos of actual humans to ensure repeatability. The images representing the computer agent were generated by a computer program called People Putty version 1.14 (Shaw, Shaw et al. 2008). People putty generates 3-dimensional life-like human characters that are highly interactive and capable of exhibiting human emotion. Three still images of the interactive character generated by People Putty were created using a screen capture software program called GrabItPro 6.02. These three images represented the computer agent expressing happy, sad, and neutral emotion. When capturing the images of the People Putty character the emotion settings within People Putty were set to: (a) ?Happy? (see Figure 16); (b) ?Sad? (see Figure 17); and neutral (see Figure 18). A group of nine graduate students were given the three images and asked to match them with the words happy, sad, and expressionless. All nine graduate students matched the three images with the corresponding emotion with 100% accuracy. 113 Figure 16. ?Happy? Emotion Settings for People Putty Character Figure 17. "Sad" Emotion Settings for People Putty Character 114 Figure 18. "Neutral" Emotion Settings for People Putty Character. Training Outcome Assessment and Guided Practice The embodied computer agent guided all participants through the software simulation while presenting multiple choice questions and instructions. The multiple choice questions were designed to measure declarative knowledge while the instructions were designed to measure immediate task performance (see Table 15). Immediately after a participant responded to a question or instruction, the agent presented personalized feedback based on their performance. In the control group the agent was always represented visually by a human face void of expression. In contrast, the empathetic treatment group received performance feedback that was emotionally varied. That is, when a participant in the treatment group responded correctly, the agent was visually 115 represented by a smiling human along with empathetic performance feedback. Conversely, when a participant in the treatment group responded incorrectly, the agent was visually represented by a frowning human face along with empathetic performance feedback (see Table 14). Figure 19. Sample Presentation Window of the Custom Software Simulation Program 116 Table 14 Performance-based Feedback for Step 9 for the Control and Treatment Groups Control Group Treatment Group Correct Incorrect Correct Incorrect Since the simulation and assessment program was highly interactive, the time it took to complete this portion of the training session varied between participants. In general, participants took between five to 15 minutes to complete the computer based simulation and assessment program. 117 118 119 120 APPENDIX E PRE-TRAINING SURVEY INSTRUMENTS 121 Pre-Training Motivation to Learn Survey Instrument 122 Pre-Training Software Self-Efficacy Survey Instrument 123 APPENDIX F POST TRAINING SURVEY INSTRUMENTS 124 Post-Training Perceptions of Computer Agent Survey Instrument (Note ? Five adjective pairs were used as a pseudo measure of empathetic emotion: (1) Compassionate-Not Compassionate; (2) Unselfish-Selfish; (3) Friendly-Unfriendly; (4) Cooperative-Competitive; and, (5) Warm-Cold) 125 Post-Training Observational Learning Processes Survey Instrument (Note ? Items 1-4 measure attention processes; items 5-8 measure retention processes; items 9-12 measure production processes; items 13-16 measure motivation processes) 126 Post-Training Software Self-Efficacy Survey Instrument 127 APPENDIX G SCRIPT OF IN-CLASS RECRUITMENT ANNOUNCEMENT 128 The lead investigator or the instructor for the course or the course instructor will address the students during a regularly scheduled class prior to giving each student a copy of the information letter. This scripted announcement will be given to extend an invitation to participate in this research study, give a brief introduction, and explain how to sign up to participate through the online reservation system. The script for this address follows: ?In an ongoing attempt to investigate improvements in online software training, you are invited to participate in a research study. You will receive extra credit toward your course grade(s) in MNGT 3140 for participating. If you choose not to participate but would like an opportunity to earn extra credit your instructor will allow you complete an alternative assignment.? ?Your decision to or not to participation will not influence your grade in this class, your relationship with the College of Business Management Department, or your relationship with Auburn University.? ?You will receive an email with a link to the participant online reservation system within the next 24 hours.? 129 APPENDIX H FOLLOW-UP EMAIL ANNOUNCEMENT FOR PARTICIPANT RECRUITMENT 130 Greetings XXX XXX! You are invited to participate in a research study designed to help establish a ?best practices? approach to developing and delivering online computer skill training. The study is being conducted by J. Ken. Corley, Doctoral Student, under the direction of R. Kelly Rainer, Jr., Ph.D. in the Auburn University Department of Management. You were selected as a possible participant because you are currently enrolled as a student in XXXX. You must be 19 years of age or older to participate in the survey. If you do not meet this minimum age requirement you must provide proof of parental consent from a parent or legal guardian PRIOR to participating in this study. If you decide to participate in this research study, please reserve a seat through our online reservation system for November 25, 26, 28 and again on December 2 & 3 from 6:00pm - 10:00 pm by clicking here http://mis.aug.edu/hci_research/index.cfm . A reservation is required due to limited seating in the computer lab. When you arrive to participate in this research study you will be asked to answer a series paper-based of survey questions, watch a short 5-7 minute software training video, and work through a software training simulation. Your total time commitment will be approximately 30 minutes. You will receive compensation for your participation in the form of extra credit toward your course grade in XXX. Please verify exactly how much extra credit your participation is worth from your instructor professor XXX PRIOR to making a decision to participate. If you have questions about this study, please ask them now or contact J. Ken. Corley or R. Kelly Rainer, Jr., Ph.D. through the contact information provide at the end of this letter. If you have questions about your rights as a research participant, you may contact the Auburn University Office of Human Subjects Research or the Institutional Review Board by phone (334)-844-5966 or e-mail at hsubjec@auburn.edu or IRBChair@auburn.edu. ___________________________________ J. "Ken." Corley II, Doctoral Student Office Phone: 706.667.4538 Mobile Phone: 706.414.3331 Email: corleij@auburn.edu 131 APPENDIX I INFORMATION LETTER FOR PARTICIPANTS 132 133