Predictors of Dissemination Success of STEM Learning Innovations: An Empirical Investigation by David Michael Bourrie A dissertation submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 2, 2014 Keywords: dissemination, engineering, educational innovations, innovation diffusion, content analysis, Delphi study Copyright 2014 by David M. Bourrie Approved by Chetan S. Sankar, Chair, COB Advisory Council Professor of IS L. Allison Jones-Farmer, C&E Smith Associate Professor, Statistics and Business Analytics Casey G. Cegielski, Associate Professor of Management Information Systems R. Kelly Rainer, Privett Professor of Management Information Systems ii Abstract Although a great deal of work has been done to develop new educational innovations, in the field of engineering, few innovations have found widespread acceptance in the classroom. Therefore, frameworks that conceptualize the interactions among the variables that influence successful dissemination are critical. In this dissertation, dissemination is defined as a process of first creating an awareness of an educational innovation, then influencing the intention to adopt, the actual adoption, and finally the routine use of that innovation (Fincher, 2000; Gravestock, 2002; Hutchinson & Huberman, 1994; King, 2003). This dissertation establishes the importance of achieving three research goals: 1) developing a framework that describes the interrelationships among the characteristics of educational innovations and an organizations? readiness to disseminate, 2) having an expert panel rank the characteristics of the educational innovations and readiness of faculty members, administrators, and students to disseminate, and 3) testing whether important characteristics of educational innovations and readiness of faculty members variables are significant predictors of successful dissemination. Three papers using different methodologies were sequentially developed to achieve the research goals: 1) a systematic literature review of recent dissemination literature, 2) a Delphi study of grant recipients who had disseminated educational innovations, and 3) a survey of electrical engineering and computer science faculty members. Analysis of the results from these papers led to the following findings: iii 1. Ease of use of an educational innovation and care about student learning outcomes were significant predictors of intention to adopt that innovation; 2. Efficacy of faculty members toward change and valence moderated the relationship between ease of use and intention to adopt educational innovations; 3. Attitude to educational innovation moderated the relationship between ease of use and intention to adopt educational innovations and moderated the relationship between intention to adopt and adoption; and 4. Awareness of others using an educational innovation was a significant predictor of adopting an educational innovation. The results of this dissertation may be leveraged by developers, faculty members, department chairs, college deans, and grant program managers, to successfully disseminate educational innovations in engineering and technology disciplines. iv Acknowledgments First and foremost, I would like to thank Dr. Chetan S. Sankar, my dissertation chair and mentor, whose wisdom in the area of educational innovations and selfless guidance has helped me immeasurably. I would also like to thank the members of my dissertation committee: Dr. L. Allison Jones-Farmer, Dr. Casey G. Cegielski, and Dr. R. Kelly Rainer for their valuable guidance through this challenging process and their influence in moving from the Master?s program to the Doctoral program. I would like to thank Dr. Rick Hanson, the outside reader for the Graduate School for his valuable feedback. I would also like to thank the Graduate School, the Department of Management, and the Department of Aviation and Supply Chain. Special thanks are due to my parents and sister for their loving support. You enabled me to move out of the restaurant business and successfully complete my degree! Finally, I would like to thank the Division of Undergraduate Education at the National Science Foundation for funding this research under grant # 1140542. Any opinions, findings, and conclusions or recommendations expressed in this dissertation are those of the authors and do not necessarily reflect the view of the National Science Foundation. v Table of Content Abstract ......................................................................................................................................... ii Acknowledgements ...................................................................................................................... iv Table of Contents .......................................................................................................................... v List of Tables .............................................................................................................................. xii List of Figures ............................................................................................................................ xiii Chapter 1. ...................................................................................................................................... 1 Introduction ....................................................................................................................... 1 Problem ............................................................................................................................. 2 Purpose .............................................................................................................................. 5 Narrowing the Scope......................................................................................................... 6 Research Design and Methods .......................................................................................... 8 Mixed Methods Research Design ......................................................................... 8 Chapter Two (Paper One) ................................................................................... 10 Chapter Three (Paper Two) ................................................................................ 11 Chapter Four (Paper Three) ................................................................................ 12 Chapter Five ........................................................................................................ 13 Definitions....................................................................................................................... 13 Chapter 2. Conceptualizing Interactions Between Characteristics and Organizational Members? Readiness To Disseminate Educational Innovations .................................................................. 14 Abstract ........................................................................................................................... 14 vi Background ......................................................................................................... 14 Purpose/Hypothesis............................................................................................. 14 Design/Method .................................................................................................... 14 Results ................................................................................................................. 15 Conclusions ......................................................................................................... 15 Introduction ..................................................................................................................... 16 Definitions....................................................................................................................... 18 Dissemination ..................................................................................................... 18 Educational Innovations...................................................................................... 18 Organizational Readiness.................................................................................... 19 Need for a Conceptual Framework on Dissemination of Educational Innovations ........ 19 Dissemination Frameworks ................................................................................ 20 Concerns-based adoption model ............................................................. 20 Diffusion of innovation theory................................................................ 21 Diffusion of service organizations .......................................................... 21 Technology acceptance model ................................................................ 23 Need for a New Conceptual Framework ............................................................ 23 Research Methodology ................................................................................................... 24 Identifying and Describing the Relevant Research ............................................. 25 Critically Appraising the Research Reports in a Systematic Matter ................... 26 Synthesizing the Findings ................................................................................... 27 Results ............................................................................................................................. 28 Critically Appraising the Research Reports in A Systematic Manner to Achieve the First Goal....................................................................................................... 28 vii Synthesizing the Findings ................................................................................... 35 Synthesis step one ................................................................................... 35 Synthesis step two ................................................................................... 37 Examples of the Framework in Practice ............................................................. 40 Moving from quadrant C/B to quadrant D/A .......................................... 40 Moving from quadrant D to quadrant A ................................................. 41 Moving from quadrant B to quadrant A ................................................. 42 Moving from quadrant C to quadrant A ................................................. 43 Moving from quadrant B to quadrant D ................................................. 44 Moving from quadrant C to quadrant B .................................................. 44 Discussion ....................................................................................................................... 45 Potential Use of the Framework by Stakeholders ............................................... 45 Limitations and Future Research .................................................................................... 46 Conclusion ...................................................................................................................... 49 Chapter 3. Identifying Characteristics of Dissemination Success Using an Expert Panel ......... 51 Abstract ........................................................................................................................... 51 Introduction ..................................................................................................................... 52 Background ..................................................................................................................... 54 Educational Innovations...................................................................................... 54 Factors that Relate to the Dissemination Process ............................................... 55 The Delphi Method ......................................................................................................... 56 Participation by Round ....................................................................................... 59 Development of the Characteristics of Dissemination Success (CODS) Framework .... 67 viii The Stages of Disseminating Educational Innovations .................................................. 67 Ranking Characteristics of Educational Innovations, Students, Faculty Members, and Administrators .............................................................................................. 69 Characteristics of the innovation ............................................................ 69 Characteristics of the students ................................................................ 70 Characteristics of faculty members ......................................................... 71 Characteristics of administrators ............................................................. 72 Limitations and Future Research .................................................................................... 73 Conclusions ..................................................................................................................... 74 Chapter 4. The Influence of Characteristics of Innovations and Readiness of Faculty Members Toward Dissemination of Educational innovations: An Empirical Study .................................. 76 Abstract ........................................................................................................................... 76 Introduction ..................................................................................................................... 77 Research Model and Hypotheses .................................................................................... 79 Parts of the CODS Framework ........................................................................... 79 Part 1: Characteristics of Educational Innovations ................................. 79 Part 2: Readiness of Faculty Members Toward Educational Innovations .............................................................................................. 80 Part 3: Dissemination .............................................................................. 81 Research Model .................................................................................................. 81 Relationship 1: Between Characteristics of Innovations and Intention to Adopt....................................................................................................... 82 Relationship 2: Between Intention to Adopt and Adoption .................... 82 Relationship 3: Between Adoption and Routine Use ............................. 83 Relationship 4: Between Readiness of Faculty Members Toward Educational Innovations and Dissemination ........................................... 83 ix Relationship 5: Moderating Relationships .............................................. 84 Methodology ................................................................................................................... 86 Questionnaire Development................................................................................ 86 Section 1: Educational Innovations ........................................................ 86 Section 2: Characteristics of Educational Innovations ........................... 87 Section 3: Readiness of Faculty Members Toward Educational Innovations .............................................................................................. 88 Section 4: Dissemination ........................................................................ 89 Section 5: Demographic and control variables ....................................... 90 Section 6: Assessment of common method bias ..................................... 91 Pilot Study ........................................................................................................... 91 IRB Approval ...................................................................................................... 92 Sample................................................................................................................. 92 Statistical Analysis .............................................................................................. 94 Results ............................................................................................................................. 95 Stage 1: Intention to Adopt ................................................................................. 95 Stage 2: Adoption ............................................................................................. 101 Stage 3: Routine Use ......................................................................................... 106 Findings and Implications ............................................................................................. 109 Ease of Use of an Educational Innovation is a Significant Predictor of Intention to Adopt that Innovation ....................................................................................... 110 Finding .................................................................................................. 110 Implication ............................................................................................ 110 Attitude to Educational Innovation was the Most Important Readiness of Faculty Member Variable .............................................................................................. 111 x Finding .................................................................................................. 111 Implication ............................................................................................ 111 Care About Student Learning Outcomes is a Significant Predictor of Intention to Adopt Educational Innovations......................................................................... 111 Finding .................................................................................................. 111 Implication ............................................................................................ 112 Efficacy of Faculty Members Toward Change and Valence Moderate the Relationship Between Was of Use and Intention to Adopt Educational Innovations ........................................................................................................ 112 Finding .................................................................................................. 112 Implication ............................................................................................ 112 Awareness of Educational Innovations Through Peers is a Significant Predictor of Faculty Members Adopting Educational Innovations ...................................... 113 Finding .................................................................................................. 113 Implication ............................................................................................ 113 Limitation, Future Research, and Conclusion............................................................... 114 Limitations and Future Research ...................................................................... 114 Conclusions ....................................................................................................... 115 Chapter 5. Major Limitations, Future Research Possibilities, and Major Contributions .......... 117 Major Limitations ......................................................................................................... 117 Future Research Possibilities ........................................................................................ 117 Dissemination of Engineering Educational Innovations ................................... 117 Dissemination of Healthcare Practices ............................................................. 118 Dissemination of Cybersecurity and Information Technology Security .......... 120 Major Contributions ...................................................................................................... 120 xi References ................................................................................................................................. 123 Appendix 2-1. PRISMA diagram summarizing the sources contributing to the systematic review ........................................................................................................................................ 154 Appendix 2-2. List of journals and conference proceeding included in this study .................. 155 Appendix 2-2. Citation for the Configurative Review .............................................................. 157 Appendix 3-1. Categorized Participant Comments from Round One of the Delphi Study ...... 162 Appendis 4-1. Characteristics of innovations and readiness variables that influence dissemination ............................................................................................................................ 195 Appendix 4-2. Survey ............................................................................................................... 197 Appendix 4-3. Approval Letter from Human Subjects Institutional Review Board ................ 207 xii List of Tables Table 2-1 Perceived characteristics that may affect dissemination ............................................ 29 Table 2-2. Vertices of the organization readiness cube .............................................................. 38 Table 3-1. The distribution of subjects and response rate for all three rounds of the study ....... 60 Table 3-2. Count of unique comments generated in round one of the Delphi study .................. 60 Table 3-3. Classifications derived from participants? comments ............................................... 61 Table 4-1. Sample demographics ................................................................................................ 93 Table 4-2. Correlation matrix ..................................................................................................... 96 Table 4-3. Regression analysis of intention to adopt .................................................................. 97 Table 4-4. Moderating effect on the relationship between ease of use and intention to adopt . 100 Table 4-5. Regression analysis of adoption .............................................................................. 102 Table 4-6. Regression analysis of routine use .......................................................................... 107 xiii List of Figures Figure 1-1: Overall Research Model ............................................................................................. 9 Figure 2-1: The Concerns-Based Adoption Model ..................................................................... 20 Figure 2-2: Conceptual model for considering the determinates of diffusion, dissemination, and implementation of innovations in health delivery and organization, bases on a systematic review of empirical research articles ...................................................................................................... 22 Figure 2-3: Conceptualizing organizational readiness for educational innovations ................... 36 Figure 2-4: Framework relating organizational readiness and characteristics of the innovation to the dissemination of innovations ................................................................................................ 39 Figure 3-1: Characteristics of Dissemination Success (CODS) Framework .............................. 68 Figure 4-1: Dissemination........................................................................................................... 81 Figure 4-2: Hypothesis 1 ............................................................................................................. 82 Figure 4-3: Theoretical model once hypothesis 2 is added to the model.................................... 83 Figure 4-4: Theoretical model once hypothesis 3 is added to the model.................................... 83 Figure 4-5: Theoretical model once hypotheses 4, 5, and 6 are added to the model .................. 84 Figure 4-6: Research model ........................................................................................................ 85 Figure 4-7: Interaction plot for the moderating effect of attitude to innovation on intention to adopt .......................................................................................................................................... 106 Figure 5-1: Comparative effectiveness research (CER) dissemination and implementation model ................................................................................................................................................... 119 Figure 5-2: Conceptualizing readiness for CER information ................................................... 119 Figure 5-3: Framework relating CER readiness and characteristics ......................................... 120 1 Chapter 1. Introduction In 1957, the Soviets successfully orbited Sputnik, which led to the passage of the National Defense Education Act. This legislation made federal funds available to improve instruction in the area of mathematics, foreign language, and sciences. Further, this legislation led to a major change in the way classes were taught in the United States including the addition of a new educational innovation, the overhead projector (Pond, 1963). Overhead projectors allowed faculty members to use pre-printed plastic sheets that could be used multiple times rather than having to manually write on the blackboard. Overhead projectors are one of the first examples of an educational innovation that successfully disseminated across the United States. With advances in technology, traditional approaches to teaching, like using an overhead projector, no longer engage students? interest. Faculty members over the years have been introducing new instructional materials and strategies and noticed that by using educational innovations in the classroom they are able to improve student motivation on both an intellectual and personal level. Some examples of educational innovations that have been introduced at the undergraduate level include active learning, computer simulations, flipped classrooms, podcasts, and spiral curricula. Unfortunately, all educational innovations do not successfully disseminate and much of the returns on investment are not transferred to society. Some of the reasons include: 1) perceived attributes of the innovation, 2) characteristics of the environment or context in which 2 potential adopters learn about innovations and make decisions, and 3) the extent of change agents? promotional efforts (Froyd, 2011; Rogers, 2003). In order to improve the dissemination of educational innovations, we must have a greater understanding of the reasons why some educational innovations successfully disseminate and others do not. Through a greater understanding of these reasons, administrators and faculty members will be able to be more effective agents of change and prepare the next generation of students for the challenges they will face. Problem The latest version of the curriculum guidelines for Information Systems (IS) disciplines is called IS 2010 (Topi et al., 2010). This curriculum guideline was a joint effort between the Association for Computing Machinery (ACM) and the Association for Information Systems (AIS). Curriculum guidelines are often the results of frequent changes in the IS field with a goal of improving the quality of education. Another key objective is to provide local administrators with the rationale to obtain the proper resources to support their program. Often administrators are not aware of the course offerings, hardware, software, classroom technology, and laboratory resources needed for a viable program (Topi et al., 2010). Many of the educational innovations that are used in IS classrooms have been created in other academic disciplines such as science, technology, engineering, and mathematics (STEM). The National Science Foundation (NSF) has funded the development of many educational innovations used in STEM classrooms today, for example course management systems and research-based instructional strategies. The NSF?s Division of Undergraduate Education created the grant program called Course, Curriculum, and Laboratory Improvement (CCLI), which in 2010 was renamed the Transforming Undergraduate Education in STEM (TUES). These grant 3 programs were created to improve the quality of STEM education through 1) the development of new educational materials; 2) the adaptation and implementation of exemplary materials, laboratory experiences, and/or educational practices that have been developed and tested at other institutions; 3) conducting research; and 4) the national dissemination of grant projects. Unfortunately, most of the innovations do not seem to be widely used in United States classrooms and the quality of mathematics and science education in the United States ranks 49th globally (World Economic Forum, 2013). This may, in part, be due to the current reward systems that are in place for faculty members that values research over teaching (Walczyk, Ramsey, & Zha, 2007). Most faculty members, except for an occasional workshop, are not exposed to pedagogy and are expected to teach with little to no training on how students learn (Loftus, 2013). Traditional lectures with PowerPoint slides are still used in the majority of IS and STEM classrooms in the United States (Macdonald, Manduca, Mogk, & Tewksbury, 2005; National Academy of Sciences, 2012; Walczyk et al., 2007). However, research-based instructional approaches to teaching that actively engage students in their own learning, such as group projects and problem-based learning, have been shown to be more effective than traditional lectures (National Academy of Sciences, 2012; Singer & Karl, 2013). Advances in the information technology and educational innovations continually inundate educators with new hardware, software, methods, and techniques that need to be evaluated to figure out whether or not they will be adopted in the classroom. Educators have a unique set of personal values, motivators, organizational policies and alliances that influence the adoption decision of educational innovations (Gillard, Nolan, & Bailey, 2008). Faculty members at institutions where student course evaluations play a role in the assessment of their teaching may be reluctant to try new, research-based teaching approaches if they expect that those 4 approaches will lead to critical evaluations (National Academy of Sciences, 2012). Gillard et al. (2008) note that some educators lag behind in adopting educational innovations and find that they have become pawns in the change process, vainly resisting the inevitable, while those on the front end of the adoption curve have eagerly embraced their role as change agents. Research regarding the dissemination of IS and engineering educational innovations is underdeveloped and can even appear to be somewhat of an afterthought (Fairweather, 2008; Hazen, Wu, & Sankar, 2012a). Several studies have identified some of the factors that influence the adoption process but the body of research regarding dissemination lacks unification (Hazen et al., 2012a; Hazen, Wu, Sankar, & Jones-Farmer, 2012b; Henderson, Finkelstein, & Beach, 2010). Henderson, Dancy, and Niewiadomska-Bugaj (2012) found that little empirical work has been done regarding the dissemination of educational innovations, especially research-based instructional strategies. Henderson and Dancy (2009) argue that current physics education research dissemination approaches, such as journal articles, conferences, and workshops, have been more successful in raising widespread awareness of new instructional practices than in helping faculty understand the underlying principles of these practices, or how to deploy them effectively. We look to fill this gap in the literature by developing a more comprehensive understanding of the factors that influence successful dissemination. Realizing the problem in disseminating educational innovations, the NSF has recently rolled out the Widening Implementation & Demonstration of Evidence Based Reforms (WIDER) program to transform institutions of higher education into supportive environments for STEM faculty members to substantially increase the use of evidence-based teaching and learning practices (National Science Foundation, 2013). The NSF?s definition of evidence-based teaching and learning practices includes such educational innovations as modern laboratory methods, 5 proven distance education methods, and approaches to motivate student interest in STEM and supporting students' efforts to succeed in STEM courses. This new NSF program has been allocated $20,000,000 to fund an estimated 30 to 50 grant projects over a four-year period. The primary goal of the WIDER program is to substantially increase the scale of improvement in higher education by the following: 1) improving student learning; 2) increasing the number of students choosing STEM majors, especially those in demographic groups that are underrepresented; 3) improving student retention in the first two years and to graduation for all STEM majors; and 4) research on dissemination. For college and university administrators to receive WIDER grants they will need to know the change theories and factors that influence the propagation and dissemination of evidence-based instructional practices. Administrators would benefit from the creation of a comprehensive framework regarding the dissemination of educational innovations that could be used as a reference while creating WIDER grant proposals. Purpose This dissertation will focus on three primary research goals that include: 1) developing a framework that describes the interrelationship among the characteristics of educational innovations and an organizations readiness to disseminate, 2) having an expert panel rank the characteristics of the educational innovations and readiness of faculty members, administrators, and students to disseminate, 3) empirically test whether important characteristics of educational innovations are significant predictors of intention to adopt and important variables that measure readiness of faculty members are significant predictors of successful dissemination. This dissertation develops a greater understanding of why some educational innovations are successfully disseminated and why others are not (Eiseman & Fairweather, 1996; Hazen et al., 2012a). 6 Narrowing the Scope The number of variables that would be needed to find all the factors that influence the dissemination of every educational innovation across all academic disciplines would be very unwieldy if it is not constrained. For example, 17 different names for computer-related majors are offered at Association to Advance Collegiate Schools of Business (AACSB) accredited undergraduate institutions (Pierson, Kruck, & Teer, 2008; Topi et al., 2010). Engineering programs traditionally consist of 22 different undergraduate majors, yet ABET certifies 83 different program areas for undergraduate studies (ABET, 2013; Yoder, 2012). The number of stakeholders involved in the dissemination process is also not manageable unless it were to be constrained because it would have to include every book publisher, innovation developer, student, faculty member, department chair, college dean, university president, and grant program manager. In this dissertation, we limit the disciplines to the areas of engineering and computer science. Due to the breadth of research that is available regarding NSF grant projects, which often includes discussions regarding dissemination, it was not realistic to look at every educational innovation that has been created. Engineering education has also become its own recognized discipline in engineering departments with pioneering programs at Purdue, Virginia Tech, Utah State University, and Clemson University (Borrego, 2007; Borrego & Bernhard, 2011; Borrego et al., 2005; Borrego, Streveler, Miller, & Smith, 2008). Finally, the opportunities to receive grant funding to conduct this research was available by studying the disciplines of engineering and computer science. For this dissertation, we limit the educational innovations to the development of new curricula, instructional materials, faculty expertise, and/or instructional strategies. Curriculum 7 development deals with overarching changes planned or implemented for an educational program or curriculum. Instructional materials are specifically designed for use in a particular course or laboratory session. The development of faculty expertise refers to methods that can be used to help faculty members improve their teaching (e.g., teaching workshops or mentoring), and instructional strategies include new approaches that enhance the transfer of knowledge (e.g., experiential learning or student centered learning). In this dissertation, we limit the stakeholders to include administrators, faculty members, and students. Borrego, Froyd, and Hall (2010) found that department chairs perceive that financial resources, faculty time and attitudes, and student satisfaction were all major considerations in the dissemination of educational innovations. The President's Council of Advisors on Science and Technology (2012) found that the administrators often lacked departmental rewards and expectations for good teaching because salaries and advancement are more closely correlated with publication rate than teaching quality. In addition, faculty members often lack knowledge of evidence-based teaching due to the lack of time and incentives to find, read, and evaluate the literature and teaching methods. Many of the educational innovations used in higher education are not mandated by the administration; therefore, faculty members are often the final hurdle in getting innovations actually adopted in the classroom. Unfortunately, faculty members often resist changing from a traditional lecture class even though there are hundreds of papers on better teaching methods and educational innovations. This has resulted in many educational innovations not being adopted or routinely used (Henderson et al., 2012). While faculty members chose educational innovations, students are the intended users of the innovations with a goal of increasing their engagement. If students resist this change or are not engaged, the educational innovation may not become routinely used out of fear of poor teacher 8 evaluations (?kerlind & Trevitt, 1999; Keeney-Kennicutt, Gunersel, & Simpson, 2008; National Academy of Sciences, 2012). Research Design and Methods Mixed Methods Research Design We chose a sequential mixed methods research design due the ?developmental? nature of this dissertation. Mixed methods research involves a research design that uses more than one research method and more than one worldview (e.g., qualitative or quantitative research approach) (Tashakkori & Teddlie, 2003a, 2003b; Venkatesh, Brown, & Bala, 2013). In a sequential mixed method research design, the research questions emerge from the inferences of the previous research method (Venkatesh et al., 2013). By using a mixed methods research design, we look to develop a deeper understanding of the dissemination research questions. The overall research design is shown in Figure 1-1 and explained below. During the qualitative research portion of this study we employ two different methods: content analysis to do a systematic review of recent dissemination literature and a Delphi study of experts that have disseminated educational innovations. In the quantitative portion of this study, we develop and administer a survey to test the significance of dissemination factors that have been identified during the qualitative portion of this study. As we perform each of the sequential research methodologies, we narrow the scope of the study. We document each of the research methodologies and results as three publishable papers (Chapters Two, Three, and Four). 9 Figure 1-1. Overall research design D e v e l o p a g r e a t e r u n d e r s t a n d i n g o f w h y s o m e e d u c a t i o n a l i n n o v a t i o n s a r e s u c c e s s f u l l y d i s s e m i n a t e d a n d w h y o t h e r s d o n o t . P u r p o s e : D e v e l o p m e n t a l S e q u e n t i a l M i x e d - M e t h o d R e s e a r c h D e s i g n S y s t e m a t i c L i t e r a t u r e R e v i e w G o a l : 1 ) T o i d e n t i f y t h e v a r i a b l e s t h a t d e s c r i b e t h e c h a r a c t e r i s t i c s o f i n n o v a t i o n s a n d r e a d i n e s s o f o r g a n i z a t i o n a l m e m b e r s t o w a r d s u c c e s s f u l d i s s e m i n a t i o n o f e d u c a t i o n a l i n n o v a t i o n s . 2 ) T o d e v e l o p a f r a m e w o r k t h a t d e s c r i b e s t h e i n t e r r e l a t i o n s h i p s a m o n g t h e c h a r a c t e r i s t i c s o f i n n o v a t i o n s a n d r e a d i n e s s o f o r g a n i z a t i o n a l m e m b e r s t o w a r d s u c c e s s f u l d i s s e m i n a t i o n o f e d u c a t i o n a l i n n o v a t i o n s . D e l p h i S t u d y G o a l : H a v e a n e x p e r t p a n e l r a n k w h i c h c h a r a c t e r i s t i c s h a v e t h e g r e a t e s t i n f l u e n c e o n t h e s u c c e s s f u l d i s s e m i n a t i o n o f e d u c a t i o n a l i n n o v a t i o n s S u r v e y G o a l : 1 ) T e s t i n g w h e t h e r i m p o r t a n t c h a r a c t e r i s t i c s o f e d u c a t i o n a l i n n o v a t i o n s a r e s i g n i f i c a n t p r e d i c t o r s o f s u c c e s s f u l d i s s e m i n a t i o n . 2 ) T e s t i n g w h e t h e r i m p o r t a n t r e a d i n e s s o f f a c u l t y m e m b e r s t o w a r d e d u c a t i o n a l i n n o v a t i o n c h a r a c t e r i s t i c s a r e s i g n i f i c a n t p r e d i c t o r s o f s u c c e s s f u l d i s s e m i n a t i o n a n d m o d e r a t e t h e r e l a t i o n s h i p b e t w e e n c h a r a c t e r i s t i c s o f e d u c a t i o n a l i n n o v a t i o n s a n d i n t e n t i o n t o a d o p t . M a j o r L i m i t a t i o n s , F u t u r e R e s e a r c h P o s s i b i l i t i e s , a n d M a j o r C o n t r i b u t i o n s o f t h i s D i s s e r t a t i o n . C h a p t e r 1 C h a p t e r 2 C h a p t e r 3 C h a p t e r 4 N a r r o w S c o p e : - D i s c i p l i n e - E d u c a t i o n a l I n n o v a t i o n s - S t a k e h o l d e r s N a r r o w S c o p e : - C o n f e r e n c e p a p e r s & J o u r n a l a r t i c l e s - E n g i n e e r i n g a n d C o m p u t e r S c i e n c e - P u b l i s h e d 2 0 0 7 - 2 0 1 2 N a r r o w S c o p e : - C C L I a n d T U E S g r a n t s - E n g i n e e r i n g a n d C o m p u t e r S c i e n c e - P I & C o - P I N a r r o w S c o p e : - E l e c t r i c a l E n g i n e e r i n g a n d C o m p u t e r S c i e n c e F a c u l t y M e m b e r s - A B E T a c c r e d i t e d s c h o o l s C h a p t e r 5 10 Chapter Two (Paper One) The first paper, entitled Conceptualizing Interactions Between Characteristics of Organizational Members? Readiness to Disseminate Educational, is a systematic review of journal articles and conference papers that discuss the dissemination of educational innovations. A systematic review is a review of literature using an explicit, rigorous, and transparent methodology (Gough, Oliver, & Thomas, 2012; Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). The first goal of this paper is to identify the variables that describe the characteristic of innovations and the readiness of organizational members (e.g., students, faculty members, and administrators) toward successful dissemination of educational innovations. The second goal of this paper is to develop a framework that describes the interrelationships among the characteristics and readiness of organizational members toward successful dissemination of educational innovations. Due to the high volume of articles on engineering educational innovations, we restricted our search to the recent literature (2007-2012). Through a literature review of 187 articles and conference papers, we identified characteristics of educational innovations, students, faculty members, and administrators that affect the dissemination process. Then a configurative review was performed on 37 of the articles, chosen at random, and developed a realist synthesis leading to a framework. A realist synthesis is concerned about the mechanisms by which an intervention works (or not) in the context of a desired outcome (Gough et al., 2012). We propose that organizational readiness for educational innovations is the result of the interrelationships among attitudes of students, faculty members, and administrators toward innovations. The interrelationship between the characteristics of an innovation and organization readiness for educational innovations determines the success or failure in disseminating an innovation. 11 Chapter Three (Paper Two) The second paper is entitled Identifying Characteristics of Dissemination Success Using and Expert Panel. The goal of this paper is to have an expert panel rank which characteristics have the most influence on the successful dissemination of educational innovations. This study utilizes the Delphi method, a survey research technique developed by the Rand Corporation in the early 1950s (Dalkey & Helmer, 1963; Malhotra, Steele, & Grover, 1994). Because of the unique characteristics of the methodology, the Delphi technique is applicable for answering multidimensional research questions that deal with uncertainty in a domain of imperfect knowledge (Cegielski, 2001; Churchman & Schainblatt, 1965). The objective of the technique is to achieve consensus among experts regarding a specific topic (Okoli & Pawlowski, 2004; Taylor & Meinhardt, 1985). In previous comparative analysis of group survey techniques, Riggs (1983) and Rohrbaugh (1979) reported that the Delphi technique achieved a greater level of accuracy than other group consensus techniques. We asked 307 TUES and CCLI Principal Investigators and Co-Principal Investigators that have received engineering and computer science grants, to develop and/or disseminate educational innovations, the following research questions in the first round of a Delphi study: 1. What characteristics of educational innovation influence successful dissemination? 2. What characteristics of students influence the successful dissemination of educational innovations? 3. What characteristics of the faculty members influence the successful dissemination of educational innovations? 4. What characteristics of the college or university administration influence the successful dissemination of educational innovations? 12 A total of 45 Principal Investigators and Co-Principal Investigators completed the first round for a response rate of 14.66 percent. Members of this panel included at least one member from 25.81 percent of all CCLI and TUES grant projects that received this survey. This paper reports the detailed analysis and results of this Delphi study. The results provide the basis for the development of the Characteristics of Dissemination Success (CODS) framework. Chapter Four (Paper Three) Paper three is entitled The Influence of Characteristics of Innovations and Readiness of Faculty Members Toward Dissemination of Educational Innovations: An Empirical Study. This paper starts by presenting a summary of the characteristics of educational innovations and faculty members that were identified in the realist synthesis and Delphi study that can lead to successful dissemination. The research questions addressed in this paper include: 1. Which of the important characteristics of the educational innovation identified by our systematic review of the literature and Delphi study are significant predictors of intention to adopt educational innovations? 2. Which of the important characteristics of the faculty members, as identified by our systematic review of the literature and Delphi study, are significant predictors of successful dissemination and moderate the relationship between characteristics of educational innovations and intention to adopt? From these research questions, we developed a theoretical research model and hypotheses based on the Characteristics of Dissemination Success (CODS) framework. To test this model, we searched for measures from education, information systems, marketing, management, psychology, and sociology literature. Over a period of five months and 13 15 meetings, the research team evaluated possible measures and developed a 116-question survey. The survey was pilot tested and improved. This survey was distributed to 4,352 faculty members at ABET certified computer science and electrical engineering programs in the United States. The response rate was 7.98% with 335 participants. Analysis was conducted using both hierarchical linear and hierarchical logistic regression using SPSS version 21.This paper reports the findings and implications of the analysis. Chapter 5 The final chapter discusses the major limitations, future research possibilities, and major contributions of this dissertation. The major limitation of this study was the sample size used in the three papers. Future research possibilities are discussed in the realm of disseminating engineering educational innovations, healthcare practices, and cybersecurity and information technology security. Major contributions of this dissertation are discussed from the viewpoint of developers of education innovations, faculty members, department chairs, college deans, and grant program managers. Definitions: Dissemination - A process of first creating an awareness of an educational innovation, then influencing the intention to adopt, the actual adoption, and finally the routine use of that innovation (Fincher, 2000; Gravestock, 2002; Hutchinson & Huberman, 1994; King, 2003). Educational innovation - encompassing any new instructional material, strategy, or pedagogy (Hazen et al., 2012a). 14 Chapter 2. Conceptualizing Interactions Between Characteristics and Organizational Members? Readiness to Disseminate Educational Innovations Abstract Background Although a great deal of work has been done to develop new educational innovations, especially in the field of engineering, few innovations have found widespread acceptance in the classroom. Therefore, a framework that conceptualizes the interactions among the variables that influence successful dissemination is critical. Purpose/Hypothesis This article proposes a framework regarding the interactions between characteristics and readiness of an organizations? faculty members, administration, and students to disseminate (become aware, intend to adopt, adopt, and use routinely) educational innovations. Design/Method One hundred eighty seven papers published in engineering education during 2007-2012 were analyzed to identify the characteristics of educational innovations that influence their eventual dissemination. A sample of these articles (37) was then analyzed by two researchers with different areas of expertise. These results were then synthesized to develop a framework. The interrater reliability was .86. 15 Results The proposed framework describes the interaction between characteristics of an educational innovation and an organizations? readiness to disseminate. Organizational readiness depends on the readiness of faculty members, administrators, and students to disseminate an innovation. The framework was then used to analyze how well the innovations discussed in the 37 articles were disseminated leading to a set of findings. Conclusions In most educational projects, the desired outcome is the successful dissemination of educational innovations; the interventions are the characteristics of the innovation; the context is the organizational readiness (readiness of faculty members, administrators, and students to disseminate innovation); and the underlying mechanism is the proposed framework that links the interventions, context and the outcome. The framework shows that for widespread dissemination to occur, educational innovators need to build positive characteristics in the innovations and need to implement them in organizations that are ready for such change. Keywords Dissemination, educational innovation, organizational readiness, framework, readiness 16 Introduction Over the past decade, major US organizations, such as the National Academy of Education (2009), National Academy of Engineering (2004, 2005), and National Science Board (2007), have recommended new educational policies to improve the quality and number of graduates in science, technology, engineering, and mathematics (STEM) undergraduate programs. One key argument in these reports is that the results of engineering education research have, for the most part, not led to widespread adoption of evidence-based teaching practices, discovery-based research courses, and research-based educational innovations within engineering education (American Society for Engineering Education, 2012; National Science Foundation, 2008; President's Council of Advisors on Science and Technology, 2012). Olds, Borrego, Besterfield-Sacre and Cox (2012) stressed the need for research on identifying barriers to dissemination of proven practices at the individual and institutional levels. Therefore, a framework that shows the interrelationships among the different variables that could lead to the successful dissemination of an innovation could be a valuable addition to the literature (Clayson, Raju, & Sankar, 2010; Donnelly, McGarr, & O?Reilly, 2011; Froyd, 2011; Hazen et al., 2012b). There have been many attempts to synthesize the complicated change processes connected with dissemination into theories, models, and frameworks (Fairweather, 2008; Hazen et al., 2012b; Henderson et al., 2010). A dissemination perspective on change is a characteristic of most prior work in engineering education (Bennedsen, 2006; Froyd, 2001; Hazen et al., 2012a; Hazen et al., 2012b; Watson, Corbett, Prather, Carpenter, & Cronk, 2010) and hence we use that perspective in this article. Henderson et al. (2010) identified a lack of meaningful roles for typical faculty members in the change process and suggested that the environments and structures that faculty work within often favor traditional teaching methods. 17 Eiseman and Fairweather (1996) suggested that dissemination should play a co-equal role with the development of new instructional materials. In a recent survey, reviewers for the National Science Foundation?s (NSF) Transforming Undergraduate Education in Science, Technology, Engineering, and Mathematics (TUES) grant program reported that one of the top three weaknesses in grant proposals was the lack of a dissemination plan that would lead to lasting changes at multiple institutions (Feser, Borrego, Pimmel, & Della-Piana, 2012) . Based on a synthesis of relevant literature we find that for most educational projects, the desired outcome is the successful dissemination of educational innovations; the interventions are the characteristics of the innovation; the context is the organizational readiness (readiness of faculty members, administrators, and students to disseminate innovation); and the underlying mechanism is the proposed framework that links the interventions, context and the outcome. In the next section, we define the terminologies of dissemination, educational innovations, and organizational readiness. Then, we review the current frameworks for dissemination (outcome) and show the need for a new framework. This leads to a discussion of the research methodology that includes both systematic and configurative reviews of literature. A systematic literature review of articles published during 2007-2013 helped identify the variables that describe the characteristics of the innovations (interventions) and the readiness of organizational members (context) to disseminate an innovation. A configurative literature review of a sample of these articles helped develop a framework that links the interventions, context, and the outcome. The framework was used to illustrate how the effort put forth to disseminate the innovations discussed in each article resulted in movement of the innovation to different quadrants of the framework. This leads to a discussion of how the framework could be of value to educational 18 innovators, faculty members, administrators, and grant managers. The paper concludes with limitations, recommendations for future research and a summary of this study?s contributions. Definitions Dissemination We define dissemination as a process of first creating an awareness of an educational innovation, then influencing the intention to adopt, the actual adoption, and finally the routine use of that innovation (Fincher, 2000; Gravestock, 2002; Hutchinson & Huberman, 1994; King, 2003). Please note that this definition is more comprehensive than those in some of the literature (e.g., Lomas (1993)) where dissemination is defined as spreading the message and does not include intention to adopt, actual adoption, and routine use by organizational members. Educational Innovations Innovations are ideas, practices, or objects that are perceived as new by an individual or group (Rogers, 2003). In this article, we classify educational innovations as development of (a) new curricula, (b) instructional materials, (c) faculty expertise, and/or (d) instructional strategies (Hazen et al., 2012a). Curriculum development deals with overarching changes planned or implemented for an educational program or curriculum, while instructional materials are specifically designed for use in a particular course or laboratory session. The development of faculty expertise refers to methods that can be used to help faculty members improve their teaching such as training workshops or mentoring. Instructional strategies, such as experiential learning or student centered learning, consist of both instructional design and pedagogy and include approaches that enhance the acquisition of knowledge (Gropper, 1974). 19 Organizational Readiness Holt el al. (2007b) synthesized a definition of readiness for change based on an exhaustive review of the readiness literature and instruments. Readiness for change was defined as: A comprehensive attitude that is influenced simultaneously by the content (i.e., what is being changed), the process (i.e., how the change is being implemented), the context (i.e., circumstances under which the change is occurring), and the individuals (i.e., characteristics of those being asked to change) involved and collectively reflects the extent to which an individual or a collection of individuals is cognitively and emotionally inclined to accept, embrace, and adopt a particular plan to purposefully alter the status quo. (Holt et al., 2007b, p. 326) Armenakis et al. (1993) states that organizational readiness, is similar to Lewin?s (1947, 1951) concept of unfreezing, which is reflected in organizational members? beliefs, attitudes, and intentions regarding the extent to which changes are needed and the organizational capacity to successfully make those changes. Creating organizational readiness involves proactive attempts by a system member to influence the beliefs, attitudes, intentions, and ultimately the behavior of the other system members (Armenakis, Harris, & Feild, 1999; Armenakis et al., 1993). In academic institutions, the three sets of important system members are: faculty members, administrators, and students. Need for a Conceptual Framework on Dissemination of Educational Innovations In this section, we review current frameworks on dissemination of engineering educational innovations and show a need to develop a new conceptual framework. The theories on dissemination originate from many different fields such as social sciences, medical education, K-12 education, and information systems. We chose to include the concerns-based adoption 20 model, diffusion of innovation theory, the model of diffusion in service organizations, and the technology acceptance model in this review since they help describe how engineering educational innovations are disseminated. Dissemination Frameworks Concerns-based adoption model. The Concerns-Based Adoption Model (CBAM) (Hall, 1974; Hall & Hord, 2006) was developed to frame the process of change in a K-12 educational setting (Figure 2-1). CBAM ?generally approaches change as a mandate from an administrator or other leader position then diffuses to the teachers as the ultimate consumer of the innovation? (Straub, 2009, p. 636). At the center of this model are three diagnostic dimensions: The Stages of Concern, the Levels of Use, and Innovation Configurations. Stages of Concern provide a way of accessing information about people's attitudes, reactions, or feelings about a new program or practice; Levels of Use utilize an interview protocol to measure user intentions in order to identify the extent to which he/she is using the innovation; and Innovation Configurations represent the patterns of use that result when different teachers implement the innovation in their classrooms. Figure 2-1: The Concerns-Based Adoption Model (Hall & Hord, 1987, p. 12) C h a n g e F a c i l i t a t o r R e s o u r c e S y s t e m I n n o v a t i o n N o n u s e r s a n d U s e r s P r o b i n g I n t e r v e n i n g S t a g e s o f C o n c e r n L e v e l s o f U s e I n n o v a t i o n C o n f i g u r a t i o n U s e r S y s t e m C u l t u r e U s e r S y s t e m C u l t u r e E n v i r o n m e n t 21 One benefit of CBAM is that Innovation Configurations can be monitored to ensure that teachers are using an innovation in the same way. Unfortunately, this model does not apply as easily to undergraduate education because educational innovations are not always mandated, with the final adoption decision being up to the instructor who is teaching a particular course. Additionally, CBAM pays relatively little attention to the students who are the end users of the innovations and disregards teachers? positive perceptions of an innovation (Straub, 2009). Diffusion of innovation theory. Diffusion of innovation theory by Rogers (1962, 2003) defines diffusion as the process by which an innovation is communicated though certain channels over time among the members of a social system. This theory outlines five sequential stages of the adoption process: knowledge or exposure to the innovation; persuasion to gain knowledge about the innovations; the decision to adopt or reject the innovation; implementation of the innovation; and confirming the decision to continue using the innovation. Rogers (2003) describes a Knowledge-Attitude-Practice gap (KAP-gap) in which knowledge of an innovation and a favorable attitude towards the innovation does not necessarily result in the adoption of the innovation. The KAP-gap has been found in engineering education research (Borrego et al., 2010) , mathematics education research (Anderson, 2011), and physics education research (Dancy & Henderson, 2008; Henderson & Dancy, 2007). KAP-gaps can be closed by a cue-to- action which is defined as ?an event occurring that at a time crystallizes a favorable attitude into overt behavioral change? (Rogers, 2003, p. 176). Diffusion in service organizations. Using a meta-narrative review approach, Greenhalgh et al. (2004) developed a model for diffusion of innovations in healthcare organizations (see Figure 2-2). Their research included reviews of articles in diverse academic disciplines including anthropology, economics, information systems, management, marketing, psychology, 22 and sociology (Greenhalgh, Robert, Bate, Macfarlane, & Kyriakidou, 2007; Greenhalgh et al., 2004; Greenhalgh et al., 2005). This model affirmed the useful list of innovation attributes that predict (but do not guarantee) successful innovations; the importance of social influence and networks; the complex and contingent nature of the adoption process, the characteristics of organizations that encourage and inhibit innovation; and the messy, stop-start, and difficult-to- research process of assimilation and routinization (Greenhalgh et al., 2004; Lomas, 1993). Figure 2-2. Conceptual model for considering the determinates of diffusion, dissemination, and implementation of innovations in health delivery and organization, bases on a systematic review of empirical research articles (Greenhalgh et al., 2004, p. 595) 23 Technology acceptance model. The Technology Acceptance Model (TAM) (Davis, 1989) was formulated by combining the diffusion of innovation theory (Rogers, 2003) with the theory of reasoned action (Fishbein & Ajzen, 1975, 1981), and the theory of planned behavior (Ajzen, 1991). The TAM model has been further refined as TAM 2, (Venkatesh & Davis, 2000); and TAM 3, (Venkatesh & Bala, 2008); the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh, Morris, Davis, & Davis, 2003); and UTAUT 2 (Venkatesh, Thong, & Xu, 2012). These models suggest that when users are presented with a new technology, a number of characteristics of the innovation influence their decision about how and when they will use it. Variations of the TAM have been used in educational research to study the integration of information and communication technology (Birch & Irvine, 2009), computer usage (Teo, 2009, 2011; Teo, Lee, & Chai, 2008), engineering education (Hazen et al., 2012a), and course management systems (Yi & Hwang, 2003). Criticism of the original TAM focuses primarily on its failure to acknowledge individual differences, such as prior experience, age, gender, and other variables that may influence attitudes about technology and consideration of organizational readiness dimension. (Agarwal & Prasad, 1999; Straub, 2009). Need for a New Conceptual Framework The above theories provide useful insights into aspects of the engineering educational dissemination process. However, in higher education the dissemination decision is often a group or organizational decision. There is often a delicate blend of the academic freedom of a faculty member to choose among the educational innovations and the readiness of an organization to disseminate the chosen innovations. There is a strong need to visualize and describe what mix of variables produce adoptable innovations in engineering education (Greenhalgh et al., 2004; Olds et al., 2012). There is a call for more systematic reviews to help advance the field by lowering 24 the barriers for both researchers and practitioners to access the literature, enabling more objective critique of past efforts, identifying gaps, and proposing new directions for research (Borrego, Foster, & Froyd, 2014). Therefore, there is a need to develop a framework that illustrates the interrelationships among the characteristics of the innovation itself and the readiness of faculty members, administrators, and students to disseminate the innovation. This led to the formulation of the two goals for this research: 1) Identify the variables that describe the characteristics and readiness of organizational members toward successful dissemination of educational innovations; and 2) Develop a framework that describes the interrelationships among these variables. Research Methodology We used the systematic literature review process espoused by Gough et al. (2012) to uncover the mechanisms that relate the outcome of interest (i.e. the successful dissemination of innovations) to its antecedents (namely the characteristics of the innovations and the readiness of faculty members, administrators, and students to disseminate an innovation) in the context of higher education. Borrego et al. (2014) state that a systematic review would help advance the field by lowering the barrier for both researchers and practitioners to access the literature, enabling more objective critique of past efforts, identifying gaps, and proposing new directions for research. Gough et al. (2012) recommend that reviewing research systematically involves three key activities: identifying and describing the relevant research, critically appraising the research reports in a systematic manner, and bringing together the findings into a coherent statement, known as synthesis. We use these three steps to describe our research methodology. 25 Identifying and Describing the Relevant Research We selected articles on dissemination of educational innovations in engineering education using a succession of inclusion and exclusion criteria. First, we searched for articles in six electronic databases (ASEE Conference Proceedings, Engineering Village, ERIC, IEEE Xplore, ingentaconnect, and ScienceDirect) and eight academic journals (Advances in Engineering Education, Computer Applications in Engineering Education, Decision Sciences Journal of Innovative Education, European Journal of Engineering Education, International Journal of Engineering Education, International Journal of Mechanical Engineering Education, Journal of Engineering Education, and Journal of Professional Issues in Engineering Education). These eight journals have historically published articles on engineering educational innovations and the dissemination process. Due to the high volume of articles on this topic, we limited our search to the most recent five-year period (2007-2012). Multiple scans of the databases and journals were conducted using different combinations of the terms ?engineering?, ?pedagogy?, ?educational?, ?innovations?, and ?dissemination?. This approach yielded 489 unique articles. Abstracts were screened using a two-step process to evaluate articles that fell outside the scope of this study and the overall quality. The first-step of the screening process evaluated whether articles where related to the diffusion, dissemination, or adoption of an engineering related educational innovation in higher education. Articles on curriculum development, developing faculty expertise, instructional materials, and/or instructional strategies were included, as were articles that discussed the impact of educational innovations on undergraduate engineering education, measured student learning in response to a new educational innovation, or discussed improving engineering instruction at a college or university. When there was a 26 question about the relevance of an article based on its abstract, we retained the article for further review. For the second-step of the screening process, papers were evaluated for worth (e.g., does the paper go beyond superficial description or commentary; the article is a competent attempt at research, enquiry, investigation, or study) (Greenhalgh et al., 2007; Greenhalgh et al., 2004). After analyzing the abstracts, 302 articles were removed leading to a revised database containing 187 articles that were then used in our content analysis. Appendix 2-1 is a PRISMA diagram summarizing the sources contributing to the systematic review. Appendix 2-2 lists the journals and conference proceedings that contained the 187 articles. Of these 187 papers, 67.9% (n = 127) were journal articles and 32.1% (n = 60) were conference papers. These articles consisted of 70 (37.4%) qualitative and case study (focus groups, qualitative interviews, case studies, comparative case studies and mixed methodology case studies) articles, 45 (24.1%) empirical (experimental, quasi-experimental, or questionnaire survey) articles, 44 (23.5%) best practice or guideline articles, 10 (5.3%) systematic review articles, 7 (3.7%) theory or conceptual framework articles, 6 (3.21%) tool/checklist/component of a model articles, and 5 articles that were categorized as other (combining multiple methods). Critically Appraising the Research Reports in a Systematic Manner A researcher read each of the 187 articles, identified the characteristics of innovations that were mentioned to be important, and listed the variables that determine the readiness of faculty members, students, and administrators toward the dissemination of innovations. Patton (2002) suggests that the reviews using qualitative data need not be exhaustive and should instead use random sampling to increase the credibility of the study. Therefore, we selected 37 articles (19.79%) randomly from the 187 articles. Citations for the 37 articles are in Appendix 2-3. 27 Two researchers read these 37 articles and identified the mechanisms that appeared to connect the intervention to the outcomes in the context of the educational organization. We used a thematic synthesis methodology which is a technique to help the researchers conduct a synthesis that is systematically grounded in the articles it reviews. One of the researchers was experienced in the research domain, with over 15 years of experience with educational innovations and dissemination and has been funded through several NSF CCLI/TUES grants. The other researcher was not an expert in the research domain, but was proficient in qualitative research methodology. Both researchers independently coded the 37 articles using a standard extraction form to match the methodology, results, discussion and conclusions sections of the articles to the type of innovation, characteristics of the innovations, attitudes of faculty members, administrators, and students to disseminate the innovations. Upon the completion of the coding process, the two researchers met to determine whether they had been able to identify similar themes in the articles. Krippendorff?s alpha coefficient for the two coders was .86, indicating that there was sufficient agreement between the two coders to justify proceeding further (Krippendorff, 2004). This process led to achievement of the first research goal. Synthesizing Our Findings We used a configurative review process (Gough et al., 2012) to answer the second research question. A realist synthesis is a method that seeks to infer a causal outcome/relationship between an intervention and an outcome of interest by understanding the underlying mechanisms that connect them and the context in which the relationship occurs (Gough et al., 2012). In this study, the desired outcome is the successful dissemination of educational innovations; the interventions are the characteristics of the innovation; the context is the organizational readiness (readiness of faculty members, administrators, and students to 28 disseminate innovation); and the underlying mechanism is the proposed framework that links the interventions, context and the outcome. Configurative reviews seek to include studies that will provide richness in terms of making distinctions and developing and exploring theory (Gough et al., 2012). In order to synthesize the results of coding, a third researcher specializing in both education and research methodology worked with the two researchers. The three researchers used the results of the configurative review of the 37 articles, used to establish interrater reliability, to develop the conceptual framework. Results Critically Appraising the Research Reports in a Systematic Manner to Achieve the First Goal Based on past literature, we initially categorized members who influence organizational readiness as faculty members and administrators because they often make the final decision on what is used in the classroom (Hazen et al., 2012b; Lee, Kozar, & Larsen, 2003). Our review of the literature revealed that students? attitudes and evaluations had an influence over whether faculty members continued to use or even adopt educational innovations. Therefore, we created students as a new member influencing organizational readiness to disseminate innovations. All the variables that define readiness of faculty members were also applicable to students, but the context was different. Earlier research by Lee et al. (2003), Compeau et al. (2007), and Hazen et al. (2012a) provided a list of characteristics of innovations that are considered valuable in dissemination as well as several variables that define the readiness of faculty members and administrators toward dissemination of innovations. We checked for these characteristics and readiness variables using 29 a systematic literature review of the 187 articles and identified several new characteristics and readiness variables that were not reported in the past studies. Quality of initial information about the innovation, awareness of innovation, learning styles of the students, and motivation of organizational members were identified as other potential new variables during the literature review. A complete list of the characteristics of the innovations and readiness variables of faculty members, administrators, and students is given in Table 2-1. This table provides a list of ? 16 characteristics needed in each innovation, ? 12 variables that describe faculty members? readiness to disseminate an innovation, ? 7 variables that describe administrators? readiness to disseminate an innovation, and ? 12 variables that describe students? readiness to disseminate an innovation. Table 2-1. Perceived characteristics that may affect dissemination (full list) (Adapted from Lee et al., 2003; Compeau et al., 2007; and Hazen, Wu, and Sankar, 2012) Characteristics Contextual Definition [sample of coded text] Characteristics of the Innovation 1. Adaptability Adaptability allows the user to modify the innovation as deemed necessary [?it is important, therefore, that the course design offer variation in large group work, small group work, pair work, and one-on-one work to maintain freshness and interest? (Kayfetz & Almeroth, 2008, p. 2)] 2. Aid Social Presence The degree to which the pedagogical innovation permits users to experience others as being psychologically present [?social interaction and communication is one of the most important components in Second Life and it can take place via different communication channels? (Pfeil, Ang, & Zaphiris, 2009, p. 224)] 3. Communicability The ease with which the results of using the pedagogical innovation can be easily described to others [?podcast format 5 (slides with original embedded audio) was deemed the most useful with 13 or the 15 students ranking it among their top three preferred formats? (Lawlor & Donnelly, 2010, p. 966)] 4. Compatibility The consistency of the pedagogical innovation with current pedagogy [?refers to consistency of the innovation with values, experiences, and needs of the potential adopter; the value of teaching innovations or improvement is most relevant across engineering education? (Borrego et al., 2010, p. 186)] 30 5. Ease of Use The degree to which a pedagogical innovation is perceived as relatively easy to use and understand [?the easiness of use of the application is another important aspect in the presented tool, given that the simplicity of the interface and the interactive help always maintains students informed about how to proceed? (P?rez Morales, Pe??n, Sevillano, & Cerra, 2009, p. 15)] 6. Enhance Image The level of increased social status received by using the pedagogical innovation [?to incorporate lasting and sustainable reform within our programs that can be a model for other engineering (and science) programs at UVM as well as elsewhere? (Hayden et al., 2011, p. 10)] 7. Enhance Visibility The observability of the pedagogical innovation [?the potential to present course material in innovative and pedagogical ways is enormous with an Learning Management System? (Christie & Jurado, 2009, p. 278)] 8. Logistical Issues (-) Usually due to geographic or time separation of resources or participants [?the students then form virtual teams with students at Corvinus University in Budapest, Hungary, and begin their collaboration during formal scheduled in- class meetings Polycom videoconferencing, and outside the class using personal Skype accounts? (Schuhmann, 2010, p. 65)] 9. Measurability The degree to which the impact of the pedagogical innovation can be assessed, in particular the ability to clearly attribute the effects to the innovation [?both qualitative and quantitative metrics paint a picture of students gaining a realistic perspective on the social process of engineering design? (Cheville & Bunting, 2011, pp. 18-19)] 10. Perceived Enjoyment An individual's perception of how enjoyable it is to use the pedagogical innovation [?to be more fun, pleasurable and challenging? (Cheung et al., 2008, p. 21)] 11. Playfulness The degree of interaction between the individual and the pedagogical innovation [?the robot will play the game of Tic-Tac-Toe with a remote human opponent? (Krishnan, Paulik, Yost, & Stoltz, 2008, p. 3)] 12. Quality of Initial Information * The degree to which the initial information regarding the pedagogical innovation is relevant, timely, complete, and appropriate in terms of amount so as to add value [?it has to be spelled out what minimum training the reviewers need to have in pedagogy and instruction and what additional skills are needed such as classroom observation and interviewing? (Bernold, 2008, p. 37)] 13. Relative Advantage The degree to which the pedagogical innovation is perceived as being better than its precursor [?the portfolio methodology improves the academic performance rate by 60% over the traditional one? (Badia et al., 2010, p. 4)] 14. Relevant to Job The ability of the pedagogical innovation to enhance the educator's performance [?the ability to solve problems has become a major prerequisite for engineering graduates that enter the workplace? (Veldman, De Wet, Ike Mokhele, & Bouwer, 2008, p. 555)] 15. Trialability The extent to which individuals are able to try the pedagogical innovation before they actually adopt it [?Most of the faculty experimenting with this new tool were optimistic about its usefulness and were excited to be part of the development process.? (Cramer et al., 2010, p. 8)] 31 16. Usability The effectiveness of use regarding the pedagogical innovation [?usability of the internet experiments achieved a much higher score? (Lang et al., 2007, p. 65)] Readiness of Faculty Members to Disseminate an Innovation 1. Anxiety to Use (-) The concerns that faculty members have when faced with using pedagogical innovations [?In fact, movement toward learner-centered instruction often produces a temporary drop in student ratings, thus having a negative effect on overall evaluation of faculty? (Walczyk et al., 2007, p. 97)] 2. Attitude to System The opinion that a faculty member holds of the pedagogical innovation [?As mentioned previously, the instructor who took over the course in fall 2008 was deeply skeptical that the active learning approach would be effective for the conceptually difficult material covered in PHYS3233. Over the course of the semester his attitude underwent a 180? shift? (Cheville & Bunting, 2011, p. 20)] 3. Awareness of Innovation * The degree of awareness from faculty members toward the pedagogical innovations [?engineering faculty and graduate students selected by them attend a 1- day (8-hour) workshop at the beginning of the semester? (Riley, Davis, Cox, & Maciukenas, 2007, p. 3)] 4. Recognition of Cultural Differences The varying intrinsic and extrinsic attributes within and between dissemination groups; accounts for the differences between different teachers, schools, ethnicities, and locations [?If the instructor has limited experience with teaching second language students and therefore do not understand the typical language patterns, fluency challenges, and register issues that second language users present, then it is predictable that the class momentum and focus will be affected? (Kayfetz & Almeroth, 2008, p. 6)] 5. Innovativeness A faculty member?s willingness to try out a pedagogical innovation [?Jack?s innovation was an informal, ?maverick? approach to online teaching as a response to his perceived limitations of the institutional LMS. He researched and set up a collaborative, wiki based platform for his teaching and learning? (Hannon, 2009, p. 19)] 6. Accommodating Learning Styles (-) Accounts for the way faculty members accommodate differences between different students, schools, ethnicities, and locations [?Traditional instructional methods are not adequate to equip engineering graduates with the knowledge, skills, and attitudes required of them in the coming decades (Felder, Woods, Stice, & Rugarcia, 2000). They are not tailored to meet the particular learning style needs of engineering students. Failure to develop appropriate pedagogical methodologies may result in poor performance, professional frustration, and loss of potentially good engineers (Felder & Silverman, 1988).? (Verzat, Byrne, & Fayolle, 2009, p. 358)] 32 7. Motivation to Innovate * The degree of desire on the part of the faculty member to innovate in the classroom [?faculty are unmotivated to adopt engineering education innovations when they perceive that teaching innovation is marginalized in promotion and tenure considerations (Borrego et al., 2010, p. 203)] 8. Receptivity to Change (-) The degree of receptivity from both faculty members toward the pedagogical innovation [?it is commonplace for faculty to resist giving over that much time, in that large a chunk, to do ethics? (Riley et al., 2007, p. 2)] 9. Self-Efficacy The belief that one has the capability to perform the pedagogical innovation [?Faculty gain a feeling of efficacy as they assess the effect on students and continue to improve their instructional practice? (Cramer et al., 2010, p. 5)] 10. Response to Social Pressure An individual's perception of what people who are important to him/her think of their use of the pedagogical innovation [?Experiences in our DLR project were extended in 2008 to develop a nanotechnology option within the CoE Department of Engineering Science and Mechanics using this spiral curriculum approach.? (Balasubramanian, Lohani, Puri, Case, & Mahajan, 2011, p. 333)] 11. Degree of Support The degree of support from faculty members toward the pedagogical innovation [?a surge of faculty support and technical assistance? (Cramer et al., 2010, p. 4)] 12. Voluntariness The degree to which use of the pedagogical innovation is perceived as being voluntary [?We do not force faculty to use the active pedagogies, but once used, most stick with it and we all try to use new active learning methods and report back to the rest of the faculty? (Borrego et al., 2010, p. 199)] Readiness of Administrators to Disseminate an Innovation 1. Providing Access The degree of simplicity in gaining access to the pedagogical innovation and all of its required components [?plans for the ELDM (Engineering Leadership Development Minor) began in 1995, courses were piloted in 1997 and the minor graduated its first students in 1999. The ELDM has since graduated over 500 students, ~90% of whom were from the college of engineering, and ~30% females.? (Schuhmann, 2010, p. 61)] 2. Developing Facilitating Conditions The degree of availability of the resources necessary to facilitate dissemination of the pedagogical innovation [?To facilitate learning, the instructor needs to provide resources that help students acquire information; examples include short videos, web sites, and readings from the textbook.? (Cheville & Bunting, 2011, p. 12)] 33 3. Management Support The degree of support provided by institutional authorities (i.e. government, school administration, etc.) [?Administrators (e.g., department heads, deans, academic vice presidents) should create incentives for faculty to develop innovative and interdisciplinary courses, to collaborate with parties outside the institution, and to develop engaging ways of teaching and assessing undergraduate science and mathematics courses? (Walczyk et al., 2007, p. 87)] 4. Motivation to Innovate * The degree of desire on the part of the administration to innovate the curriculum [?engineering enrollments are lagging, women are not significantly better represented, and under-represented groups have made little progress? (Cheville & Bunting, 2011, p. 2)] 5. Receptivity to Change (-) The degree of receptivity of the teaching environment toward pedagogical innovations [?Initial reactions were statements from engineering department chairs like, ?We should not be serving society. Aren?t we supposed to be serving industry?? ?We should not use the phrase ?serving society? in our vision statement because it sounds like we are a social non-profit organization.? and ?We will lose our funding if we adopt a vision statement about serving society.?? (Vanasupa, 2011, p. 25)] 6. Reducing Teacher/Student Turnover (-) The frequency and degree of change in the teaching staff and student population [?during the duration of our grant, the college dean of seventeen years retired, we received a new dean, had three different provosts, two of the six faculty left and were replaced by two new faculty (2006), and we lost one of those faculty members to the harsh realities of the California cost of living (2008)? (Vanasupa, 2011, p. 7)] 7. Voluntariness The degree to which use of the pedagogical innovation is perceived as being voluntary [?poor teaching evaluations during this process should not deter the faculty from using the innovations (implying that in some cases, they might be a disincentive)? (Borrego et al., 2010, p. 199)] Readiness of Students to Disseminate an Innovation 1. Anxiety to Use (-) The concerns that students have when faced with using pedagogical innovations [?another way of reducing the anxiety for students would have been to engage them more deeply in the change process? (Vanasupa, 2011, p. 24)] 2. Attitude to System The opinion that students hold of the pedagogical innovation [?The instructor reported there was initially considerable ?push-back? from students who felt they were not being ?taught? since the role of the instructor is less active under ES21C? (Cheville & Bunting, 2011, p. 17)] 3. Awareness of Innovation * The degree of awareness from students toward the pedagogical innovations [?only 5% students reported prior exposure to basic nanotechnology concepts? (Balasubramanian et al., 2011, p. 340)] 34 4. Recognition of Cultural Differences The varying intrinsic and extrinsic attributes within and between dissemination groups; accounts for the differences between different students, schools, ethnicities, and locations [?international students were more committed to the class and more diligent than some of the native speakers? (Kayfetz & Almeroth, 2008, pp. 5-6)] 5. Innovativeness A student's willingness to try out a pedagogical innovation [?Videos are entertaining and help me studying some technical concepts which are difficult to understand without a graphical representation? (Bravo, Amante, Simo, Enache, & Fernandez, 2011, p. 640)] 6. Accommodating Learning Styles (-) * Accounts for the belief differences between different teachers, schools, ethnicities, and locations [?students also recommend the use of videos, instructional web sites, slide shows, wikis and online experiments to better instruct and familiarize them with relevant information? (Kantardjieff, 2010, p. 1276)] 7. Motivation to Learn * The degree of desire on the part of the student to learn the content of an academic course [?low-cost videos are innovative teaching tools that have a positive effect on student motivation? (Bravo et al., 2011, p. 639)] 8. Receptivity to Change (-) The degree of receptivity from students toward the pedagogical innovation [?The curriculum changed while they were sophomores and many were resistant to these changes. Several students went to the professors teaching the first systems course, as well as another professor from the department, to complain about the reform.? (Hayden et al., 2011, pp. 18-19)] 9. Self-Efficacy The belief that one has the capability to perform the pedagogical innovation [?an essential aspect of the work carried out is the search for higher efficiency of the work of the students? self-preparation? (P?rez Morales et al., 2009, p. 21)] 10. Response to Social Pressure An individual's perception of what people who are important to him/her think of their use of the pedagogical innovation [?peer editing is a collaborative writing exercise where two or sometimes three or more students review and make suggestions about each author?s writing? (Kayfetz & Almeroth, 2008, p. 3)] 11. Degree of Support The degree of support from students toward the pedagogical innovation [?Interactive help and the own methodology of resolution of the exercises help students in the use of the applications? (P?rez Morales et al., 2009, p. 17)] 12. Voluntariness The degree to which use of the pedagogical innovation is perceived as being voluntary [?students were assigned at random to one of four study groups, each taught by two professors and two PhD students in electrical engineering? (Lang et al., 2007, p. 61)] Note. (-) next to a characteristic denotes that it is an inhibiting characteristic; all other characteristics are facilitating characteristics, (*) next to a characteristic denotes that this was not one of the initial 33 characteristics but was added after the aggregate review. 35 Synthesizing the Findings The findings from the configurative review were synthesized in a two-step process so as to develop a framework that was designed to achieve the second research goal. The two steps of the synthesis process were: ? Conceptualize the interrelationships among readiness of faculty members, administrators, and students thereby visualizing the organizational readiness to disseminate an innovation and ? Conceptualize the relationships between organizational readiness and the characteristics to disseminate an innovation. Synthesis step one. The variables that describe the readiness of faculty members, administrators, and students toward disseminating educational innovations are generally on a continuum, with some positively related to successful dissemination, while others are negatively related (negatively related variables are indicated in Table 2-1 with a ?-? sign). A combination of the right levels of these readiness variables among the different stakeholders can create an environment that is conducive to adopting educational innovations; similarly, the wrong combination can result in an environment in which educational innovations become very difficult to adopt. Figure 2-3 shows these combinations using a cube. The cube has three axes that represent the readiness variables of faculty members (x axis), administrators (y axis), and students (z axis), all ranging from zero to one. Although this is, to some extent, an oversimplification of these complex multivariate relationships, the figure does help us to conceptualize the combinations. The readiness of the organization to adopt an educational innovation can be thought of as the location in the cube where the level of the readiness for each 36 Figure 2-3. Conceptualizing organizational readiness for educational innovations C h a r a c t e r i s t i c s o f S t u d e n t s T o w a r d I n n o v a t i o n C h a r a c t e r i s t i c s o f F a c u l t y M e m b e r s T o w a r d I n n o v a t i o n C h a r a c t e r i s t i c s o f A d m i n i s t r a t o r s T o w a r d I n n o v a t i o n 1 . D e v e l o p i n g F a c i l i t a t i n g C o n d i t i o n s 2 . R e c e p t i v i t y t o C h a n g e ( - ) 3 . M a n a g e m e n t S u p p o r t 4 . M o t i v a t i o n t o I n n o v a t e 5 . P r o v i d i n g A c c e s s 6 . R e d u c i n g T e a c h e r / S t u d e n t T u r n o v e r ( - ) 7 . V o l u n t a r i n e s s 1 . A t t i t u d e t o S y s t e m 2 . R e c e p t i v i t y t o C h a n g e ( - ) 3 . A w a r e n e s s o f I n n o v a t i o n 4 . R e c o g n i t i o n o f C u l t u r a l D i f f e r e n c e s 4 . A c c o m m o d a t i n g L e a r n i n g S t y l e s ( - ) 4 . V o l u n t a r i n e s s 7 . M o t i v a t i o n t o L e a r n 8 . D e g r e e o f S u p p o r t 9 . A n x i e t y t o U s e 9 . I n n o v a t i v e n e s s 1 1 . S e l f - E f f i c a c y 1 2 . R e s p o n s e t o S o c i a l P r e s s u r e ( 1 , 1 , 1 ) ( 1 , 1 , 0 ) ( 1 , 0 , 0 ) ( 1 , 0 , 1 ) ( 0 , 1 , 0 ) ( 0 , 0 , 0 ) ( 0 , 1 , 1 ) ( 0 , 0 , 1 ) L e g e n d o f V e r t i c e s ( 0 , 0 , 0 ) = I n n o v a t i o n V a c u u m ( 1 , 0 , 0 ) = F a c u l t y D r i v e n ( 0 , 1 , 0 ) = A d m i n i s t r a t i o n D r i v e n ( 0 , 0 , 1 ) = S t u d e n t D r i v e n ( 1 , 1 , 0 ) = S t u d e n t R e s i s t a n t ( 1 , 0 , 1 ) = A d m i n i s t r a t i o n R e s i s t a n t ( 0 , 1 , 1 ) = F a c u l t y R e s i s t a n t ( 1 , 1 , 1 ) = I d e a l 1 . M o t i v a t i o n t o I n n o v a t e 1 . R e c e p t i v i t y t o C h a n g e ( - ) 3 . I n n o v a t i v e n e s s 4 . A t t i t u d e t o S y s t e m 5 . A w a r e n e s s o f I n n o v a t i o n 6 . D e g r e e o f S u p p o r t 7 . V o l u n t a r i n e s s 8 . R e c o g n i t i o n o f C u l t u r a l D i f f e r e n c e s 9 . A n x i e t y t o U s e 9 . S e l f - E f f i c a c y 9 . R e s p o n s e t o S o c i a l P r e s s u r e 1 2 . A c c o m m o d a t i n g L e a r n i n g S t y l e s ( - ) 1 0 1 1 C h a r a c t e r i s t i c s o f A d m i n i s t r a t o r s T o w a r d I n n o v a t i o n C h a r a c t e r i s t i c s o f F a c u l t y M e m b e r s T o w a r d I n n o v a t i o n C h a r a c t e r i s t i c s o f S t u d e n t s T o w a r d I n n o v a t i o n Note. Definitions of the characteristics appear in Table 2-1. The rankings of the characteristics are based on the frequency counts in the configurative review 37 set of stakeholders intersects. Although it is difficult to explore all the possibilities represented within the cube, we can gain useful insights by considering the situations at the extreme vertices. For convenience, we label each of the zones of the cube in terms of the distinct vertices and a brief explanation of each of the vertices is given in Table 2-2. We can define the readiness of an organization for educational innovation as some location inside the cube outlined in Figure 2-3 and Table 2-2. The organizational readiness can be assessed at the university, college, or departmental level. It is possible that an institution as a whole may have a different organizational readiness than a particular college or department within that institution. By considering the attitudes towards innovation at the department, college, or institutional level, it may be possible to identify the readiness of an organization, or a subset of an organization, to disseminate an educational innovation. Synthesis step two. We developed Figure 2-4 to show the interaction between the characteristics of the innovation and organizational readiness in order to identify whether an innovation may be most likely, more likely, least likely, or less likely to disseminate. This framework is structured in a similar manner to prior research that suggests that use of an innovation is a function of the level of innovation-value fit (poor, neutral, or good) and implementation climate (weak or strong) (Klein and Sorra (1996) and Damanpour and Wischnevsky (2006)). Figure 2-4 shows the organizational readiness on the X-axis (ranging from 0 to 1, weak to strong) and the characteristics of the innovation on the Y-axis ranging from zero to one (poor to excellent innovation). The four quadrants A, B, C, and D in this figure show the potential of an innovation to disseminate in a particular organization. Quadrant A indicates a relationship where the organizational readiness is near ideal and the characteristics of the innovation are rated 38 39 ?high.? In such situations, an educational innovation is most likely to be disseminated and adopted. Quadrant B shows a situation where the organizational readiness is near ideal, but the characteristics of the innovation is weak. Here, an innovation is more likely to be disseminated and might even be adopted and used, only to be abandoned once a newer innovation with stronger characteristics emerges. Quadrant C represents situations where both the organizational readiness and the characteristics of an innovation are rated ?low? and such an innovation is unlikely to be either disseminated or adopted. Quadrant D indicates a scenario where the organizational readiness is not ideal, but the characteristics of an innovation is rated ?high?; such an innovation may be implemented for a short time, but is less likely to be routinely adopted. Figure 2-4. Framework relating organizational readiness and characteristics of the innovation to the dissemination of innovations O r g a n i z a t i o n a l R e a d i n e s s 0 1 C h a r a c t e r i s t i c s o f t h e I n n o v a t i o n Q u a d r a n t C : L e a s t l i k e l y t o d i s s e m i n a t e Q u a d r a n t C : L e a s t l i k e l y t o d i s s e m i n a t e Q u a d r a n t D : L e s s l i k e l y t o d i s s e m i n a t e Q u a d r a n t D : L e s s l i k e l y t o d i s s e m i n a t e Q u a d r a n t A : M o s t l i k e l y t o d i s s e m i n a t e Q u a d r a n t A : M o s t l i k e l y t o d i s s e m i n a t e Q u a d r a n t B : M o r e l i k e l y t o d i s s e m i n a t e Q u a d r a n t B : M o r e l i k e l y t o d i s s e m i n a t e 1 . R e l a t i v e A d v a n t a g e 2 . R e l e v a n t t o J o b / P r o f e s s i o n 3 . A d a p t a b i l i t y 4 . E a s e o f U s e 5 . M e a s u r a b i l i t y 6 . P e r c e i v e d E n j o y m e n t 7 . U s a b i l i t y 8 . E n h a n c e I m a g e 9 . C o m m u n i c a b i l i t y 1 0 . E n h a n c e V i s i b i l i t y 1 1 . Q u a l i t y o f I n i t i a l I n f o r m a t i o n 1 2 . L o g i s t i c a l I s s u e s ( - ) 1 3 . P l a y f u l n e s s 1 4 . A i d S o c i a l P r e s e n c e 1 5 . T r i a l a b i l i t y 1 6 . C o m p a t i b i l i t y 0 1 N o t e . D e f i n i t i o n s o f t h e c h a r a c t e r i s t i c s o f t h e i n n o v a t i o n s a p p e a r i n T a b l e 2 - 1 . T h e c h a r a c t e r i s t i c s o f t h e i n n o v a t i o n s r a n k i n g i s b a s e d o n t h e f r e q u e n c y c o u n t s f r o m t h e c o n f i g u r a t i v e r e v i e w . O r g a n i z a t i o n a l r e a d i n e s s r a n g e s f r o m 0 ( 0 , 0 , 0 ) t o 1 ( 1 , 1 , 1 ) b a s e d o n F i g u r e 2 - 3 . 40 Examples of the Framework in Practice We used the 37 articles from the configurative review to analyze how well the framework described the dissemination of the innovations described in each article. The purpose of this analysis was to see whether the potential dissemination of a particular concept moved from one quadrant to another depending on the effort put forth to develop and adopt the innovation discussed in each article. Two articles were not classified since they did not deal with a particular innovation (Borrego et al., 2010; Hazen et al., 2012a). This analysis procedure provided us with some assurance that the synthesis used to develop the framework was reasonably robust. Moving from quadrant C/ B to quadrant D/ A. Sixteen of the articles focused on strong educational innovations that moved the characteristics of innovation upward on the Y- axis. These articles did not discuss the organizational members? readiness to disseminate the innovations. For example, Riley et al. (2007) discuss the development of ethics-in-a basket micro insertions in different engineering classes. The article mentions the importance of teaching ethics in engineering classes since it is emphasized by the ABET guidelines placing the teaching of the topic of ethics in quadrant C or B. The article provides several examples of how ethics can be brought into technical subject matter such as thermodynamics by developing ethics-in-basket micro insertions for this topic and disseminating them. Therefore, we classified the potential dissemination of teaching ethics in most engineering classes as in quadrant C (least likely) or B (more likely) with a possibility to move to quadrant D (less likely to disseminate) or quadrant A (most likely) once the faculty members use micro insertions innovations. Since this article did not discuss organizational readiness issues, we cannot differentiate between the quadrants C or B and D or A. 41 Other examples of movement from quadrant C/B to quadrant D/A were integration of service learning and other innovations in a curriculum (Hayden et al., 2011), implementing project based learning in secondary education (Morgan & Barroso, 2008), use of 3D virtual worlds such as Second Life (Pfeil et al., 2009), use of remote laboratories (Lang et al., 2007), reforms in doctoral engineering education (Akay, 2008), adoption of a spiral curriculum in a department (Krishnan et al., 2008), developing writing capabilities of computer science students (Kayfetz & Almeroth, 2008), development of doctoral students in an industrial program (Kolmos, Kofoed, & Du, 2008), development of learning environments that have one integrative and several smaller problems (Sanchez, Neriz, & Ramis, 2008), incorporating student designed laboratory experiments (Elahinia & Ciocanel, 2008), teaching manual technical drawing (McLaren, 2008), introducing new learning techniques to teach crystallographers (Kantardjieff, 2010), introducing collaborative assessment during first-year courses (Qualters, Sheahan, Mason, Navick, & Dixon, 2008), developing engineering leaders (Schuhmann, 2010), and using an online game-based learning environment (Cheung et al., 2008). All these articles advocate the use of exciting and interesting educational innovations, but fail to discuss the organizational readiness to implement the change process. Given that we did not have information on organizational readiness, we cannot state whether each of the innovation discussed in these articles will be less likely to disseminate (Quadrant D) or most likely to disseminate (Quadrant A). Moving from quadrant D to quadrant A. Six articles discussed innovations that used many of the characteristics shown in Table 2-1. In addition, they discussed organizational readiness variables thereby moving the dissemination of the innovation from quadrant D (less likely to disseminate) to quadrant A (most likely to disseminate). For example, Delatte, Sutton, 42 Beasley, and Bagaka's (2007) discussed the use of failure case studies in civil engineering and engineering mechanics courses. The article provided strategies such as use of workshops to train the faculty members and to develop a master plan linking courses, learning objectives, and case studies. They discussed development of a textbook based on the project and winning several awards thereby influencing the attitudes of administrators. The paper also discussed the evaluation results showing that student attitudes to learn improved as a result of this innovation. Thereby, this article showcases an innovation that moved it from quadrant D (less likely to disseminate) to quadrant A (most likely to disseminate) by addressing issues of organizational readiness. Other examples of moving from quadrant D to quadrant A were the development of technology enhanced learning in STEM disciplines (Cramer et al., 2010), use of a multimedia trainer to teach descriptive geometry (P?rez Morales et al., 2009), using podcasts to support communications skills development (Lawlor & Donnelly, 2010), applying total quality management principles (Bernold, 2008), and improving organizational readiness (Walczyk et al., 2007). It was interesting to note that even though some of these innovations were not ground- breaking or new (such as use of case studies or using podcasts), they moved to quadrant A due to strong organizational readiness. Moving from quadrant B to quadrant A. Three articles discuss situations where an innovation that was in quadrant B (more likely disseminate) moved into quadrant A (most likely to disseminate). For example, Verzat et al. (2009) discuss how the spaghetti game was used to improve team working experiences and communication skills. The first-year undergraduate students emphasized the fun and enjoyable aspects of the teams? experience. A search on google shows more than 123 million hits for the key term, ?spaghetti game engineering? showing its 43 widespread popularity. Therefore, we conclude that this innovation has moved from quadrant B to quadrant A (most likely to disseminate). Other examples of moving from quadrant B to quadrant A are the use of low cost educational videos (Bravo et al., 2011) and scheduling topics for improved student comprehension of recursion in computer science courses (Zmuda & Hatch, 2007). The major characteristics of these innovations were ease of use, simplicity, and cost effectiveness. Moving from quadrant C to quadrant A. Six articles discuss efforts by educators to move some innovations from least likely to disseminate (quadrant C) to most likely to disseminate quadrant A. For example, Cheville and Bunting (2011) report on a ten course sequence that makes changes across the program by using a development based paradigm instead of a knowledge based paradigm. They discuss efforts made to improve organizational members? readiness and characteristics of innovations so that the innovation had a probability of widespread dissemination. Thereby, they made efforts to move this innovation from quadrant C to quadrant A. It is not clear whether this innovation will remain in quadrant A given the amount of effort needed to keep it there. Other examples of movement from quadrant C to quadrant A are a final year project implemented using a Sakai virtual campus with 11 teachers (Badia et al., 2010), need for problem-based learning in South Africa (Veldman et al., 2008), increasing acceptance of a course management system (Christie & Jurado, 2009), students participating in an interdisciplinary sustainability program (Richter & Paretti, 2009), and embedding enterprise education in engineering (Handscombe, Rodriguez-Falcon, & Patterson, 2009). These innovations have many of the characteristics and are major changes initiated at these universities. 44 They seem to be implemented in an environment that is changing the organizational members? readiness to disseminate these innovations. Moving from quadrant B to quadrant D. Two articles describe a situation where an innovation moved from quadrant B (more likely to disseminate) to quadrant D (less likely to disseminate). For example, Hannon (2009) described a collaborative, Web 2.0 wiki-based platform that could be used to supplement online teaching. The wiki-based platform was set up by a faculty member to overcome the limitations of the institution?s learning management system and used successfully by a group of faculty members for two semesters before being abandoned. The primary reasons for the failure of this project were due to organizational readiness since this project lacked administrative support and the IT resources needed to maintain the wiki. Fear of student resistance, that could result in poor student evaluations for faculty members? eventually ended this project (Hannon, 2009). In another example, Vanasupa (2011) discuss a year long project based learning sequence that was implemented successfully in 2006, but began to falter due to a split in the department and a mixed belief system by 2009. Moving from quadrant C to quadrant B. Two articles discuss innovations that have moved from least likely to disseminate (quadrant C) to more likely to disseminate (quadrant B). Conlon (2008) discusses the need for ethics education to be included in all facets of engineering education and makes a strong case for improving organizational readiness in this area. But they don?t discuss any educational innovation in teaching ethics education, thereby assuming that traditional lectures will do an adequate job of covering these concepts. Gill, Sharp, Mills, and Franzway (2008) makes a persuasive case for including more women in engineering education and identify issues that deter women from joining or remaining in engineering profession. This article does not propose new educational innovations. These articles may motivate the faculty 45 members to teach ethics and encourage women in engineering for a period of time, but it is not clear whether this will be a sustained effort. The above analysis of these articles was based on two researchers? perceptions based on the information presented in the articles. This analysis may not mirror the movement of an innovation from one quadrant to another as envisioned by the article?s authors since they have access to much more information about the particular innovation and how it was disseminated. But the overall analysis shows the utility of the framework in describing the need to consider organizational readiness in addition to the characteristics of the innovation in order to achieve widespread dissemination of a particular innovation. Discussion Potential Use of the Framework by Stakeholders This framework may be beneficial to stakeholders such as educational innovators, faculty members, university administrators, and grant program managers. Educational innovators may use this framework to improve their innovations so that they excel in one or more of the identified characteristics, enabling them to market their innovations more effectively. Future research might combine these characteristics into an index that would provide a useful summary of the characteristics of new innovations. This index could be used by an innovator to communicate the benefits of his/her innovation relative to the current practice. Additionally, the developers of an innovation could use this framework to help create promotional materials for adopters who work at organizations that are ready to disseminate. This framework makes faculty members think about the readiness of their organizations to adopt and use innovations. If their organization is not ready, the faculty member may prefer to adopt only well-tested and well-known innovations, but if he or she deems their organization 46 to be ready the faculty member may be encouraged to adopt new leading-edge or even bleeding- edge innovations. For administrators, this framework could help them understand why certain educational innovations are not successfully adopted. After improving the characteristics of the innovations that are being adopted in their organization, they can enhance the readiness of organizational members by offering appropriate incentives. This framework may encourage administrators to periodically ?take the temperature? of their organization to measure progress and ensure that the organization is ready and willing for innovations to be adopted and routinely used. In addition, the framework and the characteristics of its component variables may be used to create a strategy to move towards the ideal organizational readiness zone (1,1,1). Grant program managers are also likely to find this framework useful for evaluations of proposed innovations as well as for realistic assessments of the organizational readiness of the institutions who apply for funding. Future research might develop a readiness index for an organization that could provide a quick way to estimate the likelihood of success of certain innovations within a particular organization. Grant writers could be asked to develop dissemination plans that target organizations where they intend to use the proposed educational innovations to increase the likelihood of successful adoption. This framework may help to ensure that grant funded educational innovations are disseminated for widespread routine use, rather than simply being used only at a single institution during the period for which they were funded. Limitations and Future Research The paper has several limitations. As Greenhalgh et al. (2004) states, another group of researchers might have synthesized the literature presented in this article and produced a 47 different unifying framework. The framework should therefore be seen as ?illuminating the problem better and raising areas to consider? (Gough et al., 2012, p. 613) rather than ?providing the definitive answers? (p. 614). This framework may not explain prospective change initiatives in some instances and other frameworks may have to be developed in the future. Most importantly, we may not have identified all of the characteristics of innovation and organizational readiness variables. Every reader will probably have a favorite characteristic that is not on the list. Certainly, we can think of a few of our own; however, the purpose of this article was to uncover those characteristics explicitly discussed in the engineering literature as antecedents to successful dissemination. The methods used in this study presume that the peer reviewed, published reports of educational innovations in the literature present a complete picture. However, we fully recognize that the published literature gives us a starting point, and our corpus is naturally biased towards published studies with significant effects. This publication bias is often present in meta-analytic studies and is often referred to as the file drawer effect (Rosenthal, 1979). In order to overcome this publications bias, medical journals have created quality measures to evaluate the methodology of studies (Guyatt et al., 2011; Shea et al., 2009). Similar quality measures should be developed to evaluate the methodology of engineering education studies. We also expect that selection bias may be present in this study, as we have used examples and excerpts from sampled articles to describe the largely subjective process of educational dissemination. It is also possible that other methodologies could be used to supplement this qualitative review. Future research should create case studies, similar to those developed by McKenzie, Alexander, Harper, and Anderson (2005), that study the readiness of departments and institutions to incorporate educational innovations into the curriculum and look for additional characteristics. 48 Much of the recent grant funded research regarding dissemination has focused on the faculty or course level. Future research should investigate attitudes of students and administrators on the dissemination process. The use of qualitative research methods such as Delphi studies with faculty members, administrators, and students should be used to identify any additional variables that may contribute to organizational readiness. The description of how to use the framework needs further development. The cube is essentially a black box, and we expect the readers to wonder how to specify a point within the cube. The framework is conceptually useable but not specifically useable, absent an index. The conceptual framework might actually be predictive of the likelihood of actual implementation and continuation, but this is beyond the scope of this article. Future research should also include building surveys and empirically testing the model so that an indexing scheme can be developed to evaluate the level of organizational readiness that exists at the departmental, college, or institutional level. This could result in the creation of a score card, index, or decision support tool that might be used to apply this framework in an educational setting. This is the reason we have not discussed how to operationalize the characteristics of the innovations or the variables that determine organizational readiness. Even though literature exists that operationalizes these variables, this is beyond the scope of this paper since our goal was to develop a descriptive framework. Like most frameworks developed to describe a behavioral phenomenon, our framework is an oversimplification of the complex reality of the processes involved in the dissemination and adoption of educational innovations in higher education. The framework does not account for certain difficult to measure factors such as politics, interpersonal relationships, and other similar factors. Many innovations are adopted not because they can be measured or have a clearly 49 demonstrable relative advantage, but because a friend recommended it or the adopter was pressured into it by a more senior colleague. This article is based on content analysis, which is subject to the preconceptions of those who generated the content. Consequently, the use of content analysis, particularly of papers written by individuals who lacked adequate frameworks for explaining dissemination, may lead to confirmations of ?obvious? explanations. A caveat about combining the findings of this highly diverse literature review, which included the theory of planned behavior (Ajzen, 1991), the theory of reasoned action (Fishbein & Ajzen, 1975, 1981), diffusion of innovation theory (Rogers, 1962, 2003; Rogers & Shoemaker, 1971), and CBAM (Hall, 1974; Hall & Hord, 2006), is the potential for conflicts resulting from their various epistemological and ontological assumptions. The issue of whether or not any of these assumptions were violated during the process clearly needs to be investigated further. Finally, this study focused primarily on educational innovations in engineering education. Our focus was on engineering education since reviewers from NSF?s TUES program reported that lack of dissemination was a major weakness (Feser et al., 2012), and we have not tested whether this framework could be extended beyond this discipline. Since all of our data came from the field of engineering, we are unable to generalize our results to other STEM areas. Future research should investigate whether this framework is supported in other STEM disciplines. Conclusions This synthesis of the recent literature in engineering education contributed to the development of a conceptual framework describing the dissemination process, in addition to providing a useful summary of the recent research-based findings on successful dissemination. 50 While the framework is intuitively appealing, it will be difficult to operationalize and study empirically. We recommend that empirical studies be conducted to determine the extent to which the readiness of faculty members, administrators, and students toward disseminating an innovation combines with the characteristics of the innovation to predict the success or failure of the dissemination process. The conceptual framework proposed in this paper does, however, require further research/testing/evaluation using Delphi studies/field work/surveys for triangulation and to validate its use in the future (Patton, 2002; Shenton, 2004). The contribution of this research is the derivation of a framework that achieves the two research goals. A particular strength is the emergence of students (in addition to faculty and administrators) as an important organizational member who influences organizational readiness to disseminate an innovation. In most educational projects, the desired outcome is the successful dissemination of educational innovations; the interventions are the characteristics of the innovation; the context is the organizational readiness (readiness of faculty members, administrators, and students to disseminate innovation); and the underlying mechanism is the proposed framework that links the interventions, context and the outcome. The framework shows that for widespread dissemination to occur, educational innovators need to build positive characteristics in the innovations and need to implement them in organizations that are ready for such change. 51 Chapter 3. Identifying Characteristics of Dissemination Success Using an Expert Panel Abstract Although a great deal of work has been done to develop new educational innovations, few have found widespread acceptance in the classroom. To improve the adoption of educational innovations, we need to understand why some innovations are adopted and routinely used, while others are not. This article presents an expert panel?s answer to the following question: ?What are the most important characteristics that relate to the dissemination of educational innovations?? As dissemination is a critical facet of the diffusion of an innovation, 45 researchers who received technology and engineering grants from the National Science Foundation (NSF) participated in a Delphi study designed to address this research question. In three rounds, the experts identified and ranked 11 characteristics of educational innovations, 6 characteristics of students, 13 characteristics of faculty members, and 5 characteristics of administrators that can relate to the successful dissemination of educational innovations. The results of this study led to the formation of a Characteristics of Dissemination Success (CODS) framework. This framework offers useful guidance for educational innovators seeking more widespread adoption of their innovations. Key Words: Delphi Method, Diffusion, Dissemination, Educational Innovations, Framework 52 Introduction During 2010, U.S. federal agencies invested $3.4 billion in science, technology, engineering and mathematics (STEM) education (National Science and Technology Council, 2011). The programs funded included efforts to: increase the number of students involved in degree programs and careers; develop evidence-based education models and practices; develop technical skills or knowledge in students; and increase learner engagement, interest, and value in technology areas (National Science and Technology Council, 2011). During the same period, twelve private foundations committed $506 million through the ?Investing in Innovation? fund to improve STEM education (Blankinship, 2010). However, despite the many recent advances in STEM teaching and learning know-how, it is felt by many policy makers and practitioners (and evident in accounts published in articles in academic journals) that highly effective teaching and learning practices are still not in widespread use in most institutions of higher education (National Science Foundation, 2013). Here, we define dissemination as the process of creating an awareness of the artifact of interest ? in this research, technology and engineering educational innovations ? followed by influencing the intention to adopt, actual adoption, and finally the routine use of the innovation (Fincher, 2000; Gravestock, 2002; Hutchinson & Huberman, 1994; King, 2003). In business education, the Academy of Management, the Association for Information Systems, the Decision Sciences Institute, and the Institute for Operations Research and Management Sciences (INFORMS) all have conference tracks on educational innovation. In addition, they publish academic journals such as the Academy of Management Learning and Education (AMLE), the Decision Sciences Journal of Innovative Education (DSJIE), and INFORMS Transactions on Education that disseminate research-based educational innovations. 53 Recently, several article from these journals have discussed the utility of incorporating academic games (D'Amours & R?nnqvist, 2013), active learning (Dufresne & Offstein, 2012), case studies (Erzurumlu & Rollag, 2013; Hu & Kumar, 2013; Nkhoma, Sriratanaviriyakul, Pham, & Lam, 2013), computer simulations (Leemkuil & De Jong, 2012; Legner, Estier, Avdiji, & Boillat, 2013; Snider & Balakrishnan, 2013), and experiential learning (Taras et al., 2013; Wu & Sankar, 2013) into business education. In engineering disciplines, many educational associations and government commissions, including the American Society for Engineering Education (2009, 2012), the National Academy of Education (2009), the National Academy of Sciences (2012), and the National Research Council (2011), have suggested that scenario-, problem-, and case- based teaching and learning need to be incorporated into engineering and technology classrooms. Faculty members who have used these new instructional materials and strategies in the classroom report improved student engagement and motivation on both an intellectual and personal level (Balasubramanian et al., 2011; Krishnan et al., 2008; Mykytyn, Pearson, Paul, & Mykytyn, 2008; Schunk, Pintrich, & Meece, 2008; Smith, Sheppard, Johnson, & Johnson, 2005). The President?s Council of Advisors on Science and Technology (2012) concurs, recommending the widespread adoption of such empirically validated teaching practices so as to improve the enrolment of students in STEM education. The Organisation for Economic Cooperation and Development (2010) suggests that improving all U.S. students to a baseline level of STEM proficiency could result in an increase in the gross domestic product by $72 trillion based on historical growth relationships. It is therefore critical that empirically validated educational innovations are widely diffused, disseminated, and adopted (National Science Foundation, 2013). Therefore, we formulated the research question for this study as: ?What are the most important characteristics that relate to the dissemination of educational innovations?? To answer 54 this question, we solicited, via a three-round Delphi study, the opinions of 45 faculty members who had received grants from the National Science Foundation (NSF) in the areas of engineering and technology. We synthesized their opinions and developed the Characteristics of Dissemination Success (CODS) framework. In the next section we discuss the literature regarding educational innovations and factors that relate to the dissemination process. Next, we discuss the Delphi methodology and its application within the context of the current study, after which we present the analysis of the opinions of the experts. The analysis is then used to develop the Characteristics of Dissemination Success (CODS) framework. The paper concludes with a set of recommendations for future research and a summary of this study?s contributions. Background Educational Innovations In the broadest context, Rogers (2003) describes innovations as ideas, practices, or objects that are perceived as new by an individual or group. More specifically, Tornatzky and Fleischer (1990, p. 20) describe two primary types of innovations as either product innovations, defined as ?those that are ends in themselves?, or process innovations that are ?adopted as instrumental to some other end?. Both Rogers? (2003) general and Tornatzky?s and Fleischer?s (1990) more granular characterization of innovations provide the basis from which additional researchers describe educational innovations. In pedagogical research, Hazen, Wu, and Sankar (2012a) define an educational innovation as any new instructional material, strategy, or pedagogical approach that differs from the current and previous materials or methods employed by an educator. Relating this characterization to Tornatzky's and Fleischer's (1990) product categorization, examples of educational product innovations would include multimedia resources such as podcasts, videos, laboratory simulations, and websites with subject resources (McKenzie 55 et al., 2005), while educational process innovations would include online role-playing simulations, the use of problem-based learning, and peer- or criterion-based assessments. From these definitions and examples, researchers further subdivide aspects of educational innovations into new curricula, instructional materials, faculty expertise, and/or instructional strategies. Curriculum development deals with overarching changes planned or implemented for an educational program or curriculum, while instructional materials are specifically designed for use in a particular course or laboratory session. The development of faculty expertise refers to methods that can be used to help faculty members improve their teaching such as training workshops or mentoring. Finally, instructional strategies, such as experiential learning or student centered learning, consist of both instructional design and pedagogy and include approaches that enhance the acquisition of knowledge (Gropper, 1974). Factors that Relate to the Dissemination Process A number of different theories, models, and frameworks have been proposed to describe the complicated processes associated with the dissemination of educational innovations (Fairweather, 2008; Hazen et al., 2012b; Henderson et al., 2010). Dancy and Henderson (2010) assert that current approaches to the dissemination of educational innovations, such as the publication of journal articles, and participation in conferences and workshops, are successful in raising general awareness regarding innovative instructional practices but fail to robustly support faculty seeking to adopt and implement the innovation. Borrego, Froyd and Hall (2010) concur, reporting that although 82% of department chairs surveyed were aware of seven recent educational innovations, only 47% reported any actual adoption of those seven innovations at their respective institutions. 56 These findings appear to signal a distinct disconnect between the information available regarding educational innovations and educators? subsequent adoption of those innovations. Interestingly, the department chairs that participated in the 2010 survey noted that they saw student perceptions, faculty attitudes, and administrative barriers as major considerations in educational innovation adoption decisions. A number of researchers have found evidence of significant resistance from students regarding the potential impact of dissemination of information about educational innovations (Dancy & Henderson, 2010; Hazen et al., 2012b; Henderson et al., 2010). Henderson and Dancy (2007) report student attitude to be a prominent barrier to the adoption of new instructional strategies and similar studies have also identified lack of student motivation as a significant factor in the adoption decision related to new instructional materials and strategies (Henderson & Dancy, 2007; Kolmos et al., 2008; Morgan & Barroso, 2008). Another possible reason for the apparent disconnect in the adoption of educational innovations may be a gap between the faculty member?s teaching style and his or her students? learning styles (Felder & Silverman, 1988), while faculty resistance to educational innovations has been suggested to pose a significant impediment in several recent pedagogical studies (Cheville & Bunting, 2011; Henderson et al., 2012; Riley et al., 2007). Last but not least, administration is cited by The President's Council of Advisors on Science and Technology (2012) as a barrier to educational innovation adoption, pointing out that administrators generally fail to provide faculty rewards and incentives and do not invest in classroom renovations. The Delphi Method The current study seeks to identify those characteristics that relate to the successful dissemination of educational innovations. We define ?successful dissemination? as the 57 communication of information regarding an educational innovation that results in a decision to adopt and use it in a routine manner. Methodologically, the most appropriate means to conduct this type of research is through the use of the Delphi technique. The Delphi method is a qualitative research technique developed by the Rand Corporation in the early 1950s (Dalkey & Helmer, 1963; Linstone & Turoff, 2011; Malhotra et al., 1994) and Delphi studies have been used to prioritize the qualities, attributes, and best practices of expert panels and leaders (Cegielski, Bourrie, & Hazen, 2013; Loo, 2002; Romano, 2010). Because of the unique characteristics of the methodology, the Delphi technique is well-suited to highly multidimensional research questions that deal with uncertainty in a domain of imperfect knowledge (Cegielski et al., 2013; Churchman & Schainblatt, 1965). The objective of the technique is to achieve consensus among experts regarding a specific topic (Okoli & Pawlowski, 2004; Taylor & Meinhardt, 1985). In previous comparative analyses of group survey techniques, the Delphi technique was shown to achieve a greater level of accuracy than other group consensus techniques (Linstone & Turoff, 2011; Riggs, 1983; Rohrbaugh, 1979). Operationally, the Delphi Method involves three phases: the selection of expert panelists, the collection of topic-relevant issues, and the ranking of reported issues (Cegielski et al., 2013; Okoli & Pawlowski, 2004). The term ?expert? is subjective and thus requires researchers to specify, utilizing measurable terms, what constitutes an expert for the purposes of their study. Typically, experts are selected based on factors such as years of professional experience, job or position title, level of education, and/or professional certifications. To collect topic-relevant issues, the initial round of the Delphi questionnaire is open-ended (Delbecq, Van de Ven, & Gustafson, 1975). In early research, Delphi studies were sent to between 5 and 15 experts but current standards suggest sample sizes ranging from 15-30 panelists for heterogeneous 58 populations and five to ten panelists for homogeneous populations (Linstone & Turoff, 2011; Loo, 2002; Martino, 1972). The purpose of the first questionnaire round is to aggregate information for subsequent ranking rounds of the study (Brancheau & Wetherby, 1987). In the first round, the panel of experts each contribute whatever input that they feel is pertinent to the focus question of the study (Nambisan, Agarwal, & Tanniru, 1999) and then in the second round they rank each of the issues from the first round (Paliwoda, 1983). From the data gathered in the second Delphi round, the study administrator scores the issues (typically using the weighted average method) and redistributes the results to the panelists (Nambisan et al., 1999). In the third round, as well as any subsequent rounds of the study, the experts review the group rankings and rerank the issues given the aggregated responses of the group. The process of ranking and re-ranking continues until the panel achieves a consensus (Delbecq et al., 1975; Linstone & Turoff, 2011). In the current study, we employed a three-round Delphi process to aggregate four separate consensuses regarding the research question. The three Delphi rounds were all conducted in a distributed fashion via e-mail correspondence and the online survey software Qualtrics. We began by identifying a total of 307 potential expert participants, all of whom were Principal Investigators (139 individuals) or Co-Principal Investigators (168 individuals) from the NSF?s Course, Curriculum, and Laboratory Improvements (CCLI) or Transforming Undergraduate Education in STEM (TUES) programs in the disciplines of engineering, technology, and computer science. The list of potential subjects was assembled by analyzing past and current CCLI and TUES grants and then searching the internet for Co-Principal Investigators? e-mail addresses that were not listed on the NSF website. As part of the subject 59 solicitation process for this study, each of the 307 potential panelists received an email describing the study and invited to participate by responding to the following four questions: ? What characteristics of educational innovations relate to their successful dissemination? ? What characteristics of students relate to the successful dissemination of educational innovations? ? What characteristics of faculty members relate to the successful dissemination of educational innovations? ? What characteristics of the college or university administration relate to the successful dissemination of educational innovations? Participation By Round The participants were given two weeks to respond to the initial invitation and the open- ended research questions. Forty-five responses were received for the initial round, a response rate of 14.7%, 23 of whom were Principal Investigators and 22 Co-Principal Investigators (Table 3-1). In total there were 18 non-deliverable email addresses and 15 individuals who declined to participate. Members of this panel included at least one member from 25.8% of all the CCLI and TUES grant projects that were sent the survey. A total of 593 comments were generated by this group of participants (Table 3-2). Of these, 174, 119, 165, and 135 comments were related to the characteristics of the innovation, students, faculty, and administrators, respectively. As is always the case with the Delphi technique, many of the comments generated by the members were repetitive. To synthesize the comments provided by the expert panelists, a three member committee was established consisting of one researcher with experience in the research domain and over 15 years of experience in educational innovation and dissemination who had received funding through 60 Table 3-1: The distribution of subjects and response rates for all three rounds of the study Position Title No. of Invited Participants No. of Respondents Round 1 Initial Response Rate No. of Respondents Round 2 Round 2 Response Rate No. of Respondents Round 3 Round 3 Response Rate Principal Investigator 139 23 16.75% 17 73.91% 17 100% Co-Principal Investigator 168 22 13.10% 18 81.82% 17 94.44% Total 307 45 14.66% 35 77.78% 34 97.14% Table 3-2: Count of unique comments generated in round one of the Delphi survey Total No. of Comments Generated Number of Unique Constructs Characteristics of the Innovation 174 11 Characteristics of Students 119 6 Characteristics of Faculty 165 13 Characteristics of Administration 135 4 Total 593 34 N = 45 61 several NSF CCLI/TUES grants, one researcher who was an expert in the diffusion of innovation and the Delphi approach, and one researcher who was not an expert in the research domain, but who was proficient in qualitative research methodologies. To maintain a high level of inter-rater reliability, each of the committee members individually reviewed all of the contributed comments and subsequently classified them into their own respective categories. Upon completing their individual classifications, the committee met to discuss each individual?s classification of the comments and subsequently agreed on a single comment classification schema. Table 3-3 shows examples of the comments made by the experts for each of the classifications and the resulting classification of the comments. These classifications form the basis for identifying the characteristics of the educational innovations, students, faculty members, and administration that the expert panel considered important for the successful dissemination of educational innovations. For each characteristic, a contextual definition was created based on the comments. A full list of the comments collected is available from the authors. Appendix 3-1 provides a categorized list of participants comments. Table 3-3. Classifications derived from participants? comments Example of comment by expert that illustrates the classification Characteristic Contextual Definition Characteristics of the educational innovation Must be sufficiently flexible so that instructors may tailor it to their needs, Adaptable so that an instructor can adapt to his or her style and environment, etc. Adaptability The degree to which the user can modify the pedagogical innovation as deemed necessary. Ability to communicate the results, the ability to articulate the problem addressed by the innovation, etc. Communicability The ease with which the results of using the pedagogical innovation can be easily described to others. 62 Match of innovation goal with goal of faculty and students, the innovation is not that different from what is already being done, etc. Compatibility The consistency of the pedagogical innovation with current pedagogy. Conceptually simple so that it can be easily understood by adaptors, relatively simple and user friendly, etc. Ease of Use The degree to which a pedagogical innovation is perceived as relatively easy to use and understand. Ease of adoption (how much work do "I" have to do to adopt the innovation, easy to adopt, etc. Ease of Implementation The degree to which a pedagogical innovation is perceived as relatively easy to implement and adopt. Their benefit to student learning is measureable and quantified, innovation must have solid evaluation data documenting success, statistical significance of results, etc. Measurability The degree to which the impact of the pedagogical innovation can be assessed in particular the ability to clearly attribute the effects to the innovation. Innovation is rooted in a documented need for the practice of engineering, a clear link between the innovation and a prevalent problem, etc. Practicality of the Concept The degree to which the innovation is based on a clear definition of basic terms and clear methodology applicable for the specific situation. Completeness of the ?package? on how to use the innovation, high quality descriptions, excellent formatting, visuals, etc. Quality of Initial Information The degree to which the initial information regarding the pedagogical innovation is relevant, timely, complete, and appropriate in terms of amount so as to add value. Don't require a lot of time (compared to current techniques) to prepare and administer, cost benefit, it is clearly an improvement, etc. Relative Advantage The degree to which the pedagogical innovation is perceived as being better than its predecessor. Innovation seems useful to my work and will make my job easier, professional relevancy, etc. Relevance to Job The ability of the pedagogical innovation to enhance the educator?s or student?s performance. Innovations that are based on theoretical understandings of pedagogy or learning , highly based on a strong literature review, etc. Theoretical Coherence The degree to which pedagogical innovations are based on a theoretical understanding of pedagogy or learning. 63 Characteristics of the students Diverse population of students, ethnic/social background, fluency in spoken English, etc. Cultural Differences The varying intrinsic and extrinsic attributes within and between dissemination groups; accounts for the differences between different, teachers, schools, ethnicities, and locations. Engagement with innovative products, willing to ask questions, student?s willingness to engage with others, etc. Engagement The degree to which students study or practice a subject to learn about the subject. Will the innovation advance student learning, etc. Intent to Enhance Learning The degree to which the innovation will enhance student learning. Possess the meta-cognitive traits that value improved problem solving, creativity, or complex thinking skills; It is important to form a teaching philosophy where there is a good mix of audio, visual, and interactive learning, etc. Learning Styles Accounts for the belief differences between different students, faculty, schools, ethnicities, and locations. Who would like to learn the concepts and apply them, they are not concerned about grades only, motivated to learn the material, etc. Motivation to Learn The meaningfulness, value, and benefits of academic tasks to the learner--regardless of whether or not they are intrinsically interesting. Openness to change from standard lectures and problem sets, willing to follow different paradigms, etc. Receptivity to Change The degree of receptivity from the students toward the pedagogical innovation. Characteristics of the faculty members I am looking for imaginative things, those that use technology, or are strongly visual or event things that have an entertainment factor, I hear these kinds of things, it engages me, and so I go try them in my classroom. Attitude to Innovation The opinion that a person holds toward the pedagogical innovation. Awareness of innovations through reading, conference attendance, word-of mouth; amount of media attention, especially TV. Awareness of Innovation The degree of awareness from the faculty toward the pedagogical innovation. 64 A devoted faculty who has a passion of teaching and learning and is willing to go the extra mile to ensure the integrity of the students' learning outcomes, faculty?s personal desire to improve their students? learning and success, etc. Care about Student Learning Outcomes The degree to which faculty focuses on the learning process and not the results. If they reflect approved minority/special needs status, the innovation is given consideration, etc. Cultural Acceptance of Innovation The degree to which the cultural differences of the students and faculty influence the acceptance of the pedagogical innovation. Provide logistical support to adopters; workshops, webinars, websites; etc. Degree of Support The degree of support from the educators and the students toward the pedagogical innovation. Being willing to take the risk to try something new, creativity to come up with new ideas, tolerance for early failure, etc. Innovativeness A faculty member?s willingness to try out a pedagogical innovation. Getting bored if I have to do the same thing, exactly, over and over again; funds to disseminate, the innovation clearly is going to improve an outcome that matters to me or my campus (i.e. retention, students success), etc. Motivation to be Innovative The wants, needs and set of circumstances that drive a person to be innovative. Strong ties with other faculty within their own department as well as at other institutions, faculty who participate in many newsgroups, workshops, seminars, and conferences, etc. Professional Social System A set of interrelated units that are engaged in joint problem solving to accomplish common professional goals. Openness to change, willingness to try new things, Faculty should be open-minded for such dissemination, etc. Receptivity to Change The degree of receptivity from the faculty toward the pedagogical innovation. Confidence in using the innovation, confidence in their own teaching ability, etc. Self-Efficacy The belief that one has the capability to perform the pedagogical innovation. 65 Belonging to a supportive community willing and able to try out others' innovations, faculty need to be supportive when an innovation does fail, etc. Supportive Community The degree to which the faculty community supports and encourages continuous teaching improvements regardless of measureable or successful outcomes. One needs the security of tenure to be able to devote the time and resources to such things, less desirable for non-tenured faculty to engage in introducing innovations, etc. Tenure Status Whether the individual that is adopting the pedagogical innovation is a tenured or non- tenured faculty member. Recognition that teaching is of real value, willingness to keep learning about teaching, strong commitment to teaching, etc. Value Teaching in Addition to Research The degree to which the faculty balances the priorities of teaching and research. Characteristics of the Administration Travel funds for conferences and presentations; infrastructure to support the use of the innovation; etc. Degree of Facilitating Conditions The degree of availability of resources necessary to facilitate dissemination of pedagogical innovations. Release time for instructors to learn about innovations and to create or modify their courses; creating an atmosphere of innovation in education; interest in development and support of innovations, etc. Management Support The degree of support provided by institutional authorities (i.e. government, university administration, college deans, department chairs, etc.). Promotion, tenure, and merit systems that reward innovation; recognition of the effort required to disseminate and adopt innovations; etc. Promote and Reward Innovation The degree to which the administration rewards and encourages innovation in the classroom. Open mindedness, cooperation, and collaboration; willingness to try new things, etc. Receptivity to Change The degree of receptivity from the administration toward the pedagogical innovation. To begin round 2, the specific unique results were reported, via an online survey, to the expertrs with instructions to review the characteristics and rank them in order from most important (1) to least important. Each of the participants was given two weeks to individually 66 evaluate the characteristics related to the successful dissemination of educational innovations and individually rank the associated items. The response rate for round two of the Delphi method was 73.9% (17/23) for Principal Investigators, 81.8% (18/22) for Co-Principal Investigators, and 77.8% (35/45) overall. The ranking for the overall group was computed using the weighted average method. The fact that a majority of the experts responded to the second round provides a qualitative verification that the classifications made by the researchers based on the comments in round one was reasonable. To begin round three of the Delphi survey, the participants from round two were invited to participate in an online survey and given the rankings from round two. Each of the participants was asked to review the group?s weighted average ranking of the characteristics and again rank the characteristics in light of the group rankings. The response rate for round three of the Delphi method was 100% (17/17) for Principal Investigators, 94.4% (17/18) for Co-Principal Investigators, and 97.1% (34/35) overall. The ranking of this group was computed using the weighted average method. Following the completion of the third round of the study, an inspection of the results revealed that none of the rankings of the items changed between the second and third round results. Therefore, a concordance assessment was made. The computation of Kendall?s Coefficient of Concordance (W) revealed that a consensus existed among the participants with respect to the to the round-three rankings of the importance of the items. Kendall?s W is a measure designed to determine the degree of agreement among individual sets of ranked scores (Siegel, 1956). A significant W indicates that the participants applied essentially the same standard in judging the importance of the issues and they are in consensus. For the third-round rankings of the Delphi survey, the Ws were (W(Educational innovations) = .90, p < .001, W(Students) = .84, 67 p < .001, W(Faculty) = .93, p < .001, and W(Administration) = .79, p < .001), all of which are statistically significant. An additional estimate of group consensus, the percentage of respondents whose characteristics rank matched the group rank provided further support for the assertion that each group was in accordance. Based on the Kendell?s W and the rank percentage, it was clear that consensus was achieved for all four research questions and hence no additional Delphi rounds were deemed necessary. Development of the Characteristics of Dissemination Success (CODS) Framework Based on these results of the Delphi study we developed the CODS framework shown in Figure 3-1. This figure shows the characteristics and ranks that were agreed to by the experts in round 3 of the Delphi study. The framework is described in more detail in this section. The Stages of Disseminating Educational Innovations The CODS framework is grounded in the theory of reasoned action (Fishbein & Ajzen, 1975, 1981), the theory of planned behavior (Ajzen, 1991) and other research (Moore & Benbasat, 1991; Rogers, 2003; Compeau, Meister, & Higgins, 2007). The framework shows that the characteristics of the innovation leads to intention to adopt, which then may lead to actual adoption, and then routine use. It then uses the results of the Delphi study to rank the characteristics of the innovations, students, faculty members, and administrators that relate to the dissemination process. Taylor and Todd (1995) analyzed the theory of planned behavior and found that characteristics of innovations such as relative advantage, complexity, and compatibility are antecedents to an individual?s intention to adopt, and intention to adopt is then an antecedent to actual adoption. Many researchers have pointed out that the adoption of an innovation is not in 68 Figure 3-1: Characteristics of Dissemination Success (CODS) Framework C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s C h a r a c t e r i s t i c s o f S t u d e n t s C h a r a c t e r i s t i c s o f F a c u l t y M e m b e r s C h a r a c t e r i s t i c s o f A d m i n i s t r a t o r s I n t e n t i o n t o A d o p t A c t u a l A d o p t i o n R o u t i n e U s e D e l p h i R a n k i n g F a c t o rs 1 P ro m o t e a n d R e w a rd In n o v a t i o n 2 R e c e p t i v i t y t o C h a n g e 3 M a n a g e m e n t S u p p o rt 4 D e g re e o f F a c i l i t a t i n g C o n d i t i o n s D e l p h i Ra n k i n g F a c t o rs 1 Re c e p t i v i t y t o Ch a n g e 2 Ca re a b o u t S t u d e n t L e a rn i n g O u t c o m e s 3 A t t i t u d e t o In n o v a t i o n 4 In n o v a t i v e n e s s 5 M o t i v a t i o n t o b e In n o v a t i v e 6 V a l u e T e a c h i n g i n A d d i t i o n t o Re s e a rc h 7 S e l f-E ffi c a c y 8 A w a re n e s s o f In n o v a t i o n 9 S u p p o rt i v e Co m m u n i t y 10 D e g re e o f S u p p o rt 11 P ro fe s s i o n a l S o c i a l S y s t e m 12 T e n u re S t a t u s 13 Cu l t u ra l A c c e p t a n c e o f In n o v a t i o n D e l p h i Ra n k i n g F a c t o rs 1 E n g a g e m e n t 2 M o t i v a t i o n t o L e a rn 3 In t e n t t o E n h a n c e L e a rn i n g 4 Re c e p t i v i t y t o Ch a n g e 5 L e a rn i n g S t y l e s 6 Cu l t u ra l D i ffe re n c e s D e l p h i Ra n k i n g F a c t o rs 1 Re l a t i v e A d v a n t a g e 2 E a s e o f Im p l e m e n t a t i o n 3 E a s e o f U s e 4 P ra c t i c a l i t y o f t h e Co n c e p t 5 Re l e v a n c e t o J o b 6 A d a p t a b i l i t y 7 Co m p a t i b i l i t y 8 Q u a l i t y o f In i t i a l In fo rm a t i o n 9 Co m m u n i c a b i l i t y 10 M e a s u ra b i l i t y 11 T h e o re t i c a l Co h e re n c e 69 fact the final step in the dissemination process (Froyd, 2011; Froyd, Borrego, Cutler, Henderson, & Prince, 2013; Rogers, 2003). Rather, post-adoption implementation activities directed toward embedding an adopted innovation in courses and curriculum ensure that the innovation becomes routine (Saga & Zmud, 1994). The users, environment, and culture in which the innovation is to be used have all been shown to affect intention to adopt and most are also likely to affect actual adoption and routine use (Hazen et al., 2012b; Lee et al., 2003; Rogers, 2003). Faculty members may intend to adopt an innovation, but due to tenure and promotion criteria, poor facilitating conditions, or fear of poor student evaluations due to student resistance may not actually adopt an educational innovation; an adopted innovation may not become routinely used for similar reasons. Figure 3-1 portrays the relationships among the characteristics of students, faculty members, and administrators and the intention to adopt, actual adoption, and routine uses as moderators. Ranking Characteristics of Educational Innovations, Students, Faculty Members, and Administrators We first discuss the ranking of the characteristics of educational innovations that are direct antecedents to the intention to adopt the innovations. Then we discuss the ranking of the characteristics of students, faculty members and administrators that were identified to have a moderating effect on the intention to adopt, actual adoption, and routine use. Characteristics of the innovation. Based on the 174 comments made by the experts, the researchers synthesized 11 characteristics and asked the expert panel to rank them in the second and third rounds. Relative advantage was identified as the most important characteristic, followed by ease of implementation and ease of use. Many of the characteristics identified by the experts (i.e., relative advantage, ease of use, compatibility, communicability, and 70 measurability) are similar to those discussed in the literature on the diffusion of innovations (Compeau et al., 2007; Moore & Benbasat, 1991; Rogers, 2003). One member of our expert panel noted: It?s the same characteristics specified in Roger?s diffusion of innovations theory: they have to be useful, easy to use, people must have a chance to try them at no risk, etc. It is not surprising that characteristics such as the practicality of the concept, relevance to the job, quality of initial information, and theoretical coherence appear important to the experts. They recognized that the ease of implementation and adaptability of the innovations are particularly important when integrating educational innovations in the classroom. These characteristics have also been identified as important in the education, information systems and marketing literature (Kaleelur Rahuman, Wikramanayake, & Hewagamage, 2011; Oppermann & Rasher, 1997; Ross, Rhodes, & Hastings, 2008). Characteristics of the students. The experts made 119 comments regarding the characteristics of students that relate to dissemination and the researchers classified these in terms of six characteristics. These were ranked subsequently by the experts in the following order: engagement, motivation to learn, intent to enhance learning, receptivity to change, learning styles, and cultural differences. The experts identified engagement as the most important student factor and this was mentioned in five unique comments during the first round of the Delphi study. One of our experts described it thus: Students should be active toward the direction of education. In order to improve the dissemination of educational innovations, I believe a student driven classroom environment assisted by [the] professor is important. 71 Students are the end users of any educational innovation that is taught in the classroom and students? receptivity to change and evaluations have an important influence over whether faculty members continue to use or even adopt educational innovations (Hannon, 2009). Students are often interested in student-focused, rather than teacher-focused, approaches to learning that involve more interactions with classmates and faculty members. These approaches have been shown to improve student engagement and motivate students to learn (McKenzie et al., 2005). In addition, students fall into one of sixteen different learning styles that influence how students learn (Cagiltay, 2008; Felder & Silverman, 1988; Hawk & Shah, 2007). The experts noted that finding educational innovations that are compatible with students? learning styles and cultural differences can enhance students? overall learning. Characteristics of faculty members. From the 165 comments made by the experts in this area, the researchers synthesized 13 characteristics that the experts subsequently ranked in the order shown in Figure 3-1. The highest ranked characteristic of faculty members was receptivity to change, which received 16 unique comments. One member of the expert panel commented: Some people are more amenable to change than others in any arena. Often those faculty most willing to change have both a deep-seated understanding of the content impacted by the educational innovation as well as a realization of why the educational innovation will be of benefit. Most of the faculty related characteristics break down into three common themes: the faculty member?s personal attributes (i.e., receptivity to change, innovativeness, self-efficacy, and tenure status), the faculty member?s professional network (i.e., a supportive community, professional social system, and cultural acceptance of innovation), and their attitudes regarding teaching (i.e., 72 concern for student learning outcomes and valuing teaching in addition to research). Other characteristics are awareness of innovation and degree of support. Characteristics of administrators. The researchers classified the 135 comments on this topic in terms of 4 characteristics that were then ranked by the experts. The most important characteristic here was identified as promoting and rewarding innovation in the classroom. This construct received 26 unique comments during the first round, with a panel member describing it in the following terms: Most educational innovations require substantial time, energy, and commitment on the part of the innovator. It also typically requires a collaborative effort of several education practitioners. These kinds of efforts and collaborations are not always rewarded in current workload or promotion and tenure considerations. One would not, or at least should not, expect faculty to engage in sustained practice of innovation if it is not valued in terms of workload or promotion and tenure considerations. This opinion echoes that expressed by the President's Council of Advisors on Science and Technology (2012), which identified the lack of departmental rewards and expectations for good teaching as one of the top barriers and challenges to producing more STEM graduates. The American Society for Engineering Education (2009) has suggested that hiring, tenure and promotion guidelines, policies and practices need to be reviewed to ensure that educational innovations and pedagogical preparations beyond teaching excellence need to be recognized, and rewarded. The Association to Advance Collegiate Schools of Business (AACSB) (2013) has also recognized the importance of pedagogical innovations, listing it as one of the three research outputs expected of faculty members in colleges of business across the nation. The other important characteristics identified as important for administrators were: receptivity to change, 73 management support, and facilitating conditions. The majority of faculty committees, department chairs and deans feel that management support and degree of facilitating conditions fall far short of supporting educational innovations (American Society for Engineering Education, 2012). This findings shows that in order for educational innovations to be used routinely, administrators will need to invest time in changing policies and practices to update the infrastructure of their institutions and make resources more readily available in the classrooms. Limitations and Future Research There are several limitations to this study. First, the sample consisted of a relatively homogenous group of participants, each of whom had been a recipient of a CCLI or TUES grant from the NSF in the areas of engineering and technology in the U.S. As such, the data collected may not reflect concerns that would have been expressed by faculty members in other fields. Second, while our sample was sufficiently large for a Delphi study, we did experience some subject attrition through the rounds of data collection. This attrition may have also limited the scope and perspective of the resulting data and subsequent conclusions. Third, the process of synthesizing the characteristics from the comments was inductive. While this was necessary to develop the CODS framework, this process is inevitably subjective and the data could have been synthesized differently. This limitation was addressed to some extent since the experts agreed to proceed using the characteristics identified by the researchers in rounds 2 and 3, suggesting that there was overall agreement among the researchers and the experts regarding the naming of the characteristics. Future research that operationalizes the proposed CODS framework, in part or in its entirety, will serve to allay the aforementioned limitations and enhance our understanding of the dissemination process. By operationalizing the CODS characteristics, future research may be 74 able to create a score card, index, or decision support tool that could be used to apply this framework in a prescriptive manner. Although the results of this research can be of use to scholars and practitioners, future research needs to examine specific types or categories of educational innovations (i.e., study instructional materials alone or instructional strategies alone) in order to provide more specific guidance. Using our results as a starting point, additional research may be able to develop a more refined version of the CODS framework for specific categories of educational innovations. In a similar vein, it could be interesting to conduct research that focuses specifically on students, faculty members, or administrators. There remain many unanswered questions, which we encourage other researchers to examine. Most notably, appropriate measures need to be developed for the assessment of the characteristics of the CODS model. Empirical studies examining the diffusion of innovation theory, the theory of reasoned action, the theory of planned behavior, the technology acceptance model, and the concerns-based acceptance model (CBAM) could all contribute to the development of suitable measures. Furthermore, the literature on organizational change may offer a way to operationalize measures for the characteristics of various stakeholders and facilitate the quantification of their contributions towards organizational readiness, while the levels of use of the innovation from CBAM could be applied to develop measures for the extent to which faculty members actually use new technologies in the classroom (Patton, 2002; Shenton, 2004). We intend to continue to refine the work presented here and encourage other researchers to build upon the CODS framework. Conclusions Many educators design, evaluate, and adopt various educational innovations that will help their students develop the knowledge and skills they will need to compete in today?s 75 economy. Although public and private funding has been available to develop these innovations, there is an apparent disconnect between publishing information regarding these innovations and their subsequent dissemination and adoption for use in classrooms. For example, teaching briefs have been published regularly in the Decision Sciences Journal of Innovative Education for the past ten years but few follow-up articles analyze whether they are actually being adopted in classrooms. This disconnect was addressed in this paper by seeking answers to the research question ?What are the most important characteristics that relate to the dissemination of educational innovations?? A Delphi methodology was used to elicit insights from experts regarding these characteristics. The ranking of the characteristics via a three-round Delphi survey was then formulated into a conceptual CODS framework. This paper?s major contribution is that the CODS framework is based on experts? comments and identifies the characteristics that encourage/ hinder the dissemination of educational innovations in engineering and technology disciplines. 76 Chapter 4: The Influence of Characteristics of Innovations and Readiness of Faculty Members Toward Dissemination of Educational Innovations: An Empirical Study Abstract Developers of educational innovations want to have their new educational innovations disseminated widely and used routinely. To improve dissemination results, researchers need to understand why some innovations are widely adopted and others are not. Previous research has asked Principal and Co-Principal Investigators of National Science Foundation grants to identify and rank the characteristics of innovations and faculty members that lead to successful dissemination. The purposed of this study are: 1) to test whether four characteristics of innovations are significant predictors of intention to adopt educational innovations, 2) to test whether ten readiness of faculty members toward educational innovation variables are significant predictors of dissemination success, and 3) to test whether 10 readiness of faculty members toward educational innovation variables moderate the dissemination process. In all, 335 faculty members at ABET certified computer science and electrical engineering program in the United States to participate in this study. The results of this study indicate that ease of use and care about student learning outcomes directly influence intention to adopt educational innovations. Attitude to innovation, efficacy toward change, and valence each moderate the relationship between ease of use and intention to adopt educational innovations. Awareness of an educational innovation is a direct antecedent to adoption and attitude to innovation moderates the relationship between intention to adopt and adoption. These results will help developers and 77 change agents (e.g., faculty members, department chairs, and college deans) further understand the dissemination process. Key Words: Dissemination, Educational Innovations, Readiness, Faculty Members, Survey Introduction Developers of educational innovations want to have their new curricula, instructional materials, expertise, and/or instructional strategies disseminated widely and used routinely. Dancy and Henderson (2010) assert that current approaches to the dissemination of educational innovations, such as the publication of journal articles, attending conferences, and participation in workshops, are successful in raising general awareness regarding innovative instructional practices but fail to robustly support faculty members seeking to adopt and implement the innovation. Henderson et al. (2010) found that disseminators of engineering innovations face two major issues: faculty members do not play a meaningful role in the dissemination process, and the environment and structure that faculty members work within often favor traditional lecture- based instruction. In addressing these issues, Hazen et al. (2012a) identified many of the characteristics of educational innovations, faculty adopters, and the environment that influence dissemination of innovations. Such a list helps disseminators be aware of these variables when developing their innovations. Further research led to the development of a framework which conceptualized the interactions between the characteristics of innovations and the readiness of an educational organizations? faculty members, administration, and students to disseminate educational innovations (Bourrie, Sankar, and Jones-Farmer (2014a)). This conceptualization was further 78 refined by using a Delphi study to rank the characteristics of the innovations, faculty members, administrators, and students that relate to the dissemination process (Bourrie, Cegielski, Jones- Farmer, and Sankar (2014b)). The Delphi study led to development of a Characteristics of Dissemination Success (CODS) framework which ranked the characteristics of the innovations that lead to intention to adopt, which then may lead to actual adoption, and then routine use. In addition, the CODS framework follows previous researchers? suggestions that the dissemination of innovations may be moderated by the characteristics of faculty members, administrators, and students (Hazen et al., 2012b; Rogers, 2003). This Delphi study also ranked the variables in order of importance based on the experts? opinions. Although the Delphi study results were based on the opinions of 45 Principal and Co- Principal Investigators of National Science Foundation funded projects, the ranking of the variables identified in this framework have not been empirically tested with a sample population of engineering educators. Therefore, the goal of this paper is to empirically determine the relative importance of the variables that were identified in the CODS framework. Given that there were 37 variables identified in the CODS framework, we would have required a sample size of approximately 14,600 engineering educators to test these variables using a survey methodology. Given the impracticality of obtaining responses from such a large sample of engineering educators, we constrained the list of variables to a shorter 22 variable list that included the most important and measurable variables. This led to the formation of our research model and development of a list of hypotheses (Section II). In order to test the hypotheses, we adapted previously validated measures for each of the variables and developed a 116-item questionnaire which was e-mailed to 4,352 computer science and electrical engineering educators, and received responses from 336 participants (Section III). We analyzed the 79 responses using hierarchical linear and logistic regression and tested the hypotheses (Section IV). This led to a series of findings and implications for researchers and practitioners (Section V). Finally, Section VI discusses the limitations, opportunities for future research, and overall contributions of this paper. Research Model and Hypotheses Parts of the CODS Framework The CODS framework is theoretically grounded in innovation diffusion theory (Rogers, 2003), the theory of reasoned action (Fishbein & Ajzen, 1975, 1981), and the theory of planned behavior (Ajzen, 1991). This section describes the different parts of the CODS framework that together create this framework. The parts are: characteristics of educational innovations, readiness of faculty members toward educational innovations, and dissemination. In this section, we list of variables that were included as important to each part of the CODS framework. Then, we provide theoretical justification for the relationships among the parts of the CODS framework. This discussion leads to the formation of the research model and development of hypotheses. Part 1: Characteristics of educational innovations. Roger?s (2003) diffusion of innovation theory initially identified five characteristics of innovations (relative advantage, compatibility, complexity, trialability, and observability) that influence the adoption of innovations. Further research by Moore and Benbasat (1991), Karahanna, Agarwal, and Angst (2006), and Compeau et al. (2007) has refined and expanded characteristics of innovations. Bourrie et al. (2014b) asked Principal and Co-Principal Investigators funded by National Science Foundation (NSF) grants the most important characteristics of innovations and they identified relative advantage, ease to implement, ease of use, and adaptability. We include these four 80 variables in our research model. Appendix 4-1 defines each of the four characteristics of educational innovations. Part 2: Readiness of faculty members toward educational innovations. Readiness of faculty members toward educational innovations reflects faculty members? beliefs, attitudes, and intentions regarding the extent to which educational innovations are needed and the organizational capacity to successfully disseminate educational innovations (Armenakis et al., 1993; Bourrie et al., 2014a). When these variables were presented to Principal and Co-Principal Investigators of NSF grants in a Delphi study, they identified receptivity to change, attitude to innovation, awareness of innovations, care about student learning outcomes, and motivation to innovate as important predictors of dissemination success (Bourrie et al., 2014b). Delphi participants described receptivity to change as the degree of receptivity from faculty members toward the pedagogical innovation (Bourrie et al., 2014b). In organizational change literature, receptivity to change is a very complex multi-order construct that is synonymous with the concept of readiness for change (Armenakis et al., 1993; Bartlem & Locke, 1981; Waugh, 2000; Waugh & Godfrey, 1993). Previous studies have measured faculty members? receptivity to change using multiple constructs including leadership support, openness to change, attitude to change, and characteristics of the change (Waugh, 2000; Waugh & Godfrey, 1993; Waugh & Punch, 1985, 1987). In education literature, Clarke, Ellett, Bateman, and Rugutt (1996) defined receptivity to change as one?s internal attitudes that precede the behaviors that one takes when adopting or resisting change. In this study, we identified openness to change, discrepancy, appropriateness of change, efficacy of faculty members toward change, support by principals to change, and valence as the variables to measure receptivity to change. Appendix 4-1 defines these variables. 81 Part 3: Dissemination. In engineering education, the term dissemination covers the process of creating awareness, followed by intention to adopt, actual adoption, and routine use of the innovation (Bourrie et al., 2014b; Fincher, 2000; Gravestock, 2002; Hutchinson & Huberman, 1994; King, 2003). As stated in part 2, awareness of innovations has been considered as a part of readiness of faculty members toward educational innovations. Therefore, we decided to study the stages of intention to adopt, actual adoption, and routine use in this research (Figure 4-1). Intention to adopt is defined as whether an individual, if given the opportunity, would adopt an innovation in the foreseeable future (Teo, Wei, & Benbasat, 2003). Adoption is defined as ?a decision to make full use of an innovation as the best course of action available? (Rogers, 2003, p. 21). Routine use of educational innovations is described as individuals that have mastered using an innovation and have no plans to further adapt or change that innovation (Gray, 2008; Hall & Hord, 2006; Saunders, 2012). Once an educational innovation has been adopted, the ultimate goal of dissemination is for that innovation to be routinely used (Bourrie et al., 2014b; Bourrie et al., 2014a). Figure 4-1. Dissemination I n t e n t i o n t o A d o p t A d o p t i o n R o u t i n e U s e D i s s e m i n a t i o n Research Model The three parts of the CODS framework (characteristics of innovations, readiness of faculty members toward educational innovations, and dissemination) interact in multiple ways. 82 The research model uses prior research to describe five relationships among the parts of the CODS framework. This leads to the derivation of nine hypotheses. Relationship 1: Between characteristics of innovations and intention to adopt. Research by Taylor and Todd (1995) and Hardgrave, Davis, and Riemenschneider (2003) have empirically indicated that characteristics of innovations are direct antecedents to intention to adopt an innovation (Figure 4-2). Hazen et al. (2012b) and Bourrie et al. (2014b) have postulated that faculty members? perception regarding the characteristics of educational innovations are positively related to intention to adopt. Therefore, we hypothesize: Hypothesis 1 posits a positive relationship between characteristics of educational innovations (i.e., (a) relative advantage, (b) ease to implement, (c) ease of use, and (d) adaptability) and intention to adopt educational innovations. Figure 4-2. Hypothesis 1. I n t e n t i o n t o A d o p t C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s a ) R e l a t i v e A d v a n t a g e b ) E a s e t o I m p l e m e n t c ) E a s e o f U s e d ) A d a p t a b i l i t y H 1 Relationship 2: Between intention to adopt and adoption. Empirical research related to the theories of reasoned action (Fishbein & Ajzen, 1975, 1981) and planned behavior (Ajzen, 1991) have indicated that intention to adopt is an antecedent to adoption (Figure 4-3). Moreover, theoretical and empirical research related to the different iterations of the Technology Acceptance Model (TAM) (Davis, 1989; Venkatesh & Bala, 2008; Venkatesh & Davis, 2000; Venkatesh et al., 2003; Venkatesh et al., 2012) support intention to adopt as a significant positive predictor of adoption. Therefore we hypothesize: 83 Hypothesis 2 posits a positive relationship between intention to adopt and adoption of educational innovations. Figure 4-3. Theoretical model once hypothesis 2 is added to the model. I n t e n t i o n t o A d o p t A d o p t i o n C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s a ) R e l a t i v e A d v a n t a g e b ) E a s e t o I m p l e m e n t c ) E a s e o f U s e d ) A d a p t a b i l i t y H 1 H 2 Relationship 3: Between adoption and routine use. Empirical research related to the theories of reasoned action (Fishbein & Ajzen, 1975, 1981) and planned behavior (Ajzen, 1991) have indicated that adoption is an antecedent to routine use (Figure 4-4). Further, Hazen et al. (2012b) and Bourrie et al. (2014b) have postulated that adoption is positively related to the routine use of educational innovations. Therefore, we hypothesize: Hypothesis 3 posits a positive relationship between adoption and routine use of educational innovation. Figure 4-4. Theoretical model once hypothesis 3 is added to the model. I n t e n t i o n t o A d o p t A d o p t i o n C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s a ) R e l a t i v e A d v a n t a g e b ) E a s e t o I m p l e m e n t c ) E a s e o f U s e d ) A d a p t a b i l i t y H 1 H 2 R o u t i n e U s e H 3 D i s s e m i n a t i o n Relationship 4: Between readiness of faculty members toward educational innovations and dissemination. In the organizational change literature, Armenakis et al. (1999) proposed a research model where receptivity to change is a direct antecedent to intention to adopt, adoption and institutionalization. In education literature, routine use is a commonly accepted indicator for the institutionalization of educational innovations state that readiness of 84 faculty members toward educational innovations variables are key predictors of the successful dissemination of educational innovations (Figure 4-5). Therefore, we hypothesize: Hypothesis 4 posits a significant relationship between the readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) and intention to adopt educational innovations. Hypothesis 5 posits a significant relationship between the readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) and adoption of educational innovations. Hypothesis 6 posits a significant relationship between the readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) and routine use of educational innovations. Figure 4-5. Theoretical model once hypotheses 4, 5, and 6 are added to the model. R e a d i n e s s o f F a c u l t y M e m b e r s ? T o w a r d E d u c a t i o n a l I n n o v a t i o n s a ) O p e n n e s s t o C h a n g e b ) D i s c r e p a n c y ( N e e d f o r c h a n g e ) c ) A p p r o p r i a t e n e s s o f C h a n g e d ) E f f i c a c y o f F a c u l t y M e m b e r s ? T o w a r d C h a n g e e ) S u p p o r t b y P r i n c i p a l s t o C h a n g e f ) V a l e n c e ( B e n e f i t s f r o m c h a n g e ) g ) A t t i t u d e t o I n n o v a t i o n h ) A w a r e n e s s o f I n n o v a t i o n i ) C a r e a b o u t S t u d e n t L e a r n i n g O u t c o m e j ) M o t i v a t i o n t o I n n o v a t e I n t e n t i o n t o A d o p t A d o p t i o n C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s a ) R e l a t i v e A d v a n t a g e b ) E a s e t o I m p l e m e n t c ) E a s e o f U s e d ) A d a p t a b i l i t y H 1 H 2 H 5H 4 R o u t i n e U s e H 3 H 6 D i s s e m i n a t i o n Relationship 5: Moderating relationships. In diffusion of innovation research, Rogers (2003) suggests that dissemination is moderated by the environment and culture in which the dissemination is taking place (Figure 4-6). Hazen et al. (2012b) proposed that characteristics of the adopter and characteristics of the dissemination environment moderate the dissemination 85 process. For instance, readiness of faculty members toward educational innovations variables, such as efficacy and support for principals, have been found to moderate the relationship between intention to adopt and adoption (Lee et al., 2003). Therefore, we hypothesize: Hypothesis 7 posits that readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) will moderate the relationship between characteristics of educational innovations and intention to adopt educational innovations. Hypothesis 8 posits that readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) will moderate the relationship between intention to adopt and adoption of educational innovations. Hypothesis 9 posits that readiness of faculty members toward educational innovations (i.e., (a) openness to change, (b) discrepancy, (c) appropriateness of change, etc.) will moderate the relationship between adoption and routine use of educational innovations. Figure 4-6. Research model R e a d i n e s s o f F a c u l t y M e m b e r s ? T o w a r d E d u c a t i o n a l I n n o v a t i o n s a ) O p e n n e s s t o C h a n g e b ) D i s c r e p a n c y ( N e e d f o r c h a n g e ) c ) A p p r o p r i a t e n e s s o f C h a n g e d ) E f f i c a c y o f F a c u l t y M e m b e r s ? T o w a r d C h a n g e e ) S u p p o r t b y P r i n c i p a l s t o C h a n g e f ) V a l e n c e ( B e n e f i t s f r o m c h a n g e ) g ) A t t i t u d e t o I n n o v a t i o n h ) A w a r e n e s s o f I n n o v a t i o n i ) C a r e a b o u t S t u d e n t L e a r n i n g O u t c o m e j ) M o t i v a t i o n t o I n n o v a t e I n t e n t i o n t o A d o p t A d o p t i o n C h a r a c t e r i s t i c s o f E d u c a t i o n a l I n n o v a t i o n s a ) R e l a t i v e A d v a n t a g e b ) E a s e t o I m p l e m e n t c ) E a s e o f U s e d ) A d a p t a b i l i t y H 1 H 2 H 5H 4H 7 H 8 R o u t i n e U s e H 3 H 6H 9 D i s s e m i n a t i o n 86 Methodology The research method chosen to test the above hypotheses is a survey. In the social sciences, Neuman (2006, p. 273) describes surveys as ?an appropriate research method to ask respondents about their beliefs, opinions, characteristics, and past or present behavior.? Therefore, we chose to develop a questionnaire to conduct our survey, which operationalized the variables included in the CODS framework. We searched for previously validated measures in the education, engineering, information systems, marketing, management, psychology, and sociology literature that could be adapted for use in this research. In this section, we discuss the questionnaire development in detail. Then, we discuss the pilot study that was used to establish face, and content validity. Next, we discuss the sample population used for this study. Finally, we discuss the statistical analysis procedures that were selected to analyze the data. Questionnaire Development The questionnaire created was developed over a five month period and consisted of 116 questions (Appendix 4-2). We divided the questionnaire into six sections: educational innovations, characteristics of educational innovations, readiness of faculty members? toward educational innovations, dissemination process, demographic and control variables, and assessment of common method bias. Each of these sections is described next. Section 1: Educational innovations. In the first section of this questionnaire, we asked faculty members five questions about an educational innovation that they were currently using, would like to use, or planned to use in the future in undergraduate electrical engineering or computer science courses (Appendix 4-2). Each participant was asked to describe and then classify the educational innovation as either a curriculum development, development of faculty expertise, instructional material, instructional strategy, or other type of educational innovation. 87 If a participant classified the educational innovations as other type of educational innovation, two members of the research team independently categorized these descriptions. One of the researchers was experienced in the research domain, with over 16 years of experience with educational innovations and dissemination and previously funded through several NSF grants. The other researcher was not an expert in the research domain, but was proficient in qualitative research methodologies. Section 2: Characteristics of educational innovations. In the second section of the questionnaire, 25 items measured the characteristics of educational innovations using a seven- point Likert scale where ?1 = strongly disagree? and ?7 = strongly agree? (Appendix 4-2). Relative advantage was assessed using Compeau et al.?s (2007) eight-item scale. The sixth item was reworded to eliminate reverse scoring (e.g., Using this educational innovation enhances my effectiveness in the classroom). Ease to implement was measured using the four-items ease of adoption scale by Di Benedetto, Calantone, and Zhang (2003). Ease of adoption was operationally defined as the extent to which adoption of the innovation was perceived to require effort, through items like the difficulty of technology transfer, time requirements, and problems in application in existing locations (e.g., I can foresee that some problems will happen when this educational innovation is used in our current classroom facilities). Ease of use was measured by the 6-item scale by Compeau et al. (2007) (e.g., It is easy for me to remember how to perform tasks associated with using this educational innovation). Adaptability was measured by the seven-item scale developed by Guilabert (2005) to measure perceived customization. Perceived customization was defined as the degree to which a person believed that a particular product/service or the features of that product/service were or could be customized to meet the 88 unique needs for individual customers at no additional cost (e.g., This educational innovation could meet individual students? needs very efficiently). Section 3: Readiness of faculty members? toward educational innovations. In the third section of the questionnaire, 52 items measured the readiness of faculty members toward educational innovations using a seven-point Likert scale where ?1 = strongly disagree? and ?7 = strongly agree? (Appendix 4-2). Openness to change was measured using the eight-item scale by Miller, Johnson, and Grau (1994). Four items were reworded to eliminate reverse coding (e.g., I think that the educational innovations will have a positive effect on how I perform my teaching role). Discrepancy, appropriateness of change, support by principals to change, and valence were measured using 18 items from Armanakis et al.?s (2007) Organizational Change Recipients? Beliefs (OCRBS) assessment tool (e.g., This educational innovation is the right ones for my students). Due to internal consistency issues, we dropped the third item of the valence scale (I will earn higher pay from my job after adopting this educational innovation), since faculty members often do not receive monetary gains for innovating their classrooms (President's Council of Advisors on Science and Technology, 2012). Change efficacy was measured using the six-item measure by Holt, Armenakis, Feild, and Harris (2007a). The second item was reworded to eliminate reverse scoring (e.g. I do not anticipate any problems adjusting to teaching when this educational innovation is adopted). Attitude to innovation was measured by the four- item scale developed by Agarwal and Prasad (1999). Item three was reworded to eliminate reverse scoring (e.g., I enjoy using this educational innovation). Awareness of innovations was measured by the six-item scale others? use developed by Compeau et al. (2007). Others? use referred to the degree to which potential adopters were aware of other people using the innovation (Compeau et al., 2007) (e.g., Several colleagues in my university use this educational 89 innovation). We eliminated the first item in this scale, which was consistent with Gwayi?s (2009) use of this scale in an educational context. Care about student learning outcomes was measured used the five-item scale concern about student learning developed by Hall, George, and Rutherford (1979) that was updated by Hall and Hord (2006) that is part of the CBAM. We reworded the second and fifth items after our pilot study to improve the readability of this construct (e.g., I am concerned about how this innovation affects my students). Motivation to innovate was assessed using the five-item scale by Alpkan, Bulut, Gunday, Ulusoy, and Kilic (2010) called performance-based reward systems. Section 4: Dissemination. The fourth section of the questionnaire included 5 questions to measure the dissemination process (intention to adopt, adoption, and routine use) at an individual level (Appendix 4-2). Intention to adopt was assessed using a three-item scale by Teo et al. (2003). We dropped the third item (?Please indicate the likelihood that you will use this educational innovation in the near future?) due to internal consistency issues and because we were measuring adoption and routine use in this study. Intention to adopt was rated using a seven-point Likert scale where ?1 = strongly disagree? and ?7 = strongly agree?. Adoption was assessed by using a single item based on the findings of Henderson and Dancy (2009). We asked the faculty member ?Please select the best statement that best describes your use of this educational innovation.? This item was measured using a four-point Likert scale where ?1 = I have heard the name, but do not know much else about it? and ?4 = I currently use all or part of it?. We collapsed the responses for this question and created a dichotomous variable ranging from ?0 = I have never used the innovation? and ?1 = I have used the innovation?. Routine use was assessed by using a single categorical measure based on the findings of Henderson and Dancy (2009). We asked faculty members ?How often do you use all or part of this educational 90 innovation?? This item was measured using a five-point Likert scale where ?1 = I never use it? and ?5 = ?I always use it?. We collapsed the responses for this question and created a dichotomous variable ranging from ?0 = I rarely to never use it? and ?1 = I always to sometime use it?. Section 5: Demographics and control variables. The fifth section of the questionnaire included 19 items to gather demographic information (Appendix 4-2). Five items were used as control variables (gender, nationality, department, tenure status, and percentage of teaching load) in this study since prior research has shown them to affect the adoption of innovations. Gender was dummy coded to indicate whether a faculty member was male ?0? or female ?1?. Prior research has shown that female faculty members are more willing to use interactive instructional strategies that engage students than their male counterparts (Froyd et al., 2013; Henderson et al., 2012). Nationality indicated whether a faculty member was White/Caucasian, Asian, or another nationality. Due to the small sample of faculty members that identified themselves as Black/African American, Hispanic/Latin, and Other, we combined these groups into the other nationality category. Department of the faculty member indicated whether the faculty member identified themselves as being a part of a computer science, electrical engineering, or other department. Tenure status indicated whether a faculty member identified themselves as tenured, tenure track but not yet tenured, and non-tenure track or their university does not have a tenure track system. Percentage of teaching load was measured using a dummy coded variable indicating whether faculty members teaching accounted for ?0 = 50% or more of their responsibilities? or ?1 = teaching accounts for less than half of their responsibilities?. The demographic variable with zero as the code (constant) refers to a male, White/Caucasian, tenured 91 faculty member that resides in a computer science department where teaching accounted for 50% or more of their job responsibilities. Section 6: Assessment of common method bias. The final section of the survey questionnaire included a marker variable (Appendix 4-2). Richardson, Simmering, and Sturman (2009) suggested using a marker variable as a proxy for method variance. The goal was to use a marker variable that captured the psychological influence that might lead one to answer survey questions in a way that created common method variance. To be effective, the marker variable had to be similar in format to the rest of the variables (e.g., on a 7 point Likert scale), and theoretically unrelated to the study variables. If the marker variable did not correlate with any of the other variables in the study, it indicated that faculty members were carefully reading the questions and not marked the same rating for all of the answers (e.g., yea-saying). We adapted the four-item scale by Miller and Chiodo (2008) called attitude toward the color blue and changed the referent color to green due to the high percentage of male faculty members in electrical engineering and computer science departments. Pilot Study A pilot study was conducted to test the appropriateness of the measures used in this questionnaire and the functionality of the web-based survey tool. Participants included eleven faculty members attending the 5th Annual National Engineering Mathematics Consortium in June 2013. Based on the feedback provided by the participants, several items were reworded to improve survey clarity. In addition, we decided not to use the intention to adopt measure developed by Cegielski, Jones-Farmer, Wu, and Hazen (2012) due to internal consistency issues because this measure evaluated intention to adopt at both an individual and organizational level. 92 IRB Approval On August 7, 2013, we presented our updated survey and information letter to the Auburn University Human Subjects Institutional Review Board for review. Appendix 4-3 contains the approval letter that was received August 10, 2013. Sample The sample for this study came from faculty members at ABET certified computer science and electrical engineering programs in the United States. To create the list of ABET certified computer science and electrical engineering programs, we used the ABET website to identify all of the programs across the country and then used a systematic sampling approach with a sampling interval of two (ABET, 2013). Once a certified program was identified, we searched for faculty members? contact information using the programs? website. Through this process, we accumulated a list of 4,352 faculty members from 130 universities. An invitation email to complete the survey was sent to the 4,352 faculty members. Of these, 105 emails were undeliverable, 34 emails returned out of office responses during data collection, and 16 email addresses were removed because the faculty members retired. Faculty members that chose to participate in this study were provided with a web link to an online survey. The data were collected from October 2, 2013 to December 17, 2013. Of the 4,352 faculty members that were solicited, 704 started the survey and 336 (8.00%) completed the 116 question survey. One case was eliminated because that individual was a staff member that did not teach. Therefore, the final response rate was 7.98% with 335 participants. The respondents averaged 15.73 years teaching experience (SD = 11.87), and were at their current school for 12.71 years (SD = 10.56). Table 4-1 provides demographic information for the faculty members that participated in this study. 93 Table 4-1. Sample demographics Respondents Demographics Frequency Percent Gender Male 282 84.18% Female 53 15.82% Total 335 100.00% Nationality White Caucasian 237 70.75% Asian 61 18.21% Other Nationalities 37 11.04% Total 335 100.00% Department Computer Science 136 40.60% Electrical Engineering 180 53.73% Other 19 5.67% Total 335 100.00% Tenure status Tenured 193 57.61% Tenure-track but not yet tenured 69 20.60% Non-tenure track or no tenure system 73 21.79% Total 335 100.00% Percentage of Teaching Load Teaching accounts for 50% or more of my responsibilities 220 65.67% Teaching accounts for less than half my responsibilities 115 34.33% Total 335 100.00% The participants classified twenty-two of the educational innovations as other type of educational innovation. The two researchers analyzed the descriptions provided. Kendall?s tau was calculated to determine the degree of concordance between the two coders. Based on our analysis, we found the following Kendall?s tau for the four types of educational innovations: curriculum development (Kendall?s tau = .796; p < .001), faculty expertise (Kendall?s tau = 1.00), instructional materials (Kendall?s tau = .821; p < .001), and instructional strategies 94 (Kendall?s tau = .909; p < .001). These results suggest sufficient agreement amongst the two coders. Overall, the faculty members described 55 curriculum development innovations, 10 development of faculty expertise innovations, 89 instructional material innovations, and 199 instructional strategy innovations, which were not mutually exclusive. This result is consistent with the recent emphasis on incorporating instructional strategies into engineering education (Froyd et al., 2013; Henderson & Dancy, 2007; Henderson et al., 2012). Statistical Analysis We conducted the statistical analysis using both hierarchical linear and hierarchical logistic regression. To guard against possible multicollinearity, we decided to use mean-centered scale averages for all the independent variables and intention to adopt (Cohen, Cohen, West, & Aiken, 2003). We performed the analysis in three stages with different dependent variables in the regression analysis. All of the analysis was conducted using SPSS 22. Stage one of the analysis, we used hierarchical linear regression with intention to adopt as the dependent variable. For stages two and three of the analysis, hierarchical logistic regression was conducted with adoption and routine use as the dependent variables, respectively. Variables were introduced to the model in four successive steps. In the first step of the analysis (Model 1), control variables were entered in the model. In the second step of the analysis (Model 2), the four characteristics of educational innovations were added to the model as a predictors of intention to adopt in Stage 1, intention to adopt was added to the model as a predictor of adoption in Stage 2, and adoption was added to the model as a predictor of routine use in Stage 3. In the third step of the analysis (Model 3), only the significant items identified in the second step were retained and readiness of faculty members? toward educational innovation 95 factors was added to the model. In the fourth step of the analysis (Model 4), the significant main effects identified during the second and third steps were retained and added to a series of interaction terms (consisting of the cross products of the significant items identified in Model 2 and significant readiness of faculty members toward educational innovations identified in Model 3) as predictors of the dependent variables. Interactions were only considered noteworthy if they had significant beta weights, at least one significant slope, and statistically improved the R2 of the model. Results This section describes the statistical analysis of the CODS framework. For stage one of the analysis, the dependent variable represented faculty members intention to adopt educational innovations. For stage two, the dependent variable represented whether or not the faculty member adopted the educational innovation they discussed at the beginning of the survey. For stage three, the dependent variable represented whether faculty members routinely use the educational innovation. Table 4-2 presents the Cronbach?s alphas, means, standard deviations, and intercorrelations among the variables included in this study. Stage 1: Intention to adopt Table 4-3 presents the results of the hierarchical linear regression analysis with intention to adopt educational innovations as the dependent variable. For step one of the analysis (Model 1), we found the control variables were not significantly different from our constant. For the second step of the analysis (Model 2), we added the characteristics of educational innovations to the model. Again, we found that none of the control variables were significantly different than 96 97 Table 4-3. Regression analysis of intention to adopt Variables Model 1 Model 2 Model 3 Model 4 b se b se b se b se Constant 6.12*** (0.12) 6.13*** (0.10) 6.13*** (0.09) 6.20*** (0.09) Control Variables Female 2.04 (0.154) 0.22 (0.14) 0.11 (0.12) 0.14 (0.12) Asian 0.02 (0.146) -0.17 (0.13) -0.240* (0.12) -0.242* (0.11) Other Nationalities -0.23 (0.179) -0.24 (0.16) 0.01 (0.14) 0.04 (0.14) Electrical Engineering -0.04 (0.117) 0.01 (0.10) 0.02 (0.09) -0.01 (0.09) Department Other 0.03 (0.250) -0.02 (0.22) 0.07 (0.19) 0.01 (0.19) Not Yet Tenure -0.18 (0.143) -0.11 (0.13) -0.14 (0.12) -0.16 (0.11) Non-Tenure Track -0.09 (0.141) -0.14 (0.13) -0.14 (0.11) -0.18 (0.11) Less than Half time Teaching 0.01 (0.118) 0.02 (0.11) 0.00 (0.09) 0.00 (0.09) Theorized Effects Characteristics of Innovation Relative Advantage (RA) 0.23*** (0.063) 0.01 (0.06) 0.06 (0.06) Ease to Implement 0.05 (0.045) Ease of Use (EU) 0.23*** (0.058) 0.07 (0.05) 0.06 (0.05) Adaptability 0.08 (0.046) Readiness of Faculty Members' Toward Educational Innovations Openness to Change -0.03 (0.09) Discrepancy 0.03 (0.08) Appropriateness of Change 0.11 (0.09) Efficacy of Faculty Members' Toward Change 0.21** (0.07) 0.18* (0.07) Support from Principals to Change 0.03 (0.05) Valence 0.20** (0.07) 0.15* (0.07) Attitude to Innovation 0.22** (0.07) 0.31*** (0.07) Awareness of Innovation 0.05 (0.03) Care About Student Learning Outcomes 0.11** (0.04) 0.11** (0.04) Motivation to be Innovative -0.03 (0.05) Moderating Effects RA x Efficacy of Faculty Members' Toward Change 0.12 (0.08) RA x Valence -0.01 (0.08) RA x Attitude to Innovation -0.14 (0.09) RA x Care About Student Learning Outcomes -0.02 (0.04) EU x Efficacy of Faculty Members' Toward Change -0.14** (0.05) EU x Valence -0.16** (0.06) EU x Attitude to Innovation 0.23*** (0.06) EU x Care about Student Learning Outcomes -0.05 (0.03) Adjusted R2 0.90% 20.20% 40.80% 45.30% Notes: N = 335. * = significant at 0.05 level. ** = significant at .01 level. *** = significant at .001 level. Standard errors are reported in parentheses. 98 the constant. Hypothesis 1a, which stated that relative advantage will positively influence intention to adopt educational innovations, was supported (b = .23, df = 315, p < .001). Hypothesis 1b, which posited a significant positive influence between ease to implement and intention to adopt educational innovations, was not supported (b = .05, df = 315, p = .28). Hypothesis 1c, which posited a positive relationship between ease of use and intention to adopt educational innovations, was supported (b = .23, df = 315, p < .001). Hypothesis 1d, which posited a significant association between adaptability and intention to adopt educational innovations, was not supported (b = .08, df = 315, p = .09). For the third step of the analysis (Model 3), relative advantage and ease of use were kept in the model and readiness of faculty members toward educational innovation variables were added to the model. Of the control variables, Asian faculty members (b = -.24, df ? 313, p = .048) was found significant. In addition, the results found that four out of the ten readiness of faculty members toward educational innovation variables were empirically supported. For hypothesis 4d, the results suggested a significant and positive relationship between efficacy of faculty members toward change and intention to adopt educational innovations (b = .21, df = 312, p = .004); thus, hypothesis 4d was supported. For hypothesis 4f, the results suggested a significant and positive relationship between valence and intention to adopt educational innovations (b = .20, df = 312, p = .007); therefore, hypothesis 4f was supported. For hypothesis 4g, the results suggested a significant and positive relationship between attitude to innovation and intention to adopt educational innovations (b = .22, df = 312, p = .002); therefore, hypothesis 4g was supported. For hypothesis 4i, the results suggested a significant and positive relationship between care about student learning and intention to adopt educational innovations (b = .11, df = 312, p = .002); therefore, hypothesis 2i was supported. 99 For the fourth step of the analysis (Model 4), the significant direct effects found during the analysis in steps two and three were kept in the model and we added the eight two-way interactions. Of the control variables, Asian faculty members (b = -.24, df ? 312, p = .03) were found significant. This suggested that Asian faculty members are less likely to intend to adopt educational innovations than White/Caucasian faculty members. The results found that care about student learning was a significant antecedent and positively related to intention to adopt (b = .11, df = 312, p = .003), but did not moderate the relationship between relative advantage and intention to adopt or ease of use and intention to adopt. The two-way interaction between ease of use and efficacy of faculty members toward change significantly and negatively related to intention to adopt (b = -.14, df = 312, p = .01), thus hypothesis 7d was supported, which suggested that efficacy moderated the relationship between ease of use and intention to adopt. Next, the two-way interaction between ease of use and valence significantly and negatively related to intention to adopt (b = -.16, df = 312, p = .004), thus hypothesis 7f was supported, which suggested that valence moderated the relationship between ease of use and intention to adopt. Finally, the two-way interaction between ease of use and attitude to innovation significantly and positively relates to intention to adopt (b = .23, df = 312, p < .001), thus hypothesis 7g was supported, which suggested that attitude to innovation moderated the relationship between ease of use and intention to adopt. To develop a more nuanced understanding about the nature of the interaction effects regarding the relationship between ease of use and intention to adopt an educational innovation, we followed the procedure proposed by Aiken and West (1991) by plotting the interaction diagrams shown in Table 4-4. To present the contrasts, low and high are represented by the 25th and 75th percentile. For Table 4-4A, the gap in intention to adopt educational innovations with 100 Table 4-4. Moderating effects on the relationship between ease of use and intention to adopt Note: y-axis is mean intention to adopt score based on seven-point Likert scale. * = significant at 0.05 level. ** = significant at .01 level. *** = significant at .001 level. 4.6 4.8 5 5.2 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 Low High Ease of Use Inte ntion to Ad op t A. Efficacy of faculty members toward change Low * Efficacy of faculty members toward change High 4.64.8 55.2 5.45.6 5.86 6.26.4 6.66.8 Low High Ease of Use Inte ntion to Ad op t B. Valence Low** Valence High 4.64.8 55.2 5.45.6 5.86 6.26.4 6.66.8 Low High Ease of Use Inte ntion to Ad op t C. Attitude to Innovation Low Attitude to Innovation High*** 101 low efficacy of faculty member toward change and high efficacy of faculty members toward change narrowed as ease of use improved. By following the procedure proposed by Preacher, Curran, and Bauer (2006), the simple slope for low efficacy of faculty members toward change was significantly different from zero (t(312) = 2.41, p = 0.02). This suggested that faculty members with a low level of confidence regarding change would have a greater intention to adopt educational innovations that are easier to use. In Table 4-4B, the gap in intention to adopt educational innovations with low and high valence narrowed as ease of use improved. Only the simple slope for low valence was significantly different from zero (t(312) = 2.64, p = .009). This result suggested that faculty members that did not feel they were getting anything personally out of using an educational innovation would have greater intention to adopt that innovation if it was easy to use. In Table 4-4C, the gap in intention to adopt educational innovations with low and high attitude to innovations widened as ease of use of the innovation improved. Only the simple slope for faculty members with high attitude to innovation was significantly different from zero (t(312) = 4.11, p < .001). This finding suggested that faculty members with a positive attitude toward an educational innovation would be more likely to adopt the innovation if it were easy to use. Stage 2: Adoption Table 4-5 presents the results of the hierarchical logistic regression analysis with adoption as the dependent variable. We started the analysis by entering the control variables in the model (Model 1). This model did not pass the omnibus tests of model coefficients (?2 = 13.15, df = 8, p = .11) used to establish model fit, therefore we will not discuss those results. For the second step of the analysis (Model 2), we added the mean-centered intention to adopt educational innovations to the model. Of the control variables, department of electrical 102 Table 4-5. Regression analysis of adoption Model 1 Model 2 e? e? Variables b SE b (odds ratio) b SE b (odds ratio) Constant 0.09 (0.97) 1.10 0.11 (1.00) 1.12 Control Variables Female -0.40 (0.43) 0.67 -0.27 (0.45) 0.76 Asian -0.49 (0.42) 0.61 0.49 (0.45) 0.61 Other Nationalities 0.44 (0.42) 1.55 0.31 (0.45) 1.36 Electrical Engineering 0.57 (0.31) 1.77 0.66* (0.33) 1.93 Dept. Other 0.34 (0.63) 1.41 0.50 (0.65) 1.64 Not Yet Tenure 0.77* (0.35) 2.17 0.73* (0.36) 2.07 Non-Tenure Track 0.46 (0.36) 1.59 0.42 (0.38) 1.52 Less than Half time Teaching 0.27 (0.29) 1.31 0.31 (0.31) 1.36 Theorized Effects Intention to Adopt .75*** (0.15) 2.11 Readiness of Faculty Members Toward Educational Innovations Openness to Change Discrepancy Appropriateness of Change Efficacy of Faculty Members Toward Change Support from Principals to Change Valence Attitude to Innovation Awareness of Innovation Care About Student Learning Outcomes Motivation to be Innovative Moderating Effects Intention to Adopt x Attitude to Innovation Intention to Adopt x Awareness of Innovation Log likelihood 319.31 288.68 Chi-squared (?2) 13.15 43.80*** Hit ratio (%) 80.30% 81.50% Cox & Snell R2 3.80% 12.30% Nagelkerke R2 6.10% 19.50% Notes: N = 335. * = significant at 0.05 level. ** = significant at .01 level. *** = significant at .001 level. Standard errors are reported in parentheses. 103 Table 4-5. Regression analysis of adoption continued Model 3 Model 4 e? e? Variables b SE b (odds ratio) b SE b (odds ratio) Constant 0.07 (1.15) 1.07 -0.28 (1.11) 0.75 Control Variables Female -0.15 (0.49) 0.86 -0.05 (.48) 0.95 Asian 0.02 (0.50) 1.02 -0.01 (.48) 1 Other Nationalities -0.05 (0.51) 0.95 0.17 (.49) 1.18 Electrical Engineering 0.69 (0.37) 2.00 0.65 (.36) 1.91 Dept. Other 0.36 (0.75) 1.43 0.31 (.75) 1.37 Not Yet Tenure 0.85* (0.42) 2.34 0.96* (.40) 2.62 Non-Tenure Track 0.62 (0.43) 1.87 0.63 (.42) 1.87 Less than Half time Teaching 0.35 (0.34) 1.42 0.35 (.34) 1.41 Theorized Effects Intention to Adopt 0.23 (0.19) 1.26 0.38 (.22) 1.46 Readiness of Faculty Members Toward Educational Innovations Openness to Change 0.21 (0.27) 1.24 Discrepancy -0.24 (0.22) 0.78 Appropriateness of Change 0.10 (0.32) 1.11 Efficacy of Faculty Members Toward Change 0.34 (0.24) 1.40 Support from Principals to Change -0.18 (0.19) 0.84 Valence -0.18 (0.27) 0.83 Attitude to Innovation 1.06*** (0.27) 2.90 1.24*** (.22) 3.45 Awareness of Innovation 0.37** (0.14) 1.45 0.36** (.14) 1.44 Care About Student Learning Outcomes 0.04 (0.16) 1.04 Motivation to be Innovative 0.08 (0.15) 1.08 Moderating Effects Intention to Adopt x Attitude to Innovation .26* (.11) 1.3 Intention to Adopt x Awareness of Innovation -0.05 (.12) 0.95 Log likelihood 247.07 245.17 Chi-squared (?2) 85.41*** 87.31*** Hit ratio (%) 83.30% 85.10% Cox & Snell R2 22.50% 22.90% Nagelkerke R2 35.80% 36.50% Notes: N = 335. * = significant at 0.05 level. ** = significant at .01 level. *** = significant at .001 level. Standard errors are reported in parentheses. 104 engineering (b = .66, Wald ?2 = 3.93, p = .047) and tenure track faculty that were not yet tenured (b = .73, Wald ?2 = 4.00, p = .046) were significant. The odds ratio suggested that electrical addition, the odds ratio suggested that faculty members that were tenure track but not yet tenured were 2.07 times more likely to adopt educational innovations than tenured faculty members when holding all other variables constant. Hypothesis 2 posited a significant association between intention to adopt educational innovations and adoption of educational innovations. The results suggested a significant and positive relationship between intention to adopt and adoption (b = .75, Wald ?2 = 25.49, p < .001); thus, Hypothesis 2 was supported. The odds ratio suggested that faculty members that intended to adopt educational innovations were 2.11 times more likely to adopt educational innovations when all variables are held constant. This model overall was able to classify 81.5% of those that adopted educational innovation and between 12.3% and 19.5% of the variance. For the third step of the analysis (Model 3), we added the ten mean-centered readiness of faculty members toward educational innovation variables to the model. Of the control variables, only tenure track faculty that were not yet tenured (b = .85, Wald ?2 = 4.19, p = .04) was found significant. The odds ratio suggested, when holding all other variables constant, faculty members that were tenure track but not yet tenured were 2.34 times more likely to adopt educational innovations than tenured faculty members. Once the readiness of faculty members toward educational innovation variables were entered into the model Hypothesis 2 (b = .23, Wald ?2 = 1.44, p = .23) was no longer supported. The results suggested the association between attitude toward the innovations to adoption of educational innovations was a significant and positive relationship (b = 1.06, Wald ?2 = 15.56, p < .001); thus, Hypothesis 5g was supported. The odds ratio suggested, when holding all other variables constant, faulty members with a positive 105 attitude toward an innovation would be 2.90 times more likely to adopt educational innovations. The results also suggested that the association between awareness of innovations to adoption of educational innovations was a significant and positive relationship (b=.37, Wald ?2 = 7.49, p < .01); thus, Hypothesis 5h was supported. The odds ratio suggested, when holding all other variables constant, faculty members that had an awareness of the educational innovation were 1.45 times more likely to adopt the educational innovation. This model was able to classify 83.3% of those that adopted educational innovations and between 22.5% and 35.8% of the variance. For the fourth step of the analysis (Model 4), the significant direct effects found during our analysis in steps two and three were kept in the model and we added the two interactions. Of the control variables, tenure track faculty members that are not yet tenured (b = .96, Wald ?2 = 5.80, p = .016) were found significant. This suggested that tenure track faculty members that were not yet tenured were 2.62 times more likely to adopt educational innovations that tenured faculty members. In addition, the results found that awareness of educational innovations was a significant antecedent and positively related to adoption (b = .36, Wald ?2 = 6.60, p = .01), but did not moderate the relationship between intention to adopt and adoption. The two-way interaction between intention to adopt and attitude to innovations positively related to adoption (b = .26, Wald ?2 = 5.70, p = .017), therefore hypothesis 8g was supported, which suggested that attitude to innovation moderated the relationship between intention to adopt and adoption. To develop a more nuanced understanding of the interaction effect regarding intention to adopt attitude to innovation, we plotted the interaction diagram shown in Figure 4-8. This allows the interaction plot to look similar to the linear regression interaction plots in Stage 1 of the analysis. To present the contrasts, low and high were represented by the 25th and 75th percentile. 106 The gap in log odds of adoption with low attitude to innovation and high attitude to innovation narrows as intention to adopt educational innovations increased. Figure 4-8. Interaction plot for the moderating effect of attitude to innovation on intention to adopt Stage 3: Routine Use Table 4-7 presents the results of the hierarchical logistic regression analysis with routine use as the dependent variable. The first step of the analysis (Model 1), showed the results when control variables were entered in the analysis. Of the control variables, Asian (b = -2.44, Wald ?2 = 5.62, p = .018) faculty members and teaching accounts for less than half of their responsibilities (b = .85, Wald ?2 = 6.26, p = .012) were significant. The odds ratio suggested when holding all other variables constant, Asian faculty members are .09 times less likely to routinely use educational innovations than White/Caucasian faculty members. Additionally, the odds ratio suggested when holding all other variables constant that faculty members that had teaching accounting for less than half of their responsibilities were 2.33 times more likely to -1.5 -1 -0.5 0 0.5 1 1.5 Low High Intention to Adopt Log o dd s o f A do ption Attitude to Innovation Low Attitude to Innovation High 107 Table 4-6. Regression analysis of routine use Model 1 Model 2 Model 3 Variables b SE b e? (odds ratio) b SE b e? (odds ratio) b SE b e? (odds ratio) Constant 3.11* (1.54) 22.50 1.81** (1.97) 514.57 1.46* (2.22) 230.65 Control Variables Female -1.08 (.64) 0.34 -0.87 (0.84) 0.42 -0.73 (0.99) 0.48 Asian -2.44* (1.03) 0.09 -2.73* (1.15) 0.07 -2.36 (1.30) 0.10 Other Nationalities .42 (.46) 1.52 0.15 (0.69) 1.16 0.200 (0.71) 1.22 Electrical Engineering .64 (.36) 1.89 0.440 (0.53) 1.55 0.55 (0.59) 1.73 Dept. Other .12 (.83) 1.13 0.02 (1.17) 1.02 -0.06 (1.28) 0.94 Not Yet Tenure .37 (.42) 1.45 -0.33 (0.61) 0.72 -0.22 (0.66) 0.80 Non-Tenure Track .49 (.41) 1.63 0.22 (0.62) 1.25 0.68 (0.73) 1.97 Less than Half time Teaching .85* (.34) 2.33 1.06* (0.50) 2.88 1.29* (0.55) 3.63 Theorized Effects Adoption 4.44*** (0.53) 84.55 3.98*** (0.60) 53.72 Readiness of Faculty Members' Toward Educational Innovations Openness to Change .20 (.46) 1.22 Discrepancy -.09 (.37) 0.92 Appropriateness of Change -.57 (.52) 0.57 Efficacy of Faculty Members' Toward Change .40 (.35) 1.50 Support from Principals to Change .12 (.30) 1.13 Valence .28 (.43) 1.32 Attitude to Innovation .56 (.39) 1.76 Awareness of Innovation .22 (.21) 1.25 Care About Student Learning Outcomes -.34 (.25) 0.71 Motivation to be Innovative -.02 (.28) 0.98 Log likelihood 242.78 125.14 111.82 Chi-squared (?2) 28.91*** 146.55*** 159.87*** Hit ratio (%) 86.00% 92.80% 93.70% Cox & Snell R2 8.30% 35.40% 38.00% Nagelkerke R2 14.90% 63.80% 68.30% Notes: N = 335. * = significant at 0.05 level. ** = significant at .01 level. *** = significant at .001 level. Standard errors are reported in parentheses. 108 routinely use educational innovations than faculty members whose teaching accounted for 50% or more of their responsibilities. This model overall was able to classify 86.0% of faculty members that routinely used educational innovations and between 8.3% and 14.9% of the variance. For the second step of the analysis (Model 2), we added adoption to the model. Of the control variables, Asians (b = -2.73, Wald ?2 = 5.66, p = .02) and teaching accounts for less than half of their responsibilities (b = 1.06, Wald ?2 = 4.47, p = .03) were significant. The odds ratio suggested, when holding all other variables constant, that Asian faculty members were .07 times less likely to routinely use educational innovations than White/Caucasian faculty members. Additionally, the odds ratio suggested, when holding all other variables constant, that faculty members that had teaching accounting for less than half of their responsibilities were 2.88 times more likely to routinely use educational innovations than faculty members where teaching accounted for 50% or more of their responsibilities. Hypothesis 3 posited a significant association between adoption and routine use of educational innovations. The results suggested a significant positive relationship between adoption and routine use (b = 4.44, Wald ?2 = 70.42, p < .001); thus, Hypothesis 3 was supported. The odds ratio suggested, when holding all other variables constant, faculty members that adopt educational innovations were 84.55 times more likely to routinely use educational innovations if they had adopted the innovation. This model overall was able to classify 92.8% of faculty members that routinely used educational innovations and between 35.4% and 63.8% of the variance. For the third step of the analysis (Model 3), we added readiness of faculty members toward educational innovations variables to the model. Of the control variables, teaching accounting for less than half of the faculty members responsibilities (b = 1.29, Wald ?2 = 5.56, p 109 = .02) was significant. The odds ratio suggested, when holding all other variables constant, faculty members that had teaching accounting for less than half of their responsibilities were 3.63 times more likely to routinely use educational innovations than faculty members whose teaching accounts for 50% or more of their responsibilities. Hypothesis 3 posited a significant association between adoption and routine use of educational innovations. The result suggested a significant and positive relationship between adoption and routine use (b = 3.98, Wald ?2 = 44.28, p < .001); thus, Hypothesis 3 was supported. The odds ratio suggested, when holding all other variables constant, faculty members that adopt educational innovations are 53.77 times more likely to routinely use educational innovations than faculty that did not adopt the innovation. Unfortunately, none of the readiness of faculty members toward educational innovations variables were significant, therefore we stopped the analysis. This model overall was able to classify 93.7% of faculty members that routinely used educational innovations and between 38.0% and 68.3% of the variance. Findings and Implications Analysis of the results leads to the following findings: 5. Ease of use of an educational innovation was a significant predictor of intention to adopt that innovation. 6. Attitude to educational innovation was the most important readiness of faculty member variable. 7. Care about student learning outcomes was a significant predictor of intention to adopt educational innovations. 8. Efficacy of faculty members toward change and valence moderate the relationship between ease of use and intention to adopt educational innovations. 110 9. Awareness of others using an educational innovation was a significant predictor of adopting an educational innovation. We discuss each of these findings next and their implications. Ease of use of an educational innovation was a significant predictor of intention to adopt that innovation. Finding. In this study, ease of use was the only characteristic of educational innovations that directly influences intention to adopt after readiness variables were added to the regression analysis (Table 4-3, Model 4; Hypothesis ). Prior research had identified ease of use as an important predictor of successful dissemination (Borrego et al., 2010; Bourrie et al., 2014b; Cheville & Bunting, 2011; Elahinia & Ciocanel, 2008; Hazen et al., 2012a). Implication. This result implies several steps that need to be taken by developers to improve the dissemination of educational innovations. First, developers of educational innovations need to focus on reducing the level of complexity in innovations (Borrego et al., 2010; Rogers, 2003) which will directly improve ease of use. As developers of innovations place a greater emphasis on creating innovations that are easy to use, this will also improve faculty members? perceived relative advantage (Compeau et al., 2007), which should therefore improve the widespread adoption of the educational innovations. Second, previous research by Cheville and Bunting (2011) found ease of use can depend on faculty members? level of expertise and knowledge of the educational innovation being investigated. Therefore, developers may need to show their innovations are easy-to-use through hands-on workshops so faculty members can improve their own expertise and knowledge about the innovation. 111 Attitude to educational innovation was the most important readiness of faculty member variable Finding. The attitude toward innovations by faculty members influenced the most relationships in this study (i.e., moderated the relationship between ease of use and intention to adopt innovation (Table 4-4C) and moderates the relationship between intention to adopt and adoption (Figure 4-8). Borrego et al. (2010) found faculty members? attitudes toward innovations are an important part of peers? willingness to adopt new pedagogies. Moreover, Qualters et al. (2008) found the amount of effort needed by faculty members to adjust classes was part of the attitude that faculty members develop toward new educational innovations. Prior research has suggested poor attitudes to innovations are often the result of a lack of time, training, motivation, and technological na?vet? (Bernold, 2008; Christie & Jurado, 2009; Kantardjieff, 2010; Veldman et al., 2008). Implication. This finding implies developers, department chairs, and deans need to find ways to overcome negative attitudes, if they want faculty members to successfully disseminate educational innovations. If department chairs and college deans changed promotion and tenure guidelines to include innovation in the classroom as an important criteria, it may definitely have a profound impact on faculty members? attitudes toward innovations (Tabata & Johnsrud, 2008). Care about student learning outcomes was a significant predictor of intention to adopt educational innovations Finding. This study found that faculty members who care about student learning outcomes are more likely to intend to adopt educational innovations (Table 4-3, Model 4). Bourrie et al. (2014b) found faculty members that care about student learning outcomes focused on learning rather than focused on grades. These faculty members are passionate about teaching 112 and are often focused on student success (Bourrie et al., 2014b). Furthermore, Borrego et al. (2010) found teaching is often marginalized in promotion and tenure guidelines. Implication. Promotion and tenure guidelines need to change in order to recognize the substantial time and energy needed to develop, adapt, evaluate, and disseminate innovations by faculty members into their classrooms (Veldman et al., 2008; Walczyk et al., 2007). Changing promotion and tenure guidelines is in line with recommendations by the American Society for Engineering Education (2009, 2012) and the President's Council of Advisors on Science and Technology (2012). Efficacy of faculty members toward change and valence moderate the relationship between ease of use and intention to adopt educational innovations Findings. Table 4-4A shows faculty members with low efficacy toward change (i.e., implementing educational innovations) increase their intention to adopt an educational innovation if it is easy to use. Faculty members have been shown to learn by doing (Riley et al., 2007; Riley, Davis, Jackson, & Maciukenas, 2009) and gain a feeling of efficacy as they assess the effect of the innovation on students (Cramer et al., 2010). In addition, Table 4-4B shows faculty members with low valence toward an educational innovations increase their intention to adopt an educational innovation if it is easy to use. In the organizational change literature, valence for faculty members would be the perceived benefits of implementing the educational innovations (Armenakis et al., 2007; Armenakis et al., 1999). Implications. Department chairs and college deans should conduct faculty development workshops to allow faculty members the opportunity to develop confidence toward adopting educational innovations. Futhermore, Walczyk et al. (2007) found many faculty members are unmotivated to use educational innovation since they are not tied to rewards. Faculty members 113 need to feel it is in their professional self-interest if the overall goal is for classrooms to become learning centered through increased adoption of educational innovations (Baiocco & DeWaters, 1998; Harwood, 2003; National Research Council, 1999; Walczyk et al., 2007; Wright & Sunal, 2004; Wyckoff, 2001). The American Society for Engineering Education (2009, 2012) and the President's Council of Advisors on Science and Technology (2012) have already suggested promotion and tenure requirements in engineering need to promote pedagogical innovations in the classroom. Awareness of educational innovations through peers is a significant predictor of faculty members adopting educational innovations Findings. An awareness of peers using an educational innovation was an antecedent to adopt (Table 4-5, Model 4). This finding was consistent with Walczyk et al. (2007) who found that discussions with colleagues was the most influential source consulted before adopting educational innovations. Likewise, Borrego et al. (2010) found faculty attitudes play an important role in peer willingness to adopt new pedagogies such as active learning. Implications: Faculty members should show support for their colleagues through more open discussions of teaching, which will influence the dissemination of educational innovations (Dancy and Henderson, 2004; Henderson and Dancy, 2007). Department chairs and college deans should hold faculty development workshops to discuss educational innovations that could be used in classrooms. In addition, seminars and meetings will allow faculty members to learn about innovations discussed at workshops and conferences they have not attended (Delatte et al., 2007; Riley et al., 2007). 114 Limitations, Future Research, and Conclusions Limitations and Future Research The first limitation of this study is we categorized educational innovations as curriculum development, development of faculty expertise, instructional materials, and instructional strategies. While we assert these categories of educational innovations represent a significant representation of the innovations that are available to electrical engineering and computer science faculty members, we understand that our findings cannot be generalized beyond these four categories of educational innovations. We did conduct an ANOVA between the four categories of educational innovations and found no significant differences with respect to our control and dependent variables, so we have no reason to believe that splicing the data by innovations category would have yielded different results. Future research may wish to extend our study by looking at one specific educational innovation or look at additional innovation categories. The second limitation is that the data was collected from faculty members at ABET accredited institutions that were either part of an electrical engineering or computer science department. Future research should also look to validate or extend our model using other departments in engineering and technology programs. Such investigations may use the methodology outlined in this paper to find similarities and/or differences that may exist between undergraduate institutions and community college programs. The third limitation is that the purpose of this research was not to investigate whether differences in ethnicity or tenure status influenced the successful dissemination of educational innovations. Future research should investigate what influence these factors have on the dissemination process. 115 The fourth limitation is the characteristics of educational innovations were only investigated as predictors of intention to adopt educational innovations. Due to correlations between the characteristics of innovations and the later stages of dissemination (i.e., adoption and routine use), future research should examine the degree to which intention to adopt and adoption are mediators. The fifth limitation stems from collecting self-reported, cross-sectional data from faculty members. Since the dissemination of educational innovations unfolds over time, future research could validate this model using longitudinal data. We also allowed faculty members to self-select educational innovation, which may have added bias; future research should experiment with different survey designs that could prevent the self-selection bias. Finally, the survey used in this study consisted of 116 questions. This number of questions was required because we wanted to use validated scales with excellent psychometric properties. Future research might create compressed scales that may help improve response rates. Conclusions This study uses a survey to obtain the insights of electrical engineering and computer science faculty members at ABET certified programs in the United States. This study makes several important contributions to the research literature regarding the dissemination of educational innovations. First, this study empirically investigates the impact of characteristics of educational innovations on intention to adopt and readiness of faculty members toward educational innovations on intention to adopt, adoption, and routine use. Second, this study finds that ease of use and care about student learning outcomes directly influence intention to adopt educational innovations. This study also finds that attitude to innovation, efficacy toward change, and valence each moderated the relationship between ease of use and intention to adopt 116 educational innovations. This study shows that awareness of an educational innovation is a direct antecedent to adoption. Furthermore, this study finds that attitude to innovations moderates the relationship between intention to adopt and adoption. Finally, the responses from the faculty members included the use of many educational innovations, which allows the results to have more generalizability than studies that looked at only one educational innovation implementation. 117 Chapter 5: Major Limitations, Future Research Possibilities, and Major Contributions Major Limitations A major limitation of this dissertation was the sample size used in the three papers. For the systematic literature review, 37 articles were used to perform the realist synthesis. Patton (2002) suggested that reviews using qualitative data need not be exhaustive but should instead use a random sampling to increase the credibility of the study. While this dissertation used 187 articles to identify the characteristics of innovation and readiness variables, the use of only 19.79% of the articles to identify the overall mechanism that influences the dissemination of engineering educational innovations is a major limitation. For the Delphi study, our sample size was sufficiently large with 34 respondents at the end of round three (Linstone & Turoff, 2011). Unfortunately, our initial response rate was rather low (14.66%) with an attrition rate of 24.44% over the following rounds. This resulted in a final overall response rate of 11.07% at the end of round three. For the survey study, the sample size was 335 participants with a response rate of 7.98%. While this sample size was statistically large enough to test our model, we would have preferred a response rate of at least 10%. Future Research Dissemination of engineering educational innovations. In Chapter 4, this research operationalized only a portion of the characteristics of innovations and readiness variables. Future research should operationalize the remaining 15 characteristics of innovations and 11 readiness toward educational innovation variables that pertain to faculty members that were identified in the systematic literature review and Delphi study. In addition, future research may 118 operationalize the readiness characteristics of students and administrators that were identified in the configurative review and Delphi study. Future research may create surveys to test the remaining characteristics of educational innovations, faculty members, students, and administrators. These surveys will allow a greater understanding of dissemination and provide the groundwork needed to develop a scorecard or index. Dissemination of healthcare practices. Future research can modify the CODS framework to be used in disciplines outside the area of engineering education. Last year, the School of Pharmacy at Auburn University used a modified version of the CODS framework in a five-year Agency for Healthcare Research and Quality grant proposed by Dr. Richard Hansen. In healthcare practices, dissemination and implementation research are highly relevant in conducting Comparative Effectiveness Research (CER). CER dissemination is defined as the process of making target groups aware of, accept, and use the new instructional materials or methods, with the actual usage of the innovation being the ultimate goal (Lomas, 1993). Examples of CER materials include smartphone apps, text messages, community advocates, print/video media, push emails, electronic alerts/reminders, telehealth tools, interactive training, dashboards, and customized reports. The grant team modified the methodology described in this dissertation to show the interaction of the factors that influence the stakeholder groups? use of CER information. A conceptual model for understanding dissemination and implementation of CER could help improve clinical practices. Figure 5-1 depicts the potential relationships and factors (of individuals and of CER) that may influence the use of CER information. One of the goals of this grant proposal was to refine the characteristics of CER and of the three stakeholders (patient/caregivers, provider, and policy 119 makers) and further adapt the factors identified from this dissertation on engineering educational innovations. Figure 5-1. Comparative effectiveness research (CER) dissemination and implementation model In the proposed project, representatives of each stakeholder group will be asked to compare and prioritize the characteristics. These priority scales will be synthesized to arrive at a number from 0 to 1 for each patient/caregiver, provider, and policymaker combination as shown in Figure 5-2. These will be used to develop an index (0 to 1 scale) for the readiness for CER information. The CER information readiness index will be plotted against the characteristics of CER (Figure 5-3) to develop a predictive model for CER use. Figure 5-2. Conceptualizing readiness for CER information 120 Figure 5-3. Framework relating CER readiness and characteristics Dissemination of cybersecurity and information technology security. Another area for future research is to modify the CODS framework to study the dissemination of cybersecurity and information technology security. I will soon be a faculty member at the University of South Alabama?s School of Computing, which is a National Center of Academic Excellence for Information Assurance Education (CAE/IAE), sponsored by the National Security Agency and the Department of Homeland Security. The Center for Forensics, Information Technology and Security (CFITS) in this school is dedicated to the collection and dissemination of information related to digital forensics and information technology assurance and security. Future research may identify the stakeholder and organizational readiness factors that impact the dissemination of CFITS information. Major Contributions of this Dissertation This dissertation made several major contributions that may help developers of educational innovations, faculty members, department chairs, college deans, and grant program managers. First, ease of use was identified as the most important characteristic of educational 121 innovations in Chapter 4. Making educational innovations easy to use may influence faculty members who have low efficacy toward change or low valence to adopt innovations. Developers need to provide hands-on workshops and other tutorials so that faculty members can realize that the innovation is easy to use. These workshops and tutorials may also improve faculty members? expertise and knowledge about the innovations which may help improve the adoption rate of educational innovations. Chapter 4 identified that attitude to educational innovation was the most important readiness variable for faculty members and moderated the relationship between intention to adopt and adoption. The American Society for Engineering Education (2009) and President's Council of Advisors on Science and Technology (2012) have both suggested changing promotion and tenure guidelines in universities so as to change faculty members? attitudes toward educational innovations. Finally, this dissertation created two new frameworks that help explain the dissemination of engineering educational innovations. The first framework (Chapter 2) describes the interrelations among the characteristics of innovations and the readiness of the organizations. The second framework (Chapter 3), the Characteristics of Dissemination Success (CODS), investigates whether organizational readiness factors moderate the dissemination process. Educational innovation developers may use these frameworks to improve their innovations so that they can excel in one or more of the identified characteristics, enabling their innovations to be marketed more effectively. Faculty members can use these frameworks to consider the readiness of their educational organization. If their organization is not ready to adopt and use educational innovations, the faculty member may prefer to adopt only well-tested and well-known innovations, but if their organization is deemed ready the faculty member may 122 be encouraged to adopt new leading edge or even bleeding-edge innovations. Administrators can use these frameworks to help them understand why certain educational innovations are not successfully adopted. Finally, grant program managers can also use these frameworks to evaluate proposed innovations as well as realistic assessments of the organizational readiness of institutions that are applying for funding. 123 References ABET. (2013). Find Accredited Programs Retrieved July 3, 2013, from http://main.abet.org/aps/Accreditedprogramsearch.aspx Agarwal, R., & Prasad, J. (1999). Are individual differences germane to the acceptance of new information technologies? Decision Sciences, 30(2), 361-391. Aiken, L. S., & West, S. G. (1991). Multilple regression: Testing and interpreting interactions. Thousand Oaks, CA.: Sage. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179-211. Akay, A. (2008). A renaissance in engineering Ph.D. education. European Journal of Engineering Education, 33(4), 403-413. ?kerlind, G. S., & Trevitt, A. C. (1999). Enhancing self-directed learning through educational technology: When students resist the change. Innovations in Education and Training International, 36(2), 96-105. Alpkan, L., Bulut, C., Gunday, G., Ulusoy, G., & Kilic, K. (2010). Organizational support for intrapreneurship and its interaction with human capital to enhance innovative performance. Management Decision, 48(5), 732-755. American Society for Engineering Education. (2009). Creating a culture for scholarly and systematic innovation in engineering education Retrieved June 20, 2013, from 124 http://www.asee.org/about-us/the-organization/advisory- committees/CCSSIE/CCSSIEE_Phase1Report_June2009.pdf American Society for Engineering Education. (2012). Innovation with impact Retrieved May 16, 2013, from http://www.asee.org/InnovationwithImpact Anderson, M. H. (2011). Knowledge, attitudes, and instructional practices of Michigan community college math instructors: The search for a KAP gap in collegiate math. Doctoral Dissertation, Western Michigan University. Retrieved from http://teachingcollegemath.com/files/pdf/andersen_dissertation_kap_gap_collegiate_math .pdf Armenakis, A. A., Bernerth, J. B., Pitts, J. P., & Walker, H. J. (2007). Organizational change recipients' beliefs scale: Development of an assessment instrument. The Journal of Applied Behavioral Science, 43(4), 481-505. Armenakis, A. A., Harris, S. G., & Feild, H. S. (1999). Making change permanent: A model for instituationalizing change interventions. Research in Organizational Change and Development, 12, 97-128. Armenakis, A. A., Harris, S. G., & Mossholder, K. W. (1993). Creating readiness for organizational change. Human relations, 46(6), 681-703. Association to Advance Collegiate Schools of Business. (2013). 2013 Accreditation standards Retrieved January 28, 2014, from http://aacsb.edu/accreditation/2013standards/ 125 Badia, F., Cores, F., Filella, G., Gine, F., Granollers, T., & Molto, M. (2010). Applying a methodology based on personal welfare to the final tear engineering project. in Proceedings of the Frontiers in Education Conference, Arlington, VA, United States, T1E1-T1E6. Baiocco, S. A., & DeWaters, J. N. (1998). Successful college teaching. Needham Heights, MA: Allyn & Bacon. Balasubramanian, G., Lohani, V. K., Puri, I. K., Case, S. W., & Mahajan, R. L. (2011). Nanotechnology education: First step in implementing a spiral curriculum. The International Journal of Engineering Education, 27(2), 333-353. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall. Bartlem, C. S., & Locke, E. A. (1981). The Coch and French study: A critique and reinterpretation. Human Relations, 34(7), 555-566. Bennedsen, J. (2006). The dissemination of pedagogical patterns. Computer Science Education, 16(2), 119-136. Bernold, L. E. (2008). Applying total-quality-management principles to improving engineering education. Journal of Professional Issues in Engineering Education and Practice, 134(1), 33-40. 126 Birch, A., & Irvine, V. (2009). Preservice teachers? acceptance of ICT integration in the classroom: applying the UTAUT model. Educational Media International, 46(4), 295- 315. Blankinship, D. G. (2010). Foundations offer $506 million for education innovation Retrieved November 5, 2013, from https://www.ncstem.org/blog/foundations-offer-506-million-for- education-innovation Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic literature reviews in engineering education and other developing interdisciplinary fields. Journal of Engineering Education, 103(45-76). Borrego, M. J. (2007). Development of engineering education as a rigorous discipline: A study of the publication patterns of four coalitions. Journal of Engineering Education, 96(1), 5- 18. Borrego, M. J., & Bernhard, J. (2011). The emergence of engineering education research as an internationally connected field of inquiry. Journal of Engineering Education, 100(1), 14- 47. Borrego, M. J., Froyd, J. E., & Hall, T. S. (2010). Diffusion of engineering education innovations: A survey of awareness and adoption rates in U.S. engineering departments. Journal of Engineering Education, 99(3), 185-207. Borrego, M. J., Griffin Jr, O. H., Magliaro, S., Lohani, V. K., Terpenny, J., Downey, G., . . . Walker, T. (2005). Work in progress - Developing a rigorous research foundation for 127 graduate programs in engineering education. in Proceedings of the Frontiers in EducationIndianapolis, IN, United States, F3F-9-F3F-10. Borrego, M. J., Streveler, R. A., Miller, R. L., & Smith, K. A. (2008). A new paradigm for a new field: Communicating representations of engineering education research. Journal of Engineering Education, 97(2), 147-162. Bourrie, D. M., Cegielski, C. G., Jones-Farmer, L. A., & Sankar, C. S. (2014b). What makes educational innovations stick? A Delphi Approach. Paper presented at the ASEE Southeastern Section Conference, Macon, GA. Bourrie, D. M., Sankar, C. S., & Jones-Farmer, L. A. (2014a). Conceptualizing interactions between characteristics and organizational members' readiness to disseminate educational innovations. working paper. Aviation and Supply Chain Managment. Auburn University. Brancheau, J. C., & Wetherby, J. C. (1987). Key issues in information systems managment. MIS Quarterly, 1(11), 23-46. Bravo, E., Amante, B., Simo, P., Enache, M., & Fernandez, V. (2011). Video as a new teaching tool to increase student motivation. in Proceedings of the IEEE Global Engineering Education Conference, Amman, Jordan, 638-642. Cagiltay, N. E. (2008). Using learning styles theory in engineering education. European Journal of Engineering Education, 33(4), 415-424. 128 Cegielski, C. G. (2001). A model of the factors that affect the integration of emerging information technology into corporate strategy. Unpublished Doctoral Dissertation, University of Mississippi. Cegielski, C. G., Bourrie, D. M., & Hazen, B. T. (2013). Evaluating adoption of emerging IT for corporate IT strategy: Developing a model using a qualitative method. Information Systems Management, 30(3), 235-249. Cegielski, C. G., Jones-Farmer, L. A., Wu, Y., & Hazen, B. T. (2012). Adoption of cloud computing technologies in supply chains: An organizational information processing theory approach. The International Journal of Logistics Management, 23(2), 184-211. Cheung, K. K. F., Jong, M. S. Y., Lee, F. L., Lee, J. H. M., Luk, E. T. H., Shang, J., & Wong, M. K. H. (2008). FARMTASIA: An online game-based learning environment based on the VISOLE pedagogy. Virtual Reality, 12(1), 17-25. Cheville, A., & Bunting, C. (2011). Engineering students for the 21st century: Student development through the curriculum. Advances in Engineering Education, 2(4), 1-37. Christie, M., & Jurado, R. G. (2009). Barriers to innovation in online pedagogy. European Journal of Engineering Education, 34(3), 273-279. Churchman, C. W., & Schainblatt, A. (1965). The researcher and the manager: A dialectic of implementation. Management Science, 11(4), 469-487. Clarke, J. S., Ellett, C. D., Bateman, J. M., & Rugutt, J. K. (1996). Faculty receptivity/resistance to change, personal and organizational efficacy, decision deprivation and effectiveness in 129 research 1 universities. Paper presented at the Paper presented at the Annual meeting of the Association for the Study of Higher Education, Memphis, TN. Clayson, A., Raju, P. K., & Sankar, C. (2010). National dissemination of LITEE case studies: A model. in Proceedings of the ASEE Annual Conference and Exposition, Louisville, KY, United States Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.). Mahwah, N.J.: Lawrence Erlbaum Associates, Inc. Compeau, D. R., Meister, D. B., & Higgins, C. A. (2007). From prediction to explanation: Reconceptualizing and extending the perceived characteristics of innovating. Journal of the Association for Information Systems, 8(8), 409-439. Conlon, E. (2008). The new engineer: Between employability and social responsibility. European Journal of Engineering Education, 33(2), 151-159. Cramer, S., Jeanne, R., Lafayette, M., Litkow, M. J., Smith, A. R., & Tong, L. (2010). Sustaining appropriate technology enhanced learning in STEM disciplines. In Proceeding of the ASEE Annual Conference and Exposition, Louisville, KY, United States. Retrieved from http://search.asee.org/search/fetch?url=file%3A%2F%2Flocalhost%2FE%3A%2Fsearch %2Fconference%2F32%2FAC%25202010Full1922.pdf&index=conference_papers&spa ce=129746797203605791716676178&type=application%2Fpdf&charset= D'Amours, S., & R?nnqvist, M. (2013). An educational game in collaborative logistics. INFORMS Transactions on Education, 13(2), 102-113. 130 Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458-467. Damanpour, F., & Wischnevsky, J. D. (2006). Research on innovation in organizations: Distinguishing innovation-generating from innovation-adopting organizations. Journal of Engineering and Technology Management, 23(4), 269-291. Dancy, M. H., & Henderson, C. (2008). Barriers and promises in STEM reform. Retrieved July 15, 2012, from http://www7.nationalacademies.org/bose/Dancy_Henderson_CommissionedPaper.pdf Dancy, M. H., & Henderson, C. (2010). Pedagogical practices and instructional change in physics faculty. American Journal of Physics, 78(10), 1056-1063. Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. Delatte, N., Sutton, R., Beasley, W., & Bagaka's, J. (2007). Assessing the impact of case studies on the civil engineering and engineering mechanics curriculum. Annual ASEE Conference and Exposition. Retrieved from http://search.asee.org/search/fetch?url=file%3A%2F%2Flocalhost%2FE%3A%2Fsearch %2Fconference%2F14%2FAC%25202007Full761.pdf&index=conference_papers&spac e=129746797203605791716676178&type=application%2Fpdf&charset= Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes: Glenview, IL: Scott, Foresman. 131 Di Benedetto, C. A., Calantone, R. J., & Zhang, B. (2003). International technology transfer: Model and exploratory study in the People's Republic of China. International Marketing Review, 20(4), 446-462. Donnelly, D., McGarr, O., & O?Reilly, J. (2011). A framework for teachers? integration of ICT into their classroom practice. Computers & Education, 57(2), 1469-1483. Dufresne, R. L., & Offstein, E. H. (2012). Holistic and intentional student character development process: Learning from West Point. Academy of Management Learning & Education, 11(4), 570-590. Eiseman, J., & Fairweather, J. (1996). Evaluation of the National Science Foundation Undergraduate Course and Curriculum Development Program: Final Report. Washington, DC: SRI International. Elahinia, M. H., & Ciocanel, C. (2008). A problem-solving approach for teaching engineering laboratories. International Journal of Mechanical Engineering Education, 36(2), 140- 159. Erzurumlu, S. S., & Rollag, K. (2013). Increasing student interest and engagement with business cases by turning them into consulting exercises. Decision Sciences Journal of Innovative Education, 11(4), 359-381. Fairweather, J. (2008). Linking evidence and promising practices in science, technology, engineering, and mathematics (STEM) undergraduate education. Board of Science Education, National Research Council, The National Academies, Washington, DC. 132 Felder, R. M., & Silverman, L. (1988). Learning and teaching styles in engineering education. Engineering Education, 91(1), 3-17. Felder, R. M., Woods, D. R., Stice, J. E., & Rugarcia, A. (2000). The future of engineering education: Teaching methods that work. Chemical Engineering Education, 34(1), 26-39. Feser, J., Borrego, M. J., Pimmel, R., & Della-Piana, C. K. (2012). Results from a survey of National Science Foundation Transforming Undergraduate Education in STEM (TUES) program reviewers. in Proceedings of the ASEE Annual Conference and Exposition, San Antonio, TX Fincher, S. (2000). From transfer to transformation: Towards a framework for successful dissemination of engineering education. Paper presented at the Frontiers in Education Conference, Kansas City, MO. http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=896602&url=http%3A%2F%2Fie eexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D896602 Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA.: Addison-Wesley. Fishbein, M., & Ajzen, I. (1981). Attitudes and voting behavior: An application of the theory of reasoned action. In G. M. Stephenson & J. M. Davis (Eds.), Progress in Applied Social Psychology (Vol. 1, pp. 253-313). London, UK: Wiley. Froyd, J. E. (2001). Developing a dissemination plan. in Proceedings of the Frontiers in Education Conference, Reno, NV, F2G-18-22 vol.12. 133 Froyd, J. E. (2011). Propagation and realization of educational innovations in systems of undergraduate STEM education. Characterizing the Impact of Diffusion of Engineering Education Innovations Forum, National Academy of Engineering Retrieved August 13, 2012, from http://www.nae.edu/File.aspx?id=36824 Froyd, J. E., Borrego, M., Cutler, S., Henderson, C., & Prince, M. J. (2013). Estimates of use of research-based instructional strategies in core electrical or computer engineering courses. IEEE Transactions on Education,, 56(4), 393-399. Gill, J., Sharp, R., Mills, J., & Franzway, S. (2008). I still wanna be an engineer! Women, education and the engineering profession. European Journal of Engineering Education, 33(4), 391-402. Gillard, S., Nolan, E., & Bailey, D. (2008). Ten reasons for IT educators to be early adopters of IT innovations. Journal of Information Technology Education, 7, 21-33. Gough, D., Oliver, S., & Thomas, J. (2012). An introduction to systematic reviews. Thousand Oaks, CA: Sage. Gravestock, P. (2002). Making an impact through dissemination. In C. Baume, P. Martin & M. Yorke (Eds.), Managing educational development projects: Effective management for maximum impact (pp. 109-124). London, UK: Kogan Page. Gray, J. (2008). The implementation of differentiated instruction for learning disabled students included in general education elementary classrooms. Doctoral dissertation, University of La Verne. Retrieved from http://books.google.com/books?id=hJEEmh36lAIC&printsec=frontcover&dq=inauthor: 134 %22Jayne+Gray%22&hl=en&sa=X&ei=stM-U-SkEa6- sQTqhYHgCQ&ved=0CC0Q6AEwAA#v=onepage&q&f=false Greenhalgh, T., Robert, G., Bate, P., Macfarlane, F., & Kyriakidou, O. (2007). Frontmatter, in Diffusion of innovations in health service organizations: A systematic literature review. Oxford, UK: Blackwell Publishing, Inc. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005). Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Social Science & Medicine, 61(2), 417-430. Gropper, G. L. (1974). Instructional strategies. Englewood Cliffs, NJ: Educational Technology Publications, Inc. Guilabert, M. B. (2005). Attitudes toward consumer-customized high-tech products: The role of perceived usefulness, perceived ease of use, technology readiness, and customer customization sensitivity. Doctor of Philosophy dissertation, Georgia State University. (Paper 1) Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., . . . Falck-Ytter, Y. (2011). GRADE guidelines: 5. Rating the quality of evidence?publication bias. Journal of clinical epidemiology, 64(12), 1277-1282. 135 Gwayi, S. M. (2009). Perceptions of innovation as predictors of TALULAR implementation levels among secondary school science teachers in Malawi: A diffusion of innovations perspective. Doctor of Philosophy Doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, Virginia. Hall, G., George, A. A., & Rutherford, W. L. (1979). Measuring stages of concern about the innovation: A manual for use of the SoC Questionaire (Report No. 3032). Austin, TX: University of Texas at Austin, Research and Development Center for Teacher Education (ERIC Document Reproduction Service No. ED 147 342). Hall, G. E. (1974). The concerns-based adoption model: A developmental conceptualization of the adoption process within educational institutions. Retrieved from http://www.eric.ed.gov/ERICWebPortal/detail?accno=ED111791 Hall, G. E., & Hord, S. M. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press. Hall, G. E., & Hord, S. M. (2006). Implementing change: Patterns, principles, and potholes. Boston, MA: Pearson/Allyn & Bacon. Handscombe, R. D., Rodriguez-Falcon, E., & Patterson, E. A. (2009). Embedding enterprise in engineering. International Journal of Mechanical Engineering Education, 37(4), 263- 274. Hannon, J. (2009). Breaking down online teaching: Innovation and resistance. Australasian Journal of Educational Technology, 25(1), 14-29. 136 Hardgrave, B. C., Davis, F., & Riemenschneider, C. K. (2003). Investigating determinants of software developers' intentions to follow methodologies. Journal of Management Information Systems, 20(1), 123-151. Harwood, W. S. (2003). Course enhancement: A road map for devising active-learning and inquiry-based science courses. Journal of Development Biology, 47(2-3), 213-221. Hawk, T. F., & Shah, A. J. (2007). Using learning style instruments to enhance student learning. Decision Sciences Journal of Innovative Education, 5(1), 1-19. doi: 10.1111/j.1540- 4609.2007.00125.x Hayden, N. J., Rizzo, D. M., Dewoolkar, M. M., Neumann, M. D., Lathem, S., & Sadek, A. (2011). Incorporating a systems approach into civil and environmental engineering curricula: Effect on course redesign, and student and faculty attitudes. Advances in Engineering Education, 2(2), 1-27. Hazen, B. T., Wu, Y., & Sankar, C. S. (2012a). Factors that influence dissemination in engineering education. IEEE Transactions on Education,, 55(3), 384-393. Hazen, B. T., Wu, Y., Sankar, C. S., & Jones-Farmer, L. A. (2012b). A proposed framework for educational innovation dissemination. Journal of Educational Technology Systems, 40(3), 301-321. Henderson, C., & Dancy, M. H. (2007). Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Physical Review Special Topics: Physics Education Research, 3(2), 200102. 137 Henderson, C., & Dancy, M. H. (2009). Impact of physics education research on the teaching of introductory quanititive physics in the United States. Physical Review Special Topics - Physics Education Research, 5(2), 020107-020101 - 020107-020109. Henderson, C., Dancy, M. H., & Niewiadomska-Bugaj, M. (2012). Use of research-based instructional strategies in introductory physics: Where do faculty leave the innovation- decision process? Physical Review Special Topics - Physics Education Research, 8(2), 020104. Henderson, C., Finkelstein, N., & Beach, A. (2010). Beyond dissemination in college science teaching: An introduction to four core change strategies. Journal of College Science Teaching, 39(5), 18-25. Holt, D. T., Armenakis, A. A., Feild, H. S., & Harris, S. G. (2007a). Readiness for organizational change: The systematic development of a scale. The Journal of Applied Behavioral Science, 43(2), 232-255. Holt, D. T., Armenakis, A. A., Harris, S. G., & Feild, H. S. (2007b). Toward a comprehensive definition of readiness for change: A review of research and instrumentation. Research in Organizational Change and Development, 16, 289-336. Hu, X., & Kumar, A. (2013). Case?Beijing Opera mask sales in four cities in the United States. INFORMS Transactions on Education, 13(3), 145-151. Hutchinson, J. R., & Huberman, M. (1994). Knowledge dissemination and use in science and mathematics education: A literature review. Journal of Science Education and Technology, 3(1), 27-47. 138 Kaleelur Rahuman, M. A., Wikramanayake, G. N., & Hewagamage, K. P. (2011). Case study on adaptability to ICT enabled childhood education in Sri Lanka. in Proceedings of the 2nd International Conference on Advances in ICT for Emerging RegionsColombo, Sri lanka, 112-116. Kantardjieff, K. (2010). Pushing the boundaries of technology to educate and train the next generation of crystallographers. Journal of Applied Crystallography, 43(5 Part 2), 1276- 1282. Karahanna, E., Agarwal, R., & Angst, C. M. (2006). Reconceptualizing compatibility beliefs in technology acceptance research. MIS Quarterly, 30(4), 781-804. Kayfetz, J. L., & Almeroth, K. C. (2008). Creating innovative writing instruction for computer science graduate students. in Proceedings of the 38th ASEE/IEEE Frontiers in Education Conference, Saratoga Springs, NY, United States, T4F1-T4F6. Keeney-Kennicutt, W., Gunersel, A. B., & Simpson, N. (2008). Overcoming student resistance to teachin innovation. International Journal for the Scholarship of Teaching and Learning, 2(1), 1-26. King, H. (2003). Disseminating educational developments. In P. Kahn & D. Baume (Eds.), A guide to staff and educational development (pp. 79-94). London, UK: Kogan Page. Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055-1080. 139 Kolmos, A., Kofoed, L. B., & Du, X. Y. (2008). Ph.D. students' work conditions and study environment in university- and industry-based PhD programmes. European Journal of Engineering Education, 33(5/6), 539-550. Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage. Krishnan, M., Paulik, M. J., Yost, S., & Stoltz, T. (2008). Shared projects with a multi-sub- disciplinary flavor: Providing integration and context in a new ECE spiral curriculum. in Proceedings of the ASEE/IEEE Frontiers in Education Conference, Saratoga Springs, NY, F1H1-F1H6. Lang, D., Mengelkamp, C., J?ger, R. S., Geoffroy, D., Billaud, M., & Zimmer, T. (2007). Pedagogical evaluation of remote laboratories in eMerge project. European Journal of Engineering Education, 32(1), 57-72. Lawlor, B., & Donnelly, R. (2010). Using podcasts to support communication skills development: A case study for content format preferences among postgraduate research students. Computers & Education, 54(4), 962-971. Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003). The technology acceptance model: Past, present, and future. Communications of the Association for Information Systems, 12(50), 752-780. Leemkuil, H., & De Jong, T. O. N. (2012). Adaptive advice in learning with a computer-based knowledge management simulation game. Academy of Management Learning & Education, 11(4), 653-665. 140 Legner, C., Estier, T., Avdiji, H., & Boillat, T. (2013). Designing Capstone courses in management education: Knowledge activation and integration using and ERP-based simulation game. Thirty Fourth International Conference on Information Systems. Milan. Thirty Fourth International Conference on Information Systems Retrieved January 31, 2014, from http://aisel.aisnet.org/icis2013/proceedings/ISCurriculum/8/ Lewin, K. (1947). Frontiers in group dynamics: concept, method and reality in social science; social equilibria and social change. Human relations, 1(1), 4-51. Lewin, K. (1951). Field theory in social science. New York, NY: Harper and Row. Linstone, H. A., & Turoff, M. (2011). Delphi: A brief look backward and forward. Technological Forecasting and Social Change, 78(9), 1712-1719. Loftus, M. (2013). A schools of one's own. Prism, 51(4), 45-47. Lomas, J. (1993). Diffusion, dissemination, and implementation: Who should do what? Annals of the New York Academy of Sciences, 703(1), 226-237. Loo, R. (2002). The Delphi method: a powerful tool for strategic management. Policing: An International Journal of Police Strategies & Management, 25(4), 762-769. Macdonald, R. H., Manduca, C. A., Mogk, D. W., & Tewksbury, B. J. (2005). Teaching methods in undergraduate geoscience courses: Results of the 2004 On the cutting edge survey of U. S. faculty. Journal of Geoscience Education, 53(3), 237-252. Malhotra, M. K., Steele, D. C., & Grover, V. (1994). Important strategic and tactical manufacturing issues in the 1990s. Decision Sciences, 25(2), 189-214. 141 Martino, J. P. (1972). Technological forecasting for decision making. New York, NY: Elsevier Publishing. McKenzie, J., Alexander, S., Harper, C., & Anderson, S. (2005). Dissemination, adoption, & adaptation of project innovations in higher education. A report for the Carrick Institute for Learning and Teaching in Higher Education: University of Technology Sydney. McLaren, S. (2008). Exploring perceptions and attitudes towards teaching and learning manual technical drawing in a digital age. International Journal of Technology & Design Education, 18(2), 167-188. Miller, B. K., & Chiodo, B. (2008). Academic entitlement: Adapting the equity preference questionnaire for a university setting. in Proceedings of the Southern Management Association, St. Pete Beach, FL Miller, V. D., Johnson, J. R., & Grau, J. (1994). Antecedents to willingness to participate in a planned organizational change. Journal of Applied Communication Research, 22(1), 59- 80. Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2(3), 192-222. Morgan, J., & Barroso, L. (2008). North Texas STEM center: An engineer's perspective. in Proceedings of the ASEE Annual Conference and Exposition, Pittsburg, PA. 142 Mykytyn, K., Pearson, A., Paul, S., & Mykytyn, P. P. (2008). The use of problem-based learning to enhance MIS education. Decision Sciences Journal of Innovative Education, 6(1), 89- 113. Nambisan, S., Agarwal, R., & Tanniru, M. (1999). Organizational mechanisms for enhancing user innovation in information technology. MIS Quarterly, 23(3), 365-395. National Academy of Education. (2009). Education policy white paper on science and mathematics education Retrieved August 20, 2012, from http://www.naeducation.org/xpedio/groups/naedsite/documents/webpage/NAED_080865 .pdf National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century: Washington DC: National Academies Press. National Academy of Engineering. (2005). Educating the engineer of 2020: Adapting engineering education to the new century: Washington DC: National Academies Press. National Academy of Sciences. (2012). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering. Washington, D. C.: National Academies Press. National Research Council. (1999). Transforming undergraduate education in science, mathematics, engineering, and technology. Washington, D.C.: The National Academies Press. 143 National Research Council. (2011). Promising practices in undergraduate science, technology, engineering, and mathematics education: Summary of two workshops Retrieved November 15, 2012, from http://crpde.ou.edu/sites/crpde.ou.edu/files/NRC%20Report%20on%20Promising%20Pra ctices%20in%20STEM%20Education.pdf National Science and Technology Council. (2011). The federal science, technology, engineering, and mathematics (STEM) education portfolio. Federal Inventory of STEM Education, Fast-Track Action Committee, Committee on STEM Education Retrieved November 3, 2013, from http://www.whitehouse.gov/sites/default/files/microsites/ostp/costem__federal_stem_edu cation_portfolio_report.pdf National Science Board. (2007). Moving forward to improve engineering education. Arlington, VA: National Science Foundation. National Science Foundation. (2008). Innovations in engineering education, curriculum, and infrastructure (IEECI) (NSF-08-542). Arlington, VA: National Science Foundation. National Science Foundation. (2013). Widening implementation and demonstration of evidence- based reforms (WIDER): Program soliciation Retrieved May 14, 2013, from http://www.nsf.gov/pubs/2013/nsf13552/nsf13552.htm Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches (6th ed.). Boston, MA: Allyn & Bacon. 144 Nkhoma, M., Sriratanaviriyakul, N., Pham, H. C., & Lam, T. K. (2013). Understanding the paradox of information systems case studies. Nineteenth Americas Conference on Information Systems. Chicago, IL. Nineteenth Americas Conference on Information Systems Retrieved January 30, 2014, from http://aisel.aisnet.org/amcis2013/ISEducation/GeneralPresentations/23/ OECD. (2010). OECD economic surveys: United States 2010. OECD Publishing. Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15-29. Olds, B. M., Borrego, M. J., Besterfield-Sacre, M., & Cox, M. (2012). Continuing the dialog: Possibilities for community action research in engineering education. Journal of Engineering Education, 101(3), 407-411. Oppermann, R., & Rasher, R. (1997). Adaptability and adaptivity in learning systems. In A. Behrooz (Ed.), Knowledge transfer (Vol. 2, pp. 173-179). London, UK: pAce. Paliwoda, S. J. (1983). Predicting the future using Delphi. Management Decision, 21(1), 31-38. Patton, M. Q. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: Sage Publications. P?rez Morales, M., Pe??n, P. ?., Sevillano, ?. P., & Cerra, P. P. (2009). Design and use of a multimedia trainer for the subject Descriptive Geometry. Computer Applications in Engineering Education, 17(1), 13-24. 145 Pfeil, U., Ang, C. S., & Zaphiris, P. (2009). Issues and challenges of teaching and learning in 3D virtual worlds: Real life case studies. Educational Media International, 46(3), 223-238. Pierson, J., Kruck, S., & Teer, F. (2008). Trends in names of undergraduate computer-related majors in AACSB accredited schools of business in the USA. The Journal of Computer Information Systems, 49(2), 26-31. Pond, K. S. (1963). A Language Teaching Tool: The Overhead Projector. [Article]. Modern Language Journal, 47(1), 30. Preacher, K. J., Curran, P. J., & Bauer, D. J. (2006). Computational tools for probing interaction effects in multiple linear regression, multilevel modeling, and latent curve analysis. Journal of Educational and Behavioral Statistics, 31, 437-448. President's Council of Advisors on Science and Technology. (2012). Report to the President: Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Washington DC: Executive Office of the President. Retrieved October 21, 2012, from http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-engage-to-excel- final_feb.pdf Qualters, D. M., Sheahan, T. C., Mason, E., Navick, D. S., & Dixon, M. (2008). Improving learning in first-year engineering courses through interdisciplinary collaborative assessment. Journal of Engineering Education, 97(1), 37-45. 146 Richardson, H. A., Simmering, M. J., & Sturman, M. C. (2009). A tale of three perspectives: examining post hoc statistical techniques for decteion and correction of common method variance. Organizational Research Methods, 12(4), 762-800. Richter, D. M., & Paretti, M. C. (2009). Identifying barriers to and outcomes of interdisciplinarity in the engineering classroom. European Journal of Engineering Education, 34(1), 29-45. Riggs, W. E. (1983). The Delphi technique: An experimental evaluation. Technological Forecasting and Social Change, 23(1), 89-94. Riley, K., Davis, M., Cox, A., & Maciukenas, J. (2007). "Ethics in the details": An NSF project to integrate ethics into the graduate engineering curriculum. in Proceedings of the IEEE International Professional Communication Conference, Seattle, WA Riley, K., Davis, M., Jackson, A. C., & Maciukenas, J. (2009). "Ethics in the details": Communicating engineering ethics via micro-insertion. IEEE Transactions on Professional Communication, 52(1), 95-108. doi: 10.1109/tpc.2008.2012286 Rogers, E. M. (1962). Diffusion of innovations. Glencoe, IL: Free Press. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press/Simon & Schuster. Rogers, E. M., & Shoemaker, F. F. (1971). Communications of innovations: A cross-cultural approach (2nd ed. ed.). New York: Free Press. 147 Rohrbaugh, J. (1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance, 24(1), 73-92. Romano, A. R. (2010). Malleable Delphi: Delphi research technique, its evolution, and business applications. International Review of Business Research Papers, 6(5), 235-243. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641. Ross, A. M., Rhodes, D. H., & Hastings, D. E. (2008). Defining changeability: Reconciling flexibility, adaptability, scalability, modifiability, and robustness for maintaining system lifecycle value. Systems Engineering, 11(3), 246-262. Saga, V. L., & Zmud, R. W. (1994). The nature and determinants of IT acceptance, routinization, and infusion. In L. Levine (Ed.), Diffusion, transfer, and implementation of information technology (pp. 67-86). Amsterdam: Elsevier/North-Holland. Sanchez, I., Neriz, L., & Ramis, F. (2008). Design and application of learning environments based on integrative problems. European Journal of Engineering Education, 33(4), 445- 452. Saunders, R. (2012). Assessment of professional development for teachers in the vocational education and training sector: An examination of the concerns based adoption model. Australian Journal of Education, 56(2), 182-204. 148 Schuhmann, R. J. (2010). Engineering leadership education: The search for definition and a curricular approach. Journal of STEM Education: Innovations and Research, 11(3-4), 61- 69. Schunk, D. H., Pintrich, P. R., & Meece, J. L. (2008). Motivation in education: Theory, research, and applications (3rd ed.). Upper Saddle River, NJ: Merrill Prentice Hall. Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., . . . Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of clinical epidemiology, 62(10), 1013-1020. Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22(2), 63-75. Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw- Hill. Singer, S., & Karl, S. (2013). Last word: Follow the evidence. Prism, 15(4), 92. Smith, K. A., Sheppard, S. D., Johnson, D. W., & Johnson, R. T. (2005). Pedagogies of engagement: Classroom-based practices. Journal of Engineering Education, 94(1), 87- 101. Snider, B., & Balakrishnan, J. (2013). Lessons learned from implementing web-based simulations to teach operations management concepts. INFORMS Transactions on Education, 13(3), 152-161. 149 Straub, E. T. (2009). Understanding technology adoption: Theory and future directions for informal learning. Review of Educational Research, 79(2), 625-649. Tabata, L. N., & Johnsrud, L. K. (2008). The impact of faculty attitudes toward technology, distance education, and innovation. Research in Higher Education, 49(7), 625-646. Taras, V. A. S., Caprar, D. V., Rottig, D., Sarala, R. M., Zakaria, N., Zhao, F., . . . Zengyu Huang, V. (2013). A global classroom? Evaluating the effectiveness of global virtual collaboration as a teaching tool in management education. Academy of Management Learning & Education, 12(3), 414-435. Tashakkori, A., & Teddlie, C. (2003a). Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: SAGE Publications. Tashakkori, A., & Teddlie, C. (2003b). Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. International Journal of Social Research Methodology, 6(1), 61-77. Taylor, R. E., & Meinhardt, D. J. (1985). Defining computer information needs for small business: A Delphi method. Journal of Small Business Management, 23(2), 3-10. Taylor, S., & Todd, P. (1995). Decomposition and crossover effects in the theory of planned behavior: A study of consumer adoption intentions. International Journal of Research in Marketing, 12(2), 137-155. Teo, H.-H., Wei, K.-K., & Benbasat, I. (2003). Predicting intention to adopt interorganizational linkages: An institutional perspective. MIS Quarterly, 27(1), 19-49. 150 Teo, T. (2009). The impact of subjective norm and facilitating conditions on pre-service teachers' attitude toward computer use: A structural equation modeling of an extended technology acceptance model. Journal of Educational Computing Research, 40(1), 89-109. Teo, T. (2011). Factors influencing teachers? intention to use technology: Model development and test. Computers & Education, 57(4), 2432-2440. Teo, T., Lee, C. B., & Chai, C. S. (2008). Understanding pre-service teachers' computer attitudes: Applying and extending the technology acceptance model. Journal of Computer Assisted Learning, 24(2), 128-143. Topi, H., Valacich, J., Wright, R., Kaiser, K., Nunamaker, J., Sipior, J., & Vreede, G. (2010). IS 2010: Curriculum guidelines for undergraduate degree programs in information systems. Communications of the Association for Information Systems, 26(Article 18), 359-428. Tornatzky, L. G., & Fleischer, M. (1990). The processes of technological innovation. Lexington, MA: Lexington Books. Vanasupa, L. (2011). The human dimension of systemic department-level change: A change agent's retrospective on a case of reform. Advances in Engineering Education, 2(2), 1-31. Veldman, F. J., De Wet, M. A., Ike Mokhele, N. E., & Bouwer, W. A. J. (2008). Can engineering education in South Africa afford to avoid problem-based learning as a didactic approach? European Journal of Engineering Education, 33(5/6), 551-559. Venkatesh, V., & Bala, H. (2008). Technology acceptance model 3 and a research agenda on interventions. Decision Sciences, 39(2), 273-315. 151 Venkatesh, V., Brown, S., & Bala, H. (2013). Bridging the qualitative-quantitative divide: Guidelines for conducting mixed methods research in information systems. MIS Quarterly, 37(1), 21-54. Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186-204. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. Venkatesh, V., Thong, J., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157-178. Verzat, C., Byrne, J., & Fayolle, A. (2009). Tangling with spaghetti: Pedagogical lessons from games. Academy of Management Learning & Education, 8(3), 356-369. Walczyk, J. J., Ramsey, L. L., & Zha, P. (2007). Obstacles to instructional innovation according to college science and mathematics faculty. Journal of Research in Science Teaching, 44(1), 85-106. Watson, M., Corbett, K., Prather, K., Carpenter, J., & Cronk, S. (2010). Fostering dissemination skills in STEM doctoral students: Tips for the Ph.D. student and the general impact on STEM undergraduates. in Proceedings of the ASEE Annual Conference and Exposition, Louisville, KY, United States 152 Waugh, R. (2000). Towards a model of teacher receptivity to planned system-wide educational change in a centrally controlled system. Journal of Educational Administration, 38(4), 350-367. Waugh, R., & Godfrey, J. (1993). Teacher receptivity to system-wide change in the implementation change. British Educational Research Journal, 19(5), 565-578. Waugh, R., & Punch, K. F. (1985). Teacher receptivity to system?wide change. British Educational Research Journal, 11(2), 113-121. Waugh, R., & Punch, K. F. (1987). Teacher receptivity to systemwide change in the implementation stage. Review of Educational Research, 57(3), 237-254. World Economic Forum. (2013). The global competitiveness report 2013-2014: Full data edition, from http://www3.weforum.org/docs/WEF_GlobalCompetitivenessReport_2013-14.pdf Wright, E. L., & Sunal, D. W. (2004). Reform in undergraduate science classrooms. In D. W. Sunal, E. L. Wright & J. B. Day (Eds.), Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century (pp. 33-52). Greenwich, CT: Information Age. Wu, Y., & Sankar, C. S. (2013). Impact of hands-on research experience on students? learning in an introductory management information system course. Decision Sciences Journal of Innovative Education, 11(4), 335-358. 153 Wyckoff, S. (2001). Changing the culture of undergraduate teaching: Shifting from lecture to interactive engagement and scientific reasoning. Journal of College Science Teaching, 30(5), 306-312. Yi, M. Y., & Hwang, Y. (2003). Predicting the use of web-based information systems: self- efficacy, enjoyment, learning goal orientation, and the technology acceptance model. International journal of human-computer studies, 59(4), 431-449. Yoder, B. L. (2012). Engineering by the numbers Retrieved 2013, June 30, from http://www.asee.org/papers-and-publications/publications/11-47.pdf Zmuda, M., & Hatch, M. (2007). Scheduling topics for improved student comprehension of recursion. Computers and Education, 48(2), 318-328. 154 Appendix 2-1. PRISMA diagram summarizing the sources contributing to the systematic review A d d i t i o n a l a r t i c l e s i d e n t i f i e d t h r o u g h 8 j o u r n a l s s e a r c h e d ( n = 1 6 4 ) P a p e r s i d e n t i f i e d t h r o u g h 6 d a t a b a s e s e a r c h e s ( n = 3 2 5 ) P a p e r s a f t e r d u p l i c a t e s r e m o v e d ( n = 4 8 9 ) P a p e r s i n c l u d e d f o r t h e s y s t e m a t i c r e v i e w ( n = 1 8 7 ) P a p e r s e x c l u d e d f o r n o t b e i n g a b o u t t h e d i f f u s i o n , d i s s e m i n a t i o n , o r a d o p t i o n o f a n e n g i n e e r i n g r e l a t e d e d u c a t i o n a l i n n o v a t i o n ( n = 1 9 3 ) P a p e r s e x c l u d e d f o r l a c k o f w o r t h ( n = 1 0 9 ) A f t e r p a p e r s w e r e s c r e e n e d f o r r e l e v a n c e t o t h e r e s e a r c h t o p i c ( n = 2 9 6 ) A f t e r p a p e r s w e r e s c r e e n e d f o r w o r t h ( n = 1 8 7 ) P a p e r s i n c l u d e d f o r t h e r e a l i s t s y n t h e s i s ( n = 3 7 ) I d e n t i f i c a t i o n S c r e e n i n g E l i g i b i l i t y I n c l u d e d 155 Appendix 2-2. List of journals and conference proceedings included in this study Journals or Conference Proceedings Count of Articles European Journal of Engineering Education 40 ASEE Annual Conference and Exposition 20 Advances in Engineering Education 18 International Journal of Engineering Education 15 ASEE/IEEE Frontiers in Education Conference 13 Computers and Education 9 Journal of Engineering Education 9 Annual Conference on Innovation and Technology in Computer Science Education 8 International Journal of Mechanical Engineering Education 7 Journal of STEM Education: Innovation and Research 4 Computer Applications in Engineering Education 3 International Conference on Reforming Education, Quality of Teaching and Technology-Enhanced Learning 2 Journal of Professional Issues in Engineering Education and Practice 2 Academy of Management Learning & Education 1 ACM Inroads 1 ACM Technical Symposium on Computer Science Education, SIGCSE 1 Agile Conference 1 Annual Conference of the IEEE Industrial Electronics Society, IECON 1 Annual Southeast Regional Conference, ACM SE 1 Australasian Journal of Educational Technology 1 British Journal of Educational Technology 1 Chemical Engineering Education 1 Computers and Chemical Engineering 1 Computers in Human Behavior 1 Education for Chemical Engineers 1 Educational Media International 1 Engineering Design Graphics Journal 1 Engineering Education 2010: Inspiring the Next Generation of Engineers, EE 2010 1 IEEE Global Engineering Education Conference, EDUCON 1 IEEE International Professional Communication Conference, IPCC 1 IEEE Transactions on Education 1 IEEE Transactions on Professional Communication 1 Industrial & Commercial Training 1 International Conference Information Visualisation, IV 1 International Conference on Computers in Education, ICCE 1 International Conference on Web-based Education 1 International Journal of Instructional Media 1 International Journal of Technology & Design Education 1 156 International Symposium on Wikis, WikiSym 1 Journal of Applied Crystallography 1 Journal of Engineering Design 1 Journal of Research in Science Teaching 1 Journal of Small Business and Enterprise Development 1 Liberal Education 1 Materials Research Society Symposium Proceedings 1 NordDesign 2008 Conference 1 SME Technical Papers 1 Society of Manufacturing Engineers 1 Symposium on Human Interface 1 Virtual Reality 1 Total Articles 187 157 Appendix 2-3. Citations for the configurative review. Akay, A. (2008). A renaissance in engineering PhD education. European Journal of Engineering Education, 33(4), 403-413. Badia, F., Cores, F., Filella, G., Gine, F., Granollers, T., & Molto, M. (2010). Applying a methodology based on personal welfare to the final tear engineering project. in Proceedings of the 40th Annual Frontiers in Education Conference: Celebrating Forty Years of Innovation, FIE 2010, Arlington, VA, T1E1-T1E6. Bernold, L. E. (2008). Applying total-quality-management principles to improving engineering education. Journal of Professional Issues in Engineering Education and Practice, 134(1), 33-40. Borrego, M. J., Froyd, J. E., & Hall, T. S. (2010). Diffusion of engineering education innovations: A survey of awareness and adoption rates in U.S. engineering departments. Journal of Engineering Education, 99(3), 185-207. Bravo, E., Amante, B., Simo, P., Enache, M., & Fernandez, V. (2011). Video as a new teaching tool to increase student motivation. in Proceedings of the 2011 IEEE Global Engineering Education Conference, EDUCON 2011, Amman, Jordan, (pp. 638-642). Cheung, K. K., Jong, M. S., Lee, F. L., Lee, J. H., Luk, E. T., Shang, J., & Wong, M. K. (2008). Farmtasia: An online game-based learning environment based on the VISOLE pedagogy. Virtual Reality, 12(1), 17-25. Cheville, A., & Bunting, C. (2011). Engineering students for the 21st century: Student development through the curriculum. Advances in Engineering Education, 2(4), 1-37. Christie, M., & Jurado, R.G. (2009). Barriers to innovation in online pedagogy. European Journal of Engineering Education, 34(3), 273-279. 158 Conlon, E. (2008). The new engineer: Between employability and social responsibility. European Journal of Engineering Education, 33(2), 151-159. Cramer, S., Jeanne, R., Lafayette, M., Litkow, M. J., Smith, A. R., & Tong, L. (2010). Sustaining appropriate technology enhanced learning in STEM disciplines. in Proceedings of the 2010 ASEE Annual Conference and Exposition, Louisville, KY. Delatte, N., Sutton, R., Beasley, W., & Bagaka's, J. (2007). Assessing the impact of case studies on the civil engineering and engineering mechanics curriculum. in Proceedings of the 114th Annual ASEE Conference and Exposition, 2007, Honolulu, HI. Elahinia, M. H., & Ciocanel, C. (2008). A problem-solving approach for teaching engineering laboratories. International Journal of Mechanical Engineering Education, 36(2), 140- 159. Gill, J., Sharp, R., Mills, J., & Franzway, S. (2008). I still wanna be an engineer! Women, education and the engineering profession. European Journal of Engineering Education, 33(4), 391-402. Handscombe, R. D., Rodriguez-Falcon, E., & Patterson, E. A. (2009). Embedding enterprise in engineering. International Journal of Mechanical Engineering Education, 37(4), 263- 274. Hannon, J. (2009). Breaking down online teaching: Innovation and resistance. Australasian Journal of Educational Technology, 25(1), 14-29. Hayden, N. J., Rizzo, D. M., Dewoolkar, M. M., Neumann, M. D., Lathem, S., & Sadek, A. (2011). Incorporating a systems approach into civil and environmental engineering curricula: Effect on course redesign, and student and faculty attitudes. Advances in Engineering Education, 2(2), 1-27. 159 Hazen, B. T., Wu, Y., & Sankar, C. S. (2012). Factors that influence dissemination in engineering education. Education, IEEE Transactions on, 55(3), 384-393. Kantardjieff, K. (2010). Pushing the boundaries of technology to educate and train the next generation of crystallographers. Journal of Applied Crystallography, 43(5 part 2), 1276- 1282. Kayfetz, J. L., & Almeroth, K. C. (2008). Creating innovative writing instruction for computer science graduate students. in Proceedings of the 38th ASEE/IEEE Frontiers in Education Conference, FIE 2008, Saratoga Springs, NY, T4F1-T4F6. Kolmos, A., Kofoed, L. B., & Du, X. Y. (2008). PhD students' work conditions and study environment in university- and industry-based PhD programmes. European Journal of Engineering Education, 33(5/6), 539-550. Krishnan, M., Paulik, M. J., Yost, S., & Stoltz, T. (2008). Shared projects with a multi-sub- disciplinary flavor - Providing integration and context in a new ECE spiral curriculum. in Proceedings of the 38th ASEE/IEEE Frontiers in Education Conference, FIE 2008, Saratoga Springs, NY, F1H1-F1H6. Lang, D., Mengelkamp, C., J?ger, R. S., Geoffroy, D., Billaud, M., & Zimmer, T. (2007). Pedagogical evaluation of remote laboratories in eMerge project. European Journal of Engineering Education, 32(1), 57-72. Lawlor, B., & Donnelly, R. (2010). Using podcasts to support communication skills development: A case study for content format preferences among postgraduate research students. Computers & Education, 54(4), 962-971. 160 McLaren, S. (2008). Exploring perceptions and attitudes towards teaching and learning manual technical drawing in a digital age. International Journal of Technology & Design Education, 18(2), 167-188. Morgan, J., & Barroso, L. (2008). North Texas STEM center: An engineer's perspective. in Proceedings of the 2008 ASEE Annual Conference and Exposition, Pittsburg, PA. P?rez Morales, M., Pe??n, P. ?., Sevillano, ?. P., & Cerra, P. P. (2009). Design and use of a multimedia trainer for the subject Descriptive Geometry. Computer Applications in Engineering Education, 17(1), 13-24. Pfeil, U., Ang, C. S., & Zaphiris, P. (2009). Issues and challenges of teaching and learning in 3D virtual worlds: Real life case studies. Educational Media International, 46(3), 223-238. Qualters, D. M., Sheahan, T. C., Mason, E., Navick, D. S., & Dixon, M. (2008). Improving learning in first-year engineering courses through interdisciplinary collaborative assessment. Journal of Engineering Education, 97(1), 37-45. Richter, D. M., & Paretti, M. C. (2009). Identifying barriers to and outcomes of interdisciplinarity in the engineering classroom. European Journal of Engineering Education, 34(1), 29-45. Riley, K., Davis, M., Cox, A., & Maciukenas, J. (2007). "Ethics in the details": An NSF project to integrate ethics into the graduate engineering curriculum. in Proceedings of the 2007 IEEE International Professional Communication Conference, Seattle, WA. Sanchez, I., Neriz, L., & Ramis, F. (2008). Design and application of learning environments based on integrative problems. European Journal of Engineering Education, 33(4), 445- 452. 161 Schuhmann, R. J. (2010). Engineering leadership education--The search for definition and a curricular approach. Journal of STEM Education: Innovations and Research, 11(3-4), 61- 69. Vanasupa, L. (2011). The human dimension of systemic department-level change: A change agent's retrospective on a case of reform. Advances in Engineering Education, 2(2), 1-31. Veldman, F. J., De Wet, M. A., Ike Mokhele, N. E., & Bouwer, W. A. J. (2008). Can engineering education in South Africa afford to avoid problem-based learning as a didactic approach? European Journal of Engineering Education, 33(5/6), 551-559. Verzat, C., Byrne, J., & Fayolle, A. (2009). Tangling with spaghetti: Pedagogical lessons from games. Academy of Management Learning & Education, 8(3), 356-369. Walczyk, J. J., Ramsey, L. L., & Zha, P. (2007). Obstacles to instructional innovation according to college science and mathematics faculty. Journal of Research in Science Teaching, 44(1), 85-106. Zmuda, M., & Hatch, M. (2007). Scheduling topics for improved student comprehension of recursion. Computers and Education, 48(2), 318-328. 162 Appendix 3-1. Categorized participant comments from round one of the Delphi study Characteristics of Innovation 1) Measurability (21 in Innovation ? 1 in Student ? 4 in Faculty: 26 Overall) - The degree to which the impact of the pedagogical innovation can be assessed in particular the ability to clearly attribute the effects to the innovation. ? Good evaluation data RE effectiveness ? What factors are important for successful adoption and how to measure success. ? Effectiveness in improving student learning outcomes ? effectiveness is usability ? Must have some evidence of effectiveness ? Evidence that the innovation will improve learning. ? evidence of value ? Good statistical practices. ? Assessment of results, statistical significance of results. ? Evaluation- innovation must have solid evaluation data documenting success ? Clarity of expected outcomes ? Easy to assess ? proven effectiveness ? Assessment and evaluation plan of the innovation ? Their benefit to student learning is measureable and quantified ? Validity; Scientific Impact ? Scalability of innovation outcomes 163 ? rigorous assessment that demonstrates improved student success and/or learning gains ? proof of success / - integrates assessment tools ? How much of the knowledge they retain upon completion of the course. ? the effectiveness of the assessment of the innovation ? Has evaluation data supporting its effectiveness. ? (Student) Will it advance their learning. ? (Faculty) assessment-centered ? (Faculty) willingness to assess them ? (Faculty) and ultimately report the outcomes of these assessment activities. ? (Faculty) Faculty must be knowledgeable of assessment strategies. Faculty must have a fairly strong passion and interest in student learning and must be willing to commit substantial time and energy to develop, adapt, evaluate, and disseminate innovations in STEM. 2) Relative Advantage (14 in Innovation ? 3 in Student ? 2 in Faculty ? 1 in Admin: 20 Overall) ? The degree to which the pedagogical innovation is perceived as being better than its precursor. ? Creation of an innovation that fills a void (where something does not already exist) ? I would guess it's the same characteristics specified in Rogers' diffusion of innovations theory: they have to be useful, easy to use, people must have a chance to try them at no risk, etc. 164 ? Must be attractive to potential adopters ? the Innovation seems useful to my work and will make my job easier ? Easier and faster implementation of hands-on experience; beyond-academic examples that can be observed and studied with minimal overhead of setting them up. ? don't require a lot of time (compared to current techniques) to prepare and administer ? Saving others time and money is also a big attraction ? Not too expensive ? Cost Benefit ? Value to others - potential for benefits to ? The main characteristic, in my opinion, for successful dissemination is that the educational innovation is realistically "do-able" whether that mean that an innovation is affordable or practical. ? Inexpensive to adopt ? Investigate existing programs and their success. ? saves the faculty time ? it clearly is an improvement ? (Student) Perceived benefits ? (Student) Students are most accepting of innovations that will obviously benefit them ? (Student) Educational innovations should be designed to be relevant and useful to a broad range of students. 165 ? (Faculty) Potential benefits they perceive from using the innovation ? (Faculty) saving time ? (Administration) If an educational innovation saves money without any (or at least too much) negative impact on student learning, it is more likely to be successfully disseminated. 3) Adaptability (13 in Innovation ? 2 in Faculty: 15 Overall) ? The degree to which the user can modify the pedagogical innovation as deemed necessary. ? So it needs to be general enough that there are additional examples of how it can be applied. ? Innovations that can be adapted to the particular needs and prejudices of adopters are more likely to be tried. When they can be added to existing approaches, or used to modify/supplement existing approaches, they create less disruption. ? Innovation is adaptable across different sorts of topics and teaching venues ? Ability to be tailored by adaptors so that they can get "credit" from their peers and administration for implementing the innovation ? Must be sufficiently flexible so that instructors may tailor it to their needs. ? easily transferred and adapted at other institutions ? be malleable to fit within a the new environment provided by different faculty, courses, and/or students ? Adaptable ? Adoptability and adaptability of innovation at other institutions ? have an impact on broad range of fields 166 ? Adaptable so that an instructor can adapt to his or her style and environment ? the innovation is not that different than what is already being done so it is easy to modify existing practice ? Adaptability of innovation outcomes ? (Faculty) Ability to develop or adapt the innovation's intended pedagogy ? (Faculty) Faculty must have a fairly strong passion and interest in student learning and must be willing to commit substantial time and energy to develop, adapt, evaluate, and disseminate innovations in STEM. 4) Ease of Use (11 in Innovation, 1 in Student: 12 Overall) ? The degree to which a pedagogical innovation is perceived as relatively easy to use and understand ? Ease of use in classes (fast, scale-able, students comprehend how it works) / + Ease of use by faculty (easy to und ? Conceptually simple so that it can be easily understood by adaptors. ? must be easy to use ? Ease of apply at your situation. ? easy to use ? ease of use ? Not too difficult or time-consuming to use ? how easy new people can understand how to use the innovation ? easy to understand ? relatively simple and user friendly ? Be easy for other faculty to reuse. 167 ? (Student) Is it easy for them to use. 5) Compatibility (11 in Innovation, 1 in Faculty: 12 overall) ? The consistency of the pedagogical innovation with current pedagogy. ? The main things that I think I look for when looking around for innovative approaches to teaching are examples that are from my discipline, engineering-math. / / For example, a strategy that gives an example where discussions are used to help students learn, is harder to implement in a math based engineering class. Question and answer, give and take is a great accomplishment in such classes. Rarely can I figure out a way to, or why I should have beginning students discuss whether they think the second law of thermodynamics is valid or not. / / So if the strategy illustrates with something from my field, it resonates. ? Match of innovation goal with goal of faculty and students. The innovation needs to be seen by both faculty and students as one that will help them achieve something they want. ? ability to integrate with existing program ? Educators adopt innovations when they feel comfortable with it. I wish 'comfort' meant evidence but I think it really has more to do with philosophical alignment; i.e., an instructor adopts an innovation when it looks like something that is consistent with how he or she believe their students learn best. ? Not too radical ? Goal alignment - Alignment with the goals of a potential user ? Interactiveness of the process. It is important to give students an opportunity to test their knowledge with potential award and zero penalties. This can be done in a 168 classroom setting, where students can be given credit for participation. Of course for students to feel comfortable with this idea, it is important to encourage them and help them steer a wrong answer towards the correct one. ? the innovation is not that different that what is already being done so it is easy to modify existing practice ? Easy to understand, fits within the PowerPoint framework (unfortunately). ? relatively simple and user friendly ? Fit well within the constraints of the other curriculum. ? (Faculty) Education and experience- consistent with innovation and means of implementing innovation (within "comfort zone") 6) Ease to implement (10 Overall) The degree to which a pedagogical innovation is perceived as relatively easy to implement and adopt. ? ease in implementation ? Easy to implement with minimum of advance preparation and contemporaneous overhead in implementation ? ease of adoption ? Ease of adoption ? Easy to adopt ? Easier and faster implementation of hands-on experience ? The effort it takes for adoption is also a factor. I believe instructors are less likely to adopt an innovation if it involves extensive re-design of a course that has been 169 taught for awhile. Incorporating a new project/activity is less extensive than re- designing instruction throughtout the semester. ? Not too expensive or disruptive to implement. ? The main characteristic is ease of adoption ? ease of adoption (how much work do **I** have to do to adopt the innovation) 7) Practicality of the Concept (9 overall) ? Professional relevancy ? innovation is rooted in a documented need for the practice of engineering ? There is a clear link between the innovation and a prevalent problem ? Sound instructional design MUST be included during the development of the educational innovations ? Development or adaptation of an innovation works best if the innovation is based on informed practice ? A unique idea with good planning ? Novel approach ? Practicality of the concept. Whether or not they can put the concept into practice influences how much of the knowledge they retain upon completion of the course. ? I think the first characteristic of an innovation is that is addresses a problem. 8) Communicability (5 in Innovation, 4 in Faculty: 9 Overall) The ease with which the results of using the pedagogical innovation can be easily described to others. ? effectiveness of communication 170 ? Communication- means to effectively "advertise" the innovation and invite potential adopters to join efforts ? Good, concise description / - Support from original developers, e.g. responsive to questions via email, visits, training, etc. ? ability of the faculty / innovator in making a compelling argument that influences dissemination. No innovation, regardless how unique or "innovative" it may be, will not be successfully disseminated without an effective assessment of its outcomes or without a compelling argument made in favor of the innovation. ? Structure of ideas. Well organized? Listen to peers and keep department head involved in the plan. This will help for potential institutionalization ? (Faculty) Faculty has engaging personality which exudes interest in their work, excitement about potential of innovation, excitement of dissemination opportunities ? (Faculty) ability to communicate the results ? (Faculty) Confidence and communication. It is important to be able to speak from experience and use them to explain concepts that are being taught. This directly leads to communication and command over the language. ? (Faculty) The faculty member should also have good communication skills, specifically, the ability to articulate the problem addressed by the innovation, the implementation details as well as its effectiveness. 9) Quality of Initial Information (9 Overall) The degree to which the initial information regarding the pedagogical innovation is relevant, timely, complete, and appropriate in terms of amount so as to add value. 171 ? high quality descriptions, excellent formatting, visuals, etc., Feedback about the innovation from beneficiaries (e.g., students) and those that deliver the innovation (e.g., teachers) ? Finally, I am looking for imaginative things, those that use technology, or are strongly visual or even things that have an entertainment factor. / / I hear these kind of things, it engages me, and so I go try them in my classroom ? Being "of the moment" helps getting innovations into dissemination. ? Completeness of the "package" of how to use the innovation ? the Innovation description includes scenarios in which it might be applied ? clarity ? Good, concise description ? An unique idea with good planning and plenty of supports, samples, demonstrations, and personal contacts in teaching and learning available to interested groups. ? Structure of ideas. Well organized 10) Relevance to Job (3 in innovation ? 4 student: 7 overall) The ability of the pedagogical innovation to enhance the educator?s or student?s performance. ? For example, a strategy that gives an example where discussions are used to help students learn, is harder to implement in a math based engineering class. Question and answer, give and take is a great accomplishment in such classes. Rarely can I figure out a way to, or why I should have beginning students discuss whether they think the second law of thermodynamics is valid or not. / / So if the strategy 172 illustrates with something from my field, it resonates. / / Having said that, once I have an example, I also need to hear the underlying point of the strategy, what am I trying to teach, accomplish, etc. ? Professional relevancy- innovation is rooted in a documented need for the practice of engineering- use practicing engineers in an advisory practice to keep project grounded to necessary fundamentals ? the Innovation seems useful to my work and will make my job easier / -the Innovation clearly is going to improve an outcome that matters to me or my campus--i.e. retention, student success ? (Student) Professionalism- understanding of professional need that innovation addresses in preparation for professional practice ? (Student) Goal alignment - how innovation aligns with their goals ? (Student) Students are most accepting of innovations that will obviously benefit them either in their academic careers or even more so in their pursuits after their academic careers. ? (Student) Any methods of teaching and learning to students' concerns is potential job opportunities. 11) Theoretical Coherence (2 in innovation ? 2 in faculty: 4 overall) - The degree to which pedagogical innovations are based on a theoretical understanding of pedagogy or learning. ? Innovations that are based on theoretical understandings of pedagogy or learning provide a base knowledge about what factors are important for successful adoption and how to measure success. 173 ? Highly based on a strong literature review ? (Faculty) some awareness of the STEM education literature, at least in their discipline ? (Faculty) Faculty must be informed of best practices. Faculty must have an rudimentary understanding of learning theory. Characteristics of Students 1) Receptivity to change (18 in Student ? 3 in Innovation: 21 overall) ? The degree of receptivity from the students toward the pedagogical innovation. ? Open to new ideas ? Openness to change from standard lectures and problem sets ? The innovation needs to be seen by both faculty and students as one that will help them achieve something they want. If it is perceived as an attempt to impose a foreign goal on them, it will produce reactance. Thus, it needs to recruit the faculty and students to be co-innovators with those who propose the innovation. ? Openness to change ? receptivity to try new things ? Open mindedness ? Openness to new ideas and new ways of learning ? Eagerness to try new approaches ? Should be eager to learn, willing to follow different paradigms, in other words, get trained. ? open to new methods of learning ? Innovation always appeals to students willing to take chances 174 ? Openness to new approaches ? Openness to change - willing to change ? Open minded ? Students must be open to alternative pedagogies ? willingness to embrace change, open minded ? A motivated student groups that are not only to learn new technology but also willing to explores new teaching and learning methods. ? open to Learn new things ? (Innovation) The innovation needs to be seen by both faculty and students as one that will help them achieve something they want. If it is perceived as an attempt to impose a foreign goal on them, it will produce reactance. ? (Innovation) if students don't like the new idea (e.g. active learning that requires their participation) then they will complain, and other instructors will have one more reason not to change ? (Innovation) Resistance to adoption is directly related to difficulty. 2) Motivation to Learn (2 in innovation ? 13 in student: 15 overall) ? Motivation is defined as the act or process of motivating; the condition of being motivating; a motivating force, stimulus, or influence; incentive; drive; something (such as a need or desire) that causes a person or student to act (Merriam-Webster, 1997); and the expenditure of effort to accomplish results (DuBrin, 2008) ? (Innovation) Projects/missions that can ignite students? interest ? (Innovation) Student enthusiasm and interest in the new approach. I have experienced both disinterest and enthusiasm. The changes that students talk about, 175 and agitate for are noticed by faculty, who then become more open to the new approach ? Motivated to learn ? Motivation to learn the material ? Interest in the material of study ? Willingness to learn ? Motivated ? Should be eager to learn, willing to follow different paradigms, in other words, get trained ? Who would like to learn the concepts and apply them, they are not concerned about grades only ? Interest in the topic ? Getting students invested in a project is important? something they can be a part of, see their work contribute to the project, and perhaps see some reasonable results. ? Serious about learning ? Can learn how to focus on the process of learning, rather than just the ? Students involved on the project itself, willing to participate, students showing interest, open to learn new things, responsible and good attitude 3) Engagement (5 in student ? 3 in faculty: 8 overall) ? - The degree to which students study or practice a subject to learn about the subject? ? Students should be active toward the direction of education / / In most case, students are the end point of the classroom communication (absorbing the materials), and this 176 aspect used to make the student be inactive in/out of classroom prohibiting the successful dissemination. / In order to improve the dissemination of the educational innovations, I believe a students driven classroom environment assisted by professor should be made. ? - focused on learning rather than being focused on grades / / - willing to ask questions / / - willing to allow time for innovation to fully implemented before judging results. ? * Engagement with innovative product ? Engaged ? A willingness to engage with other students since the new approaches usually involve collaboration ? (Faculty) Ability to engage others in one's own research projects and innovations ? (Faculty) Willingness to engage with others -- both the developers, other adapters, and evaluators. ? (Faculty) The key probably is the interest and engagement of the faculty in educational research, ASEE, etc. 4) Cultural Differences ? The varying intrinsic and extrinsic attributes within and between dissemination groups; accounts for the differences between different, teachers, schools, ethnicities, and locations. ? Specific characteristics that depend on the innovation, e.g., fluency in spoken English for many small-group pedagogies ? At risk students. ? Want a diverse population of students to test innovation on so that one can gain a deeper understanding on how it works. 177 ? Ethnic/social background (diversity) ? Again, interest in the topic and relevancy to the contemporary economic/environmental/societal issues. ? If they reflect approved minority/special needs status, the innovation is given consideration. ? similar demographics as the original implementation of the Innovation 5) Learning Styles ? Accounts for the belief differences between different students, faculty, schools, ethnicities, and locations. ? It is important for students to have their personal approach towards problem solving? Model of learning. It is important to form a teaching philos ophy where there is a good mix of audio, visual and interactive learning. ? Students must be open to alternative pedagogies and possess the meta-cognitive traits that value improved problem solving, creativity, or complex thinking skills that generally accompany curricular innovations. ? For example, if instructors at a particular field are challenged with getting students to visualize better, educational innovation in this area has a better chance to be disseminated than other educational innovation. 6) Enhance Learning (2 in student) ? Should this be part of the student engagement construct? ? Will it advance their learning. ? Interest- innovation must stimulate / / Curiosity- interest in new and developing challenges 178 Characteristics of Faculty 1) Receptivity to Change (2 in innovation ? 16 Faculty: 18 overall) The degree of receptivity from the faculty toward the pedagogical innovation. ? (Innovation) The innovation needs to be seen by both faculty and students as one that will help them achieve something they want. If it is perceived as an attempt to impose a foreign goal on them, it will produce reactance. Thus, it needs to recruit the faculty and students to be co-innovators with those who propose the innovation. ? (Innovation) Resistance to adoption is directly related to difficulty. ? Open to new ideas ? Faculty should be open-minded for such dissemination ? Openness to change ? receptivity to try new things / ? open mindedness ? Openness/Willingness to try something new ? open-minded, willing to try new things ? Openness to new ideas ? Openness to innovations ? open, flexible ? Openness to new ideas ? openness to new ideas ? Some people are more amenable to change that others in any arena. Often those faculty most willing to change have both a deep-seated understanding of the 179 content impacted by the educational innovation as well as a realization of why the educational innovation will be of benefit. ? A willingness to try new things ? willingness to embrace change ? The key characteristic is a grounded but open-mind ? (Student). If an instructor believes his or her students are successful under the current methods, the instructor is less likely to change. 2) Care about student learning outcomes (9 in faculty ? 2 in administration: 12 overall) The degree to which faculty focus on the learning process and not the results. ? Care about student learning outcomes ? focused on learning rather than being focused on grades ? their personal desire to improve their students' learning and student success ? Training in effective teaching and learning / 3. Perspective on higher education ? Understanding of students' needs. ? Dissatisfaction with the current level of student learning and a commitment to this learning ? Faculty must have a fairly strong passion and interest in student learning and must be willing to commit substantial time and energy to develop, adapt, evaluate, and disseminate innovations in STEM. ? A devoted faculty who has a passion of teaching and learning and is willing to go xtra mile to ensure the integrity of the students? learning outcomes 180 ? Tolerance for early failure. Related to perseverance. But also to the earlier distinction of a focus on learning rather than performance. / / A culture that supports the scholarship of learning. ? focused on learning rather than being focused on grades ? leaders who make educational innovation/student success/excellent teaching a priority in pe 3) Awareness of Innovation (2 in Innovation ? 11 in Faculty: 13 Overall) The degree of awareness from the faculty toward the pedagogical innovation. ? (Innovation) Knowledge that others are applying the innovation ? (Innovation) Somewhat familiar to those to whom it is being disseminated ? Awareness of innovations through reading, conference attendance, word-of- mouth ? Learning of the advantages of the innovation. ? Faculty will participate if the work relates well to their own teaching or research, or if it presents an interesting diversion or advance in their work. ? 1. Amount of media attention, especially daytime TV. / 2. National/International status obtained usually by free press books or commercial textbooks. / 3. Offices held in international/national professional society. / ? Venues of dissemination ? Work in process (WIP) papers and conference proceeding can be useful tool for dissemination as can onsite workshops ? Willingness to use multiple dissemination methods: demonstrations at conferences, social media, magazines articles, blogs, YouTube, etc. 181 ? Willingness to present (disseminate) in multiple venues (conferences, invited talks, workshops,?) ? Participate to as many newsgroups, workshops, seminars, and conferences. ? Amount of media attention, especially daytime TV. National/International status obtained by free press books or commercial textbooks. ? Timeline definition. Identify journals and conference proceedings. Deadlines and requirements. 4) Professional Social System ? from Rogers (3 in innovation ? 5 in faculty: 8 overall) - ? Access to networks of probable adopters. One need not be famous in a field to have access to its networks. Electronic communication has helped to make professional networks more accessible, even to peripheral players who know how to take advantage of the network. ? Good network skills. ? Strong ties to other faculty within their own department as well as at other institutions ? The extent to which the professional associations in their discipline disseminate educational innovation ? Faculty who : / - talk to their peers in about their research from as small as their department colleagues to as large as a conference. / - Participate to as many newsgroups, workshop, seminars, and conferences / - Talk to people in different fields / - organize activities to talk about educational activities ? Willingness to engage with others -- both the developers, other adapters, and evaluators 182 ? The key probably is the interest and engagement of the faculty in educational research, ASEE, etc. ? Continues communication with the other PIs of the grant. Team work effort 5) Innovativeness ? A faculty member?s willingness to try out a pedagogical innovation. ? willingness to persevere through difficulties / curiosity / ability to see the potential in things, not just their immediate value / ? being informed / - having time to explore / - being willing to invest the time to figure out the application of and educational Innovation to a specific topic / - being willing to take the risk to try something new ? Creativity to come up with new ideas, energy to try them ? open, flexible, creative, passionate ? They should have a very balanced proportion of knowledge, belief in tradition, and a sense of adventure to exploit new directions. ? Curiosity. ? Innovativeness ? An open-mind, though, because some faculty genuinely identify legitimate pedagogical problems and have identified legitimate solutions and approaches to implement the solutions. 6) Motivation to be innovative (3 in innovation ? 5 in faculty: 8 overall) Rogers Characteristics of Innovators ? Early Adopters ? (Innovation) Being paid to test the innovation. ? (Innovation) Paying them seems to work, too (of course). ? (Innovation) Best way is to get preliminary data. Seed grant funding always helps 183 ? Motivated to employ evidence based practices ? Getting bored if I have to do the same thing, exactly, over and over again. ? Funds to disseminate ? High interest projects seem to be the key (at least for my grants). Faculty will participate if the work relates well to their own teaching or research, or if it presents an interesting diversion or advance in their work. / / Untenured faculty may be "easy marks," but I've found it doesn't always work well. / / Paying faculty...the same. My current co-PIs are driven by their own interest, the possibility for publications, and the fun of working on a project that may or may not work without our constant attention. Money is not the driver for them, for sure. ? The faculty needs to be motivated. A few motivators seem to be: / / 1) saving time; perhaps the innovation provides and idea or project that can easily be incorporated into another course. (I think this is often cited as a benefit of educational innovations, but I'm not sure it actually holds true in practice). / / 2) facilitating new experiences; perhaps the innovation allows the faculty member to branch out beyond their teaching approaches and learn something new that they can then *make* their own / / 3) short term gains; e.g., stipends to use curriculum seem to be quite helpful; the ability to publish findings seems to be marginally helpful. / / 4) social gains; it seems like a tightly knit community helps to spread and foster dissemination ? The innovation clearly is going to improve an outcome that matters to me or my campus ? i.e. retention, students success 7) Self-Efficacy (1 in Innovation ? 4 in Faculty: 5 Overall) The belief that one has the capability to perform the pedagogical innovation 184 ? (Innovation) The main characteristic, in my opinion, for successful dissemination is that the educational innovation is realistically "do-able" whether that mean that an innovation is affordable or practical. ? Literacy in education (many faculty, including me, never had official training in education). ? Confidence in their own teaching ability ? Confidence to promote one's own research project and innovations ? The confidence portrayed by the faculty is often picked up by students and sets their expectations from the course. 8) Value teaching in addition to research The degree to which the faculty balances the priorities of teaching and research. ? Willingness to keep learning about teaching. / / Valuing teaching in addition to research. ? Interest in education. Many faculty has research as primary role. ? the context of their institution- research vs. teaching emphasis ? Within that group, however, there are faculty who feel a strong commitment to their teaching and these instructors are most likely to work for change. ? recognition that teaching is of real value ? A devoted faculty who has a passion of teaching and learning and is willing to go xtra mile to ensure the integrity of the students? learning outcomes ? 1. Dedication / 2. Committed to improve education innovations, committed to work with undergraduate education 185 9) User Support (3 in Innovation ? 3 in Faculty: 6 Overall) The degree to which use of the pedagogical innovation is perceived as being voluntary. ? Support- provide logistical support to adopters: 3W's- workshops, webinars, websites ? comes with High level personalized training (like interactive workshops) / - mentoring program for adopters ? An unique idea with good planning and plenty of supports, samples, demonstrations, and personal contacts in teaching and learning available to interested groups ? Access to sources of information and support: faculty development professionals, colleagues who are experienced in educational innovations ? Willingness to seek adopters, answer questions, respond to criticism, engage with interested parties, continually refine innovation. ? Continued support for assistance with the innovation. 10) Tenure status ? Use as a Control Variable ? Tenure. One needs the security of tenure to be able to devote the time and resources to such things. ? Tenure status- less desirable for non-tenured faculty member to engage in introducing innovations- that's unfortunate but true ? Rank ? Untenured faculty may be "easy marks," but I've found it doesn't always work well 186 ? Time/job assignment/focus: Faculty whose primary focus is on teaching are most likely to adopt innovations. Those whose contracts commit only 25% of time to the teaching of 4 courses over the year are much less likely to change. ? Grounded because we don't want every "innovation" that comes along to be tested by wrong-minded faculty (i.e., junior faculty who simply want to put on their resume they did a course innovation, without really having a genuine interest in improving the learning experience for students). 11) Supportive faculty community The degree to which the faculty community supports and encourages continuous teaching improvements regardless of measureable or successful outcomes. ? Sense of support from the administration for risk-taking / - Supportive community that encourages continuous teaching improvement ? Belonging to a supportive community willing and able to try out others' innovations ? That would depend upon Cooperation, and the ability of the Faculty to understand and/or accept the process(es) that was undertaken during the development of the innovations. ? Finally, faculty needs to be supportive when an innovation does fail. Like any other research, this is all experimental and things are going to go wrong from time to time. Rather than being disparaging, faculty need to be collegial and supportive, recognizing that the typical genuine faculty member is only trying to make the educational experience better for students. ? encouraging interaction among the faculty regarding best practices in education 12) Attitude to Innovation ? The opinion that a person holds for the pedagogical innovation. 187 ? (Innovation) Finally, I am looking for imaginative things, those that use technology, or are strongly visual or even things that have an entertainment factor. / / I hear these kind of things, it engages me, and so I go try them in my classroom 13) Cultural acceptance of innovation ? The degree to which the cultural differences of the students and faculty influence the acceptance of the pedagogical innovation. Characteristics of the Administration 1) Management support (1 in Innovation ? 3 in Faculty ? 22 in Admin.: 26 Overall) ? The degree of support provided by institutional authorities (i.e. government, university administration, college deans, department chairs, etc.) ? (Innovation) the Innovation seems easy to implement or if the Innovation seems difficult/politically a nightmare to implement then some advice is presented about how to implement it ? (Faculty) Feel supported by chair / dean / ? (Faculty) Sense of support from the administration for risk-taking ? (Faculty) Training in effective teaching and learning. Perspective on higher education ? Release time for instructors to learn about innovations and to create or modify their courses ? In a research institution, they say they value teaching, but faculty are not generally valued for doing a good job teaching ? Tolerance for early failure. 188 ? Provide space for faculty to talk with each other about teaching and learning / + Provide workshops / instruction / peer mentoring / peer observation for innovative pedagogy / + Are vocal in support of educational Innovation ? Willing to allow time for innovation to fully implemented before judging results. ? Support for educational activities. Are there mentorship programs? ? creating an atmosphere of innovation in education / encouraging the faculty to try new things ? Support for travel and acceptance of travel and missed class time due to dissemination opportunities. ? Invest resources in buying faculty time to learn and experiment. ? Available support. ? Support for scholarship / research. ? Supportive- provide resources and time for introduction of innovations / / Encouraging- promote faculty engagement in innovation ? Making resources available as needed for adoption (teaching assistants, summer stipends, appropriate classrooms, funding for necessary materials and supplies) / - Support for travel to professional conferences and other institutions where faculty can learn about new ideas ? Moral support for innovators, Recognition of the effort required to disseminate and adopt innovations ? leaders who make educational innovation/student success/excellent teaching a priority in pe 189 ? 1. Leadership / 2. Mission and Vision / 3. Interest in development and support of innovation / 4. Interest in sustainability of innovations / 5. Awareness of higher ed issues ? Give faculty necessary support to implement their educational innovations ? Administrators should offer support and money, and stay out of the way. ? A practice of not punishing instructors that try new approaches even if the reward structure does not encouraged them to do so. ? Release time for the early stages of adoption and recognition of the effort to do so. ? Support interdisciplinary program / Support flexible program change ? The Administration must support educational innovations. Top down curricular projects that support faculty training seem to be less successful in the long run. 2) Promote and reward innovation (26 Overall) ? The degree to which the administration rewards and encourages innovation in the classroom. ? Promotion / tenure / merit systems that reward Innovation ? The reward system that the institution has for education (i.e. does this play an important role in T&P) ? providing incentive and resources to try new ideas and technologies ? Credit for such effort in T&P process ? rewards innovation in teaching ? Rewarding- rewards engagement in innovation ? Reward for innovation, risk-taking, and high-quality teaching ? Recognition of the effort required to disseminate and adopt innovations 190 ? rewarder or educational excellence ? A willingness to change promotion and tenure so that innovation in teaching is rewarded as a field of research ? A practice of not punishing instructors that try new approaches even if the reward structure does not encouraged them to do so. ? true encouragement of innovation, backed up by time and resources / - willingness to reward and recognize faculty who take the time to implement innovations ? Potential rewards to faculty and how the innovation aligns with the rewards ? Whether or not the college and university administration value educational innovation and are willing to recognize and reward faculty who adopt educational innovations. This can include course release, stipends, publicity, etc. ? Release time for the early stages of adoption and recognition of the effort to do so. ? Most educational innovations require substantial time, energy, and commitment on the part of the innovator. It also typically requires a collaborative effort of several educator practitioners. Those kinds of efforts and collaborations are not always rewarded in current workload or promotion and tenure considerations. One would not, or at least should not, expect faculty to engage in a sustained practice of innovation if it is not valued in terms of workload or P&T considerations. ? establish and maintain reward structure for facul ? Valuing of teaching--not necessarily awards and rewards, but clear, positive encouragement and strong faculty development programs 191 ? In a research institution, they say they value teaching, but faculty are not generally valued for doing a good job teaching. / / In think quality University education survives because faculty, individually, care about doing it. ? Promotion / tenure / merit systems that reward educational research (in addition to or in place of technical research ? accepting academic research as equally valid as scientific research ? culture/context of the institution and how much it values teaching/learning ? A willingness to change promotion and tenure so that innovation in teaching is rewarded as a field of research ? I believe that the reward structures of the university matter. If promotions and raises are based primarily on basic research, the faculty is likely to dedicate most of their time to research activities. If these rewards are also used to recognize outstanding teaching, than some faculty will respond to that. There is also a need to accept educational research as part of the research work that faculty may be meaningfully engaged in ? A sincere commitment to student learning reflected in a policy to include teaching effectiveness in the reward structure and a history of actually doing this ? demonstrate real, tangible record of valuing the teaching mission, not just "saying that teaching is important" 3) Facilitating conditions (2 in Innovation ? 4 in Faculty ? 15 in Admin: 21 Overall) ? The degree of availability of resources necessary to facilitate dissemination of pedagogical innovations (i.e. travel funds for conferences and presentations, infrastructure to support the use of the innovation, etc.). 192 ? (Innovation) Minimal requirements for equipment and technology ? (Innovation) accessibility of training ? (Faculty) adequate time among other responsibilities to adopt and adapt innovation locally ? (Faculty) Sufficient time in their prep schedule to disseminate and incorporate innovations ? (Faculty) time and resources to devote to innovation ? (Faculty) willing to commit substantial time and energy to develop, adapt, evaluate, and disseminate innovations in STEM ? Information technology resources, e.g., help constructing Web sites, video demonstrations / / * Travel funds to conferences for presentations ? Openness to new ways of teaching ? Support for travel and acceptance of travel and missed class time due to dissemination opportunities. ? The institution must be willing to give resources to projects that may not be producing results on the first - third round ? When I ran the NSF project, it was hard to make any progress with other universities because there are too much obstacles in between the schools such as money issues, curriculum, definitions, distance, communications, etc. / / we were trying to be out of the conventional education method (in-classroom condition) using the distance education facilities, but there were still lots of problems that we cannot handle (mostly equipment wise). ? Cost. 193 ? financial support to participate and organize seminars ? Generous support ? Administrators should offer support and money, and stay out of the way ? Financial and logistical support for new approaches even if the amount is very modest. ? true encouragement of innovation, backed up by time and resources ? Infrastructure to support use of the innovation ? Availability of resources. This includes well-equipped teaching labs, visual aids and access to any other resources relevant to the course. ? Not much really, beyond providing the faculty member with the necessary tools and resources (assessment instruments, time, etc.) to disseminate the results ? The administrations must be willing to provide unreserved support to faculty's teaching and students' learning success. There are too many institutions focus on funding and revenues but not willing to invest in new ideas. ? Supportive in regards to laboratory infrastructure for launching grant activities. / internal funding opportunities for preliminary data collection 4) Receptivity to Change (1 in Innovation ? 9 in Admin: 10 Overall) - The degree of receptivity from the administration toward the pedagogical innovation. ? (Innovation) Resistance to adoption is directly related to difficulty. ? I believe the systematic/administrative walls between institutes prohibit the successful dissemination of educational innovations. ? Openness to change ? receptivity to try new things 194 ? Moral support for innovators / Proactive role in encouraging faculty to adopt innovations promoted by others in the same department ? Open mindedness, cooperation, and collaboration ? Culture of innovation vs., holding to tradition ? willingness to embrace change ? There are too many institutions focus on funding and revenues but not willing to invest in new ideas ? Open for institutionalization of programs 195 Appendix 4-1. Characteristics of innovations and readiness variables that influence dissemination Characteristics Contextual Definition Characteristics of the educational innovation a. Relative advantage The degree to which a pedagogical innovation is perceived as being better than its predecessor (Hazen et al., 2012a; Rogers, 2003). b. Ease to implement The degree to which a pedagogical innovation is perceived as relatively easy to implement and adopt (Bourrie et al., 2014b). c. Ease of use The degree to which a pedagogical innovation is perceived as relatively easy to use and understand (Bourrie et al., 2014b; Compeau et al., 2007; Moore & Benbasat, 1991). d. Adaptability The degree to which the user can modify the pedagogical innovation as deemed necessary (Bourrie et al., 2014a). Readiness of faculty members' toward educational innovations a. Openness to change The degree of willingness to support organizational change and positive affect toward change (Miller et al., 1994). b. Discrepancy (Need for change) The degree to which faculty members think that something is wrong and needs to be changed (Armenakis et al., 1999). c. Appropriateness of change The degree to which faculty members feel that educational innovations are an appropriate reaction to the discrepancy (Armenakis et al., 1999). d. Efficacy of faculty member toward change The degree to which faculty members believe that they are capable of implementing the new educational innovation (Bandura, 1986). e. Support by principals to change The degree to which formal and informal leaders in the organization are committed to the successful implementation and routine use of educational innovation (Armenakis et al., 1999). f. Valence (Benefits from change) The degree to which faculty members perceive the personal attractiveness and personally gain from using the proposed educational innovation (Armenakis et al., 1999). g. Attitude to innovation The opinion that a person holds toward the pedagogical innovation (Agarwal & Prasad, 1999). h. Awareness of innovation The degree of awareness from the faculty toward the pedagogical innovations (Bourrie et al., 2014b). 196 i. Care about student learning outcomes The degree to which faculty focuses on the learning process and not the results (Hall et al., 1979; Hall & Hord, 2006). j. Motivation to innovate The wants, needs, and set of circumstances that drive a person to be innovative (Bourrie et al., 2014b; Bourrie et al., 2014a). 197 Appendix 4-2. Survey Dear participant, You are invited to participate in a survey for the research entitled ?Predictors of Dissemination Success of STEM Educational Innovations? which aims at identifying the factors that predict dissemination success of STEM learning innovations. This study is being conducted by a team consisting of Drs. Sankar and Jones-Farmer and Doctoral Candidate David Bourrie from the Auburn University College of Business. Dr. Sankar is Professor in the College of Business and has taught and researched in the area of Management Information Systems for more than 28 years. Dr. Jones-Farmer is an Associate Professor in the College of Business and has taught and researched in the area of Statistics for 13 years. Your participation is very valuable to the success of the research. The participation is voluntary, if you change your mind about the participating, you can withdraw at any time. If you decide to fill out the survey, you will be asked to answer questions related to new pedagogy materials/methods. The survey should take approximately 15-25 minutes to complete. Your input is anonymous; we collect your e-mail id to ensure that we only have one entry per person. The information you provide can be only assessed by the researchers and may be used in scholarly publications. If you have any question about this study, please contact Dr. Sankar at PH: 334-844-6504 or E- Mail: sankacs@auburn.edu or Dr. Jones-Farmer at PH: 334-844-6513 or E-Mail: joneall@auburn.edu or David Bourrie at PH: 334-844-6537 or dmb0011@auburn.edu. The Auburn University Institutional Review Board has approved this document for use from August 20, 2013 to August 20, 2016, Protocol # 11-245 EX 1108. If you have questions about your rights as a research participant, you may contact the Auburn University Office of Human Subjects Research or the Institutional Review Board by phone (334)-844- 5966 or e-mail at hsubjec@auburn.edu or IRBChair@auburn.edu. HAVING READ THE INFORMATION PROVIDED, YOU MUST DECIDE IF YOU WANT TO PARTICIPATE IN THIS RESEARCH PROJECT. IF YOU DECIDE TO PARTICIPATE, PROCEED TO THE NEXT PAGE, THE DATA YOU PROVIDE WILL SERVE AS YOUR AGREEMENT TO DO SO. Thank you for your time! Sincerely, Chetan Sankar L. Allison Jones-Farmer College of Business Advisory Council C&E Smith Associate Professor of Information Systems Professor of Statistics and Business Analytics sankacs@auburn.edu ajones@auburn.edu David Bourrie Doctoral Candidate Management of Information Technology and Innovation dmb0011@auburn.edu 198 Section 1: Educational Innovations Educational innovations can be defined as any new instructional material, strategy or pedagogy. Types of educational innovations may include the development of new curricula, instructional materials, and/or instructional strategies. Curriculum development deals with overarching changes planned or implemented for an educational program or curriculum. The development of faculty expertise refers to methods that can be used to help faculty members improve their teaching such as training workshops or mentoring. Instructional materials are specifically designed for use in a particular course or laboratory session, for example learning laboratories or electronic media. Instructional strategies include new approaches that enhance the transfer of knowledge, for example experiential learning or student centered learning. 1. Please briefly describe an educational innovation that you are currently using, would like to use, or plan to use in your undergraduate classroom. [text box] 2. URL where more information on this educational innovation is available: [Text box] 3. Would classify this educational innovation as: 1. Curriculum development 2. Development of faculty expertise 3. Instructional materials 4. Instructional strategies 5. Other 4. If other, how would you classify this educational innovation? [text box] 5. This educational innovation: 1. Is new 2. Been around for some time, but new to me 3. Been around for some time 6. Please enter your email address so we can ensure that this survey only have one entry per person. [text box] 199 Section 2: Characteristics of Educational Innovations 7. Please answer the following questions with respect to the educational innovation that you described above. Please answer the following items with this scale." Scale options ranged from 1 (strongly disagree) to 7 (strongly agree). Relative advantage (RA) 8. Using this educational innovation enables me to accomplish tasks more quickly. 9 Using this educational innovation improves the quality of my classroom performance. 10. Using this educational innovation simplifies my work tasks. 11. Using this educational innovation improves my teaching performance. 12. Overall, I find using this educational innovation to be advantageous in performing my teaching related aspects of my job. 13. Using this educational innovation enhances my effectiveness in the classroom. 14. Using this educational innovation gives me greater control over my work in the classroom. 15. Using this educational innovation makes me more productive in the classroom. Difficult to Implement (DI) (Reverse score for ease to implement (EI)) 16. It is difficult for me to transfer this particular educational innovation as it is used in other classrooms. 17. It takes a long time to incorporate this educational innovation into my classroom. 18. I can foresee that some problems will happen when this educational innovation is used in our current classroom facilities. 19. I have to spend a lot of my time learning how to use this educational innovation. Ease of Use (EU) 20. I believe this educational innovation is not cumbersome to use. 21. It is easy for me to remember how to perform tasks associated with using this educational innovation. 22. I believe that it is easy to get this educational innovation to do what I want it to do. 23. Overall, I believe that this educational innovation is easy to use. 24. Learning to operate this educational innovation is easy for me. 25. This educational innovation is user friendly. 200 Adaptability (AD) 26. This educational innovation is made to suit individual needs. 27. This educational innovation can be customized to satisfy each user. 28. This educational innovation appears to be very customizable. 29. This educational innovation can be highly customized for faculty and students. 30. This educational innovation could meet individual students? needs very efficiently. 31. The technology in this educational innovation makes it very customizable to meet students? needs. 32. The features of this educational innovation make it very adaptable to many students? needs. Section 3: Readiness of Faculty Members to Disseminate Educational Innovation Appropriateness (AP) 33. I believe this educational innovation will have a favorable effect on my teaching. 34. This educational innovation will improve the learning of my students. 35. This educational innovation is the right one for my students. 36. When I think about this educational innovation, I realize it is appropriate for the classroom. Efficacy of Faculty Members Toward Change (called Change Efficacy - Holt et al. 2007) (CE) 37. I do not anticipate any problems adjusting to teaching when this educational innovation is adopted. 38. I will do well when performing many of the tasks necessary to use this educational innovation. 39. When I implement this educational innovation, I feel I can handle it with ease. 40. I have the skills that are needed to make this educational innovation work. 41. When I set my mind to it, I can learn everything that will be required when this educational innovation is adopted. 42. My past experiences make me confident that I will be able to perform successfully when this educational innovation is adopted. Valence (VA) 43. This educational innovation will benefit me. 44. With this educational innovation, I will experience more self-fulfillment in my job. 45. I will earn higher pay from my job after adopting this educational innovation. 46. Using this educational innovation in my classroom will increase my feelings of accomplishment. 201 Care about student learning outcomes (CSLO) 47. I am concerned about my students? attitudes toward this innovation. 48. I am concerned about how this innovation affects my students. 49. I am concerned about evaluating the impact of this innovation on students. 50. I would like to excite my students about their part in this new innovation. 51. I would like to use feedback from students to shape the future use of this innovation. Attitude to Innovation (AI) 52. I like using this educational innovation. 53. This educational innovation is fun to use. 54. I enjoy using this educational innovation. 55. This educational innovation provides an attractive learning environment. Others? Use of this innovations ? used to measure awareness of the innovation (OU) 56. Several colleagues in my university use this educational innovation. 57. Several colleagues in other universities use this educational innovation. 58. Faculty members in my department use this educational innovation. 59. Within my college, I am aware of many people using this educational innovation. 60. Many people in other departments in my college are using this educational innovation. 61. Many of my friends use this educational innovation. Please answer the following questions regarding educational innovations in general. Openness to Change (OC) 62. I would consider myself to be ?open? to the changes that educational innovations will bring to my work role. 63. Right now, I am not resistant to the changes brought on by educational innovations. 64. I am looking forward to the changes in my work role brought about by the implementation of educational innovations. 65. In light of changes due to educational innovations, I am not reluctant to consider changing the way I currently teach. 66. I think that the implementation of educational innovations will have a positive effect on how I accomplish my work. 67. From my perspective, educational innovations will be for the better. 68. Educational innovations will benefit me in terms of how I teach my classes. 69. I think that the educational innovations will have a positive effect on how I perform my teaching role. Discrepancy (DIS) 70. We need to change the way we do some things in the classroom. 71. We need to improve the way we teach student in this college. 202 72. Educational innovations are necessary to improve our teaching effectiveness. 73. A change is needed to improve our teaching. 74. Educational innovations will prove to be best for my students. Faculty?s Perception of Principal Support (FPPS) 75. Most of my respected peers have embraced the use of educational innovations. 76. The deans and department chairs are ?walking the talk? in terms of encouraging educational innovations. 77. The majority of my respected peers are dedicated to making the use of innovations in the classroom successful. 78. My department chair supports me in the use of educational innovations. 79. My department chair is in favor of classroom innovations. Motivation to be innovative 80. The rewards that I received or will receive are dependent on my performance in the classroom. 81. I will be appreciated by my department chair if I perform very well in the classroom. 82. Regardless of my level, I will be rewarded, if I innovate in the classroom. 83. If I am innovative and successful in teaching, I will be highly rewarded. 84. Administrators increase my job responsibilities if I innovate in the classroom. Section 4: Dissemination Intention to Adopt (Individual level) (IAI) 85. I intend to use this educational innovation in my classroom. 86. I intend to use this educational innovation whenever appropriate in my classroom. 87. Please indicate the likelihood that you will use this educational innovation in the near future. (anchored by 1 ? very improbable and 7 ? very probable) Intention to Adopt (System level) (IAS) 88. I believe that others in my field intend to use this educational innovation in their classrooms. 89. I believe that others in my field intend to use this educational innovation whenever appropriate in their classrooms. 90. I belief that the likelihood that others in my field will use this educational innovation in the near future. (anchored by 1 ? very improbable and 7 ? very probable) Actual Adoption (Individual level) (AAIND) 91. Please select the statement that best describes your use of this educational innovation 1 = I currently use all or part of it 2 = I have used all or part of it in the past 203 3 = I am familiar with it, but have never used it 4 = I?ve heard the name, but do not know much else about it Actual Adoption (System Level) (AASYS) 92. Please select the statement that best describes the use of this educational innovation by others in your field. 1 = Most have tried it 2 = Many have tried it 3 = Some have tried it 4 = Few have tried it 5 = I am not aware of others that have tried it Modification 93. To what extent did you modify the educational innovation from its original form? 1 = I developed it myself. 2 = I did not develop it, but I used it basically as described by the developer. 3 = I did not develop it, but I made some relatively minor modifications to it. 4 = I used some of the ideas from the developer, but made significant modifications. 5 = I am not familiar enough with the developer?s description to answer this question Routine Use (Individual Level) (RUI) 94. How often do your use all or part of this educational innovation? 1 = I always use it 2 = I usually use it 3 = I sometimes use it 4 = I rarely use it 5 = I never use it Routine Use (System Level) (RUS) 95. Have others in your field used all or part of this educational innovation? 1 = Most regularly use it 2 = Many regularly use it 3 = Some regularly use it 4 = Few regularly use it 5 = I am not aware that others regularly use it 204 Section 5 : Demographics Demographic questions (Currently 17 questions) 96. Which best describes your job responsibilities? 1 = Primarily teaching 2 = Teaching accounts for more than half my responsibilities 3 = Teaching accounts for about half my responsibilities. 4 = Teaching accounts for less than half of my responsibilities. 97. What is your gender? 1 = Male 2 = Female 98. Would you describe yourself as: 1. American Indian / Native American 2. Asian 3. Black / African American 4. Hispanic / Latin 5. White / Caucasian 6. Pacific Islander 7. Other 99. What is your highest earned degree? 1 = Undergraduate 2 = Masters 3 = PhD 100. What is the name of your university or college? [text box] 101. Which department are you in? 1. Aerospace 2. Architectural 3. Biomedical & Agricultural 4. Biomedical 5. Chemical 6. Civil 7. Civil/Environmental 8. Computer Engineering 9. Computer Science (inside Engineering) 10. Computer Science (outside Engineering) 11. Electrical 12. Electrical/Computer 205 13. Engineering (general) 14. Engineering Management 15. Engineering Science & Engineering Physics 16. Environmental 17. Industrial/Manufacturing 18. Mechanical 19. Metallurgical & Materials 20. Mining 21. Nuclear 22. Petroleum 23. Other 102. If other, what department are you in? [text box] 103. Which best describes your current position? 1 = Full-Time, Permanent 2 = Full-Time, Temporary 3 = Part-Time, Permanent 4 = Part-Time, Temporary 104. Which best describes your current position. 1 = Tenured 2 = Tenure-Track but not yet tenured 3 = Non-Tenure Track 4 = University does not have a tenure system 105. Years you have been tenured? (if tenured) [sliding number range or text box] 106. How many total years of faculty teaching experience do you have? [sliding number range or text box] 107. How many years have you been a member of the faculty at your current institution? [sliding number range or text box] 108.What is your current rank? 1 = Lecturer 2 = Assistant Professor 3 = Associate Professor 4 = Full professor 5 = Other 109.If other, what is your rank? 206 [text box] 114. In the course in which you are most likely to use educational innovations, what is the approximate per course enrollment? [number of students] 115. Are you a member of any of the following professional organizations? 1 = Association for Computing Machinery (ACM) 2 = American Society for Engineering Education (ASEE) 3 = Institute of Electrical and Electronics Engineers (IEEE) 116. Do you have any comments that you would like to add about any of the issues covered in this survey? [text box] Section 6: Assessment of Common Method Bias Please answer the following four questions." Scale options ranged from 1 (strongly disagree) to 7 (strongly agree). 110. I prefer green to other colors 111. I like the color green 112. I like green clothes 113. I hope my next car is green 207 Appendix 4-3. Approval letter from the Human Subjects Institutional Review Board