Development and Testing of Innovative Instructional Materials to Improve Student Learning in Engineering Classes - Case Studies, Smart Scenarios and Serious Games by Pramod Rajan A dissertation submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama December 14, 2013 Keyword: case studies, smart scenarios, serious games, learning outcomes Copyright 2013 by Pramod Rajan Approved by P. K. Raju, Chair, Thomas Walter Distinguished Professor of Mechanical Engineering Chetan S. Sankar, College of Business Advisory Council Professor of Information Systems Dan Marghitu, Professor of Mechanical Engineering David Beale, Professor of Mechanical Engineering ii Abstract Most mechanical engineering courses require students to work on over-simplified theoretical representations of real-world problems. Although this gives students an in-depth understanding of concepts and principles, they do not learn to apply these theories to solve the complex and multi-dimensional practical problems that they will need to deal with once they enter the workplace. These courses also seldom engage and motivate students. There is therefore a need to develop new instructional materials to improve student learning outcomes in engineering education. This study investigated how best to develop and test the effectiveness of case studies, smart scenarios and serious games to teach engineering concepts and improve student learning outcomes for undergraduate mechanical engineering students. The multimedia case study on identifying welding defects developed for this research study was tested with both undergraduate and graduate mechanical engineering students. The majority of the students considered the case studies to be beneficial and an effective way of linking mechanical engineering concepts to real-world issues. However, several students noted that the case studies occasionally lacked student interaction/immersion and were too rich in technical content, possibly making them overly complex for all those enrolled in freshman engineering classes. This led to the development of smart scenarios, which the majority of the students found to be both realistic and novel. They commented that the smart scenarios made them read through the material and they liked the gaming aspect of it. However, the student iii responses also indicated that the smart scenarios were tedious at times and problems needed to be broken down further to teach simpler concepts in the relatively short classroom time available. They suggested that the smart scenarios would benefit from more gaming functionality. This led to the development of a serious game to teach engineering concepts. This novel approach to teaching engineering concepts was developed in partnership with Toolwire Inc. The effectiveness of using a serious game to teach the concept of engineering design process was tested using the Presage-Pedagogy-Process-Product (4P) model in a control/experimental setting. Hierarchical multiple regression analysis was applied to investigate the inter-relationships among presage factors (gender, race and learning styles), instructional materials (pedagogy) and gains in higher order cognitive skills, concentration, student enjoyment, goal clarity (process variables) and improvement in achieving learning outcomes (product variables). All the students in the experimental group who worked with the game achieved better learning outcomes, had higher performance scores in a pasta tower design challenge, and higher perceived concentration levels. In focus group sessions, students commented that the serious game helped them understand the effect of different shapes and structures when presented with the practical challenge of designing and building pasta towers. iv Acknowledgments I would like to thank my major professor and chair of my dissertation committee, Dr. P. K. Raju, for his guidance and encouragement throughout my doctoral program. I am immensely grateful to him for providing me with such excellent facilities to conduct my research and for his trust in my ability to play a full role in his research activities. I also thank him for providing me with opportunities to work on other research projects conducted in the Laboratory for Innovative Technology and Engineering Education (LITEE) during my time at Auburn University. I would also like to thank Dr. Chetan S. Sankar for his valuable guidance and help with the evaluation model and for motivating me in the development and completion of my dissertation. I particularly wish to acknowledge and express my sincere gratitude to my committee members, Dr. David Beale and Dr. Dan Marghitu, for agreeing to serve on my committee and the support they have given me throughout my graduate studies. I would also like to thank Dr. Judith Hudson for serving as the outside reader on my committee. Dr. Barbara Kawulich, Kim Huett from the University of West Georgia, Carrollton, Georgia and Dr. Nesim Halyo and Dr. Qiang Le from Hampton University, Hampton, Virginia, have all provided invaluable help and guidance in testing and evaluating the innovative instructional materials discussed in this study. Dayvid Jones, Michael Watkins and Steve Lynch v from Toolwire Inc. provided me with excellent technological support and expertise in the development of the smart scenarios and serious game. Joseph McIntyre, Eliza Banu, Justin Bond, Ashok Manoharan and Kun-Yen Wang have all contributed to this research and I would like to thank them for their valuable support in conducting this project. Every research project relies on the generous contributions of time and effort by others, and without the input from all of those named, this project could not have reached a successful conclusion. Thanks are also due to all the mechanical engineering students who took part in this research project for the enthusiasm with which they worked with me on testing the instructional material and for patiently responding to all the surveys and focus groups. Without them, this project would not be a success. I would like to thank Auburn University and the National Science Foundation for providing me with financial assistance throughout my graduate studies at Auburn. Finally, I would like to thank my parents, P. Rajan and K. Selvakumary, and my brothers, R. Vinod and R. Radhakrishnan, for their continued support and encouragement, which have helped me to complete my PhD work. This research study is based upon work supported, in part, by National Science Foundation under Grants IIP # 1110223 and EEC# 0934800. Any opinions, findings, and conclusions or recommendations expressed in this study are those of the author and do not necessarily reflect the views of the National Science Foundation. viii Table of Contents Abstract ......................................................................................................................................... ii Acknowledgments ....................................................................................................................... iv List of Tables ................................................................................................................................ x List of Figures ............................................................................................................................. xii Chapter 1. Introduction ............................................................................................................... 1 Statement of the Problem ................................................................................................ 3 Research Questions .......................................................................................................... 5 Significance of the Study .................................................................................................. 5 Organization of the Study ................................................................................................. 5 Chapter 2. Literature Review ....................................................................................................... 7 Factors Affecting Student Learning .................................................................................. 8 Learning outcomes ............................................................................................................ 8 Multimedia case studies .................................................................................................. 10 Effectiveness of using case studies in achieving learning outcomes .................. 11 Smart Scenarios .................................................................................................................... 13 Serious Games ...................................................................................................................... 15 Effectiveness of using serious games in achieving learning outcomes .............. 17 Summary ............................................................................................................................... 21 Chapter 3. Development and Analysis of Multimedia Case Study Methodology ...................... 22 Purpose of this Study ...................................................................................................... 22 viii Research Questions ......................................................................................................... 22 Development of Automatic Weld Inspection Case Study .............................................. 23 Development of Induction Welding Case Study ............................................................ 36 Classroom implementation ............................................................................................. 39 Evaluation and Findings ................................................................................................ 41 Qualitative feedback ....................................................................................................... 47 Learning Styles of Engineering Students ........................................................................ 47 Summary ......................................................................................................................... 53 Chapter 4. Development and Analysis of Smart Scenarios ........................................................ 54 Purpose of this Study ...................................................................................................... 54 Research Questions ......................................................................................................... 54 Development of Smart Scenario ..................................................................................... 56 Classroom implementation ............................................................................................. 62 Evaluation and Findings ................................................................................................. 63 Summary ......................................................................................................................... 64 Chapter 5. Development and Analysis of Serious Games .......................................................... 65 Purpose of this Study ...................................................................................................... 65 Research Questions ......................................................................................................... 65 Design and Development of Serious Game .................................................................... 67 Classroom Implementation and Evaluation .................................................................... 77 Results and Findings from Serious Game Implementation ............................................ 96 Summary ....................................................................................................................... 111 Chapter 6. Conclusion ............................................................................................................... 112 viii Synopsis ........................................................................................................................ 112 Review of Research Findings ....................................................................................... 113 Implications................................................................................................................... 116 Limitations .................................................................................................................... 117 Suggestions for Future Research .................................................................................. 117 References ................................................................................................................................ 119 Appendix A. Detailed TAM for engineering design smart scenario ....................................... 132 Appendix B. Scenario outline/rationale for the engineering design smart scenario ................ 137 Appendix C. Detailed storyboard for engineering design smart scenario ............................... 144 Appendix D. IRB for engineering design smart scenario ........................................................ 152 Appendix E. Design flowchart of a typical building block ..................................................... 157 Appendix F. Progression matrix for the design serious game ................................................. 160 Appendix G. Design game outline ........................................................................................... 162 Appendix H. PowerPoint Presentation for Toolwire developers and managers ....................... 175 Appendix I. Storyboard for the design serious game ............................................................... 186 Appendix J. Detailed user flow experience for the design serious game ................................ 191 Appendix K. IRB for the design serious game ........................................................................ 196 Appendix L. Lecture on engineering design process ................................................................ 200 Appendix M. Statistics applied to data analysis ...................................................................... 205 ix Appendix N. Survey.................................................................................................................. 209 Appendix O. Pasta tower building activity .............................................................................. 218 Appendix P. Design process questions asked in the midterm ................................................. 224 x List of Tables 1. Analysis of past research on multimedia case studies in engineering field ........................... 12 2. Short summary of past research on serious games in engineering education ........................ 19 3. Summary of mean scores on learning constructs ................................................................... 42 4. Mean ratings to evaluation I as per case study ....................................................................... 46 5. Mean ratings to evaluation II as per case study ...................................................................... 46 6. Average score of learning style preferences of engineering students for spring and fall 2010. ........................................................................................................ 48 7. Summary of the project plan dates for the development of the smart scenarios ................... 62 8. Production milestones and schedule for the development of the design serious game??.. .71 9. Proposed measures and their Cronbach Alphas ..................................................................... 94 10. Mean comparisons of pasta tower design performance scores for gender and race ............ 96 11. Mean comparisons of concentration for gender and race .................................................... 97 12. Mean comparisons among several outcome variables of interest ........................................ 98 13. Interaction effect between gender and learning modules .................................................... 99 14. Interaction effect between race and learning modules ....................................................... 101 15. Interaction effect between learning styles and learning modules ...................................... 102 16. Interaction effect between student enjoyment and learning modules ................................. 103 17. Interaction effect between goal clarity and learning modules ........................................... 104 18. Interaction effect between concentration and learning modules ........................................ 105 xi 19. Interaction effect between higher order cognitive skills (HOCS) and learning modules .................................................................. 106 20. Most beneficial aspects of the design serious game .......................................................... 109 21. Summary of the research findings ...................................................................................... 113 xii List of Figures Figure 1. Toolwire product portfolio .......................................................................................... 15 Figure 2. Recovered patch plate weld from failed Tank 7 .......................................................... 25 Figure 3. The scene following the explosion ............................................................................. 25 Figure 4. Weld radiograph with ripples ..................................................................................... 27 Figure 5. Weld radiograph with ripples and faint indication of a lack of fusion defect ............ 28 Figure 6. Block diagram of an Automatic Weld Defect Recognition (ADR) system ............... 29 Figure 7. Block diagram of the proposed algorithm .................................................................. 31 Figure 8. Weld radiograph with the weld area and the region of interest .................................. 32 Figure 9. Pre-processed image of the region of interest showing the lack of penetration defect. ............................................................................................ 32 Figure 10. Image obtained after applying the proposed algorithm ............................................ 33 Figure 11. Induction welding operation ..................................................................................... 37 Figure 12. The number of students who benefitted from the case study ................................... 43 Figure 13.Visual representation of active-reflective learning style dimension ......................... 49 Figure 14 Visual representation of sensing-intuitive learning style dimension .......................... 50 Figure 15 Visual representation of visual-verbal learning style dimension ............................... 51 Figure 16 Visual representation of sequential-global learning style dimension ......................... 52 Figure 17. Screenshot of engineering design smart scenario ..................................................... 55 xiii Figure 18a & 18b. Different ways of interpreting the same data ............................................... 57 Figure 19. Toolwire Assessment Mapping (TAM) .................................................................... 58 Figure 20. Example of Storyboard .............................................................................................. 58 Figure 21. Smart scenario development process ........................................................................ 59 Figure 22. Example of a serious game showing its basic components ...................................... 69 Figure 23. Design process used in the game .............................................................................. 72 Figure 24. Introductory screen of the design serious game ....................................................... 73 Figure 25. Serious game user flow experience .......................................................................... 74 Figure 26. Screen shot of the lab introduction ........................................................................... 75 Figure 27. Screen shot of the building game level ..................................................................... 76 Figure 28. Screen shot of the water bridge (left) and train bridge (right) levels ....................... 77 Figure 29. 4P model with learning modules as the moderating variable ................................... 80 Figure 30. Illustration of Hypothesis 1 ....................................................................................... 90 Figure 31. Gender and pasta tower design performance scores: EDLM 1 Versus EDLM 2 .... 100 1 Chapter 1 Introduction Most mechanical engineering courses require students to work on over-simplified theoretical representations of real-world problems. Although this experience gives the students an in-depth understanding of the concept and principles, they are not trained to link the theories to solving the type of practical problems that they will occur in real life. These courses also seldom engage and motivate students (Ohland et al., 2008; Coller & Shroff, 2009). The problem of student motivation persists even today in higher education and may be a factor in issues such as student underachievement and retention (Ambrose et al., 1997). "To teach is to engage students in learning" (Christensen, 1991). Researchers agree that students need to be engaged in learning throughout their professional lives in order to perform effectively in an unknown real-world future and instructors must tailor their courses with that in mind (Bowden & Marton, 1998; Smith et al., 2005). The Accreditation Board for Engineering and Technology (ABET) has called for improved student learning outcomes relating to problem solving and real-world skills through the design of better engineering courses at universities (ABET, 2011). It has also asked instructors to try out innovative instructional methodologies or techniques in their courses to achieve the required learning outcomes. Falkenberg (2005) stressed the need for new instructional pedagogies to be developed in order to utilize information technology more effectively in engineering classrooms. The greater the level of student involvement or engagement in academic work or in the academic experience 2 of college, the greater his or her level of knowledge acquisition and general cognitive development (Pascarella & Terenzini, 1991). Both researchers stress the need for new studies on innovative learning practices and instructional methodologies to identify their impact on students' learning and to determine whether they could help address issues like student engagement and retention. Hence, this study introduces three innovative teaching methodologies, namely multimedia case studies, smart scenarios and serious games, in order to examine their potential utility in a mechanical engineering classroom. Multimedia case studies have traditionally been used as an effective instructional technique to demonstrate how real-world decisions are made, enabling students to understand the way technical needs, safety factors, financial goals and credibility issues are simultaneously considered and weighted (Shaha, 1998; Vazsonyi, 2002). Dym et al. (2005) described how case studies can be used effectively to teach engineering principles. Educational games have the potential to address a number of systemic deficiencies for five reasons: their massive reach, effective learning paradigms, enhanced brain chemistry, time on task and improved learning (Dabbagh & Menasc?, 2006; Mayo, 2007). Serious games have been proven to improve student motivation and engagement and to achieve specific learning outcomes while at the same time teaching engineering concepts (Coller & Scott, 2009; Hauge & Riedel, 2012; Okutsu et al., 2012). Initial studies have shown that serious game teaching effectiveness is 30% greater than the classic lecture method of teaching (Mayo, 2009). There is an abundance of literature showing that serious games contain the pedagogical elements needed to enhance student learning and skills (Alavi et al., 2002; Coller & Scott, 2009; Gee, 2003; Prensky, 2005, 2006) Basken (2009) reported that the Obama administration has allocated $260 million to improve student achievement in math and science through serious games and specially designed 3 television programs. The nation?s annual investment in educational technology tripled between 2002 and 2011 to $429 million (DeSantis, 2012) and the American government has earmarked $10.5 million for the development of serious games for training purposes to support better decision making (Raytheon Company, 2011). About 8,000 papers can be identified that have described the positive impacts of games on users over the past 14 years alone. Of these, only about 130 papers reported empirical evidence concerning their impacts on learning and engagement (Thomas M. Connolly et al., 2012). With recent advances in innovative instructional techniques and increasing investment in the development of serious games, it is therefore important to carry out an effective evaluation of these instructional tools to determine whether they are indeed beneficial for students. The purpose of this study was thus to investigate how best to develop innovative instructional materials like multimedia case studies, smart scenarios and serious games to teach mechanical engineering concepts. The second purpose of this study was to investigate the effectiveness of these innovative instructional materials in improving student learning. The focus throughout has been on engineering students. Statement of the Problem There is a lack of innovative instructional material to teach mechanical engineering concepts that improves student learning. This study examines the development of three types of innovative instructional materials, multimedia case studies, smart scenarios and serious games, and evaluate their effectiveness in achieving specific learning outcomes such as student performance, engagement and attitude toward engineering. The retention rate of college freshmen returning for their second year has declined between 2004 through 2008, (NCHEMS, 2008). Faculty members play a major role in student retention and can help maintain a positive learning 4 environment for students by engaging them with new multimedia technology and innovative instructional techniques such as cooperative and collaborative learning in the classroom (Lau, 2003). Today's students? desire new and innovative instructional technologies to help them become engineers but many instructors have failed to adapt to this changing educational environment. Prensky (2005) pointed out that our students are no longer the people our educational system was designed to teach. They are becoming less engaged with the education system. There has been a call for significant breakthroughs in understanding how students learn engineering so that our undergraduate and graduate programs can adequately prepare engineers to meet the needs of our changing economy and society (National Science Foundation, 2009). To address the above issues, instructors need to make their content more engaging and motivate their students, thus improving student learning and retention. Learning is more effective when it is active, experiential, problem-based and situated, providing immediate feedback (Boyle et al., 2011). Raju et al.(1999; 2004) and Sankar et al. (2010; 2001, 2011; 2008) have shown that multimedia case studies can serve as an effective pedagogical tool to improve student learning outcomes. Experiential learning has been proven to be effective in teaching engineering topics. Connolly et al. (2012) conducted a systematic literature review to show the positive impacts and outcomes of serious games with respect to learning and engagement. There is lack of research to determine the effectiveness of these instructional methodologies in mechanical engineering classes. The study presented here will show how to develop, implement and evaluate effective and innovative instructional methods to improve student learning and learning outcomes of a course. 5 Research Questions 1. How should innovative instructional methodologies, such as multimedia case studies, smart scenarios and serious games be developed and implemented for mechanical engineering classrooms? 2. How should the effectiveness of each of these innovative instructional methodologies in improving student learning be evaluated? Significance of the Study To date, there has been very little research in the field of mechanical engineering education to determine whether innovative instructional methodologies can be used to improve student learning and the learning outcomes of a course. Different universities can benefit by trying out any of the methodologies discussed in this study to teach mechanical engineering concepts in a course to improve student learning and the delivery of the instructional material to improve student engagement. The contributions of the author to this study are (1).the development of induction welding and automatic weld inspection case studies, their class implementation and evaluation of their impact on student learning and (2).the development, implementation and evaluation of the effectiveness of serious games in improving learning of engineering design concepts. Organization of the Study This dissertation is organized as follows. Chapter Two consists of a literature review which explains the factors affecting student learning and also looks at various instructional techniques that can be used to produce effective learning outcomes. It includes a description of all three types 6 of innovative instructional techniques developed and evaluated in this study. The effectiveness of both multimedia case studies and serious games in achieving learning outcomes are then reviewed based on previous research. Chapter Three presents the development and analysis of the multimedia case study methodology. The case studies developed during this study are discussed, along with the classroom implementation of the case studies and the evaluation and findings from the implementation. Chapter Four consists of an analysis of the second instructional technique, smart scenarios, focusing particularly on their development, which was based on the feedback and evaluations obtained for the case studies described in the previous chapter. The classroom implementation of the smart scenarios is presented, along with the evaluation results and findings from this implementation. Chapter Five provides an analysis of the serious games approach to teaching mechanical engineering concepts. The development of the serious game is described in detail and the classroom implementation and development of the evaluation model discussed. Both quantitative and qualitative data are analyzed and the results presented. Finally, Chapter Six provides a synopsis of the research and its findings, and discusses the implications of the study, its limitations and suggestions for future research. 7 Chapter 2 Literature Review Introduction There is a clear need for a shift in the paradigm of how engineering is being taught at universities. Lamancusa et al. (2008) suggested that this paradigm should be more industry- partnered, interdisciplinary and involve real-world problem solving. A result panel discussion among technology innovators and investors at the Milken Institute revealed that over the past decade almost $ 3 billion has been invested in the educational technology sector and over a $ 1 billion was raised in loan last year to improve education technology and create innovative and better learning processes (Nordin, 2013). Today?s industries combine the fun and engagement of video games with the academic rigor of high quality training and personnel development to create innovative simulations and solve business problems. Employees trained using simulation games have a 14% higher attainment of procedural knowledge and 20% better self-efficacy rate (www.gamessciencegroup.com). Knowing that technology can improve the student learning process and given the need for new instructional techniques, this study focuses on evaluating the effectiveness of innovative instructional techniques (multimedia case studies, smart scenarios and serious games) in improving student learning and the delivery of the instructional material in different mechanical engineering courses. The focus of the study is the development, implementation and evaluation of the above mentioned instructional techniques when used to teach mechanical engineering concepts in a classroom setting. 8 Factors Affecting Student Learning Engineering education researchers have defined student learning in many ways. Subject- based learning, cooperative learning, problem-based learning and cross disciplinary learning are different types of learning processes through which students acquire knowledge (Smith et al., 2005). Students learn most effectively when they are actively engaged in the learning process (Bonwell & Eison, 1991; Sivan et al., 2000; Smith et al., 2005). Students understand certain concepts, theories and skills in a specific way, which forms the backbone for learning a particular subject (Meyer & Land, 2003). Factors that exist prior to engagement in learning have been shown to produce varying levels of influence on the learning process. Learning styles, self-efficacy, team working skills, and problem solving skills all assist student learning (Minotti, 2005). In recent years, researchers have begun to emphasize the importance of knowledge skills and attitudes associated with teamwork and interpersonal skills. There is now a demand for improved interpersonal and team working skills in both engineering and business graduates from accrediting bodies such as ABET (Accreditation Board for Engineering and Technology) and AACSB (Association to Advance Collegiate Schools of Business). Team-based learning has been shown to produce positive effects on academic achievement, which then result in self-directed learning in engineering (Prince, 2004). Learning outcomes Learning outcomes are best measured in terms of the skills and knowledge gained by the students during the process of learning. Recently, student learning assessment has become a significant area in the field of mechanical engineering education. For example, if a student 9 performs well in a hands-on lab/project by applying the concepts learned in the course, it is clear that the student has learned something during the class period provided they do not have prior knowledge about the concepts. There has been a call to incorporate sound assessment techniques into all educational programs from accreditation boards, government and industry (McGourty et al., 1998). The ABET EC 2000 guidelines provide specific learning outcomes for engineering graduates, commonly referred to as "a-k". These are: a) the ability to apply knowledge of mathematics, science, and engineering; b) the ability to design and conduct experiments, as well as to analyze and interpret data; c) the ability to design a system, component, or process to meet desired needs; d) the ability to function on multidisciplinary teams; e) the ability to identify, formulate, and solve engineering problems; f) an understanding of professional and ethical responsibility; g) the ability to communicate effectively; h) the broad education necessary to understand the impact of engineering solutions in a global and societal context; i) a recognition of the need for, and ability to engage in, lifelong learning; j) a knowledge of contemporary issues; and 10 k) the ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. According to the ABET criteria, the measurement of student learning outcomes in a systematic and valid manner should be the focus of any institution's assessment efforts (McGourty at al, 1998). Communication, teamwork and a good understanding of ethics and professionalism have been termed "professional/process skills" and engineering?s global and societal context, lifelong learning and knowledge of contemporary issues are referred to as "awareness skills" (Shuman et al., 2005). The three instructional techniques discussed in this study will attempt to improve some of the student learning outcomes and help satisfy the ABET criteria. Innovative instructional techniques Multimedia Case Studies A case study is typically a record of a technical and/or business issue that has actually been faced by managers, together with the complex web of facts, opinions, and prejudices upon which management decisions have to depend in real-world situations. These real and particularized cases are presented to students for considered analysis, open discussion, and a final decision as to the type of action that should be taken. The course of action decided on by the students is then generally compared with the real-world action actually taken by the managers and engineers and the results of their implementation discussed. The fundamental principles underlying the case study method of teaching as summarized by Barnes et al. (1994) are: 1. The primacy of situational analysis: The analysis of a specific situation forces the student to deal with ?as is? and not the ?might be.? 11 2. The imperative of relating analysis and action: The traditional academic focus has been on knowing, while the practitioner focuses on action. The case study method of instruction seeks to combine these two activities. 3. The necessity of student involvement: The active intellectual and emotional involvement of the student is a hallmark of the case study method. That involvement offers the most dramatic visible contrast with a stereotypical lecture class. 4. A nontraditional instructor role: The instructor?s role is not so much to teach students as to encourage learning. His/her role is more of a facilitator and he/she must be both a teacher and a practitioner. 5. The development of an administrative point of view: The students develop an understanding of the problem from a holistic point of view and not from an engineer?s perspective (Raju & Sankar, 1999) Effectiveness of using case studies in achieving learning outcomes Case studies have traditionally been used to show that real-world decisions must be taken in the context of a company?s financial goals, technical needs, safety factors and credibility issues, all of which must be simultaneously considered and weighed (Shaha, 1998; Vazsonyi, 2002). For the past fifteen years, the Laboratory for Innovative Technology and Engineering Education (LITEE) at Auburn University has been producing case studies in engineering, business and technology areas and implementing them successfully them at Auburn and other universities (www.litee.org). Students react very positively to the use of these case studies in the classrooms. The case study methodology involves a great deal of interaction and is based as closely as possible on the situation as experienced by the individuals who actually dealt with the problem. Table 1 12 shows a summary of the previous research that has been published on the use of multimedia case studies in engineering classrooms and their effectiveness in achieving learning outcomes. The papers listed in the table generally engaged in mixed method analyses to show that multimedia case studies are indeed an effective instructional tool that improves learning outcomes such as higher order cognitive and team working skills. Multimedia case studies have also been shown to be a particularly effective instructional tool for female and minority students. Table 1. Analysis of past research on multimedia case studies in engineering field Author Information Title of the paper Type of Study Student learning outcomes The effect of case study on learning outcomes Positive Negative Mixed Mehta et al.(2007) Impact of multi- media case studies on improving intrinsic learning motivation of students Quantitative Perceived improvement in higher order cognitive skills X Chetan S. Sankar and P.K.Raju (2001) Use of multi-media courseware to teach real-world Decision Mixed method Skill development, self- reported learning, intrinsic learning and motivation, communication skills X Chetan S. Sankar et al. (2008) Use of case studies in engineering education: assessment of changes in cognitive skills Mixed method Cognitive skills X Chetan S. Sankar and P.K.Raju (2011) Use of Presage- Pedagogy-Process- Product model to assess the effectiveness of case study methodology in achieving learning outcomes Mixed method Higher order cognitive, skills, improvement in self-efficacy improvement in team working skills X Chetan S. Sankar et al. (2010) Developing leadership skills in introduction to engineering courses through multi-media case studies Mixed method Higher order cognitive skills, improvement in attitude towards subject matter, improvement in team working skills, impact on future work environment X 13 Smart Scenarios Smart Scenarios give students the chance to experience the ?real world? context in which technical decisions are made and tasks are assigned. Students are immersed in a virtual world where they interact with a series of characters that help to give life to a specific scenario. The end result of these interactions is that the student will be equipped with context and technical knowledge before being asked to perform ?hands-on? tasks on live equipment. Interactive scenarios are fully customizable and can be crafted for any course specific ?day-in-the-life? scenario or case study that applies to the subject matter. The hardware or software environment that is packaged with the scenario is also fully customizable and wholly integrated for a seamless experience that captures the users? interest and imagination and fosters the transfer of knowledge through hands-on activities. Learnscapes Toolwire is a leading company providing educational software. The newest product in their experiential learning portfolio is Learnscapes, a software package that allows students to step into a photo-realistic real world environment where they are surrounded by video-enabled characters Jeannie L. Pridmore, Randy V. Bradley, Nikhil Mehta (2010) Methods of Instruction and Learning Outcomes: A theoretical Analysis of Two Approaches in an Introductory Information Technology Course Mixed method Learning interest and motivation, evaluation, value of opinions of others, application, comprehension X P.K.Raju and Chetan S. Sankar (1999) Teaching Real-World Issues through Case studies Mixed method Useful, Attractive, Clear and Challenging X 14 with which they interact to gain information and solve problems as they work their way through the free form scenario. Each learnscape is carefully created with subject matter experts to ensure that both the environment and the characters with whom the student interacts are authentic, helping to create the most realistic learning environment possible. Learnscapes employ contextual learning and natural assessment to allow students to demonstrate their mastery of the subject matter. They also use a series of checkpoints and remediation steps to assess a student?s progress toward the final assessment and to ensure their exposure to pertinent learning objectives. Learnscapes may be delivered as either a standalone learning environment or coupled to create a capstone learnscape, in which learnscape ?episodes? appear each week as students? progress through a larger scenario environment. All information provided by the student in the assessment elements is captured and formatted for delivery to the course instructor for grading. Figure 1 shows Toolwire's portfolio and the spectrum of solutions they offer. 15 Figure 1. Toolwire product portfolio Serious games Introduction The commercial gaming industry is moving towards providing more education-related games that can potentially benefit higher education. At the same time, education is moving towards gaming, trying to identify the areas that can help improve student engagement, cognitive skills and retention and also improve the delivery of instructional material. (Kearney and Pivec, 2007, Quinn, 1997, p.1, Garris et al., 2002, Kearney, 2005 and Klingberg, Forssberg, & Westerberg, 2002). Just as videos, film, and even books have done in the past, video games are becoming a part of the 16 educational process. Video games offer a flexible, non-linear, learner-directed approach to learning that will become even more important in the global business and industrial environment of the 21st century. No matter how important they become, however, serious games will not replace teachers, professors and other educational facilitators. Instead, these games will become a part of the new educational toolbox. The evolution of new teaching paradigms that fully utilize video games is already underway and this will only accelerate as more and more of the so-called video game generation become teachers and professors. The power of collaboration between the gaming industry and professional educators has the potential to not only raise the bar but to move the gaming industry into an area that is more financially secure. The informed application of game based learning in teaching and learning contexts has the potential to enrich, enhance and in some cases transform the educational experience of learners. Definition of serious games Abt described serious games thus: Games may be played seriously or casually. We are concerned with serious games in the sense that these games have an explicit and carefully thought-out educational purpose and are not intended to be played primarily for amusement. This does not mean that serious games are not, or should not be, entertaining. (1987, p.9) Serious games are games or game-like interactive systems that have been developed with game technology and design principles for a primary purpose other than pure entertainment. There is still very little solid evidence of the effectiveness of games in the classroom and how serious 17 games compare to more traditional teaching methods, but although research in this area is still in its infancy, some of the early results suggest this approach shows real promise and demonstrates the potential benefits of learning through games. Nolan Bushnell, the founder of Atari and the father of the video game industry says, ?People who play video games have much better computational skills, much better logic skills, much better search and cognitive skills than kids who don't,?(Oppenheimer, 2013). The BBC?s factual entertainment department pointed out that ?People learn through games. 99% of boys and 97% of girls aged between 12-17 play video games?. They also noted that 65% of teachers were interested in the use of games in the classroom (Project Tomorrow, 2008). Serious games can be used for education at all levels, from preschool and elementary school, through middle school and high school, into higher education, and even into the job market. One game does not have to support all of these levels, but some might be able to. Prensky (2005) argued that games are good for two things. First, there are particular techniques or attributes of games that can help students learn complex material faster and understand that material better. Second, games can increase the level of engagement of the trainees so that they want to play the game and they want to learn how to successfully complete the game. Effectiveness of using serious games in achieving learning outcomes Educational gaming addresses the ABET criteria by engaging students in the learning process while meeting the following learning objectives: 3e: Ability to solve and define problems 3h: Understand the impact of engineering solutions in a global and societal context 18 3i: Recognition of the need for lifelong learning A search of the Chronicle of Higher Education archives shows over 100 articles that mention ?game-based learning? or ?games in the classroom? in the last year alone. Albers et al. (2009) stated that engineering concepts cannot be taught sufficiently in lectures alone. Students benefit from a more active learning experience. Hernandez and Davila (2010) discussed the need to develop proper engineering design skills in the student prior to the project experience, and stress the need to use educational theories (teaching styles, learning styles, etc.) to develop these skills. Many studies have demonstrated the use of simulations to improve student learning outcomes (Canon-Bowers, 2006). Educational games allow a deeper understanding of both content and concepts (Prensky, 2005). As skills and abilities are attained, the player advances through the game and increments their knowledge (Kearney and Pivec, 2007). It is often falsely assumed that the game itself will be powerful enough to cause change or learning and that the outcomes will be used automatically for decision making, but this is seldom the case (Bekebrede et al., 2005). Pivec et al. (2007) reported that 70 % of the students thought a course was successful and enjoyed taking part in the game to learn about a particular topic. By improving students? auditory and visual digit span, and thus their auditory and visual processing, their academic function relative to grade level will also improve (Jaquith, 1996). Role-playing can include all the engagement, immersion, and motivation that are inherent in the game environment (Linser, 2008). Teachers and trainers do not yet understand the use and potential of games and are thus not confident in their ability to integrate the games into their regular lessons to achieve the desired learning outcomes (Pivec, Koubek, & Dondi, 2004; Pivec, 2008). The objective is not to turn the teachers into computer game players, but to encourage them to select and implement suitable games that will support their educational objectives (Becker, 2007). 19 Unless the correct game is chosen for the selected topic and appropriate moderation and debriefing by the teacher is forthcoming, the desired learning outcome will not be achieved (Mayer and Bekebrede, 2006). The study presented here in Chapter 4 will discuss the integration of serious games into a freshman mechanical engineering course and show how it can best be implemented to achieve a specific learning outcome set by the course. Many of the prior studies in this area suffer from severe flaws related to researcher bias, short exposure time, and the lack of a control group and integration with previous research (Egenfeldt-Nielsen, 2007). It was therefore decided that the current study would expose the students to traditional lectures, active learning exercises and a serious game in order to evaluate the effectiveness of the serious game in achieving specific learning outcomes. Table 2 shows a short summary of the research papers on serious games in the engineering field and their effectiveness in achieving learning outcomes. Table 2. Short summary of past research on serious games in engineering education Author Information Type of study Type of game Perceived learning outcomes The effect of game based learning on learning outcomes POSITIVE NEGATIVE MIXED Li et al.(2013) Empirical Game-like learning systems(Simul ation-based game) Perceptions of challenge, algorithmic thinking skills, motivation and learning behavior X Lee at al. (2006) Mixed method Simulation- based game Self-achievement, self-paced and peer to peer interaction X Foss and Eikaas (2006) Empirical Dynamic simulator combine with educational game Learning experiences X Sanchez et al. (2011) Empirical Simulation- based games Score a final test X Minovic et al.(2011) Empirical Digital game The comparison of final mark based on different personality X 20 Chen et al.(2011) Empirical Game-based learning system Knowledge test (+), intention(-) and satisfaction(+) X Andres et al. (2011) Empirical Educational games in different supports Perception of students feeling for the game X Richardson et al. (2011) Mixed methods Simulation- based game Definition identification(-), sampling strategies(-), combination of statistical and mathematical analysis into procedure(+) and communication skills(-) X Coller and Shroff (2009) Empirical Serious game with simulation method Student engagement X X Saenz & Cano (2009) Empirical Simulation- based game Learning effectiveness in motivation X Al-Jibouri at al (2005) Case study Simulation- based game Final score from game ( project planning and control abilities) X Connolly et al.(2007) Conceptu al Computer games Improve on learning experience NO EVALUATION Dabbagh & Menasce (2006) Mixed method Computer- based market game Engineering related skills(-), business management skills(+) and professional skills(+) X Coller and Scott (2009) Case study Computer game Perceived importance of course X Fu et al.(2009) Empirical E-learning games Enjoyment, concentration, goal clarity, challenge X Hainey et al.(2011) Empirical Computer game Play the game over a prolonged period of time, Engaging, desire to play the game again X Gutierrez et al.(2009) Empirical Video game Spatial abilities of engineering students X Cagiltay (2007) Mixed method Computer game Problem solving, independent learning, learning by doing, application of previously learned knowledge X 21 Summary This chapter reviewed the literature on the need for new instructional methodologies, providing an overview of the factors affecting student learning and learning outcomes. In particular, the need for new instructional methodologies to teach mechanical engineering concepts was discussed and the three instructional techniques, multimedia case studies, smart scenarios and serious games that will be developed and analyzed in this research study were defined. A summary of the past research on case studies and serious games and how they affect learning outcomes was summarized in Tables 1 and 2. 22 Chapter 3 Development and Analysis of Multimedia Case Study Methodology Purpose of this Study The purpose of this study is to show the development of multimedia case studies for undergraduate and graduate mechanical engineering classes and to discuss the classroom implementation and evaluation of the case studies. Research Questions 1. How should multimedia case studies be developed and implemented for mechanical engineering classrooms? 2. How should the effectiveness of multimedia case studies in improving student learning be evaluated? Introduction One of the instructional techniques developed and tested in this research study is the multimedia case study methodology. The main purpose of developing these case studies is to teach mechanical engineering concepts to first year engineering undergraduate students and also to demonstrate the application of these concepts to solve real-world issues. This chapter presents the development, classroom implementation and evaluation details of using case studies in mechanical engineering classes. The next few sections will describe the development of 23 automatic weld inspection and induction welding case studies. The classroom implementation and the evaluation of these case studies will then be discussed. Development of Automatic Weld Inspection Case Study Introduction The Non-Destructive Evaluation (NDE) Imaging Lab at the John F. Welch Technology Center (JFWTC) in Bangalore, India, conducts advanced research in the areas of imaging for non-destructive evaluation modalities such as ultrasound, electromagnetics and X-rays, primarily for defect detection and characterization. Most NDE modalities offer either a one-dimensional signal, a 2-D image or a 3-D stack of images. The lab utilizes core competencies in the areas of NDE modalities, signal/image processing and analysis, and pattern recognition and correlation in accomplishing advanced inspection goals related to defect detection and characterization in industrial components and infrastructure. Physics-based modeling and simulation for NDE plays a vital role in understanding how various modalities (such as X-ray, eddy current, ultrasound and optics) can be used for industrial inspection applications (NDT Imaging Lab, 2013) By working with the management team at the research center, the project team identified three specific educational objectives for this case study: ? highlight the importance of weld testing in the real world and the after-effects if a weld failure occurs; ? provide information on the various types of welding defects and also suggest a process to identify welding defects in certain limiting cases; and give students an opportunity to learn 24 about non-destructive testing and the evaluation of welds using the image processing techniques typically used in automatic weld defect detection systems. Importance of the Problem: Need for Inspecting Welds Faults in welding can lead to the loss of equipment and, more importantly, lives. For example, the explosion and fire that devastated the Marcus Oil facility in Houston, Texas, in December 2004 resulted from faulty welds in a pressure vessel. Chemical Safety Board (CSB) investigators determined that the failed vessel, known as Tank 7, had been modified by Marcus Oil to install internal heating coils, as had several other pressure vessels at the facility. Following coil installation, each vessel was re-sealed by welding a steel plate over the two- foot-diameter temporary opening. The investigation revealed that Marcus Oil did not use a qualified welder or proper welding procedure to reseal the vessels and did not pressure test the vessels after the welding was completed. The investigation also showed that the weld used to close the temporary opening on Tank 7 failed during the incident because the repair weld did not meet generally accepted industry quality standards for pressure-vessel fabrication (Figure 2) (Environmental Safety and Health Advisory, 2006) 25 Figure 2. Recovered patch plate weld from failed Tank 7 The original flame-cut surface was not ground off the plate edges before the joint was re- welded, and the weld consequently did not penetrate the full thickness of the vessel head. Furthermore, the welds contained excessive porosity (holes from gas bubbles in the weld). These defects significantly degraded the strength of the weld. The fire spread back into the damaged tank and caused a violent explosion, which propelled the 25-ton vessel more than 150 feet, where it came to rest against a warehouse on an adjacent property (Figure 3) Figure 3.The scene following the explosion 26 A similar accident that highlights the importance of weld testing occurred in 1997 when the Williams-Renault racing team was put on trial for manslaughter under Italian law. They were accused of being responsible for what the prosecutor said was a faulty steering column weld on their racing car. Welding is a very large industry, and expenditure on weld testing in the U.S manufacturing, mining and construction industries totaled about $34 million for the year 2000 (American Welding Society, 2002). General Electric Aircraft Engines in Madisonville, Kentucky, produces 2.5 million radiographs per year, of which 70% are weld radiographs. The market for GE Inspection technologies X- ray testing machines is $100 million, of which 80% are utilized for testing welds. The power sector produces more than 500 welds per day. Bharat Heavy Electricals in India produces more than 1000 pipe welds each day for their boilers, nuclear reactors, and other critical applications. It is clearly necessary to inspect these welds without destroying them; non-destructive testing is vital for ensuring the quality of the welds. As size and weight decrease and the factor of safety is lowered, more and more emphasis is placed on better raw material control and higher quality of materials, manufacturing processes and workmanship. A producer of raw material or a finished product frequently does not improve quality or performance until that improvement is demanded by the customer. Pressure from the customer leads to improved design or manufacturing and non-destructive testing is frequently called on to deliver this new quality level. Non-destructive tests are used to determine the direction, amount, and gradient of stresses in mechanical parts, as applied in the field of experimental stress analysis. These play a very important role in the design of lighter, stronger, less costly and more reliable parts. 27 Problem Statement The main problems with the weld radiograph images faced by the research center were to detect defects in the presence of weld ripples and also when the defects were faint. The three types of defects identified were: lack of fusion, lack of penetration and scattered porosity. The goal was to develop an algorithm to utilize in the Automatic Defect Recognition (ADR) system. An exhaustive research of the literature related to weld ADR revealed no commercially available system that could automatically detect faint defects in the welds. Figures 4 and 5 show examples of the types of weld defect that the research center needs to be able to detect. Figure 4.Weld radiograph with ripples 28 Figure 5.Weld radiograph image with ripples and faint indication of a lack of fusion defect Solution: Automatic Defect Recognition (ADR) System Figure 6 shows a block diagram of an ADR system. The processes used in an ADR system are: preprocessing, segmentation of the defects, feature extraction, and classification of the defects. Each of these processes will be described in turn below, after which a proposed algorithm to improve the process will be presented. 29 Figure 6. Block diagram of an Automatic Defect Recognition (ADR) system(Jacobsen & Zscherpel, 1999) Preprocessing: Preprocessing prepares the acquired raw digital image for the defect detection stage by reducing noise, correcting for background trends (shading correction) and removing geometric structures that otherwise would adversely affect the defect-detection stage. Noise reduction (frame averaging, mean filter, median filter) and contrast enhancement (contrast stretching and histogram equalization) are normally performed as part of this process (Kehoe & Parker, 1990) Segmentation of the defects: The techniques used to identify objects of interest are usually referred to as segmentation techniques as they segment the foreground from the background. Morphological processing, background subtraction, profile-anomaly detection, segmentation by thresholding, edge detection, template matching and matched filters are some of the methods used for segmentation (Bovik, 2010) 30 Feature extraction: When the input data for an algorithm is too large to be processed and is suspected of being highly redundant (a great deal of data, containing relatively little information), then the input data will be transformed into a reduced representation set of features known as a features vector. Transforming the input data into a set of features is referred to as feature extraction. If the features extracted are carefully chosen, it is expected that the features set will extract the relevant information from the input data in order to perform the desired task using this reduced representation rather than the full dataset (Bovik, 2010) Extraction of defect features is one of the steps involved in weld ADR prior to defect classification, where the defects are measured. One measurement is the value of any sizeable property of the defect. A feature is a function of one or more measurements, which are registered in the computer to define the size of any significant characteristic of the defect identified. Features that serve as classifier data inputs include location, shape, length, density, aspect ratio, and roundness. Geometric feature extraction methods include techniques such as edge detection, corner/interest point detection, curve fitting or local curve estimation, model based feature detection, region detection, and feature extraction using textures. Classification of the defects: In the defect-classification stage, the defect pixels identified from the previous stage are grouped into connected regions (connectivity analysis) and their characteristics quantitatively measured, after which they are classified into different types using an expert system, and finally a pass/fail decision is made based on the defined inspection criteria. The results can be stored in a database and used for production process improvements. Artificial neural networks (ANN), fuzzy systems, and non-linear classifiers are some of the methods used to classify the defects (Bovik, 2010) 31 Proposed Algorithm Figure 7 provides an algorithm that can be used to improve the process described above. Figure 7. Block diagram of the proposed algorithm The steps in the algorithm are: ? The given digital radiographic image is analyzed and the section that does not contain the welded part is cropped to remove unwanted information. ? The image of the weld is filtered and contrast-enhanced to reduce noise and enhance the defect. ? The pre-processed image is then divided into sub-images. ? Region-growing is applied to the sub-images to focus in on the defects. ? The processed sub-images are concatenated to obtain a final image. The new algorithm was used to analyze the radiographs of multiple welds to identify 32 whether it did in fact improve the identification of faults in welds. Figures 8 and 9 show a weld radiograph image of the weld area and the region of interest containing the defects, respectively. Figure 9 reveals several defects in the weld that are not normally identifiable using traditional methods. This algorithm was used to analyze 33 radiographs and was found to be 90% efficient in detecting the defects in these radiographs. Figure 8 Weld radiograph with the weld area and the region of interest Figure 9 Pre-processed image of the region of interest showing the lack of penetration defects 33 Figure 10 shows the output image with the defects obtained after applying the algorithm. Figure 10 Image obtained after applying the proposed algorithm Outline of the Case Study and Assignments Based on a case study development format provided by the LITEE development team, a case study based on the process involved in creating the automatic weld inspection procedure described above was developed for this study. The resulting multi-media case study contains the following clickable tabs: ? Overview ? Problem Statement ? Objectives ? Credits ? Weld ADR o Introduction to ADR 34 o Need for Weld ADR o Background ? Problem o Introduction o Types of Defects o Earlier Solutions o Proposed Algorithm o Results ? Tools o Basics of ADR o Basics of Welding o Basics of Image Processing o NDT and Imaging o Glossary o References ? Assignments o Assignment 1 o Assignment 2 ? Search ? Sitemap A glossary section provides information on the technical terms used in the case study. An assignment section provides specific task for the students, who have been divided into three teams. The students are divided into teams for this assignment and discuss the following scenarios: Group A: Represents a team at the company and is required to critique and comment on the proposed algorithm 35 Group B: Represents another team at the company tasked with discovering flaws in the algorithm and coming up with new ways to detect defects. Group C: Represents a third team at the company whose members are asked to look at past research on automatic defect recognition for welds and determine the most feasible NDT approach for this application. Group D: Represents a team at the company charged with researching other types of defects in welds and determining whether the proposed algorithm also works for those types of defects. Group E: Is asked to conduct a feasibility study on the methods used for automatic weld defect detection and find out the best method suitable for the given images. The case study is now available for use in classrooms and can be obtained from www.liteecases.com. The next section describes the development of the induction welding case study. This case study is a product of a collaboration between the Laboratory for Innovative Technology in Engineering Education (LITEE), Auburn University, Auburn, AL, the Centre for Nondestructive Evaluation (CNDE) at the Indian Institute of Technology (IIT), Madras, India, and the Nondestructive Testing Laboratory (NDTL), Bharat Heavy Electricals Limited (BHEL), Tiruchirappalli, India. LITEE: LITEE was launched in 1997 and consists of faculty and students from the Colleges of Engineering and Business at Auburn University (www.litee.org). The team works with industrial partners to identify a suitable problem and bring it alive in the classroom by creating a multimedia case study. This is then tested for pedagogy and 36 content with faculty and students at different institutions. Indian Institute of Technology Madras: The IIT Madras is among the foremost Indian centers for both higher technological education and basic and applied research. The Institute hosts Centre for Non Destructive Evaluation (CNDE), a world leader in NDE research, education, training, and information. Bharat Heavy Electricals Ltd.: Bharat Heavy Electricals Limited (BHEL) is the largest engineering and manufacturing enterprise in India in the energy- related/infrastructure sector. BHEL manufactures over 180 products under 30 major product groups and caters to core sectors of the Indian economy, including power generation and transmission, transportation, telecommunications, and renewable energy, among others. The greatest strength of BHEL is its highly skilled and committed workforce of 42,600 employees. Development of the Induction Welding Case Study Today, many industrial processes utilize electromagnetic induction welding as an energy- efficient way to heat and process electrical conductive materials. BHEL welds thousands of kilometers of metal tubes and pipes every day destined for use in critical applications such as high pressure boilers. Consequently, the company requires the inspection of these welds to be both highly reliable and rapid. During welding, the component reaches temperatures of around 1300? C. The welds must then undergo a post-weld inspection before being passed as satisfactory. To perform a normal ultrasonic inspection using shear waves, the weld must cool to room temperature, after which any defective welds must be either repaired or reworked. In practice, the inspection time is longer than the welding time due to the need to cool the component completely in a controlled fashion before the inspection can be carried out. Since this process involves the mass production of pipes, the production process becomes stagnant at the inspection stage. The company 37 therefore asked the team at CNDE to carry out a feasibility study to determine the most suitable NDE method that could be implemented to reduce the inspection time and improve the production process. Figure 11 shows the actual work area where the induction coil surrounds the pipe to be welded. Figure 11 Induction welding operation Learning Objectives for the Case Study ? Explain the induction pressure butt welding process and related issues ? Underline the importance of in-situ weld inspection ? Understand the NDT techniques used for in-situ inspection of induction pressure welding ? Analyze the possible alternatives applicable to the problem Three main problems affected the inspection stage of the induction pressure welding process. The first problem involved the lengthy inspection time needed for the welds. The time taken for the cooling and inspection was considerably longer than the time for set up and welding, resulting in significant delays that BHEL was seeking to minimize. The second problem encountered by the operators at inspection was the inaccessibility of welds, with some 38 areas of piping being difficult to reach for inspection. The third problem was with the detection of pasty welds or kissing bonds. When the crack surfaces stay in very close contact with each other, the bond between the two surfaces of the crack is called a kissing bond. The welds appear to be fused and will pass an ultrasonic inspection but then fail at the work site. A lack of bonding and mismatches are the types of defects most commonly appearing on these welds. A Short Summary of the Case Study Mr. R. J. Pardikar, the Senior Deputy General Manager for the Quality/NDT division at BHEL, explained the three problems faced at the inspection stage of the induction welding process to the team working on this problem. He was not sure which would be the most suitable method to address all these problems due to the many new non-destructive testing methods now being developed. Working with all the partners in the collaboration, he asked the team to conduct a feasibility study of the various non-destructive testing methods that might be suitable and/or develop new methodologies applicable to the problems described above. The team identified three possible NDT methods, namely infrared thermography, ultrasonic testing, and digital radiography, which were then considered in more detail by conducting a feasibility study. A multimedia case study was developed during fall 2007. The multimedia case study CD- ROMs make it possible for students to visualize the case study problem and in some cases even hear the voices of those charged with making the original decisions. Photos and videos of the machinery and equipment in the actual plants were included. Assignments were provided in the case study CD-ROM. The students were divided into teams to discuss the following scenarios: Group A: Represent a non-destructive testing-infrared thermography team at BHEL. Defend 39 infrared thermography as the most suitable solution for the problems stated in the case study. Group B: Represent a quality assurance-ultrasonic testing team at BHEL. Defend ultrasonics as the most suitable solution for the problems stated in the case study. Group C: Represent a non-destructive testing digital radiography research team at BHEL. Defend digital radiography as the most suitable solution for the problems stated in the case study. Classroom Implementation The case study was used in an undergraduate/graduate level mechanical engineering course during fall 2007. The class was comprised of 12 students. A questionnaire was administered to reveal the perceptions of the students regarding the case study experience. Over two class sessions, the instructor discussed the NDT techniques and introduced the case study problem. Then, two further class periods were devoted to an analysis of the material on the CD- ROM by the students and to allow them to develop their presentations. The students then presented their recommendations for addressing the problem in a presentation. The students in Group A, who played the role of the nondestructive testing-infrared thermography team at BHEL, defended infrared thermography as the most suitable solution for the problems stated in the case study. Their argument was based on three points: 1. Since the environment is hot, the detection of defects becomes very easy using this method. 2. Faster inspection and defects are easily accessible. 3. They also researched other new alternative NDT methods including Nanometric Laser Profilometry, Laser- Scanned Penetrant Inspection (LSPI), Laser Shearography, and Computed Tomography (CT), and demonstrated that overall, thermography is a good 40 solution. The students in Group B played the role of the quality assurance ultrasonic testing team at BHEL and defended ultrasonics as the most suitable solution for the problems stated in the case study. Their arguments were based on the following points: 1. Ultrasonic guided waves, or lamb waves, are a proven way of detecting kissing bonds or pasty welds. 2. Through transmission does not require a waiting period for the weld to cool down and is both cheap and quick. 3. Much less technician training is required since the ultrasonic shear wave method is currently utilized. 4. A wireless probe would solve the accessibility problems. 5. The group used a decision support matrix to perform a feasibility study and compare it to the other two alternatives, showing the superiority of their recommendation. The students in Group C represented the non-destructive testing digital radiography research team at BHEL and thus defended digital radiography as the most suitable solution for the problems stated in the case study. Their arguments were based on the following points: 1. Real-time radiography (RTR) is the best form of digital radiography and is eminently suitable for this kind of problem. 2. With RTR, the image is available almost simultaneously as the radiation passes through the part. Lag time is less than one second. The inspection area can be shifted and an entire part inspected in seconds. 3. Although kissing bond detection is not possible in RTR, the group provided alternatives such as stressing specimens during imaging to create a space that can be detected, 41 improvements in micro- electronic sensors, and technological advances in nanotechnology. Evaluation and Findings from case study implementation A common criticism of using new methodologies for teaching is that their effectiveness is not measured. Therefore, the effectiveness of using the induction welding case study was evaluated by asking the students to complete a questionnaire that included both quantitative and qualitative questions (Raju & Sankar 1999). Four students reported having only school experience of the engineering field. Three students had experience with engineering through co-ops or internships, two students reported having work experience in the engineering field, and three students had experience both as interns or co-op students and through employment in the field. A quantitative analysis of the results is shown in Table 3. It shows that the students reported more than average (3.0) satisfaction on all the constructs, which indicates that the students found the case study to be relevant in learning the subject materials and that it improved their higher-order cognitive skills, encouraged a positive attitude toward engineering, reduced their negative attitudes toward engineering, and improved their team building and communication skills. The qualitative analysis (Figure 12) showed eight students found the use of case studies to be beneficial, particularly because of the group work and applicability to real life situations. One student stated that the use of case studies had been somewhat beneficial to his/her learning in the course. Eight students indicated a preference for working in groups, while two students preferred to solve problems alone. One student reported a preference for working alone, then with a group later, while another student indicated no preference for either working alone or in groups, commenting that either was fine with him/her. 42 Table 3. Summary of Mean Scores on Learning Constructs Scale: 1 ?Strongly disagree; 3 ? Neither agree nor disagree; 5 ? Strongly agree Two students thought that the information from this course would be helpful in their future work, while two students stated that the course information will help them in their future engineering work. Another two mentioned that the topic of the case study, defect testing, was an important aspect of the course that would be useful in the future, while one student commented that information from this course would help him/her in the biomedical field. Another student considered that this course would help him/her in the use of NDT methods. One student responded ?probably all applications? would be helpful. Three students indicated that they found the presentations most interesting, while two students singled out the acoustics and testing as being most interesting to them. Two other students liked the practical application of course concepts, and another two students found the NDT methods to be most interesting. One student mentioned the case studies as most interesting, while two other students indicated that everything was interesting to them. Four students found the use of presentations (both PowerPoint and student presentations) to be most helpful. One of these students also mentioned the combination of lecture and PowerPoint presentations as being the most helpful to his/her learning. Two students found the Variable N Mean Standard Deviation General interest in subject 12 3.2 0.36 Relevance of subject 12 4.2 0.72 Cognitive skills 12 3.7 0.66 Positive attitude 12 3.4 0.45 Negative attitude 12 2.8 0.56 Team building 12 3.7 0.71 Communication 12 3.4 0.86 43 case studies to be the most helpful aspect of the course, while other students found homework (1 mention), independent projects (1), and examples in class (1) to be most helpful to their learning the material. One student indicated that speaking in front of the class had improved his/her communication/speaking skills, and another student noted that everything in the course had been helpful to his/her learning process. Figure 12. The number of students who benefitted from the case study In response to the question: ?How helpful did you find the use of student groups/teams in solving the problems presented in the case studies?? three students did not respond to the question. The other nine felt that the use of groups or teams to solve problems had been helpful. Three students noted that working in groups generated more ideas than working alone. Other reasons students gave for finding the use of groups helpful were that it gave them a greater level of confidence, increased their understanding of the subject, improved their teamwork skills, enhanced their decision making, and made the work more applicable to future work environments. 44 Student reactions to the case study Students generally found the use of case study beneficial, as shown by some of their comments: ?It was helpful in seeing a real world example of how topics learned in class can be applied in real-world [situations].? ?It was very beneficial as it uses ?real-world? experiences to show the importance of applying these technologies to industry.? ?I enjoy being able to review the material at my own pace. Detailed images are helpful.? ?The case studies were really useful in knowing the problems and other issues that the real world is facing. And it helps us to relate to the things that we learn in the class.? ?It was very beneficial through gaining information in one on one experience with classmates. It was difficult in some ways to defend an idea that was not our own.? ?I believe using lectures and PowerPoint presentations are great. However, using hands-on case studies and group projects to apply the knowledge taught in these is the best way to teach analytically thinking engineers.? 45 Evaluation of STS-51L challenger case study The highlights of the STS-51L case study are: ? Historical details leading to Challenger accident (from 1971 to 1986) are brought alive using textual materials, photos, and videos. ? Consequences of choosing an engineering design and learn from the failures of the design. ? Evaluate several options by applying principles of ethics, engineering design considerations, and statistical methods to test data. We consider the implementation of STS-51 L case study in a mechanical engineering classroom. ? Students were randomly assigned to one of two engineering design courses: an intervention class (N=31) in which case studies was used, or a comparison class (N=31) in which traditional practices was followed. Comparative data were collected from the students regarding perceived learning and problem-solving techniques. ? Student responses was collected from the experimental class regarding STS-51L case study. ? Responses to electronic journal prompts was collected from the experimental class on STS- 51L case study to capture problem-solving processes. Two individual evaluation forms were used to evaluate ST-51L case: ? Evaluation I: 24 bipolar descriptors on a 5-point continuum (Constructs: Interesting, Valuable, Instructionally Helpful, Relevant). ? Evaluation II: 16 evaluation items asking students to rate extent of agreement on a 5-point Likert scale (Constructs: Skill Development, Self-Reported Learning, Motivation, Communication Skills, and Learn from Fellow Students). 46 Table 4.Mean ratings to evaluation I as per case study Table 5. Mean ratings to evaluation II as per case study Implications about mean ratings to evaluation I and II ? All the ratings in table 4 and 5 were highly favorable (above the neutral 3.0 rating), indicating students? approval of the case study instructional methodology. ? Evaluations indicated that the students found the STS-51L case study to be particularly relevant and offered opportunities to learn from peers. Comments regarding the STS-51L challenger case study: ? Described as intriguing, exciting, and interesting. Interesting Important Instructionally Helpful Relevant STS-51L 3.8 4.2 4.0 4.3 Growth of perceived skills Self- reported learning Motivation Communication Peer learning STS- 51L 4.2 4.2 4.2 3.5 4.1 47 ? Patterns of strengths: (1) the incorporation of ethical issues into an engineering design decision and (2) the ability to connect personally to the study. ? ?This case study gave me a new criterion by which to evaluate design. I learned to identify and scrutinize ethical decisions.? ? ?It was the most personal case study because of growing up during the time of the Challenger accident.? Qualitative Feedback Some of the students who worked with the case studies found them to be very rich in technical content and extremely beneficial, but at the same time they thought the case studies could be overwhelming, with a lot of text. Several students also suggested that the case studies needed more interaction/immersion and more pictures and videos, which aligns with the results obtained from the learning styles survey. In the next section the learning styles of the cohort of students used in this study is described in detail. Learning Styles of Engineering Students Learning styles are indicators of how students perceive, interact with and respond to a particular environment. Learning styles give an idea of the students? cognitive, affective and psychological behaviors. In engineering, the way students take in and process information often differs from the teaching styles of the professors. This mismatch in learning and teaching styles can result in a loss of attention and motivation in the class, which can lead to poor class grades and retention issues. To overcome these problems instructors need to find the right balance between teaching and learning styles of the students. 48 The Index of Learning Styles is a learning style model designed to capture the most important learning style differences among engineering students. It is a self-scoring questionnaire for assessing preference on four dimensions of the Felder-Silverman model. The four dimensions are active-reflective, sensing-intuitive, visual-verbal, and sequential-global. In spring 2010, the Introduction to Mechanical Engineering course students at two universities, Auburn and Hampton, worked on three LITEE case studies (the STS-51L Challenger, Lorn Manufacturing, and Chick-fil-A cases). The Index of Learning Styles (ILS) survey was implemented in spring and fall 2010 for the 215 students who had enrolled in the classes at Auburn and Hampton Universities over the two semesters to capture the different learning styles of the engineering students. Table 6 shows the means and standard deviation of the student learning style for all four of the dimensions. Active learners tend to learn and retain information by doing something active. They like applications and group work more than reflective learners. Reflective learners prefer to think it through first. They tend to read and memorize information. The analysis revealed that more than 70 % of the students were active learners. Figure 13 shows a visual representation of the Active-Reflective learning style Table 6: Average score of learning style preferences of engineering students for spring and fall 2010 Learning Styles Descriptive Statistics N Minimum Maximum Mean Std. Deviation Active ? Reflective 215 -11 9 -2.04 4.533 Sensing ?Intuitive 215 -11 11 -2.45 5.607 Visual -Verbal 215 -11 9 -5.70 4.521 Sequential ? Global 215 -11 11 -1.75 4.449 49 Figure 13. Visual representation of active?reflective learning style dimension (n=215) Sensing learners prefer courses with real-world connections, whereas intuitive learners do not like courses that require a great deal of memorization and routine calculations. Intuitive learners are innovators, whereas sensing learners like to solve problems using well-established methods and dislike complications and surprises. The analysis revealed that more than 70 % of the sample population was sensing learners. Figure 14 shows a visual representation of the sensing- intuitive learning style dimension for engineering students at Auburn and Hampton. 50 Figure 14. Visual representation of sensing-intuitive learning style dimension (n=215) Visual learners prefer pictures, diagrams and demonstrations when learning course material and concepts, whereas verbal learners prefer written or spoken words. The analysis showed that more than 90% of all the students preferred visual learning. Figure 15 shows a visual representation of the visual-verbal learning style dimension. 51 Figure 15. Visual representation of visual-verbal learning style dimension (n=215) Sequential learners prefer to learn in logical sequences of steps whereas global learners like to look at the big picture first. Global learners absorb material in large leaps, whereas sequential learners like to find solutions in a stepwise manner. Seventy percent of the students who took the Index of Learning Styles survey were sequential learners. Figure 16 shows a visual representation of the sequential-global learning style dimension. All the results obtained in this learning styles survey analysis agree with the results reported by Felder at al. (1988, 2005) in 52 engineering classrooms. Consequently, based on these results it is safe to say that the majority of engineering students prefer an active-sensing-visual-sequential style of learning. Figure 16. Visual representation of sequential-global learning style dimension (n=215) Linking the qualitative feedback on case studies to the students? learning styles The students commented that the case studies are overwhelming with lot of text and need more pictures and videos which align with the learning styles results that 90% of the students were found to be visual learners. 53 The students also commented that the case studies lack interaction/immersion which aligns with the active-reflective learning style dimension. Seventy percent of the ENGR 1110 students were inclined to active style of learning. We also observed that the case studies take a global approach, looking at the big picture first and then provide all the details of the case. The student ability to understand the case study is tested through the assignments section which comes in the end. This aligns with the sequential-global learning style dimension. Seventy percent of the ENGR 1110 students were shown to sequential learners who like to learn in logical sequence of steps. Summary This chapter discussed the development of an automatic weld inspection and induction welding case study and the classroom implementation of the new case study in a freshman Introduction to Mechanical Engineering class at Auburn. Results and findings related to the use of case studies and the learning style preferences of engineering students at Auburn and Hampton universities were also discussed. The majority of the students enjoyed the case studies and considered that they help them learn new material in an interesting and engaging way, but would prefer more interaction/immersion to be included by incorporating more pictures and videos, which aligns with their preference of being active-sensing-visual-sequential learners. 54 Chapter 4 Smart Scenarios Purpose of this study The purpose of the study was to create more interactive/immersive instructional material based on the feedback obtained from the case study evaluation reported in the previous chapter. The development of smart scenarios for undergraduate mechanical engineering classes are described, along with their classroom implementation and evaluation. Research Questions 1. How should smart scenarios be developed and implemented for mechanical engineering classrooms? 2. How should the effectiveness of smart scenarios in improving student learning be evaluated? Introduction Smart scenarios provide students with an immersive environment where they interact with a series of avatar characters as they work through a scenario. Like interactive scenarios, these are fully customizable and can be crafted to a specific course topic and objective. Smart scenarios are designed to help students integrate and assimilate information, rather than to establish a context for hands-on technical tasks. In smart scenarios, students interact with characters to gain information, validate ideas, and advance their knowledge before being assessed. Smart scenarios employ natural assessments that deliver assessments in a variety of real world contexts such as integrated discussions, email replies, presentations to colleagues, team meetings, and the creation 55 of executive briefings. Natural assessments allow students to demonstrate their command of the topics addressed in the same way that they might in a true to life situation by articulating their knowledge to their virtual ?boss?, colleagues or others within the scenario. All information provided by the student in the assessment elements is captured and formatted for delivery to the course instructor for grading. A screen shot of a typical smart scenario is shown in Figure 17. Figure 17. Screenshot of engineering design smart scenario Toolwire, a company that specializes in immersive virtual learning labs, joined forces with LITEE to develop a pilot project during July-November 2010 to verify the feasibility of developing serious games for use in higher education. This resulted in developing a pilot version of an engineering design smart scenario supplemented by information contained in the Challenger STS 51-L case study. This chapter describes in detail the process of developing the smart scenario, its implementation in the classroom, and the feedback obtained from students. 56 Development of Smart Scenario The initial step in the pilot project was for Auburn University faculty members and graduate students to participate in a series of weekly conference calls with the instructional material developers and project managers at Toolwire during August and September 2010 to discuss the learning objectives and develop a timeline. The learning objectives for the Engineering Design Smart Scenario were: ? Understand the eight steps of engineering design: problem definition, concept formation, concept evaluation, concept evaluation, detailed design, prototyping, testing, and send to production. ? Define each step. ? Illustrate each step using the Challenger STS 51-L case study. ? Analyze the data presented that shows the test results of all shuttle launches before STS 51-L with the temperature and the number of failures in the O-ring. Different graphs are presented in the smart scenario: Figure 18a depicts only failures with temperature and it is difficult to correlate temperature and O-ring erosion from this information because of the severe erosion at 75?F (STS 61A). Figure 18b provides a complete plot of all failures with temperature and uses a logistic regression. This shows that the probability of failure is close to 1 at temperatures below 35?F (Pinkus et al., 1997). 57 Figures 18a and 18b: Different ways of interpreting the same data The pilot version followed Toolwire?s instructional design architecture to create the ensuing unifying storyline. A Toolwire Assignment Map (TAM) was created and edited extensively by the LITEE team (Pramod Rajan, P. K. Raju, Chetan S. Sankar) so that the STS 51- L case study information and multiple-choice questions could be included (Figure 19). The external evaluator, Barbara Kuwalich from University of West Georgia, also provided assistance in modifying the TAM so that formative assessment questions could be included in the smart scenario. Once the TAM was complete, Toolwire created a detailed storyboard. This included the development of ten scenes, with each scene emphasizing certain learning objectives, and ways to transition smoothly from one topic to the next. The LITEE team revised this extensively and included additional information from the STS 51-L case study to clarify the design principles, and to ensure that the assessment questions were appropriate and that all the learning objectives were included in the design. An example of the storyboard is shown in Figure 20. 58 Figure 19: Toolwire Assignment Map (TAM) Figure 20: Example of Storyboard Once the storyboards had been agreed upon by all the partners, Toolwire developed the design of the smart scenario further, with points being allocated for answering appropriate questions in the multiple-choice questions. An example of a screenshot from the resulting smart ENGR1110 Scenario 1: The Importance Of Engineering Design In Product Safety Scene 7: Student?s Cubicle on Thursday Location Actor Interaction Action Notes Responder1 All of this data is interesting. I?m getting somewhere in my work on this, but there are still a few questions I need to ask. OnClick Enable Scene7_Whiteboard Whiteboard in student?s cube is clickable to reveal work, which includes the 8 steps and graph mentioned in Scene 5. VL: Alessa?s office Middle Right (bg_COO_office.psd) Alessa Giamo (ch_TheresaW- Phone.psd, no headset) Enable Update Alessa dialog: Hi! How are you today? Enable Responder2 Remove existing female character from the scene. Toolwire Assignment Map (TAM): September 2, 2010 TOOLWIRE ASSIGNMENT DESCRIPTIONS: Engineering Design Principles & Written Communication Week One: Importance of Engineering Design in Product Safety Toolwire Smart Scenario: Importance of Engineering Design in Product Safety DESCRIPTION: In this Smart Scenario students will take on the role of a newly hired mechanical engineer with Lunar Aerospace. Our fictitious company, Lunar Aerospace, has been a key contributor to the space program in the United States and in this Smart Scenario students will be introduced to the company, its projects, and their colleagues. Additionally, this SmartScenario will serve to set context for additional Smart Scenarios which follow. As a new mechanical engineer with Lunar Aerospace, the student will participate in trainings, discussions, conversations, and presentations as he/she learns about the new company and what it means to be an engineer. The focus of this SmartScenario will be on the importance of understanding key engineering design principles. Knowledge about design principles will be gained throughout the Smart Scenario as the student participates in ?new hire? training as well as through meetings and conversations he/she will have with work colleagues. Upon completing this SmartScenario students will be able to do the following: Areas Explored Through Assessment Questions in the SmartScenario. The Smartscenario will provide an ability for students to answer these questions, to score them, and then to provide the best answer: ? Draw a flowchart that explains the design process. ? Describe each element of the design process. ? How design process was used in STS 51-L. o Testing for low temperatures; provides chart showing different temperatures & O-ring erosion; o Illustrating results when proper analysis was not performed; o Showing graph that includes no erosion & with erosion; illustrates how the difference in depiction leads to different conclusions; o Showing the need for adding a plate in the space shuttle that clearly indicated when the shuttle should not be launched ? Discuss the reasons why STS 51-L was launched & compare with answers given by SmartScenario 59 scenario is shown as Figure 17. This pilot version of the smart scenario was tested in an Introduction to Engineering course at Auburn University during November 2010. Activities Performed During the Development of Smart Scenarios Design and Development Process Figure 21 shows the development process for smart scenarios. This process was created by Toolwire Inc. Figure 21: Smart scenario development process 60 Summary of the Development Process 1. The ENGR 1110 - Introduction to Mechanical Engineering Course has been used to test case studies previously. The STS-51L case study developed by the LITEE lab was therefore chosen to be converted to an engineering design smart scenario. 2. The learning objectives of the engineering design smart scenario were a. Understand the eight steps of engineering design: problem definition, concept formation, concept evaluation, concept evaluation, detailed design, prototyping, testing, and send to production. b. Define each step. c. Illustrate each step from the Challenger STS 51-L case study. d. Analyze the data presented that shows the test results of all shuttle launches before STS 51-L, including the temperature and number of failures in the O-ring. 3. After the learning objectives had been set, a Toolwire Assignment Map (TAM) was created for the smart scenario. The detailed TAM for the engineering design smart scenario is provided in Appendix A. 4. The TAM was reviewed by the LITEE team and feedback was given to Toolwire. The TAM was finalized on September 7th, 2011. 5. The next step in the process was the development of the scenario outline/rationale. This was the first high level design document from Toolwire and was delivered to the LITEE team on September 10th, 2011 for feedback. The scenario outline/rationale for the engineering design smart scenario is shown in Appendix B. 61 6. After several revisions of the outline within the LITEE team, the feedback was provided to Toolwire on the scenario outline/rationale which lead to the development of the detailed storyboard. The team participated in weekly teleconferences to discuss the feedback. 7. Toolwire delivered the detailed storyboard on September 23rd, 2011 to the LITEE team for feedback. The detailed storyboard included sketches of visual designs and layouts for all screens, navigation, description of interactions, description of animation, video, and audio components. The detailed storyboard for the engineering design smart scenario is in Appendix C. 8. The detailed storyboard was approved and finalized based on the feedback and Toolwire started developing the alpha version of the smart scenario on September 29th, 2011. 9. Toolwire provided the alpha version of the smart scenario. With the help of extensive feedback from the LITEE team, Toolwire developed the beta version and presented it for another round of feedback and approval 10. Finally, the beta version was tested in the ENGR 1110 class on October 30th, 2011 Table 9 shows a summary of the project plan dates for the development of the smart scenario. 62 Table 7: Summary of the project plan dates for the development of Smart Scenarios SMART SCENARIO PROJECT PLAN DATES September 7th, 2010 Toolwire Assignment Map (TAM) finalized September 7th, 2010 Development of the initial scenario outline/rationale begins September 10th, 2010 Scenario outline/rationale delivered to the LITEE team September 13th, 2010 Scenario outline/rationale feedback due from LITEE team September 13th, 2010 Development of the detailed storyboards begins September 23rd, 2010 Storyboards delivered to the LITEE team September 28th, 2010 Storyboard comments due from LITEE team September 29th, 2010 Development of the Smart Scenarios begins October 21st, 2010 Alpha version of the Smart Scenarios delivered to the LITEE team October 25th, 2010 Feedback on Alpha version due from LITEE team October 27th, 2010 Beta version of the Smart Scenarios delivered to the LITEE team October 28th, 2010 Feedback on Beta version due from LITEE team October 29th, 2010 Finalized version of the Smart Scenarios to be delivered October 30th, 2010 Students go through Smart Scenarios and feedback is collected Classroom Implementation This pilot version of the smart scenario was tested in an Introduction to Engineering course at Auburn University during November 2010. Overall, 69 students worked with the Engineering Design Smart Scenario, of whom 57 were male and 12 were female. Prior to collecting data for this study, the evaluation team, formed by Chetan S. Sankar, Justin Bond from Auburn 63 University, Auburn, Alabama and Barbara Kuwalich, Kim Huett from University of West Georgia, Carrollton, Georgia, obtained permission from the Institutional Review Board (IRB) at Auburn University (Appendix D) Evaluation and Findings The survey was developed by the evaluation team. Of the 69 undergraduate students who participated, 70% expressed an interest in working with such instructional material in the future. Sixty-two percent of the 69 students who used the design smart scenario perceived that they read through the required material deeply, 30% considered it improved their thinking skills, and 26% thought that they had become more conversant with the technical information. Eighty percent of the students perceived this to be a different method of learning and 26% found it to be realistic, while 40% of the students regarded the smart scenario as an innovative method to learn engineering design and 54% of the students preferred the gaming aspect that was included in the pilot study. However, 66% of the students thought that the Smart Scenario could be made easier to navigate and should include video and audio material, 44% wanted the scenario to be made more like a game, and 42% expressed a need to simplify the user interface. Thirty percent of the students would have been willing to pay extra fees to work with this type of instructional material. Two instructors who used the smart scenarios also commented on the experience: Instructor 1: I think they are very well created and designed. Overall, I think this is actually another huge improvement to the case study. The biggest advantage of this gaming style learning is that student has to keep their attention on all materials since there are multiple mini tests/quizzes. The questions are challenging enough for me. The single and multiple choice questions are nice and balanced. 64 Instructor 2: The students appeared to be genuinely interested in the Toolwire Smart Scenario. While there were a few issues brought up by multiple students, I believe the interactive computer format provided deeper engagement in the material than that of a lecture session. Summary This chapter presented the design and development process for smart scenarios. Based on the feedback received from a classroom implementation, additional steps are needed to create a learning game that fully meets the students? needs. Key elements that emerged from the evaluation are that future development should reduce the complexity/length of the smart scenario and that both audio and video components should be incorporated to move away from the cartoon depiction and enhance the gaming functionality and multi-path nature of the game. While the product used for this pilot was very well received, this feedback obtained will serve to guide the creation of the next version of these learning games. 65 Chapter 5 Development and Analysis of the Serious Game Purpose of this Study The purpose of this study was to showcase the design and development of an engineering design serious game incorporating feedback received from the smart scenario implementation described in Chapter 4. The second purpose of this study was to develop an evaluation model to assess the effectiveness of the serious game when implemented in an Introduction to Mechanical Engineering class at Auburn University. Research Questions 1. How should serious games be developed and implemented for mechanical engineering classrooms? 2. How should the effectiveness of serious games in improving student learning be evaluated? Introduction Serious games are the third innovative instructional technique to be considered in this research study. One of the most critical items of feedback received from the evaluation of the smart scenario developed and tested in the previous chapter was that the students would like to see greater gaming functionality and more interactivity in their instructional material. The students also suggested that the focus should be on one concept at a time instead of teaching a whole case study through the gaming approach, which they found overwhelming. Based on this feedback, the concept of serious games was selected to provide a step by step introduction to the 66 engineering design process for undergraduate engineering students. Once again, Toolwire worked with the LITEE team to design and develop a serious game to teach the engineering design process to a mechanical engineering class at Auburn. The LITEE team consisted of two groups, the design and development group and the evaluation group. The design and development group is formed by the author and Joseph McIntyre under the leadership of Dr. P. K. Raju. The design and development of the design serious game needed a collaborative teamwork with Toolwire developers via teleconference call, emails, and sharing of drop box. The Toolwire?s developer group was headed by Dayvid Jones and Michael Watkins. My role in the LITEE team development group had the following responsibilities: 1. Creating PowerPoint presentations for Toolwire developers demonstrating the user flow experience of the serious game. 2. Providing the required technical content and calculations for analyzing and simulating the tower building performance in the serious game. 3. Providing feedback on the technical content documents developed by Toolwire. 4. Helped the evaluation group in developing the 4P model to check the effectiveness of the serious game. 5. Helped in the implementation of serious game in the introduction to engineering class and collecting data. The evaluation group comprised of Dr. Chetan S. Sankar and Justin Bond from Auburn University, Dr. Barbara Kuwalich and Kim Huett from University of West Georgia. They were responsible for obtaining the IRB approval at Auburn University. The evaluation team was also 67 responsible in designing, implementing and analyzing the qualitative and quantitative aspects of this study. Design and Development of a Serious Game Definition of a Serious Game A serious game has a major theme or emphasis that is the focus of the majority of the concepts addressed by the building blocks that constitute the game. Each game will tell a story that enables the distinct parts to be knitted together to provide a credible gaming environment for the student. Each game contains multiple building blocks, of which one theme/ emphasis will form the majority. Each game will finish with a climax that requires students to make a major decision that will form the endpoint of the game. The user profile of an individual student may be remembered by the system and appropriate statistics brought up when he or she moves on to the next game. Some games might flow one to the next and there may be a continuing story line. Other games might be completely independent of each other. A serious game is expected to consist of multiple acts, each of which will contain a collection of scenes. A scene will provide opportunities for the student to acquire, demonstrate, or apply knowledge regarding a particular concept and will contain multiple building blocks. A building block describes a nugget or area of information and is the smallest element of a game. Each building block may define a concept, show an application of the concept to a scenario, and/or allow the student to apply the concept to advance the overall story. A concept is the particular knowledge a student is expected to gain using a building block. An example of a 68 concept might be ?audience in communication, or testing in design.? There is generally a mentor, a game character who instructs the user and provides feedback on the appropriateness of the user?s answers to assessment questions. Figure 22 shows the basic components of a serious game and Appendix E shows the design flowchart of a typical building block. 69 Figure 22. Example of a serious game showing its basic components Design and Development of an Engineering Design Serious Game The three main objectives of an engineering design serious game are: ? To provide students with an opportunity to learn about the engineering design process in an interesting and engaging gaming environment. ? To help students understand the intricacies involved in designing engineering structures in an interactive manner. ? To demonstrate the effectiveness of the use of serious games in teaching engineering concepts. A progression matrix displays the progression in the game scenes and activities in a schematic form. The progression matrix for the design serious game is provided in Appendix F and shows the different Scenes, Building Blocks (BB) and Acts involved in the design game. The design game outline developed is listed in Appendix G. This explains in detail all the elements of the design serious game. A simple PowerPoint presentation was made to the Toolwire developers and managers explaining how the game should look. This presentation is reproduced in Appendix H. Based on this outline, progression matrix and presentation, Toolwire developed the initial storyboard for the design serious game (Appendix I). The BOX 2D physics flash engine was used 70 to develop this game. The LITEE-Toolwire team participated in weekly teleconferences and continuous feedback was obtained after each stage from the students to ensure the learning objectives and goals of the game were being adhered to. The production milestones and schedule are shown in Table 8. Table 8. Production Milestones and Schedule for the development of the design serious game Engineering Design Serious Game The serious game is designed to teach students about the engineering design process. The engineering design process used in the development of this game is shown in Figure 23. Figure 23. Design process used in the game 71 The design process chart is based on Pahl and Beitz?s model of the design process (Pahl et al., 2007), focusing particularly on the first seven steps of the design process. The game is entitled 'Engineering Heights: The Design Process in Action'. A screen shot of the first screen of the game is shown in Figure 24. Stage Number Description Due Date Stage 1 + Warm-Up/Tutorial: Competition Goal & Design Process prototype with still photo mentors reviewing three failed designs. There is very little user action. It is mainly comprised of animation and video with next buttons. Jun 25th Stage 2 + Tinkering/Simulation: Five different screens, each focusing on the design process. These include ? Height, Concept Evaluation, Concept Selection, Detailed Design, and Testing. Players go through each screen/process and make selections. Players can go back at any time and make changes. The last process tests the load bearing of the player?s structure in a simulation of the bridge building goal. (This is not the game and so doesn?t include dragging to build or physics. Art is rough but includes different styles for testing.) July 16th Stage 3 + Competition/ Building game prototype. Includes dragging and building a structure. Physics are in place. Rough art. The game engine will include these features: - Load testing. - Cost. - Weight. July 30th During Stage 3 + Test prototypes and define the game. Create a final game design doc/use case that includes goals, intermittent gratification (collectables), winning, risk/losing, etc. Another round of art, but not final. July 30th Stage 4 + Gamification. Add game elements, risk, intermittent gratification, etc. Add train track bridge animation and effects. Additional Level progression (Lookout Tower, Water Tower, Antenna Tower) August 20th Stage 5 + Put all stages/prototypes together. Use rough art for Intro and Instructions. Skin simulation and competition with agreed art, may not be final yet. September 3rd Stage 6 + Intro/Splash movie or animation. Video characters. Final art, effects and video. Game Sound FX. September 17th Stage 7 + Completion sharing. October 1st Stage 8 + Testing of Beta, bug fixes, and changes. October 15th Final/ Release build + Final release and testing October 22nd 72 Figure 24. Introductory screen of the design serious game The example of building a structure to support a rail road bridge was chosen as the students' final task as the principles involved corresponded to those in the pasta tower building activity performed as a part of the ENGR 1110 course. Figure 25 shows a basic block diagram of the game user flow experience, described briefly in turn below. A more detailed user flow experience diagram is provided in Appendix J. 73 Figure 25. Serious game user flow experience Overview This section of the game defines the overall goal of the game, which was to teach students about the engineering design process. It also highlights the need for a formal design process by presenting examples of failed bridges. The overview introduces students to basic construction materials like beams and joints that are required to build their structure. Lab Introduction In this section the student reviewed in detail each of the core engineering design process steps. Once they have learned these steps, they had a chance to design a structure and make decisions that affect the weight, cost and the load capacity of their structure. This is done by selecting from a set of pre-defined shape structures, different materials and beam and joint choices. The game simulates their tower and shows the estimated load that their structure will withstand. A screen shot of the lab introduction is shown in Figure 26. 74 Figure 26. Screen shot of lab introduction The Building Game In this part of the game the students were guided on how to use the tools and screen areas to design, build and test a structure from scratch. This is like a tutorial in that they have to join the dots and learn how to build their structure and then use the tools to test their structure. Several tool tips were used to convey this message to the students. There are a number of different goals for the students within this building game level. A screen shot of the building game level is shown in Figure 27. 75 Figure 27. Screen shot of the building game level The Main Game The main game consists of three levels. The first level is a simple test tower where the students are given some constraints on weight, cost, and load to build their tower. The second level is a water tower level where they have to build a tower to hold a water tank at the top of their tower. The third level is a train bridge level where the students have to build a structure to support a train bridge. The difficulty increases as the students? progress through different levels. The game also allots a score for each finished level as a measure of the students' performance. Screen shots of the water and train bridge levels are shown in Figure 28. 76 Water bridge level Train bridge level Figure 28. Screen shots of the water bridge (left) and the train bridge (right) levels Classroom Implementation and Evaluation Prior to collecting data for this study the evaluation team obtained permission from the Institutional Review Board (IRB) at Auburn University (Appendix K).The goal of this study was to develop and test a research model that investigates the effect of a design serious game on gaining higher order cognitive skills, concentration, goal clarity, and student enjoyment. Any differences based on gender, race and learning style will also be tested for. Two learning modules were used in this study. The first module is called the Engineering Design Learning Module 1 (EDLM 1). In this module the students were exposed to a lecture on engineering design process (Appendix L) and an active learning exercise (Statistics Applied to Data Analysis (Appendix M). The second module is called the Engineering Design Learning Module 2 (EDLM 2). In this module the students were exposed to the same lecture on engineering design process and a design simulation game. In the lecture the instructor used a PowerPoint presentation to 77 teach the engineering design process. In the active learning exercise the students collected data using LABVIEW software and LEGO robots. They performed some simple statistical analysis on the data using Excel. In the design simulation game the students were allowed to play a serious game which emphasized the engineering design process using a tower building example. Both sections participated in the Pasta Tower challenge at the end of the course. The instructional material covered in the control and experimental sections is explained below Control Section The control section performed Engineering Design Learning Module 1 (EDLM 1), which covered the following instructional material: 1. Lecture on engineering design process by the instructor 2. Active Learning Exercise (Statistics Applied to Data Analysis or Lecture on statistics/unit conversion), and Experimental Section The experimental section performed Engineering Design Learning Module 2 (EDLM 2), which covered the following instructional material: 1. Lecture on engineering design process by the instructor 2. Design Simulation Game (Engineering Heights: Design Process in Action) Previous researchers have shown that personal factors/characteristics can influence a students? approach to learning and learning outcomes (JB Biggs, 1970, 1987, 1992; Dart et al., 78 2000). The 4P model was first used by Sankar et al. (2011) and was derived by studying the learning approach models used by Biggs (1989) and Nemanich et al. (2009) The model proposes that presage conditions and learning modules (both pedagogy factors) combine to create the approach a student takes in their learning (a process factor), which in turn influences the improvement in achieving outcomes (a product factor). This implies that for different learning modules or instructional methods used to teach engineering design process concepts, the improvement in achieving outcomes may be different based on the presage and process factors. Figure 29 shows the 4P model, with learning modules as the moderating variable. Evaluation model The four P variables, presage, pedagogy, process and product, are shown in Figure 29, along with the factors that comprise them. Presage Presage factors exist prior to the engagement that affects the learning process. The presage factors considered in this model are gender, race, and learning style. These factors interact with the learning module to affect the process and the learning outcomes. The presage factors usually constitute the independent variables in the 4P model. 79 Presage Pedagogy Process Product Hypothesis 1 Hypothesis7 Hypothesis 2 Hypothesis 6 Hypothesis 3 Hypothesis 5 Hypothesis 4 Figure 29. 4P model with learning modules as the moderating variable Gender Although there is a general impression that the majority of gamers are teenage boys, the data suggest otherwise. In 2011, 47% of game players were women and 29% were over the age 50 (Entertainment Software Association, 2011). Several studies have been conducted on how males and females perform in a gaming versus a non-gaming environment. Brown et al. (1997) studied the effect of gender on video game (Pong) performance and found that in all three experiments Gender Race Learning style Goal Clarity Performance Perceived Learning Higher Order Cognitive Skills Concentration Learning Modules Perceived Stickiness Improvement in achieving outcomes Student Enjoyment Difficulty with subject matter Attitude Perceived Usefulness 80 conducted, both males and females showed significant improvement in performance over trials but overall males outperformed females in the first two experiments. Males and females like to see different aspects in a game. (Schell, 2008). Gender was therefore selected as the first presage factor and a question on gender was be included in the survey (Appendix N) administered at the end of the learning module. Race In the U.S., African-Americans constitute one of the largest minority groups who are significantly underrepresented in engineering. Education researchers and academicians have long debated the reasons for the differing academic achievements of African-Americans and other ethnic groups (Banks, 1988; Chubin et al., 2005). ScoreNetworks, an American firm that measures online game use, confirms that players are beginning to resemble the general (American) population. On average, 8.9% of players on the Top 10 gaming sites are African American, 4.2% are Asian and 79.3% are white (Corti, 2006). Therefore, race was chosen as the second presage factor. Race of the students was recorded by their answers to a survey (Appendix N) administered at the end of the learning module. Learning Styles The main goal of any learning module is to help students learn effectively and efficiently. Learning styles affect how students learn and engage with a particular topic or concept. Different students have different learning styles, and the Index of learning styles questionnaire developed by Felder & Soloman (1991a) was be used to measure the learning styles of the students participating in this study. The Index of Learning Styles (ILS) is a forty-four item forced-choice 81 instrument that is used to assess preferences on four scales: active & reflective, sensing & intuitive, visual & verbal, and sequential & global, as described in Chapter 4. It is available from the website: http://www.engr.ncsu.edu/learningstyles/ilsweb.html. The conventional lecture-based teaching approach in engineering education favors intuitive, verbal, reflective, and sequential learners, while the majority of engineering students are actually sensing, visual, active, and sequential in style (Felder & Brent, 2005). Therefore, the third presage factor for this study was chosen to be learning styles. Pedagogy The two learning modules used to teach the engineering design process are: Engineering Design Learning Module 1 (EDLM 1), consisting of a lecture on the engineering design process and an active learning exercise entitled "Statistics Applied to Data Analysis,? after which the students followed the engineering design process to build a tower using pasta and masking tape. Engineering Design Learning Module 2 (EDLM 2), consisting of a lecture on the engineering design process and a serious game entitled "Engineering Heights: The Design Process in Action" where the students built a series of towers of increasing difficulty, after which they followed the engineering design process to build a tower using pasta and masking tape. The control section performed EDLM 1 and the experimental section performed EDLM 2. 82 Process The heart of the teaching/learning system is the process level, where the learning related activity either produces or does not produce the desired outcomes (Biggs et al., 2001). Process incorporates the student?s learning experience (Nemanich et al., 2009). Higher Order Cognitive Skills Higher order cognitive skills include problem formulation, critical reasoning, decision making, problem solving and reasoning. The ABET (2009) 3(e) criterion states that students need to be able to identify, formulate, and solve engineering problems at the end of their education. Feng et al (2007) examined the differences in spatial attention and cognition between males and females following 10 hours of practice with either an action or a non-action game. They found that performance of the game group improved substantially compared to the control group and females benefitted more than males. Hwang et al (2013) also reported that women had a higher cognitive load and more competition anxiety from playing competitive games compared to men. Therefore, higher order cognitive skills were chosen as the first process factor. The questions measuring higher order cognitive skills were taken from Hingorani et al. (1998) and are shown later, in Table 9. Concentration Concentration on a task is nothing but a persistent shift of attention to this task. Thus the task should be able to mentally load students' cognition and also be perceptually incentive. The task requires such concentration that only a very select range of information can be allowed into awareness (Csikszentmihalyi, 1990). Perceived web skills and postive challenges are positively 83 related to shopping enjoyment and concentratrion of online consumers (Jung et al., 2008). Zhang et al. (2006) indicate that cognitive concentration has a significant influence on perceived usefulness and perceived ease of use. Therefore, concentration was selected as the second process factor. The questions measuring concentration are taken from Koufaris (2002). These questions are also shown later, in Table 9. Goal Clarity International Organization for Standardization states that usability related to games encompasses measures such as effectiveness and efficiency. Effectiveness is defined as the accuracy and completeness with which users achieve set goals and efficiency represents the resources expended to achieve those goals. There should be a clear overriding goal for the game that is presented early on in the process (Clanton, 1998; Malone 1982). Guo et al., (2009) measured goal clarity in the context of online shopping and found that having clear goals and rapid feedback both had a positive effect on the online shopping experience. Chen (2006) measured goal clarity in a study of web users? online behavior and found that web users generally experience positive moods and better engagement on the web with clear goals. The questions measuring goal clarity are taken from Guo et al., (2009) and are shown later, in Table 9. Student Enjoyment Davis (1992) defined enjoyment as the extent to which the activity of using the computer is perceived to be enjoyable in its own right, quite apart from any performance consequences that may be anticipated. Agarwal et al (2000) define cognitive absorption or heightened enjoyment as the pleasurable aspects of an interaction. Nemanich at al. (2009) reported that student enjoyment is positively related to the relevance of the course content and learning environment. The questions 84 measuring student enjoyment are taken from Nemanich at al., (2009) and are shown later, in Table 9. Product Product is the outcome of learning. Product factors are indicators of the knowledge, skills, and behaviors students have gained by participating in the learning process. Five product factors were identified in the research model and these are explained below. Performance There are two measures of performance: 1. Pasta tower performance factor The performance formula for the pasta tower is based on the Euler equation for buckling in a simply supported column. The simply supported column is a good first approximation for the tower as both the top and bottom of the tower are free to rotate when loaded. The Euler buckling equation states that the load at which buckling occurs is inversely proportional to the square of the height of the column. As the column gets higher, the load required to buckle the column reduces with the square of the height. The tower performance formula used for this measure is directly proportional to the square of the height in order to reward the groups that took the more difficult design task of making a tall tower. The square of the height is multiplied by the maximum load supported without failure to reward the builders of the strongest tower. The fraction of weight taken up by the supplies used to build the tower measures the efficiency of material usage in the design. The tower performance 85 number then measures the height, strength, and efficiency of the pasta tower in a manner that follows the buckling behavior of columns. The students are asked to build a tower using pasta and masking tape. There are specific goals and requirements for the tower. For example the tower must be at least 12 inches in height. The formula used to calculate the pasta tower performance factor is given as Pasta tower performance factor = Tower height2 x (Weight of supplies/Tower weight) x Load supported by tower. The students have to fill out a tower design worksheet showing their design process while building the tower. The instructions and the tower design work sheet for the pasta tower building activity is shown in Appendix O. 2. Engineering design process exam score A set of questions designed to test the students on their understanding of the engineering design process was included in the midterm exam. The questions are shown in Appendix P. Perceived Subject Matter Learning A learning module may change the students? perceived subject matter learning while meeting its goals. Rossiou et al (2008) evaluated the effectiveness of an online multi-player game to teach students how to use a recursive algorithm and found that participants who played the game performed better in the final exam and also on recursive problems than those who participated in tutoring and did not receive the game. Vahed (2008) found that the web-based format game significantly impacted the students? knowledge of and attitudes to tooth morphology. Fu et al. 86 (2009) measured knowledge improvement in different online games and defined it as measurement of e-learning games. Therefore, perceived subject matter learning was chosen as the second product factor in the research model. The questions measuring perceived subject matter learning were taken from Alavi et al. (2002) and are shown later, in Table 9. Attitude The attitude of the student encompasses both the student?s attitude toward the subject being taught and whether the student believes she or he will be able to learn the material. This includes emotional response to learning, confidence in learning new materials, responsibility, accomplishment, and understanding of cross-disciplinary work, all of which contribute to team working skills and higher order cognitive skills (Santhanam et al., 2008). Cybinski and Selvanathan (2005) found there was a relationship between student enjoyment of a statistics course and student attitudes towards the subject matter. Therefore, attitude was chosen to be the third product factor in the research model. The questions measuring attitude are taken from Hingorani et al. (1998) and are shown later, in Table 9. Difficulty with subject Some students have difficulty in understanding subject matter when learning new concepts/ideas. When a student thinks a subject matter is difficult, then they will have trouble working on homework or a real world problem related to it. For example, Smith (1946) showed how difficulty of subject matter affects learning general science. Therefore, difficulty with subject matter was chosen to the fourth product factor in the research model. The questions measuring difficulty with subject were taken from Sankar et al. (2010) and are shown later, in Table 9. 87 Perceived Usefulness Perceived usefulness is measured in many information technology studies. Within the context of their study, Davis et al. (1989) defined perceived usefulness as ?the degree to which a person believes that using a particular system would enhance his or her job performance.? The Technology Acceptance Model (Davis et al, 1989) hypothesizes that intention to use a particular technology is predicted by its usefulness and ease of use. Agarwal et al (2000) found that perceived usefulness of an information technology has a positive effect on behavioral intention to use the information technology. In the current study, perceived usefulness was applied to both engineering design learning modules and was defined as the degree to which a person believes that using the engineering design learning module would enhance his or her achieving outcomes. Therefore perceived usefulness was chosen to be the fifth factor in the research model. The questions measuring perceived usefulness are taken from Malhotra et al. (2003) and are shown later, in Table 9. Perceived Stickiness Wu et al. (2009) described stickiness in the context of online games, defining it as ?the gamers? willingness to return to and prolong their duration of each stay in the online game.? In the context of website usage, Lin (2007) defines stickiness as ?the user?s willingness to return to and prolong his/her duration of stay on a website.? Lin also explains that other studies have shown that an individual?s perception of a website?s value significantly influences their intention to return to that website. These findings are based on the Technology Acceptance Model (Davis, 1989), which suggests that a user?s positive attitude towards an information system (IS) will influence his/her intention to use that IS. Additionally, stickiness can be measured objectively during game play by 88 examining the time spent by students on the game and the number of times the students return to the game. The questions measuring perceived stickiness are taken from Lin (2007) and are shown later, in Table 9. Hypotheses The hypotheses guiding this research were derived based on the expected relationships among the variables in this model, with the instructional methodologies being the moderating variable. In general, females like to learn by example and tend to prefer entertainment that connects meaningfully to the real-world, while males prefer the trial and error method and like competition and mastering things (Schell, 2008). A study by Hwang et al. (2013) found that women have a higher cognitive load and more competition anxiety while playing competitive games and both males and females showed a positive attitude and intentions to play the game. Hence, Hypothesis 1: When using Engineering Design Learning Module 1 (EDLM 1), the improvement in achieving outcomes experienced by females compared to males will be different than when students are taught using Engineering Design Learning Module 2 (EDLM 2). An illustration of Hypothesis 1 is shown in Figure 30. This shows that the female students would exhibit a higher improvement in achievement outcome using the Engineering Design Learning Module 2 (EDLM 2) than female students using the Engineering Design Learning Module 1 (EDLM 1). 89 Fig 30. Illustration of Hypothesis 1 One of the major reasons cited for the difference in academic achievements of African- Americans compared to other ethnic groups is the difference in the cognitive or learning styles of African-American students compared to other groups (Witken, 1962; Witken et al., 1974; Ramirez & Price-Williams, 1974). Many African-American learners tend to view objects in their environment in their entirety rather than in isolated parts; seem to prefer intuitive to deductive or inductive reasoning; tend to approximate concepts of space, number, and time rather than aiming at exactness or complete accuracy; prefer to attend to people stimuli rather than nonsocial or object stimuli; and tend to rely on nonverbal as well as verbal communication (stimuli, 1988, 1992, 1995; Heath, 1982, 1983; Hilliard, 1976; Jones, 2002). This difference in learning style has been attributed to social and cultural contexts, economically challenged situations, and differences in the social, cultural and family values in which they were brought up rather than genetic/biological/racial reasons (Bransford et al., 2000; McPhail, 2002; Kerns, 2005). Hence, Male Female Im pro ve me nt in a chi ev ing outc om es Gender EDLM 2 ( Game) EDLM 1 ( No Game) 90 Hypothesis 2: When using Engineering Design Learning Module 1 (EDLM 1), the improvement in achievement outcomes experienced by minorities compared to Caucasians will be different when students are taught using the Engineering Design Learning Module 2 (EDLM 2). The conventional lecture-based teaching approach in engineering education favors intuitive, verbal, reflective, and sequential learners (Felder & Brent, 2005) and the use of hypermedia instruction has been shown to decrease performance disparities among students (Zywno & Wallen, 2001). Hence, Hypothesis 3: The improvement in achievement outcomes based on learning styles (sensing/intuitive, visual/verbal, active/reflective, global/sequential) will be stronger among students using the Engineering Design Learning Module 2 (EDLM 2) than among students using Engineering Design Learning Module 1(EDLM 1) Student enjoyment is the aspect of intrinsic motivation related to experiencing joy or pleasure as enjoyment of the course. Students describe it in terms of its being ?enjoyable,? ?fun,? and ?my favorite?. Past research has shown that student enjoyment is positively related to the type of learning environment and student learning performance (Nemanich et al, 2009). Hence, Hypothesis 4: The relationship between perceived student enjoyment and improvement in achievement outcomes will be more pronounced among students using the Engineering Design Learning Module 2(EDLM 2) than among the students using the Engineering Design Learning Module 1(EDLM 1) An activity must have relatively clear goals (Csikszentmihalyi, 1988, p.32). Locke et al., (1981) found that specific and challenging goals led to higher performance than easy goals in 90% of laboratory and field studies. Earley et al., (1990) found that process and outcome feedback both interact with goal setting to enhance performance. Hence, 91 Hypothesis 5: The relationship between perceived goal clarity and improvement in achievement outcomes will be more pronounced among students using the Engineering Design Learning Module 2 (EDLM 2) than among students using the Engineering Design Learning Module 1 (EDLM 1). Concentration has been identified as a critical factor in several online consumer behavior studies. It has been found to positively influence overall experience of computer users (Noval et al., 1998) and their intention to use a system repeatedly (Webster et al., 1993). Agarwal et al. (2000) studied the impact of concentration on the perceived usefulness and ease of use of the internet. Hence, Hypothesis 6: The relationship between concentration and improvement in achievement outcomes will be more pronounced among students using the Engineering Design Learning Module 2 (EDLM 2) than among students using Engineering Design Learning Module 1 (EDLM 1). Several theories have been suggested to explain the development and improvement of higher order cognitive skills (Kolb, 1984; Spiro et al., 1991). According to Kolb (1984), learning can be improved by a feed-oriented engaging process. Higher order cognitive skills also represent an engaging process that requires the individuals to derive abstract concepts and new knowledge from existing information by identifying, integrating, analyzing and evaluating the information. Barlett et al. (2009) studied the impact of computer games on cognitive performance and found that players reached a limit on cognitive measure after 4 trials. In addition, they reported that the cognitive benefit from playing video games occurs independently of violent and non-violent game content. Hence, Hypothesis 7: The relationship between improvement in higher order cognitive skills and improvement in achievement outcomes will be more pronounced among students using the 92 Engineering Design Learning Module 2 (EDLM 2) than among students using Engineering Design Learning Module 1 (EDLM 1). Experimental Design and Measures This study was conducted during classroom lectures and laboratory sessions in undergraduate Introduction to Engineering courses held at Auburn University. This course was selected because it is the first engineering course that students are required to study in the engineering curriculum. The subjects of the study included 90 undergraduate students in fall 2012, 97 in spring 2013 and 18 in summer 2013 (a total of 205 students) enrolled in the course Introduction to Mechanical Engineering-ENGR 1110. Auburn University is a land grant university situated in a rural area with a majority of white students. At Auburn University, the same instructor taught three sections using EDLM 1 (control section) and another three sections using EDLM 2 (experimental section). Two external evaluators, Barbara Kuwalich and Kim Huett from University of West Georgia, Carrollton, Georgia, used the same instruments to collect quantitative and qualitative assessment data. Existing instruments from the literature were used to measure the variables shown in Figure 29. Table 9 lists the proposed measures for each variable used in this study, which have all been adopted from the results of past research projects (Hingorani et al., 1998, Koufaris, 2002, Guo & Klien, 2009, Alavi et al., 2002, Sankar et al., 2010, Malhotra & Galletta, 2003). 93 Table 9. Proposed measures and their cronbach alphas Constructs/ Items Measures Process Variables 1. Higher order cognitive skills (?=0.885) Perceived measures of higher order cognitive skills used by Hingorani et al., (1998) ? The instructional materials in the engineering design learning module helped me identify engineering tools that will assist me in decision-making. ? In this engineering design learning module I learned how to inter-relate important topics and ideas using the instructional materials. ? In this engineering design learning module I learned how to identify various alternatives/solutions to a problem using the instructional materials ? The instructional materials in this engineering design learning module improved my problem solving skills ? I learned how to sort relevant from irrelevant facts using the instructional materials in this engineering design learning module. 2. Concentration (? =0.863) Perceived measures of concentration used by Koufaris (2002) ? I was absorbed intensely in the engineering design learning module. ? My attention was focused on the engineering design learning module. ? I concentrated fully on the engineering design learning module ? I was deeply engrossed in the engineering design learning module 3. Goal Clarity (?=0.858) Perceived measures of goal clarity used by Guo & Klien (2009) ? I knew clearly what I wanted to do in the engineering design learning module. ? I had a strong sense of what I wanted to do in the engineering design learning module. ? I know what I wanted to achieve in the engineering design learning module. ? My goals were clearly defined in the engineering design learning module. 4. Student Enjoyment (?=0.899) Perceived measures of student enjoyment used by Nemanich et al (2009) ? The learning module has been enjoyable ? This was one of my favorite learning modules ? I had fun working on this learning module ? I enjoyed many aspects of this learning module Product Variables 1.Performance Demonstrated measures of performance ? Pasta tower performance score ? Exam score for the design process questions 2. Perceived Subject Matter Learning (?=0.894) Perceived measures of perceived subject matter learning used by Alavi et al.,(2002) ? I became more interested in the concept of engineering design process ? I gained a good understanding of the concept of engineering design process ? I learned to identify central ideas in the area of engineering design process ? I developed the ability to communicate clearly about the concept of engineering design process ? I was stimulated to do additional work in the area of "engineering design process ? I found the engineering design learning module to be a good learning experience 94 3. Attitude (?=0.851) Perceived measures of attitude used by Sankar et al., (2010) ? From my experience in this engineering design learning module I believe engineering is irrelevant to my life ? This engineering design learning module has increased my appreciation for engineering. ? From the engineering design learning module experience I think engineering is highly technical. ? This engineering design learning module has shown me that I can learn Engineering. ? Engineering skills learned in this engineering design learning module will make me more employable. ? The engineering design learning module was integrated in a way that made it easier to learn new engineering concepts. ? The engineering design learning module emotionally engaged me in learning the topics. 4. Difficulty with subject (?=0.701) Perceived measures used by Sankar et al., (2010) ? From my experience in this engineering design learning module I believe engineering is a subject learned quickly by most people ? Based on my experience in this engineering design learning module I believe I have trouble understanding engineering because of the way I think. ? My experience in this engineering design learning module has shown me that engineering concepts are easy to understand. ? I felt that performing this engineering design learning module was stressful. ? As a result of participating in this engineering design learning module I believe learning engineering requires a great deal of discipline. ? This engineering design learning module failed to expand my working knowledge of what goes on in engineering. 5. Perceived Usefulness (?=0.917) ? Perceived measures of usefulness used by Malhotra and Galletta, 2003 ? ? Using the engineering design learning module improved my performance ? Using the engineering design learning module enabled me to accomplish my tasks more quickly ? I found the engineering design learning module useful ? Using the engineering design learning module increased my productivity ? Using the engineering design learning module enhanced my effectiveness Using the engineering design module made it easier to do my work 6. Perceived Stickiness (?=0.842) Perceived measures of stickiness used by Lin(2007) ? I would stay longer on this learning module than others ? I intend to prolong my staying on this learning module ? I would visit this learning module as often as I can ? I intend to link to this learning module when I am studying design process 95 Results and Findings from Serious Game Implementation Descriptive statistics Independent t-tests (i.e., mean comparisons) were utilized to report the means and standard deviation of individual students, males and females and Caucasians and minorities. The data in the following tables represent the results of the mean comparisons for several variables of interest across the control (EDLM1: non-game) and experimental groups (EDLM 2: game). P-values that were significant at the .05 level or smaller are highlighted in bold numbers and reported. A further explanation is given for each of the significant results. Table 10. Mean comparisons of pasta tower design performance scores (first performance measure) for gender and race Variable M SD N t df P 1 All students -3.071 220 .002** Control 3858.37 2658.34 89 Experimental 5188.11 3806.63 134 2 Females -3.332 37 .002** Control 2264.74 1908.36 14 Experimental 5686.31 3552.22 25 3 Males -1.901 181.98 .059 Control 4155.84 2682.13 75 Experimental 5073.84 3869.11 109 4 Caucasians -2.842 192.99 .005** Control 4018.70 2640.04 78 Experimental 5357.52 3938.67 117 5 Minorities -1.553 25 .133 Control 2721.45 2626.41 11 Experimental 4244.39 2419.45 16 Note. *p<.05. **p<.01. ***p<.001 Three significant results were observed in Table 10. On discovering that the overall mean pasta tower design performance scores of the control (?=3858.37) and experimental (?=5188.11) groups were significantly different, (p=.002), additional analyses revealed that the relevant subgroups were females and Caucasians. Specifically, females in the control group (?=2264.74) 96 had significantly lower scores than females in the experimental group (?=5686.31), (p=.002). In the Caucasian subgroup, those in the control group (?=4018.7) had significantly lower scores than those in the experimental group (?=5357.52), (p=.005). These results could be interpreted to suggest that females and Caucasians responded positively to the serious game with regard to their pasta tower design performance score. Table 11. Mean comparisons of concentration for gender and race. Variable M SD N t df p 1 All students -2.093 174 .038* Control 3.45 .884 62 Experimental 3.72 .760 114 2 Males -1.617 143 .108 Control 3.44 .930 52 Experimental 3.67 .744 93 3 Females -1.461 29 .155 Control 3.47 .628 10 Experimental 3.90 .819 21 4 Minority -1.120 21 .275 Control 3.58 .451 9 Experimental 3.91 .794 14 5 Caucasian -1.869 151 .064 Control 3.42 .939 53 Experimental 3.69 .755 100 Note. *p<.05. **p<.01. ***p<.001 The results in Table 11 revealed only one significant difference between mean scores. The overall mean perceived concentration scores of the control (?=3.45) and experimental (?=3.72) groups were significantly different, with the experimental group displaying higher scores, (p=.038). All other gender and race subgroups showed no significant mean differences between the control and experimental groups. Table 12 shows the mean comparisons for the remaining outcome variables of interest. Contrary to expectations, there were no significant mean differences between the control and 97 experimental groups for exam score, attitude, difficulty with subject matter, goal clarity, student enjoyment, perceived subject matter learning, perceived stickiness or higher order cognitive skills in either the overall student group, or within the race and gender subgroups. Table 12. Mean comparisons among several outcome variables of interest Variable M SD N t df P 1 Exam Score -1.101 222 .272 Control 4.06 .9811 92 Experimental 4.19 .8049 132 2 Attitude -.122 174 .903 Control 3.50 0.67 62 Experimental 3.51 0.75 114 3 Difficulty with subject matter -1.486 174 .139 Control 2.64 0.75 62 Experimental 2.83 0.83 114 4 Goal Clarity 0.320 154.82 0.750 Control 3.58 0.62 62 Experimental 3.54 0.81 113 5 Student Enjoyment 0.592 88 .555 Control 3.44 0.67 17 Experimental 3.30 0.95 73 6 Perceived Subject Matter Learning 1.617 88 .109 Control 3.68 0.56 17 Experimental 3.35 0.80 73 7 Perceived Stickiness 1.002 88 .319 Control 3.26 0.65 17 Experimental 3.04 0.86 73 8 Higher Order Cognitive Skills 0.164 174 .870 Control 3.45 0.65 62 Experimental 3.43 0.78 114 Note. *p<.05. **p<.01. ***p<.001 Results of Hierarchical Multiple Regression Analysis In order to test the hypothesis proposed in the evaluation model, hierarchical multiple regression analysis was applied. Before analyzing the data it was checked for multi-collinearity. The Pearson coefficient and the variance inflation factor (VIF) suggested that the variables needed 98 to be centered, so all the independent variables in the model were centered before performing the analysis. Effect of Gender The results of the interaction effects between gender and learning modules are shown below in Table 13. The interaction effect between gender and the learning modules as it relates to explaining the pasta tower design performance scores (? = -.133, p = .044) was significant. Table 13. Interaction effect between gender and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulnes s (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML) Attitude (ATT) N 223 224 176 176 90 90 176 Gender .044 .080 .011 -.015 .038 -.011 -.082 Learning Modules (EDLM 1 vs. EDLM 2) .19** .078 .112 -.074 -.105 -.169 .007 Gender ? Learning Modules -.133* -.021 -.080 .003 .022 .034 -.017 F 4.217** .889 1.100 0.323 0.409 0.880 0.439 ?R2 .018 .000 .006 .000 .000 .001 .000 Adjusted R2 .042 -.001 .002 -.012 -.020 -.004 -.010 Note: All tests are two-tailed.*p<.05, **p<.01, ***p<.001. Interaction effects explained 1.8% of the variance in the pasta tower design performance scores (?R2=.018, p=.044). A simple slope analysis (Figure 30) showed that, the pasta tower design performance scores of females were significantly higher than the pasta tower design performance scores of males in classes using the Engineering Design Learning Module 2 (EDLM 2) (5686.31 versus 5073.84). However, the pasta tower design performance scores of females were 99 not significantly higher than males in classes using Engineering Design Learning Module 1 (EDLM 1) (2264.74 versus 4155.84). Figure 31. Gender and pasta tower design performance scores: EDLM 1 Versus EDLM 2 This result implies that compared to Engineering Design Learning Module 1 (EDLM 1), the improvement in pasta tower design performance scores experienced by females compared to males will increase when the students are exposed to Engineering Design Learning Module 2 (EDLM 2). Therefore Hypotheses 1 was supported. There was no interaction effect between gender and learning modules, explaining the other outcome variables. A limitation of this interaction analysis is that female scores used in this analysis are not a true indication of their individual scores as they are obtained from a team score. The pasta tower design lab is done in teams. EDLM 2 (Game) EDLM 1(No Game) 2200 2700 3200 3700 4200 4700 5200 5700 M A L E F E M A L E PASTA TOWER PERFORMACE SCORE GENDER 100 Effect of Race The results of the interaction effects between race and learning modules are shown in Table 14. The interaction between race and learning module, explaining the outcome variables, was not significant so Hypothesis 2 was not supported. Table 14. Interaction effect between race and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 222 223 176 176 90 90 176 Race -.113 -.112 .088 .032 .037 -.101 .057 Learning Modules (EDLM 1 vs. EDLM 2) .194** .082 .113 -.073 -.103 -.165 .011 Race ? Learning Modules .009 .059 .084 .023 .073 .134 -.026 F 3.878 1.646 1.638 .401 .599 1.342 0.226 ?R2 .000 .003 .007 .001 .004 .014 .001 Adjusted R2 .038 .009 .011 -.010 -.014 .011 -.013 Note: All tests are two-tailed.*p<.05, **p<.01, ***p<.001. Effect of learning style The results of the interaction effects between learning styles and learning modules are shown in Table 15. The interaction effect between learning styles and learning modules, as explaining the outcome variables, was not significant, so Hypothesis 3 was not supported. 101 Table 15. Interaction effect between learning styles and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 114 117 112 112 50 50 112 Learning Styles Active ? Reflective .127 -.071 .016 -.018 .097 .228 -.038 Sensing ? Intuitive .198 .019 -.118 -.194* -.137 -.313* -.109 Visual ? Verbal .261** -.108 -.090 -.103 -.16 -.153 -.148 Sequential ? Global -.060 .246** -.125 -.096 -.091 -.021 -.047 Learning Modules (EDLM 1 vs EDLM 2) Active ? Reflective .241** .138 .109 -.086 -.132 -.221 .066 Sensing ? Intuitive .243** .140 .106 -.090 -.108 -.213 .063 Visual ? Verbal .248** .134 .111 -.084 -.11 -.198 .068 Sequential ? Global .254** .099 .122 -.076 -.103 -.226 -.071 Learning Styles X Learning modules Active ? Reflective -.043 .091 .150 .140 .289 .250 .174 Sensing ? Intuitive .121 .079 -.106 -.127 .089 .114 -.117 Visual ? Verbal .142 .075 -.035 -.160 -.052 -.205 -.161? Sequential ? Global .021 -.091 -.011 -.048 -.740 -.104 -.039 F Active ? Reflective 3.047* 1.276 1.309 .988 2.159 3.825* 1.321 Sensing ? Intuitive 4.576** .981 1.429 2.409 .445 2.215 1.151 Visual ? Verbal 6.150*** 1.474 .807 1.736 .714 2.267 2.132 Sequential ? Global 2.441 3.268** 1.047 .737 .530 0.939 0.321 ?R2 Active ? Reflective .002 .008 .022 .020 .071 .053 .030 Sensing ? Intuitive .014 .006 .011 .016 .007 .011 .014 Visual ? Verbal .020 .006 .001 .026 .002 .037 .026 Sequential ? Global .000 .008 .000 .002 .003 .005 .001 Adjusted R2 Active ? Reflective .052 .007 .008 .000 .066 .147 .009 Sensing ? Intuitive .087 -.001 .011 .037 -.035 .069 .004 Visual ? Verbal .120 .012 -.005 .019 -.018 .072 .030 Sequential ? Global .037 .055 .001 -.007 -.030 -.004 -.019 Note: All tests are two-tailed.*p<.05, **p<.01, ***p<.001. 102 Effect of student enjoyment The results of interaction effects between student enjoyment and learning modules are shown in Table 16. The interaction effect between student enjoyment and learning modules, as explaining the outcome variables was not significant, so Hypothesis 4 was not supported. Table 16. Interaction effect between student enjoyment and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 88 84 90 90 90 90 90 Student Enjoyment .152 0.010 .254? .876*** .892*** .809*** .800*** Learning Modules (EDLM 1 vs. EDLM 2) .130 .206 .103 -.041 -.046 -.121** -.055 Student Enjoyment ? Learning Modules -.119 .022 -.166 .011 -.073 .033 .047 F .728 1.225 1.138 105.690*** 72.928*** 73.389*** 67.853*** ?R2 .008 .000 .014 .000 .003 .001 .001 Adjusted R2 -.009 .008 .005 .779 .708 .709 .693 Note: All tests are two-tailed. *p<.05, **p<.01, ***p<.001. Effect of goal clarity The results of interaction effects between goal clarity and learning modules are shown in Table 17. The interaction effect between goal clarity and learning modules, as explaining the outcome variables, was not significant, so Hypothesis 5 was not supported. 103 Table 17. Interaction effect between goal clarity and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 165 167 175 175 90 90 175 Goal Clarity .076 .027 .019 .536*** .563*** .746*** .541*** Learning Modules (EDLM 1 vs. EDLM 2) .240** .161* .107 -.064 -.062 -.110 .019 Goal Clarity ? Learning Modules -.099 .049 .023 -.060 -.023 -.057 -.045 F 3.524* 1.697 .723 21.463*** 12.589*** 30.45*** 21.858*** ?R2 .000 .002 .000 .003 .000 .001 .002 Adjusted R2 .044 .012 -.005 .261 .281 .498 .264 Note: All tests are two-tailed. *p<.05, **p<.01, ***p<.001. Effect of concentration The results of the interaction effects between concentration and learning modules are shown in Table 18. The interaction effect between concentration and learning modules, as explaining perceived usefulness (? = .152, p = .005), was significant and positive. The interaction effect explained 2.3% of the variance in perceived usefulness (?R2=.023, p=.005). The interaction effect between concentration and learning modules, as explaining attitudes towards engineering (? = .165, p = .002) was also significant and positive, explaining 2.7 % of the variance in attitude towards engineering (?R2=.027, p=.002). Thus, Hypothesis 6 was supported. 104 Table 18. Interaction effect between concentration and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 165 167 176 176 90 90 176 Concentration .047 .058 .110 .710*** .652*** .672*** .714*** Learning Modules (EDLM 1 vs. EDLM 2) .227** .152? .094 -.173** -.162* -.220** -.090 Concentration ? Learning Modules -.049 -.005 -.015 .152** .008 .074 .165** F 3.475** 1.665 1.444 60.162*** 22.302*** 32.240*** 60.918*** ?R2 .002 .000 .000 .023 .000 .004 .027 Adjusted R2 .043 .012 .008 .504 .418 .513 .507 Note: All tests are two-tailed. *p<.05, **p<.01, ***p<.001. Effect of higher order cognitive skills The results of interaction effects between higher order cognitive skills and learning modules are shown in Table 19. The interaction effect between higher order cognitive skills and learning modules, as explaining the outcome variables was not significant, so Hypothesis 7 was not supported. 105 Table 19. Interaction effect between higher order cognitive skills (HOCS) and learning modules Dependent variable: Learning Outcomes Variables Pasta tower Design Performance Scores Exam Score Difficulty with subject (DIFF) Perceived Usefulness (PU) Perceived Stickiness (PS) Perceived Subject Matter Learning (PSML Attitude (ATT) N 165 167 176 176 90 90 176 HOCS -.125 -.027 .089 .805*** .777*** .898*** .835*** Learning Modules (EDLM 1 vs. EDLM 2) .223** .162* .113 -.063 -.011 -.060 .020 HOCS ? Learning Modules .160 -.041 -.028 .475 -.038 -.035 -.020 F 4.774** 1.640 1.152 113.556*** 37.367*** 101.641*** 126.705*** ?R2 .021 .002 .001 .000 .001 .001 .000 Adjusted R2 .065 .011 .003 .659 .551 .772 .683 Note: All tests are two-tailed. *p<.05, **p<.01, ***p<.001. Qualitative evaluation Two external evaluators, Barbara Kuwalich and Kim Huett from University of West Georgia, Carrollton, Georgia, conducted focus group discussions with the sections who played the serious game and reported their results. There were three main evaluation questions which needed to be answered during the focus groups. They are: 1. To what extent is the engineering design serious game effective in promoting learning? 2. To what extent does use of the engineering design serious game among students in an introductory undergraduate engineering course improve student understanding of the engineering design process? 3. What are student perceptions of the engineering design serious game as an instructional method? 106 General Perceptions of the Engineering Design Learning Module (EDLM) Students said the two-week design module ?had good flow.? One Auburn student commented, ?The last two weeks have been the most mechanical of the whole semester. We?re actually building stuff as opposed to computer-based, software activities.? A self-described hands- on learner at Auburn enjoyed the two-week module, and others felt ?engaged in what [they] were doing.? Serious game feedback Students generally had positive things to say about the serious game and provided useful feedback suggesting improvements. One student said, ?I did like the game, even though it was annoying sometimes. Overall, it was pretty cool.? Another student said, ?The objectives were incredibly clear: they wanted us to think like an engineer and solve a problem. It gave clear specifications. But if we were incorrect in our design, it was vague in what we should go and correct.? Several students appreciated learning about the business or fiscal aspects of the job of engineering. One said, ?This is what you have to work with. You have to make it work, or you don?t have a job.? What has the serious game taught? Students learned about the engineering design process from the serious game. One student said the serious game modeled ?how you have to have a bunch of different ideas and decide which works best for the task.? For another, the choice of material for each scenario was a new concept 107 that he learned. Several students appreciated having cost limits: ?I liked how you have a price range because in the real world, there?s not an endless supply of money.? Another student learned the value of the ?planning process.? It was better to plan your structure and carefully build it than to rush through it and make mistakes that you would have to repair. Another student built on this idea when he said, ?In the real world, you probably won?t get a second chance to build that bridge.? Effectiveness of Serious Game in Promoting Learning and Understanding of the Engineering Design Process One of the most effective aspects of the serious game was that it allowed students to test their prototypes prior to actually building them. They felt that the serious game provided a simulation that mirrored a real work setting. For some students, the serious game stimulated their thinking about whether they had made the right decision to go into engineering as a career. The content of the serious game was not necessarily directly related to all students? majors, and there was a difference in how students from different majors perceived the serious games usefulness. For some students, the serious game provided a confirmation that they had made the right career choice. Student Perceptions of the Serious Game as an Instructional Method Students felt that the computer activity gave them control of their learning and provided them with a feedback mechanism to help them improve their design skills. Participating in the serious game made students actively and consciously think about the design process prior to doing the pasta tower exercise. The planning aspect of the serious game was also appreciated by students, 108 who recognized how important it is to try pilot test an idea prior to implementation, particularly when lives are at stake. Most Beneficial Aspects Learned Through the Engineering Design Serious Game Table 20 shows the most beneficial aspects of the design serious game based on the student comments made during the focus group session. The main ideas have been categorized based on their comments. Table 20. Most beneficial aspects of the design serious game Main Idea Supporting Explanation or Student Comments Effective Learning Experience ? Many students felt they learned from the serious game. ? A new engineer ?would have learned from the game, and the learning would have helped in the Pasta Tower [lab].? ? The game helped other students with the Pasta Tower lab. ? ?There was a carryover from the game to the real-life activity with the Pasta Tower.? ? The serious game provided a challenging, compelling learning experience Scaffolding of Student Learning ? The serious game provided an ability to design for the subsequent hands-on Pasta Tower lab. ? One student remarked, ?The game [was] helpful to someone doing a follow-up project like a pasta tower.? ? ?The SIM was good as a warm up to later classes that go into greater depth related to structural concepts.? Hands-on Learning ? The serious game allowed students to ?work with stuff and put things together? without having to ?deal with the added stress of doing the math, which hinders it.? Natural and Fun Way to Learn ? The serious game taught principles of building in a fun and enjoyable way. ? One student ?liked building things and seeing them fall.? ? ?This is a natural way of learning. ? The SERIOUS GAME was ?an overall positive experience.? Clear Learning Objectives ? The serious game established clear objectives for students? behavior in the environment. ? ?They wanted us to think like an engineer and solve a problem.? Simple Format ? The serious game presented the learning experience in a simple, straightforward format. ? The navigation was ?easy to figure out.? Immediate, Persistent Feedback ? The serious game gave immediate feedback on students? designs (in contrast to the Pasta Tower activity). 109 Important Subject Matter: Structural Concepts ? The serious game provided good introduction for learning to build structures with consideration for price and weight, making the experience simulate a real- world design project. ? It presented multiple structures for students to play with, allowing for the development of conceptual learning of physics. ? It appealed in particular to civil engineering majors, whose focus is on the building of static structures; one such major described the serious game as ?really cool.? Important Subject Matter: Engineering Design Process ? The serious game provided a good foundation for teaching students the engineering design process. ? One student said the serious game modeled ?how you have to have a bunch of different ideas and decide which works best for task.? Appealing Format for Other Simulations ? The serious game presented learning in an environment that students would like to use for learning other engineering concepts. ? Many Auburn students indicated they would enjoy learning other engineering concepts in the serious game environment Student Creativity ? One student noted how the SERIOUS GAME contrasted with other labs where specific step-by-step procedures are given: he appreciated the more open-ended, exploratory feeling evoked by the serious game. Student Self-teaching ? The serious game supported student self-teaching; one student said, ?Wisdom comes from experience, and I now have a better foundation of how to build a structure.? Any-pace Learning ? The serious game supported students? ability to work at their own pace. Visual Learning ? The serious game appealed to students? visual sense. Experiential Learning ? The serious game appealed to students? preference to learn by doing. Relevance of Game Format to Modern Students ? The SERIOUS game presented the material in a game format, which appeals to the students? need to win. ? The format was ?better than lectures.? ? Students play serious games similar to this serious game voluntarily. Real-World Transfer ? The serious game provided a realistic picture of the limitations of engineering design problems with its parameters of cost and weight. ? Students appreciated learning about the ?business or fiscal aspects of the job.? ? It was more real-world, engaging, and interactive than textbook reading. ? One student likened the serious game to job training and said, ?It?s much better than someone telling you how to do something and a lot more interesting.? Learning to Fail ? The serious game supported student willingness to ?test and not fear failure? since it is a simulation. ? With the pasta tower, by contrast, ?you could only test once.? ? The SERIOUS GAME supported the trial and error appropriate for an introductory course. ? ?You try and fail, but a little explanation would keep you engaged more instead of getting angry and fed up.? Trying Out Engineering ? The serious game assisted students in ?trying out engineering;? one student said, ?If you didn?t enjoy this type of thing, a sim like this teaches engineering majors about what they like.? ? One student said the serious game was an effective tool for helping them to learn whether they like engineering and feel they have made the right choice in major. 110 Summary This chapter presented the design and development of the serious game to teach the engineering design process to engineering class at Auburn University. Some of the important documents which were developed during the design and development of the serious game are available in the appendices. The use of the Presage-Pedagogy-Process-Product (4P) model to investigate the effectiveness of the serious game in achieving the learning outcomes was also discussed. The classroom implementation and evaluation methodology was described and the 4P model applied to investigate the effects of gender, race and learning styles on learning outcomes moderated by the learning modules. Both quantitative and qualitative analyses were conducted to more completely understand the effectiveness of the game. 111 Chapter 6 Conclusions Synopsis This chapter presents a summary of the research conducted, a review of the research findings, a discussion of its limitations and suggestions for future research. To fully understand students? learning achievement, a thorough assessment of their learning outcomes is critical. The Accreditation Board for Engineering and Technology (ABET) calls for a rigorous assessment of the student learning outcomes in all engineering programs. There is a general lack of student engagement and motivation in many engineering classes, two of the major factors affecting student retention issues in many universities. These factors come to the fore when considering the type of instructional materials presented to the students in engineering classes in any attempt to improve student learning outcomes. The flow of instructional activities in engineering classes follows a teach/teach/teach/test/test/test pattern without considering different forms of assessments and learning styles (Behrens, 2009). There is also a strong recommendation from ABET that the use of technology to teach classes be increased. This research study has described the development process for three different types of innovative instructional materials, namely multimedia case studies, smart scenarios and serious games, which can be used to teach engineering concepts in mechanical engineering classrooms. It has also shown the effectiveness of case studies in achieving learning outcomes in terms of student perceptions and their ability to engage and motivate students. The design and development of an engineering design serious game was also 112 presented, along with a comprehensive evaluation of its effectiveness in improving student learning outcomes using the Presage- Pedagogy-Process-Product (4P) model. Review of Research Findings In the past, multimedia case studies have been shown to be a very effective way of teaching real-world decision making problems in engineering and business courses (Raju et al., 1999; Sankar et al., 2001). Table 21 summarizes the research findings from the implementation of multimedia case studies, smart scenarios and serious games in engineering classrooms. Table 21. Summary of the research findings Instructional Methodology Research Findings Multimedia case studies ? Induction welding case study ? STS-51 L case study All the ratings in tables 3, 4 and 5 were highly favorable (above the neutral 3.0 rating), indicating students? approval of the case study instructional methodology. Student comments about the case studies showed that they had a positive experience with the case studies and it linked the engineering concepts taught in the class to the real-world problem Smart Scenario ? Design smart scenario Majority of the students thought that smart scenarios are an innovative way to teach engineering concepts and made them read through the material. They also found it to be realistic and different Students suggested that smart scenarios needed more gaming functionality while teaching simpler concepts Serious game ? Engineering Heights: Design Process in Action Students who worked with the game in EDLM 2 performed better in the pasta tower design challenge than the students who did not work on the game in EDLM 1. The perceived concentration level of the students who worked with game in EDLM 2 was significantly higher than the students who did not work on the game in EDLM 1 113 The multimedia case study on identifying welding defects developed for this research study were tested with undergraduate and graduate mechanical engineering students and proved to be a very popular and effective instructional technique. The majority of the students considered the case studies to be beneficial and an effective way of linking mechanical engineering concepts to real-world issues. However, some students did note that the case studies occasionally lacked student interaction/immersion and were too rich in technical content, possibly making them overly complex for all those enrolled in freshman engineering classes. As a result, it was decided to create a set of smart scenarios incorporating the feedback from the case studies in partnership with a commercial educational software company. The student responses to the smart scenarios showed that the majority of the students found them to be realistic and different. They commented that the smart scenarios made them read through the material and they liked the gaming aspect of it. However, the student responses also indicated that the smart scenarios were tedious at times, needed to break problems down further to teach simpler concepts in the relatively short classroom time available, and would benefit from more gaming functionality. Based on this feedback, the team worked with the same company as before to design, develop and test a serious game capable of teaching the concepts involved in the engineering design process to engineering students. The serious game was tested in a control/experimental setting where the students in the control section were exposed to a lecture about the engineering design process and an active learning exercise (?Statistics Applied to Data Analysis?). In the experimental set up, the students were exposed to the same lecture about the engineering design process and participated in a serious game (?Engineering Heights?). Both groups then took part in a pasta tower building challenge. 114 The students in the experimental group had significantly higher pasta tower design performance scores than the control group. This result was supported by student comments from the focus group sessions, where they commented that the serious game helped them understand the effect of different shapes and structures when designing and building pasta towers. Further analysis revealed that females in the experimental group (5686.31) had significantly higher mean pasta tower design performance scores than males in the experimental group (5073.84). This finding was consistent with the findings of a study by Schell (2008) that revealed that female students like to learn by example and prefer games that simulate real-world experience. In addition, Hwang et al. (2013) noted that females showed a positive attitude and intention to play a serious game and they experienced higher cognitive load and competition. This result involving females may provide valuable insights for future implementations of serious games involving women in science, technology, engineering, and math (STEM) education. Caucasians members of the experimental group (5357.52) had significantly higher pasta tower design performance scores than in the control group (4018.70). These results suggest that females and Caucasians both responded positively to the serious game with regard to their pasta tower design performance score. The students in the experimental group (3.72) had significantly higher mean perceived levels of concentration compared to the students in the control group (3.45). This result may suggest that students in the experimental group had higher levels of concentration while playing the serious game than the students in the control section taking part in the active learning exercise. This is consistent with studies by Jung et al. (2008) and Zhang et al. (2006), who reported that concentration is positively related to perceived web skills, perceived usefulness and ease of use. 115 Implications This research study has implications for students, educators, researchers, and funding agencies, all of whom could benefit from the products and results of this study. The engineering students can use the multimedia case studies to learn how to solve real-world issues and thus prepare themselves before they join the work force. The students can also use the engineering design serious game to learn about the design process in a fun and engaging manner. The results of this study have clearly demonstrated that the case studies and serious games are effective instructional techniques that can significantly improve student learning outcomes in engineering classrooms. Engineering faculty members who have had no experience with using case studies in their classrooms assume that they are just a short write-up of an engineering problem with a unique solution, not appreciating their scope and complexity, the range of skills that students must exercise in completing them, and the level of student involvement that can be achieved using this approach. This study may go some way towards addressing these misperceptions and encourage faculty to consider incorporating multimedia case studies in their classrooms in order to increase their students? engagement in learning and expose them to real-world decision making environments. The serious game developed for this study was shown to be a very effective instructional tool to introduce students to the engineering design process. In view of the positive benefits that can be achieved using serious games, it is critical for industries and other funding agencies to encourage engineering professors to develop more innovative serious games and implement them in engineering classrooms to improve student learning and performance. Game developers and researchers need to develop a deeper understanding of the process of designing a serious game and 116 the importance of collecting continuous feedback from the students during the design and development stage. Limitations The data used to evaluate the serious game in this research study was gathered from students attending a single university over three semesters. The generalizability of the results would be enhanced if the experiment were to be repeated over several semesters and at other institutions of higher education. The testing of the serious game concept was limited to a single topic, the engineering design process. It would be interesting to discover whether these results are repeated with other engineering concepts and subjects. In testing the 4P research model only two objective performance measures were examined. Another design project that also utilizes the engineering design process would substantially strengthen the performance results. Suggestions for Future Research This research study presents a number of possibilities for future research that could help develop a better understanding of the impact of innovative instructional materials on student learning outcomes. The following research topic areas could be considered as follow-up studies related to this project. 1. The research study presented here focuses on an undergraduate student population for the sake of better sample size and data. Recent case studies developed at the Laboratory for Innovative Technology and Engineering Education have the ability to extend this approach to improve higher order cognitive and team working skills in graduate programs. 117 2. A longitudinal study could be conducted comparing serious games, case studies and traditional lecture methodologies to determine how they affect student learning outcomes. 3. The design process used to develop this serious game could be generalized and the game used in cross disciplinary programs and business schools to reveal the effect of the serious game on non-engineering students. 118 References ABET. (2011). Criteria for Accrediting Engineering Programs. Baltimore, MD : ABET Engineering Accreditation Commission. Agarwal, R., & Karahanna, E. (2000). Time flies when you're having fun: cognitive absorption and beliefs about information technology usage 1. MIS quarterly, 24(4), 665-694. Alavi, M., Marakas, G. M., & Yoo, Y. (2002). A Comparative Study of Distributed Learning Environments on Learning Outcomes. Information Systems Research, 13(4), 404-415. American Welding Society, (2002). Welding-Related Expenditures, Investments and Productivity Measurement in U.S. Manufacturing, Construction, and Mining Industries. Retrieved on May 2006 from http://files.aws.org/research/HIM.pdf Anderson, J. (1988). Cognitive styles and multicultural populations. Journal of Teacher Education, 39, 2-9. Anderson, J. (1992). Acknowledging the learning styles of diverse populations: Implications for instructional design. Teaching for Diversity. San Francisco: Jossey-Bass. Anderson, J. (1995). Toward a framework for matching teaching and learning styles for diverse populations. In R. Sims & S. Sims (Eds.), The importance of learning styles. (pp.69) Westport, CT: Greenwood Publishing Group. Barlett, C. P., Vowels, C. L., Shanteau, J., Crow, J., & Miller, T. (2009). The effect of violent and non-violent computer games on cognitive performance. Computers in Human Behavior, 21(1), 96?102. Barnes, L. B., Christensen, C. R., & Hansen, A. J. (1994). Teaching and the case method: Text, cases, and readings: Harvard Business Press. 119 Basken, P. (2009, November 23). Obama Touts TV and Video Games as Teachers of Math and Science Skills, The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Obama-Touts-TVVideo-Ga/49247/ Biggs, J. (1970). Personality correlates of certain dimensions of study behaviour. Australian Journal of Psychology, 22(3), 287-297. Biggs, J. (1982). Student motivation and study strategies in university and college of advanced education populations. Higher Education Research & Development 1(1): 33-55. Biggs, J. (1987). Student Approaches to Learning and Studying. Research Monograph: ERIC. Biggs, J. (1992). Why and how do Hong Kong students learn?: Using the learning and study process questionnaires: Faculty of Education, University of Hong Kong. Biggs, J. B. (1989). Approaches to the enhancement of tertiary teaching. Higher Education Research and Development, 8(1), 7-25. Biggs, J. B., Kember, D., & Leung, D. Y. P. (2001). The revised two factor study process questionnaire: R-SPQ-2F. British Journal of Educational Psychology, 71, 133-149. Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom: School of Education and Human Development, George Washington University Washington, DC. Bovik, A. C. (2010). Handbook of image and video processing: Access Online via Elsevier. Bowden, J. A., & Marton, F. (1998). The university of learning: Psychology Press. Boyle, E., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2(2), 69-74. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds). (2000). How people learn: Brain, mind, experience, and school, Washington D.C.: National Academy Press. 120 Brown, R. M., Hall, L., Holtzer, R., Brown, S., & Brown, N. (1997). Gender and Video Game Performance. Sex Roles, 36(11-12), 793-812. doi: 10.1023/A:1025631307585. Cagiltay, N. E. (2007). Teaching software engineering by means of computer-game development: Challenges and opportunities. British Journal of Educational Technology, 38(3), 405-415. doi: 10.1111/j.1467-8535.2007.00705.x Chen, W. F., Wu, W.H., Chuang, T.Y., & Chou, P.N. (2011). The Effect of Varied Game-based Learning Systems in Engineering Education: An Experimental Study. International Journal of Engineering Education, 27(3), 482-487. Christensen, C. R. (1991). Education for Judgment: The Artistry of Discussion Leadership: ERIC. Coller, B.D., & Shernoff, D. J. (2009). Video game-based education in mechanical engineering: A look at student engagement. International Journal of Engineering Education, 25(2), 308-317 Coller, B. D., & Scott, M. J. (2009). Effectiveness of using a video game to teach a course in mechanical engineering. Computers & Education, 53(3), 900-912. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi: 10.1016/j.compedu.2012.03.004 Connolly, T. M., Stansfield, M., & Hainey, T. (2007). An application of games?based learning within software engineering. British Journal of Educational Technology, 38(3), 416-428. 121 Corti, K. (2006). Games-based learning: A serious business application. Informe de Pixel Learning, 34(6), 1-20. Csikszentmihalyi, M. (1998). Finding flow: The psychology of engagement with everyday life. New York: Basic Books. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper & Row. Cybinski, P., & Selvanathan, S. (2005). Learning experience and learning effectiveness in undergraduate statistics: Modeling performance in traditional and flexible learning environments. Decision Sciences Journal of Innovative Education, 3, 251?271. Dabbagh, N., & Menasc?, D. A. (2006). Student perceptions of engineering entrepreneurship: An exploratory study. JOURNAL OF ENGINEERING EDUCATION-WASHINGTON-, 95(2), 153. Dart, B. C., Burnett, P. C., Purdie, N., Boulton-Lewis, G., Campbell, J., & Smith, D. (2000). Students' conceptions of learning, the classroom environment, and approaches to learning. The Journal of Educational Research, 93(4), 262-270. Davis, F. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340. DeSantis, N. (2012, March 18). A Boom Time for Education Start-Ups, The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/A-Boom-Time-for- Education/131229/ Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 94(1), 103-120. 122 Earley, P. C., Northcraft, G. B., Lee, C., & Lituchy, T. R. (1990). Impact of process and outcome feedback on the relation of goal setting to task performance. Academy of Management Journal, 33(1), 87-105. Entertainment Software Association. (2011). Essential facts about the computer and video game industry. Retrieved from http://www.theesa.com/facts/pdfs/ESA_EF_2012.pdf Environmental Safety and Health Advisory. (2006). Explosion and fire at Texas chemical plant result from faulty welds. Retrieved from http://hss.energy.gov/CSA/csp/advisory/sad_2006-03.pdf Falkenburg, D. R. (2005). Information Technology in Support of Engineering Education: Lessons from the Greenfield Coalition. Educating the Engineer of 2020: Adapting Engineering Education to the New Century, 69. Felder, R .M. & Silverman, L. K. (1988). Learning and teaching styles in engineering education. Engineering Education, 78(7): 674-681. Felder, R. M., & Soloman, B. (1991a). Index of learning styles questionnaire. Retrieved from http://www.engr.ncsu.edu/learningstyles/ilsweb.html Felder, R. M. & Brent, R. (2005). Understanding student differences. Journal of Engineering Education. 94(1): 57-72. Feng, J., Spence, I., & Pratt, J. (2007). Playing an action video game reduces gender differences in spatial cognition. Psychological Science, 18(10), 850?855. Foss, B. A., & Eikaas, T. I. (2006). Game Play in Engineering Education Concept and Experimental Results. International Journal of Engineering Education, 22(5), 1043- 1052. 123 Fu, F.-L., Su, R.-C., & Yu, S.-C. (2009). EGameFlow: A scale to measure learners? enjoyment of e-learning games. Computers & Education, 52(1), 101-112. doi: 10.1016/j.compedu.2008.07.004 Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20-20. Guo, Y. M., & Klein, B. D. (2009). Beyond the test of the four channel model of flow in the context of online shopping. Communications of the Association for Information Systems, 24(1), 48. Hainey, T., Connolly, T. M., Stansfield, M., & Boyle, E. A. (2011). Evaluation of a game to teach requirements collection and analysis in software engineering at tertiary education level. Computers & Education, 56(1), 21-35. Hauge, J. B., & Riedel, J. C. (2012). Evaluation of Simulation Games for Teaching Engineering and Manufacturing. Procedia Computer Science, 15, 210-220. Heath, S. B. (1982). Questioning at home and at school: A comparative study. In G. Spindler (Ed.), Doing the ethnography of schooling: Educational anthropology in action. Prospect Heights, IL: Waveland Press. Heath, S. B. (1983). Ways with words: Language, life, and work in communities and classroom. New York: Cambridge University Press. Hernandez, N. V., & Rangel, G. D. (2010). Improving engineering design education: From skills to educational objectives. In Proceedings of the ASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, Vol 6. Montreal, August 15-18, 2010, pp. 729-736. 124 Hilliard, A. G. (1976). Alternatives to I.Q. testing: An approach to the identification of gifted minority children. Sacramento, CA: California State Department of Education. Hingorani, K., Sankar, C. S., & Kramer, S. W. (1998). Teaching project management through an information technology-based method. Project Management Journal, 29, 10-21. Hwang, M. Y., Hong, J. C., Cheng, H. Y., Peng, Y. C., & Wu, N. C. (2013). Gender differences in cognitive load and competition anxiety affect 6th grade students' attitude toward playing and intention to play at a sequential or synchronous game. Computers & Education, 60(1), 254-263. J. S?nchez, S. D.-C., G. Far?as, S. Dormido and F. Godoy. (2011). Understanding automatic control concepts by playing games. International Journal of Engineering Education, 27(3), 528-534. Jacobsen, C., & Zscherpel, U. (1999). Automated evaluation of digitized radiographs with neuronal methods. Journal of Nondestructive Testing & Ultrasonics(Germany), 7(2). Jones, L. (2002). Making it on broken promises: Leading African American male scholars confront the culture of higher education, Sterling, VA: Stylus Publishing. Jung, Y. H., Perez-Mira, B., & Wiley-Patton, S. (2009). Consumer adoption of mobile TV: Examining psychological flow and media content. Computers in Human Behavior, 25(1), 123-129. Kearney, P. (2005). Cognitive callisthenics: Do fps computer games enhance the player's cognitive abilities? Paper presented at the DiGRA 2005 Changing Views: Worlds in Play International Conference, Vancouver, Canada. 125 Kearney, P. & Pivec, M. (2007). Recursive loops of game based learning. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications. Vancouver, 2007, pp. 2546 ? 2553 Kerns, S.E. (2005). Keeping us on the same page, Journal of Engineering Education, 93(2), 205- 205 Kehoe, A., & Parker, G. (1990). Image processing for industrial radiographic inspection: image enhancement. British Journal of Non-Destructive Testing, 32(4), 183-190. Lamancusa, J. S., Zayas, J. L., Soyster, A. L., Morell, L., & Jorgensen, J. (2008). 2006 Bernard M. Gordon Prize Lecture*: The Learning Factory: Industry?Partnered Active Learning. Journal of Engineering Education, 97(1), 5-11. Lau, L. K. (2003). Institutional factors affecting student retention. Education-indianapolis,, 124(1), 126-136. Li, Z.-Z., Cheng, Y.-B., & Liu, C.-C. (2013). A constructionism framework for designing game- like learning systems: Its effect on different learners. British Journal of Educational Technology, 44(2), 208-224. doi: 10.1111/j.1467-8535.2012.01305. Lin, J. C.-C. (2007). Online stickiness: its antecedents and effect on purchasing intention. Behaviour & information technology, 26(6), 507-516. doi: 10.1080/01449290600740843 Locke, E. A., Shaw, K. N., Saari, L. M., & Latham, G. P. (1981). Goal setting and task performance: 1969?1980. Psychological Bulletin, 90(1), 125-152. NationalScienceFoundation. (2009). Innovations in Engineering Education, Curriculum, and Infrastructure (IEECI): National Science Foundation. 126 Nemanich, L., Banks, M., & Vera, D. (2009). Enhancing knowledge transfer in classroom versus online settings: The interplay among instructor, student, content, and context. Decision Sciences Journal of Innovative Education, 7(1), 123-148. Malhotra, Y., & Galletta, D. (2003). A multidimensional commitment model of volitional systems adoption and usage behavior. Journal of Management Information Systems, 22(1), 117-151. Maria Angeles Andreu-Andres, M. G.-C. (2011). Perceptions of Gaming as Experiential Learning by Engineering Students. International Journal of Engineering Education, 27(4), 795-804. Mayo, M. J. (2007). Games for science and engineering education. Communications of the ACM, 50(7), 30-35. Mayo, M. J. (2009). Video games: A route to large-scale STEM education? Science, 323(5910), 79-82. McGourty, J., Sebastian, C., & Swart, W. (1998). Developing a Comprehensive Assessment Program for Engineering Education*. Journal of Engineering Education, 87(4), 355-361. McPhail, I.P. (2002). Culture, style, and cognition. In Jones, L (Ed), Making it on broken promises: Leading African American male scholars confront the culture of higher education. Sterling, VA: Stylus Publishing. Mehta, A., Clayton, H., & Sankar, C. S. (2007). Impact of Multi-Media Case Studies on Improving Intrinsic Learning Motivation of Students. Journal of Educational Technology Systems, 36(1), 79-103. Meyer, J., & Land, R. (2003). Threshold concepts and troublesome knowledge: linkages to ways of thinking and practising within the disciplines: University of Edinburgh. 127 Minotti, J. L. (2005). Effects of learning-style-based homework prescriptions on the achievement and attitudes of middle school students. NASSP Bulletin, 89(642), 67-89. Minovic, M., Milovanovic, M., Kovacevic, I., Minovic, J., & Starcevic, D. (2011). Game design as a learning tool for the course of computer Networks. The International journal of engineering education, 27(3), 498-508. NCHEMS. (2008). Retention rates-First-time college freshman returning their second year. Retrieved May 26, 2011 http://www.higheredinfo.org/dbrowser/index.php?submeasure=223&year=2008&level=n ation&mode=graph&state=0 Okutsu, M., DeLaurentis, D., Brophy, S., & Lambert, J. (2012). Teaching an aerospace engineering design course via virtual worlds: A comparative assessment of learning outcomes. Computers & Education. Oppenheimer, T. (2013). Learning through Gaming. Retrieved July 17, 2013, from http://reason.com/reasontv/2013/06/17/father-of-video-games-nolan-bushnell-tal Pahl, G., Beitz, W., Schulz, H. J., & Jarecki, U. (2007). Engineering design: a systematic approach: Springer. Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students: Findings and insights from twenty years of research: ERIC. Prensky, M. (2005). " Engage Me or Enrage Me": What Today's Learners Demand. Educause review, 40(5), 60. Prensky, M. (2006). Don't Bother Me, Mom, I'm Learning!: How Computer and Video Games are Preparing Your Kids for 21st Century Success and how You Can Help! : Paragon House St. Paul. 128 Pridmore, J. L., Bradley, R. V., & Mehta, N. (2010). Methods of Instruction and Learning Outcomes: A Theoretical Analysis of Two Approaches in an Introductory Information Technology Course*. Decision Sciences Journal of Innovative Education, 8(2), 289-311. Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223-231. Raju, & Sankar, C. S. (1999). Teaching Real?World Issues through Case Studies*. Journal of Engineering Education, 88(4), 501-508. Raju, P. K., Sankar, C. S., & Xue, Y. (2004). A curriculum to enhance decision-making skills of technical personnel working in teams. European Journal of Engineering Education, 29(3), 437-450. doi: 10.1080/03043790310001658578 Sankar, C. S., Kawulich, B., Clayton, H., & Raju, P. (2010). Developing Leadership Skills in" Introduction to Engineering Courses" through Multi-Media Case Studies. Journal of STEM Education: Innovations and Research, 11(3), 34-60. Sankar, C. S., & Raju, P. (2001). Use of multi-media courseware to teach real-world decision making skills. Information Technology and Management, 2(4), 443-457. Sankar, C. S., & Raju, P. (2011). Use of Presage-Pedagogy-Process-Product Model to Assess the Effectiveness of Case Study Methodology in Achieving Learning Outcomes. Journal of STEM Education: Innovations and Research, 12(7), 45-56. Sankar, C. S., Varma, V., & Raju, P. (2008). Use of case studies in engineering education: Assessment of changes in cognitive skills. Journal of Professional Issues in Engineering Education and Practice, 134(3), 287-296. Shaha, S. H. (1998). Integrated outcomes: where CIOs need to be thinking. Health management technology, 19(10), 44. 129 Shuman, L. J., Besterfield?Sacre, M., & McGourty, J. (2005). The ABET ?Professional skills?? Can they be taught? Can they be assessed? Journal of Engineering Education, 94(1), 41- 55. Sivan, A., Leung, R. W., Woon, C.-c., & Kember, D. (2000). An implementation of active learning and its effect on the quality of student learning. Innovations in Education and Teaching International, 37(4), 381-389. Smith, K. A., Sheppard, S. D., Johnson, D. W., & Johnson, R. T. (2005). Pedagogies of Engagement: Classroom-Based Practices. Journal of Engineering Education, 94(1), 87- 101. Vazsonyi, A. (2002). Which Door Has the Cadillac: Adventures of a Real-liefe Mathematician: iUniverse. Wu, J. H., Wang, S. C., & Tsai, H. H. (2009). Why Do Players Stick to a Specific Online Game? The Users and Gratifications Perspective. 130 APPENDICES 131 APPENDIX A DETAILED TAM FOR ENGINEERING DESIGN SMART SCENARIO 132 ENGR 1110: Introduction to Engineering Toolwire Assignment Mapping (TAM): September 2, 2010 TOOLWIRE ASSIGNMENT DESCRIPTIONS: Engineering Design Principles & Written Communication Week One: Importance of Engineering Design in Product Safety Toolwire SmartScenario: Importance of Engineering Design in Product Safety DESCRIPTION: In this SmartScenario students will take on the role of a newly hired mechanical engineer with Lunar Aerospace. Our fictitious company, Lunar Aerospace, has been a key contributor to the space program in the United States and in this SmartScenario students will be introduced to the company, its projects, and their colleagues. Additionally, this SmartScenario will serve to set context for additional SmartScenarios which follow. As a new mechanical engineer with Lunar Aerospace, the student will participate in trainings, discussions, conversations, and presentations as they learn about their new company and what it means to be an engineer. The focus of this SmartScenario will be on the importance of understanding key engineering design principles. Knowledge about design principles will be gained throughout the SmartScenario as the student participates in ?new hire? training as well as through meetings and conversations they will have with their work colleagues. Upon completing this SmartScenario students will be able to do the following: OBJECTIVES: ? Understand the eight steps of engineering design ? Explain the importance of engineering design in solving real-world problems.(Demonstrate a design failure example which brings out the importance of engineering design and then relating relevant areas in STS-51 L case to engineering design) ? Understand impact of political and financial pressures on design. (Example of a design failure which brings out the financial importance while designing and then relating financial details on STS 51-L case for design) ? Analyze data presented in a graphical form ? Perform hands-on exercises to show that these concepts have been mastered Areas Explored Through Assessment Questions in the SmartScenario. The Smartscenario will provide an ability for students to answer these questions, to score them, and then to provide the best answer.: ? Draw a flowchart that explains the design process ? Describe each element of the design process ? How design process was used in STS 51-L o testing for low temperatures; chart showing different temperatures & O-ring erosion; 133 o Proper analysis was not performed. o Show graph that includes no erosion & with erosion; how the difference in depiction leads to different conclusions o Need for adding shims between the outer clevis flange and the tang of each field joint in the space shuttle that clearly indicated when the shuttle should not be launched ? Discuss the reasons why STS 51-L was launched & compare with answers given by SmartScenario Active learning Exercise Week 1: Work as a team to design an effective seal when connecting two hose pipes (provide specs) and develop a procedure to test your design. Critique your design and state what parts of the design flowchart were followed and which ones were not used? Input these in SmartScenario. Learning Objectives ? Discuss the key elements involved in designing an effective seal with PVC pipes and testing it. ? What are the possible causes of design failure when testing the above seal? ? What are the factors which contribute for a good/best design of an effective seal? Week Two: Engineering Communications Toolwire SmartScenario: Mastering Engineering Communication DESCRIPTION: This SmartScenario will be a continuation of the topics initially presented in week three with the student building upon the knowledge gained about communication and presentation styles as they prepare to present at a departmental ?brown bag? lunch and learn. This SmartScenario will focus on ways of organizing a presentation, as well as how to deal with the anxiety that sometimes accompanies public speaking. Upon completing this SmartScenario students will be able to do the following: OBJECTIVES: ? Understand importance of communication ? Show importance of audience analysis and audience-centered communication ? Evaluate documents used in communicating design concepts in real-world and point out errors/ problems (use slides from Challenger) ? Understand the best process for creating engineering communication o 1.) Plan the sequence of your presentation ? Sequencing 134 ? Clustering o 2.) Support your points with evidence ? What counts as evidence (graphs, charts, figures, data) o 3.) Make sure your materials are clear ? Do they proceed logically from one point to the next? ? Do you transition well by summarizing one point, then linking it to the next? ? Are you using audience-appropriate language? Does your audience know the technical terms you are using, or should you use less jargon? o 4.) Check your content for accuracy ? Understand the best way to create slide presentations o Organization: ? 1.) Introduction/title slide ? 2.) Overview slide ? 3.) Content slides ? 4.) Summary slide ? 5.) Questions slide o Types of slides: ? Traditional bullet points: ? Use no more than 6 points per slide ? Use no more than 6 words per point ? Anything more, and your audience will read your slide instead of listening to you. Even with this reduced content, some audience members will still just read your bullet points, so an even better presentation style is: ? Assertion-Evidence ? Make an assertion in the heading of the slide ? Provide graphic evidence (tables, charts, graphs, pictures) in the body of the slide. That way the audience has to actually listen to you in order to get the information they need. Of course, this means you also have to really know your stuff; you don?t have bullet points to fall back on! Areas Explored Through Assessment Questions in the SmartScenario. Students will be able to enter their presentation in SmartScenario & it will evaluate & provide them feedback on the following: ? What are the steps for planning a presentation? ? What is sequencing? (Putting ideas into logical order) What is clustering? (Grouping like ideas) ? What is the best way to organize a presentation? ? What are the two most common types of slides? What are the strengths/weaknesses of each? ? How many bullets should be in a presentation slide? 135 ? How many words should be used in a sentence in a slide? Active Learning Exercise: 1. A ?colleague? in SmartSceario presents the student with a the set of slides used by MTI?s engineers in the STS 51-L case study and asks for the student?s help revising them. a. Ask students to critique the slides; input these in SmartScenario & the system will provide the points that were missed 2. Then, students actually revise these slides & input this in SmartSceario. Extra Credit: Develop slides to communicate the design of the seal for garden hose and state conditions when the seal does not perform as designed. Who is the audience for this presentation? 136 APPENDIX B SOME SCENES FROM SCENARIO OUTLINE/RATIONALE FOR THE ENGINEERING DESIGN SMART SCENARIO 137 AUBUR N UNIVERS ITY ENGR1110 ? SCE NAR IO RATIONA LE TOOL W IRE DEVEL OP MENT SP ECIF IC A TION Revision History Table TABLE OF CONTENTS About this document ............................................................................................ 2 Unifying Scenario Storyline .................................................................................. 2 Characters Used Throughout Scenarios............................................................... 2 Scenarios............................................................................................................... 2 Scenario 1 ? The Importance of Engineering Design in Product Safety .......... 2 Scene 1 ? Classroom at Lunar Aerospace ..................................................... 2 Scene 2 ? Student?s Cubicle at Lunar ........................................................... 2 Scene 3 ? Lunar Outpost Cafe ....................................................................... 2 Scene 4 ? Tuesday Afternoon?s Class............................................................ 2 Scene 5 ? Student?s Cubicle .......................................................................... 2 Scene 6 ? Lunar?s Shop ................................................................................. 2 Scene 7 ? Student?s Desk on Thursday......................................................... 2 Scene 8 ? Lunar Cafe Thursday Afternoon ................................................... 2 Scene 9 ? Friday Morning Presentation in Class .......................................... 2 Version Revision Summary Date Editor 1.0 Initial Cut 09.10.10 Paul Angileri 138 Scene 10 ? Brock?s Office .............................................................................. 2 Scenario 2 ? Mastering Engineering Communication .......................................2 Scene 1 ? Orientation Classroom at Lunar Aerospace ................................. 2 Scene 2 ? Lunar Cafe..................................................................................... 2 Scene 3 ? Presentation Class ........................................................................ 2 Scene 4 ? Nitin?s Cubicle ............................................................................... 2 Scene 5 ? John Cho?s Class ........................................................................... 2 Scene 6 ? Tim?s Cubicle Wednesday Morning .............................................. 2 1. ABOUT THIS DOCUMENT The goal of this document is to ensure all stakeholders understand the products Toolwire is developing as early as possible. This is an early phase high level design document, and as such, subject to change. The Scenario back-story, characters, topics and objectives, notepad items and student calls-to- action are defined here. These high level definitions help the SME and ID write better dialog for the characters, the artist create better imagery and ensures all stakeholders agree we are headed in same direction. 2. UNIFYING SCENARIO STORYLINE Lunar Aerospace, LLC is a vital contributor to NASA?s space programs in the United States. Based in Alabama, Lunar oversees various engineering projects related to space flight systems and research and development (R&D). Lunar has recently taken on several recently graduated mechanical engineers who are progressing through their undergraduate curriculum. Lunar is going through a hiring phase, giving many local graduates the chance to get their feet wet with some of the most challenging engineering projects in the industry. There are several weeks of ramping that have to be completed to integrate new hires into the projects, but also to assess their academic progress. You are one of Lunar?s new employees, and you will be sitting through the weeks of orientation exercises and activities to prepare you. Your first two weeks are jam-packed with immersive work and interaction with Lunar?s trainers and engineering staff. CHARACTERS USED THROUGHOUT SCENARIOS These are the likely primary characters used throughout the Scenarios. As we develop the storyboards, characters and story often evolve or change. Characters may be introduced, or even removed, to aid in student learning or storyline progression. Name Position Description / Personality Brock Washington Trainer Brock is a former NASA engineer that has worked on the space shuttle project. Several years ago he left NASA to become a trainer with Lunar to develop their young engineers. 139 Jim Holley Boss Jim has been at Lunar his whole career. He?s put many years of service in at the company, and has his finger on the pulse of everything the company does. He?s a tough leader but encourages the new hires to share ideas, and go through their own trial and error process. Shauna Harris Teacher Assistant Shauna is in the process of getting her advanced degree in mechanical engineering, and is acting as Brock?s teaching assistant. She takes over for him at different points in Lunar?s new hire training. Alessa Giamo Mentor Alessa has her undergraduate engineering degree and a few years of experience under her belt. She helps Shauna with new hires. John Chu High-Grade Engineer John has a robotics engineering background, and has earned Lunar several significant patents of late. He is a very capable engineer in the field, and has a knack for communicating his ideas and concepts clearly to Lunar?s staff. Tim Sibrian Grad Student Tim got into engineering through his love of automobiles. As he approaches his graduate degree, he is anxious to start sinking his teeth into some of the more complex projects at Lunar?s facility. Nitin Shachandra New Hire Nitin is a new engineer who wants to become an automated systems specialist. He is excited to be working around Lunar?s talented staff, and is looking forward to a fruitful career there. Jorg Aakre New Hire Hailing from Norway, Jorg is a new engineer fascinated by the field. He?s bookish and can always be found in the shop late at night working on his ideas. 3. SCENARIOS 3.1.SCENARIO 1 ? THE IMPORTANCE OF ENGINEERING DESIGN IN PRODUCT SAFETY Storyline It?s your first week at Lunar Aerospace and orientation has begun. Over the course of this week you will rehearse key engineering concepts in your orientation classes and interpersonal communications. You will also get your first shot at a sample engineering problem. You are about to meet your instructor, your boss, and fellow co-learners. A teachers assistant and mentor are available for added consultation. You?ll integrate with Lunar?s team and get to work on the engineering problem. 140 At the end of this scenario, the student will be able to: ? Give the eight steps of engineering design. ? Demonstrate the importance of engineering design in solving real-world problems through a design failure example, and then relating relevant areas in STS-51 L case to engineering design. ? Explain the impact of political and financial pressures on design through a design failure example which covers financial importance while designing and then relating financial details on the STS 51-L case for design ? Analyze data presented in a graphical form and demonstrate interpretation against the STS 51-L case. ? Successfully complete hands-on exercises to show mastery of concepts. Characters ? Brock Staddington ? Jim Holley ? Shauna Harris ? Alessa Giamo ? Nitin Shachandra ? Jorg Aakre 3.1.1. SCENE 1 ? CLASSROOM AT LUNAR AEROSPACE ? The student meets with Nitin and Jorg, and goes through an orientation led by Brock. Jim also makes an appearance to go over his role and how the new hires will be reporting to him. o Narrator: The first day of work for you at Lunar Aerospace has come and it?s time for orientation. You and a couple other new hires sit in Lunar?s main classroom waiting for the session to begin. In addition to your orientation into Lunar over the coming few weeks will be your integration with a project to solve an engineering problem. You will need to apply what you?ve learned thus far to solving that problem. o Brock: I?d like to give a hearty welcome to our new hires. I think you?ll find Lunar Aerospace a fun and challenging place to work, and we?re excited to have you. In front of each of you is your schedule of activities for the week and other information like the location of your cube, etcetera. o Jim: Hello, my name is Jim. Lunar has been my home for my entire career. I think all of you will enjoy your time here. I?m the shop boss so I have oversight over all of Lunar?s projects. I run a tight ship ?round here, but I have no problem letting you grow into your roles. If you have any questions don?t hesitate to ask. o B: Thanks Jim. Well with that, why don?t you all introduce yourselves? o Nitin: Hello everyone. I?m Nitin and I just graduated last week. I look forward to my time here with Lunar, and my goal is to become an automated systems specialist. o Jorg: Hallo! I?m Jorg. I?m excited to be here. I?ve always loved engineering and feel very lucky to be working on these space projects! o (student): Hi. I am a new graduate as well. I too am excited to be able to have a hand in Lunar?s space projects. I can?t wait to get going! 141 o B: Excellent, good to hear. We use the industry standard design process. You?ll find a sheet on the STS 51-L engineering problem in your packet, and another with some test data on its design. You?re free to look those over and begin thinking about solutions. o Ji: Thanks Brock. Throughout the week you?ll be earning points on some impromptu tests. The subject of each test will be revealed when you are being tested. You also have a presentation to give this Friday about the STS 51-L problem, which you will also earn points on. We collaborate here at Lunar, but we also think a little competition doesn?t hurt. For now we?ll let you get to your cubes to get set up and complete the basic items on your list. o Narrator: The orientation class adjourns for now, and you go to your cube to complete your new hire worksheet. ? Advance to Scene 2. 3.1.2. SCENE 2 ? STUDENT?S CUBICLE AT LUNAR ? The student is at their new cubicle at Lunar setting things up later the same day. ? Student?s computer screen has new email from Jim on it. The email asks the student to type the design process in proper order, start to finish. o 10 points are awarded to the student for completing this successfully. ? This can be a drag-n-drop. ? This would be a TW self-grading step. ? The student reviews the papers from class sitting on the desk, and can flip between each of the three: the design process, the STS 51-L problem, and the data graphs ? STS 51-L problem: o Narrator: This engineering problem has some key points that need solving. You will find out over the course of the week more information about the nature of this problem. ? Data graphs: o Narrator: These graphs contain the measured data from STS 51-L tests. These graphs provide some insight as to what may be the fault points or other issues with the STS 51-L problem. ? Advance to Scene 3. 3.1.3. SCENE 3 ? LUNAR OUTPOST CAFE ? It?s Tuesday and the student is at a table in the cafe with Nitin and Jorg. They are discussing STS 51- L and their thoughts. o Narrator: You are meeting Nitin and Jorg in the cafe to talk about the STS 51-L problem. Each of you has reviewed the information from class, and you all have to begin thinking about what solutions there might be. o (student): ?Morning guys. What do you think of this problem they gave us? o Jorg: Very interesting. I was up most of the evening mulling it over. I did as much fitting of information into the design process 142 as I could. o Nitin: I think it?s deceptively simple. I took a different tack and looked at the data and what problems they were still having. o (s): I reviewed it as well. I don?t know that I went as far as either of you did. What concerns me are details about the cost of the parts. And, is there any bureaucratic red tape here, since this is a NASA problem? o J: Whoa! You can bet there is. And they will make you follow all of the design steps! o N: How far into the design process did you get Jorg? o J: I was only able to step 3, but of course we still need more information I think. We shouldn?t go too far otherwise our solution will probably be wrong. o (s): I think this is all good stuff for us to bring into class later on. o N: I agree. Let?s try and get more information and data later today. o J: (whispering) Hey did you hear about this Maximillian? o N: Yeah! Someone in my group was telling me about him. He?s pretty eccentric apparently. o J: I guess he?s the top engineer at Lunar. People rarely see him supposedly. His doctorates are in Painting and Engineering. o (s): Sounds like a pretty avant-garde guy. I?d like to meet him. o J: He?s supposedly hard to find, and rumor is he works from his own ship in geosynchronous orbit. o (s): Well that explains why no one sees him. o J: He makes appearances now and then. Supposedly. o Narrator: The three of you decide to go into class with a series of questions for solving the sample problem. You break for the morning and will rejoin in class in the afternoon. ? Advance to Scene 4. 143 APPENDIX C DETAILED STORYBOARD FOR THE SCENES IN APPENDIX B FOR ENGINEERING DESIGN SMART SCENARIO 144 Scenario 1 ? The Importance of Engineering Design In Product Safety Storyline It?s your first week at Lunar Aerospace and orientation has begun. Over the course of this week you will rehearse key engineering concepts in your orientation classes and interpersonal communications. You will also get your first shot at a sample engineering problem. You are about to meet your instructor, your boss, and fellow co-learners. A teachers assistant and mentor are available for added consultation. You?ll integrate with Lunar?s team and get to work on the engineering problem. At the end of this scenario, the student will be able to: ? Understand the eight steps of engineering design ? Define each step ? Illustrate each step from STS 51-L case study ? Analyze data presented in Table 1 in a graphical form ? Perform hands-on exercises to show that these concepts have been mastered Characters ? Brock Washington ? Jim Holley ? Shauna Harris ? Alessa Giamo ? Nitin Shachandra ? Jorg Aakre 145 ENGR1110 Scenario 1: The Importance Of Engineering Design In Product Safety Scene 1: Classroom at Lunar Aerospace Location Actor Interaction Action Notes Classroom at Lunar Image: bg_conference_ room_wood.ps d Enable Narrator Use table and chairs from ch_andrea_writing.psd Narrator Narrator dialog: The first day of work for you at Lunar Aerospace has come and it?s time for orientation. You and a couple other new hires sit in Lunar?s main classroom waiting for the session to begin. In addition to your orientation into Lunar over the coming few weeks will also be your integration with a project to solve an engineering problem. You will need to apply what you?ve learned thus far to solving that problem. Enable Brock Learner is seated in the room waiting for class to begin. Student, Nitin, and Jorg sit around a table facing Brock at the front of the room. Narrator dialog pops in at start, and out when reader is done and clicks. Brock Washington (ch_cisco-male- 5.psd, no headset) Enable Update Brock dialog: I?d like to give a hearty welcome to our new hires. I think you?ll find Lunar Aerospace a fun and challenging place to work, and we?re excited to have you. In front of each of you is your schedule of activities for the week and other information like the location of your cube, etcetera. Enable Jim Jim Holley (ch_borislav.ps d, hands out) Enable Update Jim dialog: Hello, my name is Jim. Lunar has been my home for my entire career. I think all of you will enjoy your time here. I?m the shop boss so I have oversight over all of Lunar?s projects. I run a tight ship ?round here, but I have no problem letting you grow into your roles. If you have any questions don?t hesitate to ask. Enable Brock Brock Washington (ch_cisco-male- 5.psd, no headset) Enable Update Brock dialog: Thanks Jim. Well with that, why don?t you all introduce yourselves? Enable Nitin Nitin Shachandra (ch_nitin_rearleft _sittin g) Enabl e Update Nitin dialog: Hello everyone. I?m Nitin and I just graduated last week. I look forward to my time here with Lunar, and my goal is to become an automated systems specialist. Enable Jorg Jorg Aakre (ch_jorg_rearrigh t _sittin g) Enabl e Update Jorg dialog: Hallo! I?m Jorg. I?m excited to be here. I?ve always loved engineering and feel very lucky to be working on these space projects! Enable Responder1 146 Brock Washington (ch_cisco-male- 5.psd, no headset) Enable Update Brock dialog: Excellent, good to hear. We use the industry standard design process. You?ll find a sheet on the STS 51- L engineering problem in your packet, and another with some test data on its design. You?re free to look those over and begin thinking about solutions. Enable Jim Jim Holley (ch_borislav.ps d, hands out) Enable Update Jim dialog: Thanks Brock. Throughout the week you?ll be earning points on some impromptu tests. The subject of each test will be revealed when you are being tested. You also have a presentation to give this Friday about the STS 51-L problem, which you will also earn points on. We collaborate here at Lunar, but we also think a little competition doesn?t hurt. For now we?ll let you get to your cubes to get set up and complete the basic items on your list. Enable Narrator Narrator Narrator dialog: The orientation class adjourns for now, and you go to your cube to complete your new hire worksheet. Advance to Scene 2. Responder1 Hi. I am a new graduate as well. I too am excited to be able to have a hand in Lunar?s space projects. I can?t wait to get going! Enable Brock 147 ENGR1110 Scenario 1: The Importance Of Engineering Design In Product Safety Scene 2: Student?s Cubicle at Lunar Location Actor Interaction Action Notes Student?s cubicle Image: bg_desktop 3.psd Enable Narrator Narrator Narrator dialog: You get to your cubicle and there?s a lot left to do. No doubt you have some emails waiting, and there may be a couple more objectives to make note of. The engineering problem will also need to be reviewed sooner than later. OnClick Scene2_Email1 =True Remove piggy bank, seal picture, calendar, and pager from scene, and add in a marker whiteboard behind the monitor. Email1 Enabled From: Jim Holley To: New Hires CC: Montana Jones Subject: Pop Quiz Hello everyone. Now that you?ve been introduced to the environment, I want to have you review some of the resources we?ve provided you. For 10 points, locate the steps of the engineering design process and sort them into the proper sequential order. Send your response to me by the end of the day. Don?t forget that we have all you new hires competing for points! One of our engineers from last quarter?s orientation, Montana Jones, is the current record holder for week 1 with a 94%! He holds the Lunar Aerospace trophy of achievement; it?s your job to take ownership of that trophy! Welcome to Lunar Aerospace! Jim Holley Boss Extraordinaire ATTACHMENT (01_design_principles.docx) OnClick Scene2_Email2 =True Student?s computer screen has new email from Jim on it. The email asks the student to type the design process in proper order, start to finish. 10 points are awarded to the student for completing this successfully. This is a TW self-grading step. Email2 Enabled From: Schwartzpunkt University Subject: SPAM >>> EARN YOUR ENGINEERING DEGREE IN 6 MONTHS !!1! Blah meaningless graphics in side margins of every website blah bl- blah we charge $10,000 per class blah get federal financing now! Sincerely yours, Automated computers in warehouses running ponzi schemes OnClick Scene2_STS51L =True This is a satirical aside for the student as they move through the scenario. STS 51L Problem Enabled Narrator dialog:This engineering problem has some key points that need solving. You will find out over the course of the week more information about the nature of this problem. OnClick Scene2_DataGr aphs=True One set of papers on the desk that when clicked shows the engineering diagram and key points. Narrator bubble appears under or above the diagram. 148 DataGraphs Enabled Narrator dialog:These graphs contain the measured data from STS 51-L tests. These graphs provide some insight as to what may be the fault points or other issues with the STS 51-L problem. If Scene2 _Emails ==True && Scene2 _STS51 L==Tru e && Scene2_DataGr aphs==True Then Advance to Scene 3 Another set of papers on the desk that when clicked shows data graphs from the STS 51-L problem. Narrator bubble appears under or above the graphs. 149 ENGR1110 Scenario 1: The Importance Of Engineering Design In Product Safety Scene 3: Lunar Outpost Cafe Location Actor Interaction Action Notes Lunar Outpost Cafe Image: bg_cafeteria_ LunarAero.psd Enable Narrator Narrator Narrator dialog: You are meeting Nitin and Jorg in the cafe to talk about the STS 51-L problem. Each of you has reviewed the information from class, and you all have to begin thinking about what solutions there might be. Enable Responder1 Learner is in the caf? and runs into Nitin and Jorg. Nitin and Jorg are at the table facing the student, not each other. Responder1 ?Morning guys. What do you think of this problem they gave us? Enable Jorg Jorg Aakre (ch_brett_co nf- room.psd) Enable Update Jorg dialog: Very interesting. I was up most of the evening mulling it over. I did as much fitting of information into the design process as I could. I feel good about my chances, since I got those first 10 points from Jim. Enable Nitin Nitin Shachandra (ch_henrik- side.psd, facing front) Enable Update Nitin dialog: I got my points too. I think this is a deceptively simple problem. I took a different tack and looked at the data and what problems they were still having. Enable Responder2 Responder2 I reviewed it as well. I don?t know that I went as far as either of you did. What concerns me are details about the cost of the parts. And, is there any bureaucratic red tape here, since this is a NASA problem? Enable Jorg Jorg Aakre (ch_brett_co nf- room.psd) Enable Update Jorg dialog: Whoa! You can bet there is. And they will make you follow all of the design steps! Enable Nitin Nitin Shachandra (ch_henrik- side.psd, facing front) Enable Update Nitin dialog: How far into the design process did you get Jorg? Enable Jorg Jorg Aakre (ch_brett_co nf- room.psd) Enable Update Jorg dialog: I was only able to step 3, but of course we still need more information I think. We shouldn?t go too far otherwise our solution will probably be wrong. Enable Responder3 Responder3 I think this is all good stuff for us to bring into class later on. Enable Nitin Nitin Shachandra (ch_henrik- side.psd, facing front) Enable Update Nitin dialog: I agree. Let?s try and get more information and data later today. Enable Jorg 150 Jorg Aakre (ch_brett_conf - room.psd) Enable Update Jorg dialog: (whispering) Hey did you hear about this Maximillian? Enable Nitin Responder4 You mean Montana? I haven?t heard of a Maximillian. Enable Jorg Nitin Shachandra (ch_henrik- side.psd, facing front) Enable Update Nitin dialog: Yeah! Someone in my group was telling me about him. He?s pretty eccentric apparently. Enable Jorg Jorg Aakre (ch_brett_conf - room.psd) Enable Update Jorg dialog: I guess he?s the top engineer at Lunar. People rarely see him supposedly. His doctorates are in Painting and Engineering. Enable Responder5 Responder5 Sounds like a pretty avant-garde guy. I?d like to meet him. Enable Jorg Jorg Aakre (ch_brett_conf - room.psd) Enable Update Jorg dialog: He?s supposedly hard to find, and rumor is he works from his own ship in geosynchronous orbit. Enable Responder6 Responder6 Well that explains why no one sees him. Enable Jorg Jorg Aakre (ch_brett_conf - room.psd) Enable Update Jorg dialog: He makes appearances now and then. Supposedly. Enable Narrator Narrator Narrator dialog: The three of you decide to go into class with a series of questions for solving the sample problem. You break for the morning and will rejoin in class in the afternoon. Advance to Scene 4. 151 APPENDIX D IRB APPROVAL FOR ENGINEERING DESIGN SMART SCENARIO 152 153 154 155 156 APPENDIX E DESIGN FLOWCHART OF A BUILDING BLOCK 157 158 159 APPENDIX F PROGRESSION MATRIX FOR THE DESIGN SERIOUS GAME 160 9 8 9 7 8 9 I 2 3 4 SCENES BB1 - BB5 Assessment questions with remediation on engineering design ACT I ? Define the term design ? Importance of design ? Need for minorities and females in design ? Complexity of design ACT II & III ? Overview of design process ? Understanding the first step of design process(problem definition) BB6, BB7 6 5 7 Assessment questions with remediation on design process and problem definition ACT IV ? Understanding concept formation ? Tower design activity with qualitative evaluation ACT V ? Understanding concept evaluation ? Tower design activity with quantitative evaluation (with actual numbers) ACT VI ? Understanding concept selection ? Students have to give reasons for all the selections 8 9 BB8 ? BB12 BB13 - 15 BB16, BB17 Assessment questions with remediation on concept formation and tower design specifications Assessment questions with remediation on concept evaluation with simulations Assessment questions with remediation on concept selection BB8 ? BB12 BB13 -15 BB16, BB17 Assessment questions with remediation on concept formation and tower design specifications Assessment questions with remediation on concept evaluation with simulations Assessment questions with remediation on concept selection ? Conversion of stated goals in to measureable engineering parameters. ? Identify Engineering parameters and their effects on cost weight and strength. ? Constant and variable parameters ? Compare their parameter list to the ideal list (2nd apprentice) ? Qualitative evaluation of tower cost, weight and strength ? Quantitative evaluation of tower cost, weight and strength (numbers) ? Compare concepts with the ideal solution (2nd apprentice) BB13 - 15 BB16, BB17 Assessment questions with remediation on concept evaluation with simulations Assessment questions with remediation on concept selection BB16, BB17 Assessment questions with remediation on concept selection 161 APPENDIX G DESIGN GAME OUTLINE 162 Design Game outline January 20, 2012 Pramod & Joseph Game focuses on tower example with multiple choice selections for each design step activity. Story Idea: Seeking to turn their talent and love for tinkering into designing things of real use to people the student has come to Mr. I. K. Brunel?s workshop, the place the designers of the things that fascinate you got their start. The workshop is a unique looking place that is part shipyard, rail yard, construction firm, and factory. The student has been recruited to be part of Mr. Brunel?s design competition team. Act I: Learning Objectives: Define the term ?Design? Show the importance of design The student is welcomed to Brunel?s by Mr. Brunel himself as his new apprentice. He says that being an apprentice is a bit old fashioned but you are here to learn how to do design not just study it. Scene 1: Brunel begins to tell the student what systematic or scientific design is. BB1: Brunel asks the student ?What is systematic or scientific design??(Student preconception question) a) Making clothes b) Arranging furniture and artwork in a room c) Creating artwork d) Making the most useful thing with the least cost. Brunel asks the student what is engineering, answer ?The science by which the properties of matter and the sources of energy in nature are made useful to man in structures, machines, and products.? Webster?s Unabridged Dictionary Brunel asks the student what is the difference between an engineer and a scientist. The student gets a multiple choice answer selection, answer is ?A Scientist studies what is. An Engineer creates that which has not been.? Theodore Von Karman. What will you be doing in systematic or scientific design? Creating new things that are useful in a rigorous systematic (scientific) way. 163 Conversation continues with more of what design is, importance of design to final product success, and needing to plan how you go about the design. Systematic or Scientific Design (Definition): This is the process of devising a system, component, or process to meet desired needs. It is a decision making process (often iterative) in which the basic sciences, mathematics, and engineering sciences are applied to convert resources optimally to meet a stated objective. Among the fundamental elements of the design process are the establishment of objectives and criteria, synthesis, analysis, construction, testing, and evaluation. It is essential to include a variety of realistic constraints such as economic factors, safety, reliability, aesthetics, ethics, and social impact. Accreditation Board for Engineering and Technology (ABET, 1996). Question activity: What is engineering design? a) It is a decision making process Remediate: Yes, you make decisions but what is your goal? b) Designing a new system or a component Remediate: Yes, you are making new things but how are you doing the creation? c) Creating systems to fit human needs Remediate: Yes, human factor is important but it is just a part of engineering design. d) All the above Correct answer!!! e) None of the above Remediate: Really!! None of the statements above apply to design. Act II: Learning Objective: Provide overview of process of systematic or scientific design Scene 2: Provide overview of systematic or scientific design process BB2(BB6):Design process overview. 2nd apprentice shows student the design process diagram and says how anything can be designed with it. 2nd apprentice takes the student through diagram pointing out that the arrows don?t just flow down the diagram but can go back up to any step. 164 Design is thus a primary component of product development. The attention given to a product during the design stage has a direct bearing on the future costs and the performance of a product. Good design includes mechanisms to predict and correct failures before their occurrence. With no investment of time and money in systematic design, the costs associated with failure escalate rapidly. Simply designing a product or system is often no longer sufficient. The design process must be iterated often in order to improve quality, reduce costs, and prevent failure. The safety of a product or a system must be considered, as should public opinion. If the design process is not taken seriously, products will not sell, businesses will collapse, and competitors will thrive. To compete effectively with others, good design techniques must be implemented as a tool of continuous improvement. The design process consists of several distinct steps which are shown in the flow chart in Figure 1. Figure 1: Elements in the Design process 165 Simply designing a product or system is often no longer sufficient. The design process must be iterated often in order to improve quality, reduce costs, and prevent failure. The safety of a product or a system must be considered, as should public opinion. If the design process is not taken seriously, products will not sell, businesses will collapse, and competitors will thrive. To compete effectively with others, good design techniques must be implemented as a tool of continuous improvement. The design process consists of several distinct steps which are shown in the flow chart in Figure 1. BB3: Problem Definition: We start the design process by determining just what you are trying to achieve and what you have to work with. The stated need or problem is converted from human terms into measureable quantities. The importance of the parts of the problem to reaching the desired goal are determined. BB4: Concept Formation: Now you know what the problem is, you apply your knowledge and imagination to create solutions to it. Different concepts/alternatives to solve the problem are created by applying science and engineering theories to generate valid solutions to the problem. Not only is the acquisition of the technical theories important, the engineers? imagination is also essential in the design process. BB5: Concept Evaluation: To focus in on a workable solution to the problem you think through the strengths and weaknesses of each of the solutions you thought up. The concepts created to solve the problem are checked for feasibility when cost, technological limitations, legal issues, environmental impacts, and time are taken into account. BB6: Concept Selection: Based upon the evaluation results the concept that solves the problem and is most feasible is selected for further development. BB7: Detailed Design: The selected concept is now developed in detail, from the overall shape to the size and placement of the smallest screw. BB8: Prototyping: During prototyping, either full-size, scaled, and/or virtual models of the product are built to further determine the merit of the idea and to test different aspects of the design. 166 BB9: Testing: The prototype is tested to see if the design meets the specifications and success goals from the problem definition step. BB10: Send to production: Once testing proves that the product is of acceptable quality, the product can be produced for the customer. BB11: Question Activity: Question 1. What is the first step in the design process? a. Concept formation. b. Concept selection. c. Problem definition. (Answer) d. Testing. Question 2. When do the designers? know they have succeeded? a. Problem definition. b. Concept evaluation. c. Prototyping and testing. (Answer) d. Send to production. BB12: Pick and Place activity: Place process steps in the correct order on a blank diagram. 167 Act III: Understand the first step of the design process (problem definition) by going through a simulated example. Learning Objective: Explain conversion of stated goals into measureable design parameters. Explain what are design parameters and their effects on each other, weight, strength, cost, etc. Explain problem decomposition procedure. Scene 3: Tower Design Contest BB 13: Here is your first design problem. The problem for this year?s design competition is to make a tower that will hold up as much weight as possible. The tower also must resist side loading and earthquakes. The tower is to be three feet tall. It can be made of steel, concrete, or wood. The tower can weigh no more than 22 lb. The tower can cost at most $86. The tower must support a minimum16,000 lb. Now you get to define the design problem from what was stated. Scoring for the contest will give preference to designs that hold up the most weight for the least cost and weight. BB14: Problem Definition Explanation: When defining a problem requiring a new design, it is crucial that the critical characteristics or constraints be determined and documented. For example if a team is asked to design a car navigation system, they need to know the physical size constraints before the design process can proceed. The team may believe that the ideal location for this device is in the dashboard of the car. However, the initial models of the navigation system might be too large for the dashboard. Therefore, steps must be taken to either reduce the size of the navigation system or relocate pieces of it to other areas of the vehicle. Without a consideration of these constraints early on, significant amounts of time and money may be wasted. Game note: A problem is defined by asking multiple questions so that the scope of the problem is revealed. Student is now presented with the task of defining the problem in measureable terms. Student creates problem definition statement by choosing measureable properties and setting their importance. Brunel asks questions about what measureable performance goals the tower must meet. If student gets them wrong 2nd apprentice helps for remediation. 168 Question activity: Multiple choice questions in text to test understanding of concepts. Question 1: What is the simplest statement of what the design is to do? Ans: a. Hold a weight above the ground. Correct b. Be a 3 ft tall tower. Remediation: What does tower mean? c. Be 3 ft tall. Remediation: Is that all it is to do? Question 2: What is the measurable design goal for ?hold a weight?? a. Force carried by the tower. Correct. b. Volume of building material. Remediation Questions: Filling a volume holds a weight? All objects have volume but does that resist the force at the top of the tower? c. Stability. Remediation Question: The hold part of ?hold a weight? does indicate the tower should not wobble but what is being held stably? Ans: the force of weight. Question 3: What are the specified amounts in the problem statement? Ans: a. 3 feet in height. b. Weight. Remediation: Is the size of the weight actually specified? Ans: No. c. ?As much as possible?. Remediation: That is a goal not a specific amount. 169 BB15: Competitive activity: Start tower problem. Select and order design parameters for successful tower design. Compare parameter list with 2nd Apprentice?s to see why the order is important. Design parameters list: Weight, Strength (Beams & Joints), Structure (Shape), Cost, Appearance 2nd Apprentice?s list: (Correct Answer) Structure, Strength, Weight, Cost, Appearance Brunel: The order can change depending on the problem. Brunel: You have the measure of the problem, state what is its essence as simple as you can. Student chooses simplest problem definition from list. 2nd apprentice helps with hints if needed. Act IV: Understanding the second step in the design process (Concept Formation) Learning Objectives: Explain what problem solution concepts are. Explain concept creation procedures. Scene 4: Start Tower Design Concept Formation BB16: Introduction to Concept Formation and Evaluation Activity: Game choice/interface introduction (Design Game introduction exercise.docx) where user sees how design choices effect problem outcome. Scripting will determine when to start activity, before, in parallel, or after the dialogue for BB17-18. 170 BB17: Concept Formation: Once the problem has been identified, engineers must develop different concepts/alternatives to solve the problem. These concepts could be derived by creatively applying their knowledge of science and engineering theories to generate valid solutions to the problem. Not only is the acquisition of the technical theories important, the engineers? imagination is also essential in the design process. One well established method of developing valuable ideas is through brainstorming, an extremely useful tool for idea generation. This allows a team of people to rapidly suggest and reject ideas in a manner that inspires and encourages all involved. At this stage, no ideas should be evaluated in detail. In order for brainstorming to bring successful results, there are a few guidelines that must be adhered to. First, all team members must have a positive outlook. Negative thoughts or comments should be restricted. Participants must be willing to hear any and every idea despite its possible absurdity. The selection of a facilitator or session coordinator who is responsible for hindering the stating and development of negative comments is very helpful. The facilitator also coordinates idea development by establishing a sequence around the room which allows everyone the opportunity to speak. This speaking sequence, if properly enforced by the session coordinator, ensures the equalization of overpowering or outspoken team members. During this brainstorming session, the ideas mentioned must also be captured using videotapes, audiotapes, or well-written notes. These captured concepts will be evaluated at another time in detail. Abiding by these guidelines will create an atmosphere of encouragement and acceptance. BB18: Concept Evaluation: Having developed some apparently feasible ideas for solving a problem, these ideas must be reduced in number based on factors such as cost, technological limitations, legal issues, environmental impacts, and time. This is an important step in estimating the value of the solutions the team has come up with. Without some estimate of how much money the company can earn from the product, many ideas may not be excluded as they should be. For example, a company may estimate that a new product design could generate revenues of $1 million over its lifetime. If it costs more than $10 million to produce, then the net loss on the product will be $9 million and the idea is not feasible. In other instances, certain technologies may need to be developed before the product can be made economically. The costs associated with research and development of the technology may exceed the value of the product. However, an idea that was conceived years ago might become economically viable with new technology and can benefit the company at the present time. 171 Brunel: Now that you have your problem in measureable terms, how can you achieve those measures. 2nd apprentice and Brunel explain Concept Formation formal definition and practice. Apply your creativity and knowledge to form concepts that might achieve the design goals. Here are a set of concepts for you to choose from. Which do you think look like they could work? You will get qualitative evaluation information about each possible choice. Student selects tower design details. Choice effects weight, cost, and tower strength. Student sees cost and weight display change with each selection. Strength is shown in test phase. All choices result in a completed tower of specified height. BB19: Tower Detailed Design and Prototyping Competitive activity: Part 1: Student selects tower shape and material choices. Choices are evaluated for weight and cost. If cost over budget or weight over limit design choices must be redone. Student can choose to change choices to optimize tower design or keep current choices. Please refer to the Design_game_tower_activity_09_29_11.ppt in Toolwire Delivery Files folder in dropbox Student performs Tower Design activity from start to Screen 2. BB20: Detailed Design: Once several feasible ideas have been evaluated, the best concepts are selected. An enormous amount of time is spent on determining the specific characteristics of each piece of the product. The anticipated specifications are usually communicated through the use of engineering drawings and specification sheets (often called ?spec sheets?). Part 2: Student selects beam length and joint type choices. Choices are evaluated for weight and cost. Load estimate is given. If cost over budget or weight over limit design choices must be redone. Student can choose to change choices to optimize tower design or keep current choices. Please refer to the Design_game_tower_activity_09_29_11.ppt in Toolwire Delivery Files folder in dropbox Student performs Tower Design activity from Screen 2 to Screen 3. 172 BB21: Prototyping: With detailed drawings and specifications completed, the product can be submitted to the prototype stage. During prototyping, either full-size, scaled, and/or virtual models of the product are built to further determine the merit of the idea and to test different aspects of the design Questions concerning applicable engineering theories, costs, construction time, etc. need to be answered. BB22: Tower load test Part 3: Student selects test order. Animations of tests are shown listing maximum loads tower will support and withstand. Please refer to the Design_game_tower_activity_09_29_11.ppt in Toolwire Delivery Files folder in dropbox Student performs Tower Design activity from Screen 3 to End. BB23: Testing: Testing of the prototypes is the next phase. Prototypes may go through several design iterations before the final prototypes are made. The final prototypes should be very close to the target product. Characteristics such as appearance, materials, and performance will be matched closely with the expected production line item. To check product performance, testing must be conducted to ensure that the product meets explicit specifications. Specifications give the testing process the benchmarks necessary to evaluate the product. Student selects order of load tests for tower. Student sees effect of choices on tower strength. Choice affects which tower properties shown. Choosing complex tests first make diagnosing tower failures more difficult. Scene 5: Final Design Selection BB24: Send to production: If testing proves that the product is of acceptable quality, then the product can enter the production phase. Thoughts of the production phase likely begin during the detailed design stage. As engineers become more experienced, they will consider not only the design of the product components, but also how the components will be made. Given two equal possibilities for product construction, the one that is proven or easier to manufacture might be the best alternative. Developing some idea of how a product should be manufactured, often as early as the design stage, can help to speed up the design process. 173 The user now decides if the tower design is worth preparing for the competition. User finalizes design or starts over. Once user finalizes choice they will see the score comparing their design to the optimum design. Score computed from Success equation. Success= 0.15(optimum weight / tower weight) + .35(optimum tower cost / tower cost) +.3(tower vertical load/spec. load)+.1(tower side load/spec side load)+.1(earthquake pass(1) fail(0)) 174 APPENDIX H POWERPOINT PRESENTATION FOR TOOLWIRE DEVELOPERS AND MANAGERS 175 176 177 178 179 180 181 182 183 184 185 APPENDIX I ILLUSTRATION OF STORYBOARD FOR THE DESIGN SERIOUS GAME 186 187 188 189 190 APPENDIX J DETAILED USER FLOW EXPERIENCE FOR ALL LEVELS OF THE DESIGN SERIOUS GAME 191 Overview User flow experience 192 Lab Introduction user flow experience 193 Building game user flow experience 194 Main Game user flow experience 195 APPENDIX K IRB FOR TESTING THE DESIGN SERIOUS GAME 196 197 198 199 APPENDIX L LECTURE ON ENGINEERING DESIGN PROCESS 200 201 202 203 204 APPENDIX M STATISTICS APPLIED TO DATA ANALYSIS 205 ENGR 1110 Fall 2012 Statistics Applied to Data Analysis 50 points The analysis of collected data is a common activity in the practice of engineering. In professional practice data is not a list of numbers in a textbook problem. Data is a measure of the problems you must overcome or of your success. In this lab exercise you will be collecting information on the duration of light falling on the sensor of the robot. The light is controlled by a card with a hole that the light passes through. The card is spun by a motor. The motor speed is controlled by the volume of sound in the room. Divide into your robot lab teams and complete the following tasks. Task 1: Control Sound Selection Select a song to provide the sound to control the speed of the motor rotating the card. The sound source needs to be close to the sound sensor. Select a song with noticeable volume changes for best results. If you do not have a music playing device select a song on the computer and use the speakers. Test the effect of you song on the speed of the card by directly observing the speed of rotation while the song plays. If there is not a noticeable change in the speed of rotation move the sound source closer to the senor or change songs. Use the orange button on the robot control brick to start the disk spinning and the dark grey button to stop it. Task 2: Data Collection Turn on the light and adjust the disk until the hole is in front of the light. Position the light sensor of the data collection robot so it is in the light shining through the hole. On the data collection robot press the orange button and select Data Log from the menu choices. It is in the same list as NXT programs and software. After selecting Data Log select Ambient light from the list of data types. Set the sensor port to the port where the light sensor is connected. Then select Done. The robot is now set to record data from the light sensor. Press the orange button to begin recording data. Press the dark grey button to stop collecting data. After you have stopped press the orange button to save the data file. The file name will be OBD#.log. The file name is automatically created by the robot. Start your music and then start the disk spinning. Start data collection and collect data for two minutes. Stop the disk and music. Turn off the light. Take the data collection robot to your table and plug it into the PC. You will need to open LabView 2009. Open a new VI. Target the VI to the NXT. Select Tools from the list at the top of the VI. Select NXT Tools from the drop down menu. Select NXT terminal from the slide out menu. The NXT terminal window will open. Select the OBD#.log file with the highest number. The OBD files are numbered as they are created; the highest number is the most recent. Click the Save to PC button to send the OBD file to the PC. Once you have your file return the robot to experimental apparatus. Task 3: Statistics Applied to Data Analysis Use Excel to open you OBD file. It will be listed as a text file in the Excel open window. The first column of numbers is the time from start that the data was collected in milliseconds. The second column of numbers is the reading from the light sensor in percent saturation. A reading of zero is no light on the 206 sensor. A reading of 100 is bright light on the sensor. Plot the data on an XY scatter with only markers graph. Look at where the peaks are in your data. Record the width in seconds of each peak. Record the spacing in seconds between the peaks. You will produce two lists of numbers. Label the width numbers ?Peak Width? and the spacing numbers ?Peak Spacing?. Compute the mean, median, and standard deviation of a sample for both the Peak Width and Peak Spacing. Task 4: Report of Findings As a group write a one page single spaced report on your activities during this lab. You need to describe the following in your report: The experimental apparatus used to produce the data. How you controlled the speed of the disk. What occurred during the data collection. The results produced by your statistical analysis of the data. Submit your report and Excel file through Canvas. Each team member is to submit a copy of the report and Excel file. Bonus 10 points: Disk rotation speed. Compute the speed of the disk in rotations per minute from your data. The hole allows the light to reach the sensor once per revolution. Compute the mean, median, and standard deviation of the RPM. Show the computation in your Excel sheet. List your RPM finding in your report. 207 Grading Rubric Points 10 7.5 5 2.5 0 Report Format Report Language Correct Format Professional Language and Grammar Minor format errors Misspellings Multiple format errors Unprofessional Language Ignored format Poor Grammar Report less than 1 page Difficult to Understand Points 10 7.5 5 2.5 0 Question 1 Complete description of topic easily understood by reader. Some missing details but description of topic understandable. Description is becoming vague to reader. Many gaps in description of topic. Missing topic. Question 2 Complete description of topic easily understood by reader. Some missing details but description of topic understandable. Description is becoming vague to reader. Many gaps in description of topic. Missing topic. Question 3 Complete description of topic easily understood by reader. Some missing details but description of topic understandable. Description is becoming vague to reader. Many gaps in description of topic. Missing topic. Question 4 Complete description of topic easily understood by reader. Some missing details but description of topic understandable. Description is becoming vague to reader. Many gaps in description of topic. Missing topic. 208 APPENDIX N SURVEY 209 Design Activity Questionnaire ? Spring 2013 Engineering design learning module includes materials covered in class/ lab and the following activities that you performed: 1. Lecture on engineering design process (PowerPoint presentation done by the instructor) 2. Active learning exercise (Either analyzing statistics with a robot light sensor, or review statistics/unit conversion lecture, or performing design simulation exercise) Q1 Please enter the number provided by your instructor Q2 Select the name of your university ? Auburn University ? Hampton University Q3 Please select your lab section from the list below ? 13 ? 14 ? 15 ? 21 ? 24 ? 33 ? 37 Q4 Gender ? Male ? Female Q5 Major ? Business or sub-discipline ? Engineering or sub-discipline ? Other (please list) ____________________ Q6 Status ? Freshmen ? Sophomore ? Junior ? Senior ? Graduate Student Q7 Race ? White ? African-American ? Hispanic 210 ? Asian-American ? American Indian ? Other Q8 Please rate the degree to which you agree or disagree with the following statements in this questionnaire by bubbling in or clicking on the response according to the following 5-point scale 211 Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree I was absorbed intensely in the engineering design learning module. ? ? ? ? ? My attention was focused on the engineering design learning module ? ? ? ? ? I concentrated fully on the engineering design learning module ? ? ? ? ? I was deeply engrossed in the engineering design learning module ? ? ? ? ? Using the engineering design learning module improved my performance ? ? ? ? ? Using the engineering design learning module enabled me to accomplish my tasks more quickly ? ? ? ? ? I found the engineering design learning module useful ? ? ? ? ? Using the engineering design learning module increased my productivity ? ? ? ? ? Using the engineering design learning module enhanced my effectiveness ? ? ? ? ? Using the engineering design module made it easier to do my work ? ? ? ? ? My goals were clearly defined in the engineering design learning module ? ? ? ? ? I knew clearly what I wanted to do in the engineering design learning module. ? ? ? ? ? I had a strong sense of what I wanted to do in the engineering design learning module ? ? ? ? ? 212 I know what I wanted to achieve in the engineering design learning module ? ? ? ? ? I became more interested in the concept of engineering design process ? ? ? ? ? I gained a good understanding of the concept of engineering design process ? ? ? ? ? I learned to identify central ideas in the area of engineering design process ? ? ? ? ? I developed the ability to communicate clearly about the concept of engineering design process ? ? ? ? ? I was stimulated to do additional work in the area of engineering design process ? ? ? ? ? I found the engineering design learning module to be a good learning experience ? ? ? ? ? Q9 Please rate the degree to which you agree or disagree with the following statements in this questionnaire by bubbling in or clicking on the response according to the following 5-point scale 213 Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree From my experience in this engineering design learning module I believe engineering is a subject learned quickly by most people ? ? ? ? ? Based on my experience in this engineering design learning module I believe I have trouble understanding engineering because of the way I think ? ? ? ? ? My experience in this engineering design learning module has shown me that engineering concepts are easy to understand ? ? ? ? ? From my experience in this engineering design learning module I believe engineering is irrelevant to my life ? ? ? ? ? I felt that completing this engineering design learning module was stressful ? ? ? ? ? As a result of participating in this engineering design learning module I believe learning engineering requires a great deal of discipline ? ? ? ? ? This engineering design learning module failed to expand my working knowledge of what goes on in engineering ? ? ? ? ? This engineering design learning module has increased my appreciation for engineering ? ? ? ? ? From the engineering design learning module experience I think engineering is highly technical ? ? ? ? ? 214 During this engineering design learning module I felt insecure when I had to do engineering homework ? ? ? ? ? This engineering design learning module has shown me that I can learn Engineering ? ? ? ? ? Engineering skills learned in this engineering design learning module will make me more employable ? ? ? ? ? The instructional materials in this engineering design learning module helped me identify engineering tools that will assist me in decision-making ? ? ? ? ? In this engineering design learning module I learned how to inter- relate important topics and ideas using the instructional materials ? ? ? ? ? In this engineering design learning module I learned how to identify various alternatives/ solutions to a problem using the instructional materials ? ? ? ? ? The instructional materials in this engineering design learning module improved my problem solving skills ? ? ? ? ? I learned how to sort relevant from irrelevant facts using the instructional materials in this engineering design learning module ? ? ? ? ? 215 The engineering design learning module was integrated in a way that made it easier to learn new engineering concepts ? ? ? ? ? The engineering design learning module emotionally engaged me in learning the topics ? ? ? ? ? Q10 Please rate the degree to which you agree or disagree with the following statements in this questionnaire by bubbling in or clicking on the response according to the following 5-point scale Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Learning to use the engineering design learning module for performance-based activities is easy for me ? ? ? ? ? I find the engineering design learning module flexible to interact in performing work-related tasks and activities ? ? ? ? ? I find it easy to get the engineering design learning module to do what I want to do in performing work- related activities ? ? ? ? ? It is easy for me to become skillful at using the engineering design learning module in work-related activities ? ? ? ? ? I find the engineering design learning module easy to use at work ? ? ? ? ? My interaction with the engineering design learning module at work is clear and understandable ? ? ? ? ? 216 Q11 Please rate the degree to which you agree or disagree with the following statements in this questionnaire by bubbling in or clicking on the response according to the following 5-point scale Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree The learning module has been enjoyable ? ? ? ? ? I enjoyed many aspects of this learning module ? ? ? ? ? This was one of my favorite learning modules ? ? ? ? ? I had fun working on this learning module ? ? ? ? ? Q12 Please rate the degree to which you agree or disagree with the following statements in this questionnaire by bubbling in or clicking on the response according to the following 5-point scale Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree I would stay longer on this learning module than others ? ? ? ? ? I intend to prolong my staying on this learning module ? ? ? ? ? I would visit this learning module as often as I can ? ? ? ? ? I intend to link to this learning module when I am studying design process ? ? ? ? ? Q13 Thank you for completing this survey. Please click the next button to submit your responses. 217 APPENDIX O PASTA TOWER BUILDING ACTIVITY 218 Pasta Tower Design Lab You will be divided into teams. You are to design a structure to support the greatest load possible at the greatest height using the listed materials, tools, and criteria. Your design work will be in four phases each producing its own type of deliverable information. Phase 1: Design (50 minutes ) Design a tower to meet the following criteria using the provided tools and materials. Use the Tower Design Worksheet to record your team?s application of the design process. You will turn in your team?s Tower Design Worksheet at the next lab meeting. The criteria you must meet are: 1. The structure will be able to support weights that are loaded on top your structure. 2. The tape is to be used to hold the structure together. 3. The structure is to be at least 12 inches tall when completed. 4. The structure must remain standing when the top of the structure is displaced 2 inches latterly. If the structure falls over 5 points will be deducted from your teams grade. 5. The structure is to support the most weight possible at the greatest height possible. The team whose structure has the greatest pound-inch2 rating will receive 5 bonus points. The winning team will be determined by multiplying the height to the bottom of the weight by the load at failure, pound-inch2 rating =load x height2. The materials you have to use are: 1. One 16oz box of spaghetti. 2. One roll of masking tape. 3. 120 minutes of time. The tools you can use are: 1. One pair of scissors. Do not cut the spaghetti with the scissors. 2. One pair of pliers. Use the wire cutter at the pivot end of the jaws to cut the spaghetti. 3. A tape measure. 4. A Sharpe marker. Phase 2: Construction (50 minutes ) 219 Your group will construct the structure you designed in Phase 1. Document any changes you make to the design as you construct the structure on your sketch. Phase 3: Testing ( 40 minutes ) Your group will test your design by loading weight onto it until it fails. Record the amount of weight loaded at the time of failure and the behavior of the structure that caused the failure. The team that designs a structure that has the highest Tower Performance number will receive 5 bonus points. Tower Performance = Tower Height2 x (Supplies weight/Tower weight) x Load Supported by Tower. Give your design notes to the TA at the end of testing so he can make a copy of it. Turn in your teams Tower Performance Data Sheet. Phase 4: Design Performance Report Writing Start Your team will produce a 1? page long single spaced report describing: 1. How you went about the design process explaining the reasons for your design decisions. 2. The form of the structure you designed in detail. 3. The construction process, with any changes you made to the original design and problems encountered while constructing your structure. 4. The performance of the structure when loaded and how it failed. Give the pound-inch2 rating of your structure. Use your design notes to write your report. Submit your report to Canvas. Each member of your team is to submit a copy of the report. The reports are due two days after your lab meeting. Tower Performance Data Sheet Team Member ID Numbers from Canvas grades page. ______________________ _____________________ ______________________ _____________________ ______________________ _____________________ Tower height _______________ 220 Tower weight_______________ Supplies Weight_______________ Load Supported by Tower_____________ Tower Performance calculation Tower Height2 x (Supplies weight/Tower weight) x Load Supported by Tower = Tower Performance ___________2 x (_____________ / __________) x _____________________ = ________________ 221 TOWER DESIGN WORKSHEET Tower Design Worksheet Engr. 1110 Fall 2012 Team Members _______________________________ ______________________________ _______________________________ ______________________________ _______________________________ ______________________________ 1. Problem Definition Define the need or problem in measureable terms. State your problem as simply as possible. 2. Concept Formation Think of at least three Concepts that meet the requirements from the Problem Definition. Write a short description of each concept and make a simple sketch of it. Concept 1: Concept 2: Concept 3: Concept 4: 3. Concept Evaluation List each Concept?s strengths and weaknesses. Do the Concepts meet the Problem?s engineering requirements? How do the Concepts? performances compare to each other? What effects the performance of the Concepts? Do any of the Concepts solve the problem? Do you have the time, tools, and resources the Concepts need? Concept 1 Evaluation: 222 Concept 2 Evaluation: Concept 3 Evaluation: Concept 4 Evaluation: 4. Concept Selection Which Concept or Concepts best meet the requirements? Which Concept can you implement with the resources you have? If no concept meets all the requirements alone, combine the concepts that meet only part of the requirements. If none of the current concepts really work, go back and make new concepts or refine the problem definition. You can ask for another worksheet if your team needs it. Selected Concept: 5. Detailed Design Now you apply engineering science to take your concept from idea to reality. Where do the physical parts go? How big are the parts? What materials are used to make the parts? How are the parts made? What specifications must the parts meet to perform their functions? How are the parts put together and in what order? The Detailed Design answers all questions about the final design solution. Give a description and a sketch of your design. Leave room to record changes you may make while constructing your tower. 223 APPENDIX P DESIGN PROCESS QUESTIONS ASKED IN THE MIDTERM EXAM 219 1. What is the first step in the Design Process? a. Concept Evaluation. b. Prototyping. c. Testing. d. Problem Definition. 2. What limits design freedom as you progress through a design project? a. Concept selection errors. b. Management?s financing decisions. c. Design team personalities. d. Previously made design decisions. 3. In trying to produce the best solution to the design problem what may it be necessary to do after each step of the design process? a. Improve the appearance of the product to please management. b. Reduce the cost to get financing. c. Re-specify the problem using knowledge gained. d. Conduct a focus group interview of the customers. 4. What is always missing at the beginning of a design problem? a. Problem information. b. Testing facilities. c. Manufacturing equipment. d. A design team. 4. During which step of the design process is creativity used to apply to Sciences to produce solutions to the design problem? a. Concept Evaluation b. Concept Formation c. Detailed Design d. Concept Selection Contents Chapter 1 ....................................................................................................................................................... 1 Introduction ............................................................................................................................................... 1 219 Group E: Is asked to conduct a feasibility study on the methods used for automatic weld defect detection and find out the best method suitable for the given images. ............................................... 35 Summary ................................................................................................................................................. 64 Effectiveness of Serious Game in Promoting Learning and Understanding of the Engineering Design Process ..................................................................................................................................... 107 Student Perceptions of the Serious Game as an Instructional Method .......................................... 107 Game focuses on tower example with multiple choice selections for each design step activity. ............. 162 Act I: Learning Objectives: Define the term ?Design?...................................................................... 162 Scene 1: Brunel begins to tell the student what systematic or scientific design is. ........................... 162 Act II: Learning Objective: Provide overview of process of systematic or scientific design ............. 163 Scene 2: Provide overview of systematic or scientificdesign process.................................................... 163 Figure 1: Elements in the Design process ................................................................................................ 164 BB11: ........................................................................................................................................................ 166 Question Activity: ..................................................................................................................................... 166 Question 1. What is the first step in the design process? .......................................................................... 166 Question 2. When do the designers? know they have succeeded? ........................................................... 166 Explain problem decomposition procedure. ............................................................................................. 167 Scene 3: Tower Design Contest .............................................................................................................. 167 BB 13: Here is your first design problem. ............................................................................................ 167 Question activity: Multiple choice questions in text to test understanding of concepts. ......................... 168 Question 1: What is the simplest statement of what the design is to do? ................................................ 168 Ans: a. Hold a weight above the ground. .............................................................................................. 168 Correct ...................................................................................................................................................... 168 Ans: a. 3 feet in height. .......................................................................................................................... 168 Act IV: Understanding the second step in the design process (Concept Formation)........................ 169 Learning Objectives: .............................................................................................................................. 169 Explain what problem solution concepts are. ........................................................................................... 169 Explain concept creation procedures. ...................................................................................................... 169 Scene 4: Start Tower Design Concept Formation ............................................................................... 169 BB16: Introduction to Concept Formation and Evaluation ............................................................... 169 Activity: Game choice/interface introduction (Design Game introduction exercise.docx) where user sees how design choices effect problem outcome. Scripting will determine when to start activity, before, in parallel, or after the dialogue for BB17-18. .............................................................................................. 169 219 BB19: Tower Detailed Design and Prototyping ................................................................................... 171 Competitive activity: ................................................................................................................................. 171 Student performs Tower Design activity from start to Screen 2. .............................................................. 171 BB22: Tower load test ............................................................................................................................ 172 Student selects order of load tests for tower. ............................................................................................ 172 Scene 5: Final Design Selection ............................................................................................................. 172 BB24: Send to production: .................................................................................................................... 172 If testing proves that the product is of acceptable quality, then the product can enter the production phase. Thoughts of the production phase likely begin during the detailed design stage. As engineers become more experienced, they will consider not only the design of the product components, but also how the components will be made. Given two equal possibilities for product construction, the one that is proven or easier to manufacture might be the best alternative. Developing some idea of how a product should be manufactured, often as early as the design stage, can help to speed up the design process. ..................... 172 The user now decides if the tower design is worth preparing for the competition. User finalizes design or starts over. Once user finalizes choice they will see the score comparing their design to the optimum design. Score computed from Success equation. ..................................................................................... 173 Success= .................................................................................................................................................... 173 0.15(optimum weight / tower weight) + .35(optimum tower cost / tower cost) +.3(tower vertical load/spec. load)+.1(tower side load/spec side load)+.1(earthquake pass(1) fail(0)) ................................. 173