An Investigation into the Benefits of Design Science Research for the Development of Wicked Educational Information Systems: A Case Study

Abstract

Wicked educational information systems (IS) address problems with unstable educational requirements, in educational environments that are ill-defined, where complex interactions among subcomponents of the problem and its solution exist. In this dissertation, a design science research methodology (DSRM) was used to evaluate projects that developed, implemented, tested, and evaluated two wicked educational IS: multimedia case studies and serious games. The original DSRM model was changed so that it could be used as the theoretical framework for this dissertation. An analysis of each project using this model answered the research question: how can a design science research methodology lead to the development of wicked educational IS? In project one, multimedia case studies were implemented over a three year period in an Introduction to Engineering course with 696 students who were divided into experimental (multi-media case studies) and control (round table discussion) groups. A presage-pedagogy-process-product (4P) model was used to develop hypotheses. Survey data were collected using validated scales from literature and hierarchical linear regression analyses were conducted to test the hypothesized relationships of the 4P model. External evaluators collected qualitative data in the form of open-ended survey responses and focus group feedback. Concurrent triangulation was applied to compare quantitative and qualitative results, revealing that female and minority students using a multimedia case study methodology earn higher grades than those using a round table discussion methodology. In project two, a serious game was developed by a company, in cooperation with a university research laboratory, to increase student immersion in the topic of engineering design methodology. Students simulated building a water tower and a train bridge with constraints placed on the weight, cost, and load of their structure. As students progressed through the game, the level of difficulty increased. The project was implemented during three concurrent semesters of an Introduction to Engineering course with 238 students using experimental (serious games) versus control (lecture) groups. Hypotheses were developed using the 4-P model, survey data were collected using validated instruments, and analyzed using hierarchical linear regression analyses. External evaluators collected qualitative feedback in the form of open-ended survey responses and focus group feedback. Findings suggest that female students using a serious game methodology earn higher scores than those engaged in traditional lectures. In addition, students using a serious game experience higher levels of goal clarity compared to students in traditional classroom settings. Analysis of each project using the DSRM model revealed that emphasis on planning, communication, and rigorous evaluation provide significant benefits. These findings led to development of a refined DSRM for wicked educational IS that provides detailed guidelines, via step-by-step recommendations, for potential developers of wicked educational IS.