AADMLSS Practice: A Culturally Relevant Algebra Tutor Except where reference is made to the work of others, the work described in this thesis is my own or was done in collaboration with my advisory committee. This thesis does not include proprietary or classi ed information. Lingyan Wang Certi cate of Approval: Kai H. Chang Professor Computer Science and Software Engi- neering Juan E. Gilbert, Chair Professor Computer Science and Software Engi- neering Cheryl Seals Associate Professor Computer Science and Software Engi- neering George T. Flowers Dean Graduate School AADMLSS Practice: A Culturally Relevant Algebra Tutor Lingyan Wang A Thesis Submitted to the Graduate Faculty of Auburn University in Partial Ful llment of the Requirements for the Degree of Master of Science Auburn, Alabama Dec. 18, 2009 Vita Lingyan Wang, daughter of Fengzhe Hu and Daji Wang, wife of Dr. Shaoen Wu, mother of David Sixing Wu, was born in Sichuan, China. She is a master student in the Computer Science and Software Engineering Department at Auburn University. Ms. Wang received a Master of Library and Information Science degree from Chinese Academy of Science in 2007. She received a Bachelor of Computer Management Information System from University of Electronic Science and Technology of China in 2002, an Associate diploma from Sichuan Normal University in 1998. iii Thesis Abstract AADMLSS Practice: A Culturally Relevant Algebra Tutor Lingyan Wang Master of Science, Dec. 18, 2009 (M.S., Chinese Academy of Science, 2007) (B.S., University of Electronic Science and Technology of China, 2002) 63 Typed Pages Directed by Juan E. Gilbert This thesis presents an online web-based culturally relevant algebra tutor sys- tem. It is designed for African American students to improve their mathematical skills through practice on this platform. As a critical part of a culturally relevant tutor framework AADMLSS, this system particularly focuses on practicing algebra. But it can also easily be adapted to other courses with proper changes in problem repository and problem solving engine. This platform consists of two interface modules at the front-end and one database at the back-end. One of interface modules, called Algebra Problems Management, is designed for instructors to manage algebra problem pool and to track the activities that students perform in solving algebra problems. The other interface module, Algebra Problems Practice, is shared by instructors and stu- dents. This module is the environment where students can extract an algebra problem and practice solving this problem through interaction with a 3D virtual tutor. This module is hooked into Algebra Problems Management module as one of its elements so that instructors can preview the environment a student might be in when solving an algebra problem. With practice on this online web system that is tailored with iv the familiar cultural elements to students, they are expected to improve their ability and skills in mathematics. An extensive usability experiment is also conducted to evaluate e ectiveness, e ciency and user satisfaction of this system. Experimental result shows that AADMLSS Practice is e ective and e cient in interface usability and functionalities helping student practice their algebra problems. v Acknowledgments I would like to express my deepest gratitude to my advisor, Dr. Juan E. Gilbert, for his patient guidance, continued encouragement and nancial support throughout my graduate studies. He has been extremely helpful to me. I would also like to thank my graduate committee members, Dr. Chang and Dr. Seals for their reviewing and advising e orts. Additional thanks must also be given to the members of the HCCL lab for advice, opinions, and support. Special thanks go to my family for believing in me and encouraging me in my decision to pursue my goals, particularly to my husband Shaoen Wu for his continuous support. This work is dedicated to my lovely son David Sixing Wu. vi Style manual or journal used Journal of Approximation Theory (together with the style known as \aums"). Bibliograpy follows van Leunen?s A Handbook for Scholars. Computer software used The document preparation package TEX (speci cally LATEX) together with the departmental style- le aums.sty. vii Table of Contents List of Figures x 1 Introduction 1 2 Related Work 4 2.1 Culturally Relevant Instruction . . . . . . . . . . . . . . . . . . . . . 4 2.2 Adaptive instruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 De nition of Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 Animated Tutors/Agents . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.5 AADMLSS System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.5.1 System Structure of AADMLSS . . . . . . . . . . . . . . . . . 8 3 AADMLSS Practice Design 11 3.1 Architecture of AADMLSS Practice . . . . . . . . . . . . . . . . . . . 11 3.2 Algebra Problem Management . . . . . . . . . . . . . . . . . . . . . . 12 3.2.1 System Structure of APM . . . . . . . . . . . . . . . . . . . . 12 3.2.2 Process Flow of APM . . . . . . . . . . . . . . . . . . . . . . . 15 3.3 Algebra Problem Practice . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3.1 System Structure of APP . . . . . . . . . . . . . . . . . . . . 18 3.3.2 Process Flow of APP . . . . . . . . . . . . . . . . . . . . . . . 20 3.4 back-end Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.1 E-R Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.2 Tables in Database . . . . . . . . . . . . . . . . . . . . . . . . 26 3.5 User interface design . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4 Evaluation Methodology 34 4.1 Environment and tools . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Scenarios/Tasks in Evaluation . . . . . . . . . . . . . . . . . . . . . . 35 4.3.1 Scenario/Task 1: Add an algebra word problem into AADMLSS Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3.2 Scenario/Task 2: Browse, edit and preview the algebra prob- lem(s) just added . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3.3 Scenario/Task 3: Solve algebra problems . . . . . . . . . . . . 37 4.3.4 Scenario/Task 4: Browse interactive data . . . . . . . . . . . . 38 viii 4.4 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5 Experimental Results and Discussion 41 5.1 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2 Interface Results . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6 Conclusion 47 Bibliography 48 Appendix A 50 Appendix B 51 ix List of Figures 2.1 System Structure of AADMLSS . . . . . . . . . . . . . . . . . . . . . 8 3.1 Architecture of AADMLSS Practice . . . . . . . . . . . . . . . . . . . 12 3.2 System Structure of APM . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3 Process Flow of APM . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.4 System Structure of APP . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.5 Process Flow of APP . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.6 E-R Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.7 APP?s Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.8 APM?s Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.9 Instructors? Login Interface . . . . . . . . . . . . . . . . . . . . . . . 29 3.10 Students?s Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.11 Error Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.12 List Problems Interface . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.13 List Students? Interaction Interface . . . . . . . . . . . . . . . . . . . 32 3.14 Solving Equation Interface . . . . . . . . . . . . . . . . . . . . . . . . 32 5.1 Age Distribution of Participants . . . . . . . . . . . . . . . . . . . . . 42 5.2 System Usability Score . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.3 Average Score of Each Item in Post-survey Questionnaire . . . . . . . 46 x Chapter 1 Introduction Achievement in mathematics for African American students is a serious concern in mathematics education. The 7th National Assessment of Educational Progress (NAEP) administered a study in 1996. NAEP reported that White students scored higher than African American students at all grade levels [1,2]. Results from NAEP showed that African American and Hispanic students consistently scored below White students. At the fourth grade level, 28% of White students are at or above the pro - cient achievement level, while 8% of Hispanic students and only 5% of African Amer- ican students are at or above the pro cient achievement level. Studies show that the access to algebra instruction in middle school can help students complete higher levels of math in high school and college to gain access to important technological elds of study at advanced levels [3]. Furthermore, the issue of math and youth has a far- reaching and devastating impact on students throughout their lives in other aspects, including limited opportunities in higher education, employment, and earnings [4]. Although progresses have been made in closing this gap between white students and minorities in mathematics achievement, there is still much work to do. One way of addressing this issue is through designing innovative pedagogies that use culture to improve mathematics achievement amongst African Americans, essentially, integrat- ing culture into the mathematics curriculum to promote students? mathematical un- derstanding. African American Distributed Multiple Learning Styles System, called AADMLSS, was created for this purpose. AADMLSS is funded by NSF (National Science Foundation) and a collaboration research project among several educational 1 institutions, including Auburn University, Clemson University, Boston University, Portland State University, and Texas A&M University, with the goal of advancing African American communities through the use of innovative information technolo- gies. It is an online educational system that incorporates culture and sophisticated instructional tools into learning environments attempting to improve a student in learning experience and academic performance [5]. AADMLSS utilizes a hierarchical model to organize the educational material contained in the system. There are three layers in a hierarchical structure, starting from top to bottom: Courses, Module, and Concept. The Courses represent the highest layer in the hierarchy. The middle layer, that is, the Module, can be viewed as instructors of the course. Each instructor adopts his/her own unique teaching style to teach the course. The Concept represents the lowest layer in the hierarchy, as a lesson in the course. Currently, AADMLSS contains a course for Algebra. Additional courses can be added later. A more detailed elaboration of the system structure is discussed in the next chapter. The Algebra course consists of three components: instruction, assess- ment, and practice. The practice of the Algebra course, named AADMLSS Practice in the remaining of the thesis, is one part of AADMLSS. AADMLSS Practice is also the core of this thesis work. With the completion of AADMLSS Practice, this thesis contributes in three aspects. The rst is a platform for instructors to manage an alge- bra problem pool and to preview the procedures or the environment where a student solves algebra problems. Instructors can add new problems, edit or delete existing problems. The second contribution is a platform that allows students to practice solving algebra problems anytime and anywhere. The third is the connection be- tween the above two platforms that tracks students? learning patterns and di culties they encounter in solving algebra problems. To evaluate the usability of this system, 2 an extensive study is conducted. The results indicate that AADMLSS Practice is a system more fun, more helpful and easier for students to learn algebra. The remainder of this thesis is organized as following: Chapter 2 provides a de nition of culture, the introduction of AADMLSS, a virtual agent technology and a review of research related to virtual agent systems and interactive applications. Chapter 3 presents a thorough description of this thesis project: the architecture of AADMLSS Practice and its logical and physical designs. Chapter 4 describes the methodology of the usability study on this system: data collection, setting, subjects, scenarios and procedures. Chapter 5 discusses the main outcomes of the usability experiment. Finally, Chapter 6 summarizes the work and discusses improvements and recommendations for the future. 3 Chapter 2 Related Work This chapter describes culturally relevant instruction, adaptive instruction, and presents a de nition of culture. After that, animated tutors/agents are introduced. We review some researches related to virtual agent systems and interactive applica- tions in education. Then, we explain the details of AADMLSS: system structure and three components: Instruction, Assessment, and Practice. 2.1 Culturally Relevant Instruction Culturally relevant instruction as a method of teaching African American stu- dents can improve their school success. It empowers students intellectually, socially, emotionally, and politically by using cultural relevance to impart knowledge, skills, and attitudes [6]. In culturally relevant instruction, ethnomathematics is the study of the relation- ship between mathematics and culture. It refers to a broad cluster of ideas ranging from distinct numerical and mathematical systems to multicultural mathematics ed- ucation. Ethnomathematics contributes both to the understanding of culture and the understanding of mathematics, but mainly to appreciating the connections be- tween these two [7]. Several educators and researchers sharing or supporting this concept [8{10] argue that the implementation of culture into the mathematics class- room assists students in making connections among mathematic concepts and pro- motes students? understanding of mathematics. When culture is integrated into the 4 mathematics curriculum, ethnomathematics becomes a tool valuable in mathematics classrooms to help students make connections and develop deeper mathematical un- derstanding [11]. At the heart of culturally relevant instruction is the culture of the learner. In order to use ethnomathematics to teach African American students, before an instructional program that is relevant to students is designed, their predominant culture must be identi ed. Some recent research studies reveal that mathematic instruction for African Americans promotes their learning when the course content takes into consideration rap music and video games prevailing in African American culture. The content used within AADMLSS, the parent project of this thesis work, is collected from African American educators who teach primarily African American students and by research teams of the AADMLSS. In AADMLSS Practice, an algebra tutor is created in the appearance of an African American instructor. In the case where culture is incor- porated into the mathematics learning, e.g. ethnomathematics, culturally relevant instruction can be shown anywhere within AADMLSS system. 2.2 Adaptive instruction Adaptive instruction refers to the instruction that is tailored to the learning style for each learner. In some studies, it has produced promising results [12{14] in accommodating personalized learning styles. Adaptive instruction is used in the form of Many-to-One (M-1) or Multiple Instructor Single Learner (MISL) instructional model in real practice. The M-1 instructional model refers to the relationship between instructors and learners. In this model, there can be many educators per learner so that a single learner can have access to multiple pedagogical approaches. 5 2.3 De nition of Culture Culture is most commonly used in two basic senses. One is an integrated col- lection of human knowledge, belief and behavior that depends upon the ability to think and learn. "Our perceptions of objects and events in the natural world are strongly dependent on our store of prior knowledge ... we view the world through a pair of ?conceptual goggles? ", according to Mintzes [15]. The other is a set of shared attitudes, values, goals and practices that characterize an institution, organization or group. Rogo says culture is not a biological trait, but a dynamic repertoire of practices that are in uenced by an individual?s prolonged participation in cultural communities [16]. It has a signi cant impact on conceptual goggles in speci c com- munities. Once someone is identi ed with a speci c culture, this is manifested in many ways, e.g. appearance, thought and lifestyle. AADMLSS de nes culture in detail through two dimensions: who we are and what we do. Who we are refers to our physical and/or observable attributes, such as appearance including height, weight, skin tone and eye color, and other attributes, e.g. age, gender and class. The second dimension is what we do, which includes daily activities and regular participation we practice, such as music, religious beliefs and political a liation. 2.4 Animated Tutors/Agents Animated tutors appear on the screen as embodied characters that exhibit var- ious types of human-like behaviors, such as speech, emotions, and gestures e.g. eye, head and body movements. It is widely used in web-based instructions. Lester and 6 Stone claim that the presence of an animated agent in an interactive learning envi- ronment can positively a ect students? perceptions of how well they can learn in this environment [17]. Numerous tutors have been developed and designed in education. Nel is an interactive physics tutor designed by a mechanical engineering professor at Auburn University to teach introductory physics. It guides students through the systematic steps of solving problems by emulating a one-on-one tutoring session with a professor. Cognitive tutor by Carnegie is an intelligent instructional software program, designed as an object for a curriculum to be used in a classroom environment where it combines computer-based training, text, and teacher-led classroom instruction that supports guided learning [18]. Algebrator, an intelligent algebra tutor, was designed to teach pre-college algebra. A student can enter any symbolic problem. The tutor will then solve the problem step-by-step and provide context-sensitive explanations [19]. In this thesis, we adopt the animated tutor technology into Algebra Problem Practice (APP) platform. The creation and implementation of a web-based mathe- matical learning environment that incorporates animated agents/tutors may lead to improvement in mathematics achievement and attitudes [20]. Results of statistical tests hint that the presence of animated agents in web-based mathematics education and positive reinforcement may have led to the improvement of students? attitudes toward mathematics and mathematics achievement [21]. 2.5 AADMLSS System AADMLSS has a 3D and multimedia game-like environment with rap lyrics and animated African American characters used as conversational tutors, which integrates 7 Lessons Courses Module Module Concept Concept Concept Concept Instruction Assessment Practice Instructors Courses Figure 2.1: System Structure of AADMLSS a familiar culture into the learning of algebra concepts and provides the look and feel of an urban setting. 2.5.1 System Structure of AADMLSS As shown in Figure 2.1, AADMLSS models three levels to organize the edu- cational material in a system: Courses, Module, and Concept. The system may have a set of courses. Currently this system provides an Algebra course for learn- ers. Each course includes many lessons that can be further divided into one or more concepts, which can be viewed as Lessons. Concepts are the smallest unit of instruc- tion. Between courses and concepts, there are Modules that are M-1 relationship instructional model, in which many instructors are available to one student. The primary advantage of this adaptive model allows the student to identify a teaching style with which s/he feels most comfortable. An instructional unit, or a Concept, has three sub-components: Instruction, Assessment, and Practice described in detail in the following. 8 Instruction component of AADMLSS is a great tool for presenting the algebraic material to the student in a culturally relevant manner. This system resembles a typical urban environment that most African American kids would recognize, such as the candy store, fruit stand and murals like real world entities. For example, the rst algebra lesson covers simple linear equations. After the student is asked to enter his or her name e.g. Nicole as an avatar to interact with 3D environment, the Instruction component begins when Nicole enters the candy store to buy candy, the total price of her purchase is given and Nicole need to calculate the price of a single candy bar by presenting the solution to this linear equation using rap lyrics. The student is able to pause and repeat the animation as needed anytime. This continues until the student has interacted with each of the worlds within the environment. When a student completes the Instruction component, the student is guided into the Practice component. The Practice component aims to provide chances for the student to practice and reinforce what was learned from what was demonstrated in the Instruction. More details are discussed in the next chapter. After the Practice component is completed, the student is given an assessment. The assessment component provides a typical algebra quiz to evaluate the student?s ability to master the concepts covered in the Instruction and Practice components. It has a typical computer-based testing interface or a simple paper-pencil assessment. The student can interact with this assessment by using keyboard and mouse. The assessment has a pass threshold set by the instructor, which is around 80%, based on Mastery Learning Theory. Particularly, if the student scores equal to or above the set threshold, s/he moves on to the next logical algebra lesson with the same instructor model as in the current algebra lesson. However, if the student scores below the threshold, s/he will be assigned the same algebra lesson with a di erent instructor style using the M-1 adaptive instruction model. 9 For every algebra lesson, the procedure can be repeated from the Instruction com- ponent to the Assessment component. The Instruction and the Practice components both use animation technologies to train the student. In the Instruction component, the student interacts with this system in a highly engaging 3D animation game-like environment with rap lyrics and sounds. In the Practice component, the student interact with a tutor through typing. The Assessment component of AADMLSS is paper-pencil or computer-based testing. In the Assessment component, no advanced technologies are used. In closing, this section presents some relevant concepts that are involved in the system, introduces virtual agent technology, and provides applications of current vir- tual agent systems. We mainly overview the system structure of AADMLSS and describe its components in detail: Instruction, Assessment, and Practice. In the next section, the AADMLSS Practice component, the core of this thesis work, is described at length. 10 Chapter 3 AADMLSS Practice Design AADMLSS Practice, the core of this thesis work, is the component in AADMLSS system where a student can practice what s/he learned in the Instructional component . This chapter presents the architecture of AADMLSS Practice, system structures, process ows, user interface design and database related design: E-R diagram and tables. 3.1 Architecture of AADMLSS Practice Figure 3.1 shows the architecture of AADMLSS Practice. It consists of two components: Algebra Problems Management, named APM, and Algebra Problems Practice, dubbed APP. APM is the interface platform in AADMLSS Practice on which instructors can manipulate algebra problems repository. APP allows algebra problems practice with a culturally relevant tutor and logs the interactive behaviors in solving a problem into the database repository. Through "Preview" function in APM, instructors can also experience exactly the environment where students practice algebra problems in APP except there is no interactive data to be stored into the repository. Therefore, APM is the interface for instructors only. But, APP is the one for students and also the Preview subcomponent for instructors. AADMLSS Practice has a typical Back-Front framework. Users interact with APM and APP as the interfaces to the system. The back-end, the database repos- itory, processes the user inputs from the front-end and responds to users? request if 11 User Repository AADMLSS Practice Algebra Problems Management(APM) Algebra Problems Practice(APP) Preview Front-end Back-end Figure 3.1: Architecture of AADMLSS Practice necessary. The user interface in AADMLSS Practice is a web based system. There- fore, an inexpensive computing equipment running web server and MySQL database server is enough to support this system. Expanding to hundreds of classes simply requires the scaling of server hardware and network environment. Since AADMLSS Practice is a web-based application, anyone familiar with a web browser can easily use this system. 3.2 Algebra Problem Management 3.2.1 System Structure of APM Algebra Problem Management (APM) is the user interface for instructors in AADMLSS Practice. Figure 3.2 depicts its system structure. APM consists of four modules: User Authentication, Add Problem, List Problems and Trace Student?s Interaction. 12 Algebra Problems Management(APM) User Authenticationn Logout AddProblem Edit Login Delete Preview ListProblems TraceStudents?Interaction Figure 3.2: System Structure of APM User authentication is used to determine whether users can access web pages, what data sources can be accessed in algebra problems repository and whether users can be guaranteed to swift passage from one place to the next. It also performs the security gate keeping for access to any internal web pages: if a user types in the web browser any other internal page web link without passing authentication, s/he will be redirected to the login web page for authentication rst. To carry user identity information for both security and logging reason, user authentication relies on browser cookies because most modern browsers can support cookies. A cookie, like a key, can enable a user to access from web page to web page in the site by memorizing the user?s unique information through his/her whole session. Without a cookie, every time the user opens a new web page even in the same session, the web server treats him/her like a completely new visitor. There are two types of cookies: session cookies and persistent cookies. Session cookies are temporary and are erased when clients close the browser at the end of their sur ng session. When they visit that particular site at the next time it will not recognize them and will 13 treat them as completely new visitors as there is nothing in their browser to let the site know that they have visited before. Persistent cookies remain on client machine?s cache on the hard drive until they are erased or they expire. Persistent cookies can be valid for months or years depending on how long the visited web site has programmed the cookie to last. Therefore, session cookie provides better security than permanent cookie while o ering the same convenience. For security reason, session cookie is adopted in the design of this work. APM uses session cookies, which are valid and alive only for current session. Users? username information and passwords, which are entered though HTML forms in login web page, are stored in session cookies and are encrypted when they are passed back and forth between clients and servers. For example, when the user enters her username, e.g. "wly" and password, e.g. "111111", "wly" and "111111" can be stored temporarily in the browser cookie. To use session cookies, it is required that browsers enable cookies. Add Problem is one of primary modules of AADMLSS Practice. It provides the interface for instructors to add more algebra questions into the question repository to be used by other models, e.g. List Problem module and APP. In this module, there are multiple mandatory elds that instructors have to input in adding a question, such as Title, Problem and Equation of a problem. Another primary module of AADMLSS Practice is List Problems, which allows instructors to browse all problems existing in the current problem repository. This function is useful when an instructor intends to add, update or delete a problem. It consists of three sub modules: Edit, Delete, and Preview. Through Edit, an instructor is able to update some elds of existing problems, such as title, problem and equation. The second one is Delete, which allows instructors to remove problems they don?t want any more. Preview is the third submodule in List problems module. It emulates the exact situation where a student experience in the APP platform in 14 solving a speci c algebra problem. Through Preview, an instructor can obtain the idea of what environments and steps a student might go through to solve a speci c problem. Trace Students? Interaction module is convenient for instructors to track students? activities in solving problems in APP platform. This module shows the log of each single step or input of a student in solving a problem. For example, who solves what problems at what time? What inputs does the user type? What solutions are provided? Reviewing those activities can provide more information to instructors: 1) learning patterns of students, 2) di culties that they had in solving algebra problems and 3) if a problem is too easy or hard for most students. Then, an instructor could improve her or his teaching by observing these feedbacks. Also, they could promptly adjust some problems for students to practice better. 3.2.2 Process Flow of APM In above section, the structure of APM is described. This section illustrates process ows in APM that are depicted in Figure 3.3. Four primary processes are identi ed: 1. Entering username and passwords ! Adding problem ! Listing problems ! Previewing ! Logout. 2. Entering username and passwords ! Logout. 3. Entering username and passwords ! Listing students? interaction ! Logout. 4. Entering username and passwords ! Listing problems ! Editing or Deleting a problem ! Logout. 15 Above processes all start with users entering the username and password assigned before they can use this system. After the server authenticates a user, s/he can then use this system. The user may go through a portion of or all processes, but they all end up with the Logout process. Users can leave this site any time once they login. Further details about each process ow are explained in the following. Users, who are our target users in APM, refer to instructors, especially, those teaching mathematics. The process "Entering username and passwords ! Adding problem ! Listing problems ! Previewing ! Logout" is a short path for the rst time use of this system and also a main process in the system. If the system is being used for the rst time, where there is no algebra problem resource available for an instructor, s/he has to add a problem. After the instructor?s identity is authenti- cated, s/he can add algebra problems into the speci c repository. If s/he completes the adding, s/he can review what was just added. In adding problems, some elds are mandatory. An instructor might forget some elds and directly click the add button. The system will remind the instructor of those ignored elds. The process "Entering username and passwords ! Logout" is a very special process: it is only used for an instructor to verify if her/his username and password are correct. After that, s/he leaves from this system. The third scenario is "Entering username and passwords! Listing students? interaction!Logout process". In order to trace students? activities in solving problems in APP platform, all their activities are recorded as a log in the database. This process provides information for instructors to understand students? learning pattern, to know whether those problems are proper for them or not, and to nd di culties that students encounter when practicing with those problems. An- other process is "Entering username and passwords!Listing problems!Editing or Deleting a problem!Logout". In this process, the instructor can edit and/or delete a problem. To let the instructor pay more attention to the deletion of a problem, 16 Entering username and passwords Existing username and passwords users Adding problem N Y Each item added is not empty? Listing problems N Y Previewing problem Editing a problem Edit or cancel a problem? Y Y N Listing students? interactions Y Deleting a problem Delete a problem? Confirm to delete a problem? N If finish adding a problem, show an alert to tell a instructor add a problem successfully Logout Y N User can logout any time End Y Process Decision Parallel Process Notation Terminator Y Add or cancel adding a problem? Y N Each item edited is not empty? Y Y N Figure 3.3: Process Flow of APM 17 Algebra Problems Practice(APP) User Authentication Logout Help Check EquationLogin Get Problem Solve ProblemTutor Solve EquationRead Problem Notification Provide Help Figure 3.4: System Structure of APP an alert is shown to ask the instructor to con rm this deletion before it really takes e ect. Editing a problem has the same completion check for all elds as in adding a problem. 3.3 Algebra Problem Practice In addition to APM, Algebra Problem Practice (APP) is the other part of the interface in AADMLSS Practice. It is mainly used by students to practice their ability in solving algebra problems that are added into the database by instructors in advance through APM module. But, it is also used by an instructor to preview the environment a student might experience in solving a problem. 3.3.1 System Structure of APP The system structure of APP is plotted in Figure 3.4. APP includes four main modules: User Authentication, Tutor, Solve Problem and Help. Every module but Help consists of more than two sub modules. 18 User authentication performs the same function as that of APM. The only dif- ference is that the server authenticates the user from a di erent table in the database that is dedicated to students, not instructors. Tutor in APP is actually a culturally relevant virtual tutor. It is a 3D agent with voice interaction ability. SitePal, a speaking avatar platform developed by Oddcast, provides a rich environment for creating animated characters via the Internet. It also provides programming APIs for voice based interaction. To facilitate the conversa- tional interaction between students and the tutor in our system, SitePal animated agents are embedded in APP interface. SitePal?s Dynamic Text-to-Speech (TTS) engine enables the tutor to speak dynamically upon the input text when a scene is played. We use JavaScript to control what and when the tutor speaks. Based on this feature, three main functions are provided by the tutor: reading a problem, no- ti cation and providing Help if necessary. When an algebra is extracted from the database, the tutor reads out the algebra problem for the student, which is also dis- played in text on the web page. By reading the problem, the student might be able to concentrate on the algebra problem. Through noti cation, the tutor reminds the student of what s/he should do next. For example, after a scene is downloaded, the tutor starts with the welcome and introduction about this platform, and then noti es the student to enter an equation for this problem. If the equation is correct, it tells her/him to solve the equation. If the student has any trouble in using the platform, the tutor can assist her/him at any time through the Help function. Students can ask for help by just typing the word "help". When entering an equation, if the student cannot provide one correct equation, the tutor is able to aid her/him with some hints. This also occurs if there is no any input from a student for a predetermined period. Solve problem module is the key module. It includes three sub modules. The rst one is the Get problem module. It extracts an algebra problem in a random way 19 from the problem repository for the virtual tutor to read and for the student to solve each time. Check equation module is to compare students? input equation to the correct equation associated with the problem and stored beforehand by an instructor through APM when the problem was added. The result from the check determines whether the student is allowed to move to the next step to solve a problem. The last submodule is Solve equation. It performs the procedure of solving the equation and checks if the solution input by the student is correct or not. It gives hints and noti cations until the problem is solved successfully. Help module supports students in using the system and in solving a problem, like a handbook. It determines when and what help is provided. The di erence between this module and the help function in Tutor module is that Help module is the engine to interpret and form the help information, but the Tutor speaks out this information through its TTS engine. 3.3.2 Process Flow of APP Figure 3.5 depicts process ows in APP. There are four main processes or sce- narios. 1. Entering username and passwords!Displaying a problem!Entering an equa- tion ! Solving the equation ! Logout,. 2. Entering username and passwords ! Tutor ! Welcome ! Logout. 3. Entering username and passwords!Tutor!Welcome!Reading a problem ! Logout. 4. Entering username and passwords ! Tutor ! Welcome ! Reading and Dis- playing a problem ! Remind entering an equation ! Provide hints for the 20 Entering username and passwords Existing username and passwords users Displaying a problem N Y Equation is correct? Entering an equation N Y Solving the equation Y N Help Y Logout User can logout any time End Y Entering is correct? Y Go to next problem? Reading a problem Y Reading a problem? Tutor Y N Welcome Reading Remind entering an equation N Provide hints for the equation N Enter helpProvide help? Y N Y N Tutor can read the problem for a student any time Process Decision Parallel Process Notation Terminator Y Figure 3.5: Process Flow of APP 21 equation v Entering an equation!Solving the equation!Enter help!Solv- ing the equation ! Logout. Authentication is always the rst step for all ows in the APP platform. Users might go through partial or all steps, but they all end up with Logout process. Users can leave from this site at any time in the middle of a process. Users can ask the tutor for help to understand how to use this platform and how to interact with the system at any time. Users can also request the tutor to read the problem at any time in solving the problem. In the APP platform, users refer to students who practice solving algebra prob- lems, especially, for African American students. The process "Entering username and passwords ! Displaying a problem ! Entering an equation ! Solving the equation ! Logout" is a basic process in the system. It provides a direct way to access this platform regardless of what the Tutor is speaking. After the student?s identity is veri ed, an algebra problem is extracted randomly from the algebra repository and displayed for the student to practice. Then, the system prompts the student to enter an equation for the algebra problem displayed. If any input is typed, the system compares the equation that instructors added through the APM platform to what the student inputs, which is done by a check equation process. If her/his input is correct, the correct equation will appear and then the system prompts the student to provide solutions to the equation. Based on knowledge about basic algebra, three- step of solving the equation is adopted. Namely, most algebra problems usually go through a maximum of three steps to be solved. The scenario "Entering username and passwords ! Tutor ! Welcome ! Lo- gout" is a very special process. Suppose the instructor only wants to verify her/his 22 username and password and have a look at what the system looks like, after that, s/he leaves this system. The third one is the process "Entering username and passwords! Tutor ! Welcome!Reading a problem or Help!Logout process". In this situation, it aims to allow the tutor to read a problem for the student. Or the student only wants to know how to use the system with the tutor. During the whole process, reading a problem or asking for help is available anytime for the student. The process "Entering username and passwords!Tutor!Welcome!Reading and Displaying a problem ! Remind entering an equation ! Provide hints for the equation!Entering an equation!Solving the equation!Enter help!Solving the equation!Logout" is almost the complete scenario that a student may go through. If the student follows the tutor?s noti cation to interact with APP platform until the end, s/he experiences this process. Considering the student may have di culties in solving a particular problem, the tutor allows the student to try to enter her/his equation three times. After that and if s/he still cannot provide a correct equation for the algebra problem, the tutor will aid the student and provide some hints for her/him. While solving the algebra problem, the tutor allows the student to enter "help" when s/he needs help. For example, for the equation "2x+3=5", after the student types "help", the tutor will suggest to her/him with a hint like "would you try to use subtract?". After solving the problem, the student can determine whether s/he goes to the next problem or not. If s/he does, this process is repeated. If not, s/he can choose to logout of the system. All interactive information that students input into the system is stored in the database for instructors to trace students? records in APM platform. 23 3.4 back-end Database MySQL is the database management system used in AADMLSS Practice. The database in AADMLSS Practice implements a design that stores students? and in- structors? account information, an algebra problems pool and students? interactive data with APP platform. In general, the back end database serves as a repository that enables AADMLSS Practice to interact with instructors and students. A database named AADMLSS was created on MySQL Server on the host lets.cse.eng.auburn.edu at Auburn University for this purpose. 3.4.1 E-R Diagram The major goal of AADMLSS Practice is to provide a platform where instruc- tors can manage algebra problems and review students? activities in solving algebra problems and where students can practice what they learn in the classroom through solving algebra problems. There are three entities identi ed in this system: Instructor, Student and Prob- lem. The E-R diagram for the database of this system is depicted in Fig . Instructor entity represents an instructor?s identity information. It has attributes: ID, username and passwords. Problem represents an algebra problem and its associated equation. It has four attributes: PID, title, problem and equation. Student entity represents students? identity information. There are three attributes: SID, username and pass- words. In addition, to store students? interactive information, there are ve proper- ties, including SID, PID, equationInput, solutionInput and date. Between Problem and Instructor, there exists a many-to-one relationship: an instructor can add many problems to the database. But one problem can be added by only one instructor. 24 1, 1 1, N equation adds instructor ID username PID equation N, 1 interacts Cardinality 1 One N Many problem problem student 1, N title username passwords SID PID SID equationInput solu tionInput Date Attribute Relationship ID Primary key passwords Figure 3.6: E-R Diagram 25 FieldName Type Length NULL Primary Key ID int 3 N Y username char 15 N N passwords char 20 N N Table 3.1: Instructor Table There is a many-to-many relationship between Student and Problem. This relation- ship manifests one student can play with many algebra problems. Also, one problem can be accessed by many students. 3.4.2 Tables in Database Based upon above E-R diagram, in the AADMLSS database, four tables are created respectively for Instructor, Student, Problem and Interaction. Table 3.1 shows the details for Instructor table including attributes and their properties. Student and Problem tables have similar structures but di erent at- tributes. Instructor, Student and Problem tables have ID, PID and SID respectively as primary keys. In the Interaction table, the primary key consists of SID, PID and Date as shown in Table 3.2. Given a student may enter equations and solutions many times before s/he nally solve an algebra problem, "text" in the Interaction table is regarded as the data type for equationInput and solutionInput in both elds. The maximum length of text depends on the amount of available memory and the size of the communications bu ers. Both elds need enough size to store the data. This system does not request instructors to enter ID, PID and SID in Instructor, Student and Problem tables respectively. But ID, PID and SID are very important, which are primary keys. These tables utilize AutoIncrement function to automatically and implicitly generate sequential numbers for those IDs, e.g. 1, 2, 3, 4 and so on. 26 FieldName Type Length NULL Primary Key SID int 5 N Y PID int 4 N Y equationInput text N solutionInput text N Date varchar 17 N Y Table 3.2: Interaction Table 3.5 User interface design A good user interface helps users understand the problem domain to work with the application without having to read the manuals or receive training. From the analysis of AADMLSS Practice System, several key features are designed for this purpose. First, the overall user interface design is a user-centered design that focuses on the users? tasks and goals. In Figure 3.7 and Figure 3.8, a multimodal agent-student interaction is adopted as a framework for tutoring through the use of a text-to-speech engine, and keyboard/mouse input. The animated agent (the tutor) uses speech and body language to emulate the human interactions that a student would experience when working with a tutor in real life. Once the practice begins, an algebraic problem in text format is presented to the student. The student can choose to read the problem at any time. The tutor prompts the student to enter the algebraic equation for the algebra problem. Then the tutor evaluates the student?s input and a spoken dialogue with the student begins to guide the student in solving the problem. This layout presenting a familiar environment where a student practices a basic algebraic problem increases the system?s usability. Second, the overall user interface layout is very simple, which communicates clearly with our target users. For example, in Figure 3.9, the login interface only provides user authentication. Instructors and students both cannot change their 27 Figure 3.7: APP?s Interface Figure 3.8: APM?s Interface 28 Figure 3.9: Instructors? Login Interface usernames and passwords by themselves because their usernames and passwords are given in advance, which is associated with their school?s usernames and passwords, like global identity. In this way, they are not required to give private information such as an email address or SSN. Another feature is the consistency that enables users to build an accurate mental model of the way the user interface works. In ADDMLSS Practice system, two interfaces (APM and APP) use the same color, the same hyperlink color, the same background and the same layout. For example, in Figure 3.9 and Figure 3.10, both user Login interfaces in APM and APP are in the same style. This consistency relieves users from the trouble of adapting to di erent interfaces and the cost for training. User friendly, which means that a computer program or web site is easy for users to use, is one of several features in our user interface design. For example, in Figure 3.11, if the instructor makes an error, the system will provide information about how to correct the error. If the instructor leaves some elds blank when s/he 29 Figure 3.10: Students?s Interface adds a problem, the system will display information with letters in red e.g. "Error, please enter a title" to remind her/him of the issue. Also, it is very convenient for the instructor to browse the details of the algebra problems in the database that are listed with ID and title as shown in Figure 3.12 or the interactive information from students. Readability is another critical feature in user interface design. It can be de ned as easy to read. The procedures where students solve a problem will be stored into the database, including equations and solutions, which can be reviewed later by in- structors. Suppose there is no "/" added to distinguish those inputs in Figure 3.13, instructors may be confused with multiple inputs running together. Figure 3.14 is also a good example for improving readability. While solving equation, symbols "- ", "+","*" and "/"used in the problem solving procedure are more meaningful and readable for students. 30 Figure 3.11: Error Interface Figure 3.12: List Problems Interface 31 Figure 3.13: List Students? Interaction Interface Figure 3.14: Solving Equation Interface 32 Interaction is another feature in this system. Interaction is the ability for a student to be able to converse with the tutor. AADMLSS Practice is interactive. The tutor asks a question; the student responds; the tutor then provides feedback. For example, the tutor asks the student to solve "Jane spent $42 for shoes. This was $14 less than twice what she spent for a blouse. How much was the blouse?". If the student enters an incorrect equation e.g. "2x + 14 = 42", the tutor says "would you try one more time?" to give the student more time to think. In a summary, this chapter presents a thorough and detailed description of the design of AADMLSS Practice. The architecture of this system includes two interfaces: APM and APP, and one database. In front-end interface design, two main function modules, APM and APP, consist of multiple submodules. To exemplify the scenarios in real world, di erent process ows are described. APM and APP both have four primary process ows. In back-end design, AADMLSS database is built. Three entities are identi ed in the E-R diagram design: Instructor, Student and Problem. Four tables are created to cover these entities and their relationships. In addition, the key features, e.g. consistency, readability and other features, of the user interface design in this system are analyzed. 33 Chapter 4 Evaluation Methodology The usability evaluation is a critical part of a Human Computer Interaction system, which reveals the e ectiveness of a system interacting with human beings. This chapter is dedicated to the experiment settings of the usability evaluation of AADMLSS Practice. Four sections are covered: environment and tools, scenar- ios/tasks, participants and data collection. 4.1 Environment and tools To guarantee the study is performed successfully, a variety of equipments, soft- wares and technologies were used in this study. There were two computers to access Internet with Mozilla Firefox and Internet Explore. Before the study was performed, an algebra database was created on lets.cse.eng.auburn.edu server to store user ac- counts and algebra problems and to support students interacting with the tutor. Each participant was assigned one account that was created in advance. The source code of the system was uploaded to the web server directory so that participants start the study through the web. At the beginning of the testing session, the web browser?s ad- dresses were set to http://lets.cse.eng.auburn.edu/ AADMLSS/Practice/teacher/login.php and http://lets.cse.eng.auburn.edu/ AADMLSS/Practice/student/slogin.php respec- tively on two di erent computers. The computers utilized in the study are in Auburn University network domain because the prototype of the study is only open for Auburn University students this time. Other material involved includes the information letter 34 that was o ered to the participants of the study and the survey forms given to the participant before and after the study (See Appendix A and B). Microsoft Excel was used to manipulate and analyze the experiment results. 4.2 Participants Mass e-mails were sent to AU students across the university with the invitation to recruit participants for the study. Participation was voluntary. The participants are Auburn University students who are 19 or older. Part of this population is graduate teaching assistants who are or were teaching and tutoring students. They have more experience with respect to creating algebra problems and in using a system to teach students. We also have some students perform the student role in solving some of these algebra problems in the database. All participators are randomly selected from Auburn University students who agree to serve in the study. All students, both undergraduate and graduate, were accepted. 4.3 Scenarios/Tasks in Evaluation AADMLSS Practice usability evaluation is to verify whether the design and implementation of this system meet the preset usability goals and users? satisfaction. For example, is the system easy to use? Is the animated tutor helpful for students learning basic algebra? Is the style and color of the system consistent? To cover the main functionalities of the system, four testing scenarios were developed for the usability evaluation. 35 4.3.1 Scenario/Task 1: Add an algebra word problem into AADMLSS Practice Expected steps: Login in the instructors? interface with assigned usernames and passwords and click "login" button or hit "Enter" key. Click "Add a Question" link. Fill in the Add a Question form and click "Add Question" button. Click "OK" button to approve the problem just being added. Click "Logout" link to log out the system. 4.3.2 Scenario/Task 2: Browse, edit and preview the algebra problem(s) just added Expected steps: Login in the instructors? interface with assigned usernames and passwords and click "Login" button or hit "Enter" key. Click "List Questions" link to browse all algebra problems. Click the title just added in task 1 to browse the speci c problem. Click "edit" link to update the problem if necessary. Click "OK" or "Cancel" button to update/keep the problem. Click "preview" link to experience how students will interact with the tutor. Close the current window. 36 Click "Logout" link to log out of the system. 4.3.3 Scenario/Task 3: Solve algebra problems Expected steps: Login in the students? interface with assigned usernames and passwords and click "login" button or hit "Enter" key. Listen to the tutor?s speech and/or click "Read The Problem" button. Follow the tutor?s instruction to enter the equation for the displayed algebra problem. Click "Check Equation" button or hit "Enter" to check the correctness of the input equation. If the equation is not correct, follow the tutor?s instruction to reenter a di erent equation and repeat last step. If the equation is correct, enter one solution with commands, such as "add 3 from both sides", "subtract 3 from both sides", and "divide 3 from both sides", etc. to solve the equation. After entering the solution, click "Solve Equation" button or hit "enter" to let the tutor check the solution. If each step of solving the equation is not correct, follow the tutor?s instruction to reenter a di erent solution and repeat the last step. Or enter "help" to get hints. If each step of solving the equation is correct, click "Go To The Next Problem" button to repeat task 3. Or close the current window to log out of the system. In task 3, students can click "Help Commands" menu to get help at any time. 37 4.3.4 Scenario/Task 4: Browse interactive data Expected steps: Login in the instructors? interface with assigned usernames and passwords and click "login" button or hit "Enter" key. Click "List Students? Interaction" link. Click one of the following: "User", "Problem", "EquationInput", "SolutionIn- put" or "Date" to browse the data ordered in di erent elds. Click "Logout" link to log out of the system. 4.4 Data Collection E ectiveness, e ciency and user satisfaction in the usability evaluation of the AADMLSS Practice system were measured. To measure e ectiveness, we recorded whether a participant completed a task successfully. To measure e ciency, we recorded the time spent for a participant to nish all tasks. To measure user satisfaction, we collected information through a post-survey. At the beginning of testing the system, a pre-survey was given to participants, because di erent groups of people have variant features, such as skills of using a computer, familiarity with tutoring systems and math knowledge. In order to validate the results of the survey, demographic information was also collected in the pre-survey (See Appendix A). The pre-survey included two main parts. The rst part asked for demographical information with respect to the participant, such as age, gender, and other related information. The second part in the survey asked for the familiarity with online tutoring systems. It is important to collect this information because the more comfortable the subjects were with web-based tutoring systems, especially with 38 math learning tutoring system in general, the more likely they were to succeed in using AADMLSS Practice. After testing the system, that is, nishing all tasks, participants were asked to ll out a post-survey (See Appendix B) to collect information and opinions about the system, which was used to measure user satisfaction. The post- survey consisted of three main parts. The rst part of the survey asked for opinions about the overall system, such as consistency, whether contextual helps are provided, and other related information. The second part of the survey asked for opinions on the tutor?s response and instruction. The third part of the survey asked whether or not the way of the design in solving the algebra problem is straightforward and simple. Users? satisfaction measurement largely depends on an evaluation score scale. It discovers overall users? satisfaction. In system usability, a seven-level scale was chosen to measure users? satisfaction, because it can tell more precisely how users feel about the system. A score of each question ranges from 0 to 7. If users are not willing to answer the question, NA (not aplicable) is for this purpose. The lowest is 0. The highest score is 7. Following is an example: To what extent did the system understand what you typed? Never: 0 1 2 3 4 5 6 7 : Always NA To calculate the nal score, 1) sum what each question gets; 2) divide the sum of the scores by 7; 3)multiply the results of the second step by 10. Since there are total 10 questions, the total score falls into between 0 and 100. It reveals the overall users? satisfaction. The higher the total score, the more likely the users are satis ed with the design of the system. This chapter summarizes the experimental methodology, environment and sce- narios. In order to truly understand what users expect about the system and whether 39 they are satis ed with it, the usability evaluation is utilized to validate the assump- tions made by application developers and designers in this chapter. The whole pro- cedure about before, during and after the testing of the system is explained. The following chapter discusses the usability evaluation results from the experiments. 40 Chapter 5 Experimental Results and Discussion This chapter presents and analyzes the major outcomes of the usability evaluation experiments. First presented is the information of participants in the usability study. Then the evaluation result of user interface is discussed. Finally we disclose the strength and weakness of this system based on the experiment result. 5.1 Experimental Results 5.1.1 Participants As shown in Table 5.1, 26 students participated in AADMLSS evaluation. They represent di erent demographic backgrounds: Chinese, Indian, South Korean, Japanese and Native-American students. Among these students, there were 21 males, 5 females and 1 participant who did not indicate his age; this participant?s information was re- moved from the nal analysis because the participants must be 19 years old or older. The age of the participants ranged from 19 to 34 with a mean of 26.42, shown in Figure 5.1. There were 12 participants who are Graduate Teaching Assistant (GTA) teaching or tutoring some courses or seminar accounting to 48% of all participants. Gender Number Percentage Females 5 20% Males 20 80% Total 25 100% Table 5.1: Gender Distribution of Participants 41 Figure 5.1: Age Distribution of Participants 5.1.2 Interface Results Table 5.2 summarizes the system usability score and the time used to complete all four tasks by each participant. We could observe that 99% of the participants were able to nish all four tasks: 1) Add an algebra word problem into AADMLSS Practice; 2) Browse, edit and preview the algebra word problem just being added; 3) Solve the algebra word problem; and 4) Browse interactive data. One participant failed to nish task 3. Therefore his time used to nish all task was zero. The average task completion time is 9.41 minutes with SD of 5.57. Participant ]16 spent 34 minutes on nishing all tasks without any real teaching and online tutorial experience. During his testing, he complained of "no voice I cannot hear from the tutor!!!". This may have been caused by a network problem since the voice is transmitted through Sitepal, not locally through Auburn University?s network domain. Figure 5.3 furthermore reveals the system usability score where the y-axis represents the scaled score while the x-axis shows di erent participants by numbers. The mean of the overall system 42 Figure 5.2: System Usability Score usability score is 73.81 out of 100 (SD =15.26). 52% of the participants? scores were higher than average. Participant ]6 got the highest score (96.28) with experience in using online learning tools to create instruction, e.g. blackboard. Participant ]19 also got a higher score (90.34) who had rich experience in creating online tutorial learning tools and using online tools to learn. He said "the system works very well". However, participant ]18 scored the lowest in the survey and failed task ]3. That is due to the technical di culties the participant was facing. The fact that almost all participants can nish the all four talks shows that this system is very e ective. As for the e ciency of this system, 64% of the participants spent less than the average time of 9.41 minutes. This implies our system is e cient to some degree. 5.2 Discussion In this section, we discuss the strengths and weaknesses of our system observed from the evaluation surveys. Figure 5.3 presents the result of the averaged score for 43 Participant No Score Total Time (min) Note 1 62.21 6 2 56.55 7 3 61.35 14 4 68.79 8 5 93.30 7 6 96.28 7.5 7 89.57 9 8 94.99 9 9 63.78 10 10 80.82 5.5 11 76.55 9.5 12 68.57 7.67 13 85.10 7 14 75.98 6 15 59.66 6 16 82.21 34 17 71.99 8 18 36.18 0 Failed in task 3 19 90.34 10 20 49.36 5 21 94.07 7 22 68.96 12 23 76.82 9 24 83.02 12 25 58.74 10.1 AVERAGE 73.81 9.41 STDDEV 15.26 5.57 Table 5.2: Usability Evaluation Data of AADMLSS Practice 44 each question in the post-survey. The Y-axis represents participants? average scores while the X-axis plots the questions on the survey by question numbers. From this gure, most participants claimed the strength of the system as following: 1. the design is consistent in color and layout; 2. the system is easy to use; 3. it is straightforward to start using it; 4. it is fast to learn how to use the system; 5. the number of steps to solve an equation is reasonable; 6. it is straightforward to solve the equation; 7. the system?s prompts are helpful; Despite the above strengths, some weaknesses is also identi ed from this study. They are: 1. Most participants complained that the tutor cannot speak and read. This de- pends on the network bandwidth as described above. 2. With incorrect equation, no better suggestion is provided. This problem could be addressed with more contextual hints associated with each question. But this may request new technology to extract question speci c information. 3. Tutor?s feedback is not enough. This could be solved with more frequent tutor voice hints at di erent moments. 45 Figure 5.3: Average Score of Each Item in Post-survey Questionnaire 46 Chapter 6 Conclusion In this thesis work, a web-based algebra teaching system is designed, analyzed and evaluated. This system is dedicated to African American students to practice solving algebra problems in order to improve their mathematical skills. This platform includes two modules for interface at the front-end and one database at the back-end. One interface module is designed for instructors to manage an algebra problem pool, to preview the environment a student might be in when solving an algebra problem and to track the activities that students perform in solving algebra problems. The second interface module provides the platform where a student can extract an algebra problem and practice solving this problem through the interaction with a 3Dvirtual tutor. Practicing on this online web system that is tailored with the familiar cultural elements to students, they are expected to improve their ability to comprehend in math. A comprehensive usability evaluation is performed to observe user experience with the system. Results from this experiment show that AADMLSS Practice is e ective and e cient in usability. 47 Bibliography [1] M. Strutchens and E. Silver, \NAEP Findings Regarding Race/Ethnicity: Stu- dents? Performance, School Experiences, and Attitudes and Beliefs," Seventh Mathematics Assessment of The National Assessment of Educational Progress, Reston, VA: National Council of Teachers of Mathematics, 2000. [2] F. R. Moses-Snipes, \The E ect of African Culture on African American Stu- dents? Achievement on Selected Geometry Topics in the Elementary Mathematics Classroom," The Negro Educational Review, vol. 56, no. 2, pp. 147-166, 2005. [3] N.Nasir, A.S. Rosebe B. Warren and C.D.Lee, \Learning as a cultural process: Achieving equity through diversity," Handbook of the learning sciences. New York: Cambridge University Press. [4] C.D. Lee, \MInterrogating race and ethnicity as constructs in the examination of cultural processes in developmental research," Human Development, 45(4), 282-290, 2002. [5] N. Parks, T. Simmons, K. Sapp and J.E. Gilbert, \Culturally In uenced E- Learning: An Introduction to AADMLSS," Hin Proc. E-Learn World Conference on E-Learning in Corporate, Government, Healthcare and Higher Education, pp. 1960-1965, 2003. [6] Mary Stone Hanley, \A Culturally Relevant Les- son for African American Students," Available: http://www.newhorizons.org/strategies/multicultural/hanley2.htm. [7] Ethnomathematics, Available: http://en.wikipedia.org/wiki/Ethnomathematics. [8] I. Abdal-Haqq, Culturally Responsive Curriculum, ERIC Digest, pp. 1-4, 1994. [9] J. Barta and D. Schelling, \Games We Play: Connecting Mathematics and Cul- ture in The Classroom," Teaching Children Mathematics, vol. 4, no. 7, pp. 388- 393, 1998. [10] V. Katz, \Ethnomathematics in the Classroom," For the Learning of Mathemat- ics, vol. 14, no. 2, pp. 26-30, 1994. 48 [11] J. Masingila and K. King, \Multicultural and Gender Equity in the Mathematics Classroom: The Gift of Diversity," Reston, VA: National Council of Teachers of Mathematics, 1997. [12] K. Dunn and R. Dunn, \Teaching Learners through Their Individual Learning Styles: A Practical Approach," Reston, VA: Reston Publishing, 1978. [13] J. E. Gilbert, \Case Based Reasoning Applied to Instruction Method Selection for Intelligent Tutoring Systems," Rin Workshop proceedings of ITS?2000: Fifth International Conference on Intelligent Tutoring Systems, Montreal, CA. pp. 11-15, 2000. [14] J. E. Gilbert and C. Y. Han, \Arthur: A Personalized Instructional System," Journal of Computing In Higher Education, vol. 14, no. 1, Norris Publishers, Amherst, MA, 2002. [15] J. Mintzes and J. H. Wanderse, \Reform and Innovation in Science Teaching: A Human Constructivist View," Teaching Science for Understanding; A Human Constructivist View, pp. 29-58, San Diego, CA: Academic Press, 1997. [16] B. Rogo , \The Cultural Nature of Human Development," New York: Oxford University Press, 2003. [17] J. C. Lester and B. A. Stone, \Increasing believability in animated pedagogical agents," In Proceedings of the First International Conference on Autonomous Agents, pages 16-21, 1997. [18] V.Aleven and KR Koedinger, \An e ective meta-cognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor," In Cognitive Science, 26, page 147-179, 2002. [19] Algebra Tutor, In Tech Trends, 43, pages 4-40, 2007. [20] A. Baylor and Y. Kim, \The role of gender and ethnicity in pedagogical agent perception," In the World Conference on E-learning in Corporate, Government, Healthcare, and Higher Education, 2003. [21] A. W. Lodree and J. E. Gilbert, \E ects of Animated Agents in Post-secondary Matheimatics," In i-manager?s Journal of Educational Technology, vol. 5, no. 1, 2008. 49 Appendix A Pre-Survey 1. ID: 2. Age: 3. Gender: 4. Major: 5. Race/Ethnicity: 6. Citizenship: 7. Are you a graduate teaching assistant (GTA)? Yes No 8. Is English your native or second language? Native language Second language 9. Have you ever taught a course or seminar? Yes No 10. Have you ever used a tutoring service before? Yes No 11. If yes, has it been for a math based course? Yes No 12. Have you ever used an online (web based) learning tool to create instruction, e.g. Blackboard? Yes No 13. Have you ever used an online (web based) learning tool to learn something? Yes No 50 Appendix B Post-Survey Please try to answer the following questions. Please check one answer for each ques- tion. 1. To what extent did you think the preview function of the system is useful? Never: 1 2 3 4 5 6 7 :Always NA 2. To what extent did the system understand what you typed? Never: 1 2 3 4 5 6 7 :Always NA 3. To what extent was the system?s help and prompts easy to understand? Di cult: 1 2 3 4 5 6 7 :Easy NA 4. To what extent did you know what to type in response to the system?s prompts? Never: 1 2 3 4 5 6 7 :Always NA 5. To what extent did the system behave the way you expected? Never: 1 2 3 4 5 6 7 :Always NA 6. Which most appropriately re ect your impression of using this system? Overall reactions to the system: 6.1 Terrible: 1 2 3 4 5 6 7 :Wonderful NA 6.2 Frustrating: 1 2 3 4 5 6 7 : Satisfying NA 6.3 Dull: 1 2 3 4 5 6 7 : Stimulating NA 6.4 Di cult: 1 2 3 4 5 6 7 :Easy NA 6.5 Rigid: 1 2 3 4 5 6 7 : Flexible NA 51 7. How helpful were the suggestions that were given by the tutor? Not Helpful: 1 2 3 4 5 6 7 :Very Helpful NA 8. After using this system, how likely are you to use it or a similar system again? Unlikely: 1 2 3 4 5 6 7 :Likely NA 9. Which most appropriately re ect your impressions of using this system? Getting Started 9.1 Di cult: 1 2 3 4 5 6 7 :Easy NA Time to learn to use the system 9.1.1 Slow: 1 2 3 4 5 6 7 :Fast NA Discovering features 9.2 Di cult: 1 2 3 4 5 6 7 :Easy NA Keeping consistency 9.2.1 Never: 1 2 3 4 5 6 7 :Always NA The way to solve the equation in a straight-forward manner 9.3 Never: 1 2 3 4 5 6 7 :Always NA Number of steps per equation 9.3.1 Too Many: 1 2 3 4 5 6 7 : Just right NA Steps to complete solving the equation follow a logical sequence 9.3.2 Never: 1 2 3 4 5 6 7 :Always NA Feedback on the completion of the steps 9.4 Unclear: 1 2 3 4 5 6 7 :Clear NA 10. How much of your understanding of linear equations improve after using this system? Not At All: 1 2 3 4 5 6 7 :Very Much 11. Please provide any additional comments that you have about the system: 52 Note: After the experiment, please don?t mention this experiment to any other students who will do this experiment later! Thank you very much! 53