i DESIGNING FOR THE IMPROVEMENT OF OPERATOR SITUATION AWARENSS IN AUTOMATION SYSTEMS Except where reference is made to the work of others, the work described in this dissertation is my own or was done in collaboration with my advisory committee. This dissertation does not include proprietary or classified information. ______________________________________ Jeannie L. Pridmore Certificate of Approval: ______________________ ______________________ R. Kelly Rainer Terry Anthony Byrd, Chair Professor Professor Management Management ______________________ ______________________ Thomas Marshall George T. Flowers Associate Professor Interim Dean Management Graduate School ii DESIGNING FOR THE IMPROVEMENT OF OPERATOR SITUATION AWARENSS IN AUTOMATION SYSTEMS Jeannie L. Pridmore A Dissertation Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Auburn University December 17, 2007 iii DESIGNING FOR THE IMPROVEMENT OF OPERATOR SITUATION AWARENSS IN AUTOMATION SYSTEMS Jeannie L. Pridmore Permission is granted to Auburn University to make copies of this dissertation at its discretion, upon request of individuals or institutions and at their expense. The author reserves all publication rights. ___________________ Signature of Author ___________________ Date of Graduation iv VITA Jeannie L. Pridmore was born in Opelika, Alabama on April 20, 1974, the daughter of Mr. And Mrs. George T. Pridmore. She graduated from Auburn High School in 1992. In 1996, she graduated from Auburn University with a Bachelor?s in Chemical Engineering. In 2000, she graduated from Troy University with a Master?s of Business Administration. She worked for MeadWestvaco for ten years as a controls engineer, a business analyst, and a six-sigma instructor and project leader. Upon completion of this dissertation, Jeannie will begin her career as an assistant professor at Loyola College in Baltimore. v DISSERTATION ABSTRACT DESIGNING FOR THE IMPROVEMENT OF OPERATOR SITUATION AWARENSS IN AUTOMATION SYSTEMS Jeannie L. Pridmore Doctor of Philosophy, December 17, 2007 (BS, Auburn University, 1996) (MBA, Troy University, 2000) 161 Typed Pages Directed by Terry Byrd The enhancement of operator situation awareness (SA) has become a major design goal for automation systems, in a wide variety of fields, including aviation, aerospace, nuclear plants, and advanced manufacturing systems. One can easily grasp that SA has always been needed in order for people to perform tasks effectively and efficiently. For many years, having good SA was largely a matter of learning what to watch and listen for in the field and learning what cues to recognize in the physical process, in other words, training and field experience. Today, operators rely on the principles and design of human computer interaction (HCI) to observe and comprehend the overwhelming amount of process data that can vary rapidly. It is widely accepted that vi more data does not equate to more information. In many cases, instead of assisting in this issue, automation has only worsened the problem (Endsley & Kiris, 1995; Sarter & Woods, 1995). Human factors research involves discovering and applying information about human behavior, abilities, limitations, and other characteristics to the design of machines, systems, and environments for productive, safe, comfortable, and effective human use (Chapanis, 1985). This research paradigm seeks to determine relationships between stimuli presented to a human and the action they take. To apply human factors in HCI design, one must investigate the system?s users, how they think, use technology, and interpret their environment. The design of the interface will determine the extent, to which SA and information processing are optimized, and ultimately, the efficiency and effectiveness of an operator and automation system. The proposed designs will be evaluated by conducting a laboratory experiment using a microworld (MW) environment that will assess each interface design according to the ideas proposed by the SA theory. An overview is first provided to highlight the study and the type of data that will be collected from each experiment. Next, the MW methodology that will be used for the proposed study is explained along with a description of the interface to be used in the study, and the specific tasks the study participants will be asked to complete. vii ACKNOWLEDGMENTS It is such a blessing to be able to write this section. Various individuals played vital roles in this accomplishment, and I welcome them to share in this with me. First, I express my deepest gratitude to my major professor, Dr. Terry A. Byrd. You were encouraging, in your own way, and you challenged me to be better than I thought I could be. Thank you for your support through all the ups and downs and for not giving up on me. I would also like to thank the other two members of my committee, Dr. R. Kelly Rainer and Dr. Thomas Marshall, and my outside reader, Dr. Jefferson Jones for their thoroughness, patience, and career advice. Second, I would like to thank to the operation managers at MeadWestvaco's Mahrt Mill for allowing me to conduct my research at its site. I especially owe a great deal of thanks to Mr. Charles Hodge and Mrs. Blanche Smith. Thank you for your support and encouragement and for helping me to grow professionally. Last, but certainly not least, to my wonderful parents and sister. Thank you, for your patience, support, and understanding. I know that at times it was as difficult for you as it was for me. Through it all, you continued to trust and believe in my ability. I thank God for you everyday. You were never selfish and always giving to make sure that I could focus on my coursework and research projects. I will never forget the sacrifices you have made for me. I could not have accomplished this without your constant love and support. viii Style manual or journal used: Guide to Preparation and Submission of Theses and Dissertations and APA Computer software used: Microsoft Word ix TABLE OF CONTENTS LIST OF FIGURES...........................................................................................................xii LIST OF TABLES............................................................................................................xv I. INTRODUCTION......................................................................................................1 Problem Statement..................................................................................................3 Theoretical Underpinnings......................................................................................6 Situation Awareness....................................................................................6 Human Factors and HCI..............................................................................9 Automation................................................................................................10 Research Objects and Plan........................................................................14 Research Approach...................................................................................15 Limitations and Assumptions................................................................................17 Expected Contributions.........................................................................................17 Summary...............................................................................................................18 II. LITERATURE REVIEW.........................................................................................19 Related Research...................................................................................................20 Automation................................................................................................20 Situational Awareness...............................................................................22 Information Process and SA...........................................................25 Working Memory, Attention, and Perception................................27 x Long-term Memory, Mental Models and SA................................29 Projection and SA..........................................................................31 Creating Automation Interfaces to Support SA.....................................................33 Perception: Structuring Information and Providing Visual Cues..............35 Comprehension: Development of Effective Mental Models....................37 Projection: Feedback.................................................................................39 Research Model for Each Level of SA..................................................................41 Evaluating Operator Interface...............................................................................42 Objective Measures of SA.........................................................................43 Situation Awareness Global Assessment Technique....................43 Behavioral Measures for Inferring SA..........................................44 Subjective Measures of SA......................................................................45 Situation Awareness Rating Technique........................................45 Post Trial Participant Subjective SA Questionnaire.....................46 Summary..............................................................................................................46 III. RESEARCH METHODLOGY................................................................................48 Research Design....................................................................................................48 Microword Experiments and Other Simulation Environments.............................49 Tractability................................................................................................50 Realism......................................................................................................51 Engagement...............................................................................................51 Experiment Overview..........................................................................................52 Study Participants......................................................................................54 xi Study Procedure........................................................................................54 Plan of Analysis....................................................................................................56 Summary...............................................................................................................56 IV. ANALYSES AND RESULTS.................................................................................62 Microworld Experiment Dimensions....................................................................63 Tractability................................................................................................63 Realism......................................................................................................64 Engagement...............................................................................................64 Experiment Testing...............................................................................................65 Participants and Randomization...............................................................65 Data Analysis........................................................................................................68 Demographic Variables.............................................................................68 Experiments...............................................................................................72 SAGAT..........................................................................................72 PSAQ.............................................................................................81 Performance Measure....................................................................86 Hypothesis Testing....................................................................................89 Hypothesis 1a................................................................................89 Hypothesis 1b................................................................................89 Hypothesis 1c................................................................................90 Hypothesis 2a................................................................................90 Hypothesis 2b................................................................................90 Hypothesis 3a................................................................................91 xii Hypothesis 3b................................................................................91 Summary of Hypothesis Testing...........................................................................92 V. DISCUSSION AND IMPLICATIONS...................................................................95 Findings.................................................................................................................96 The Operator?s Ability to Perceive and Interface Design.........................97 The Operator?s Ability to Comprehend and Interface Design..................99 The Operator?s Ability to Project and Interface Design.........................100 The Operator?s Performance and Interface Design.................................102 Implications.........................................................................................................103 Research Implications.............................................................................103 Managerial Implications..........................................................................104 Limitations..............................................................................................106 Future Research Suggestions..................................................................107 Conclusion...........................................................................................................108 BIBLIOGRAPHY...........................................................................................................111 APPENDICES.................................................................................................................130 APPENDIX A: Instruments................................................................................131 APPENDIX B: Experiment Overview................................................................140 APPENDIX C: Information Sheet......................................................................144 xiii LIST OF FIGURES FIGURE 1.1........................................................................................................................8 Model of SA in Dynamic DM (Endsley, 1995b) FIGURE1.2.......................................................................................................................11 Adopted from Hodge 2002 FIGURE 1.3......................................................................................................................13 Seridan?s Model of Human Supervisory Control FIGURE 1.4......................................................................................................................14 General Research Model FIGURE 2.1......................................................................................................................21 Process Control Interfaces FIGURE 2.2......................................................................................................................23 Model of SA in Dynamic (Endsley, 1995b) FIGURE 2.3......................................................................................................................26 Wicken?s Information Processing Model, 1992 FIGURE 2.4......................................................................................................................42 Research Model FIGURE 3.1......................................................................................................................50 Relationship between three dimensions of simulated task environments FIGURE 3.2A....................................................................................................................58 Interface #1 (Traditional Design) process not running FIGURE 3.2B....................................................................................................................58 Interface #1 (Traditional Design) process running in an abnormal state FIGURE 3.3A....................................................................................................................59 Interface #2 (Perception Design) process not running FIGURE 3.3B....................................................................................................................59 Interface #1 (Perception Design) process running in an abnormal state xiv FIGURE 3.4A....................................................................................................................60 Interface #3 (Comprehension Design) process not running FIGURE 3.4B.......???????????????????????????.60 Interface #3 (Comprehension Design) process running in an abnormal state FIGURE 3.5A....................................................................................................................61 Interface #2 (Projection Design) process not running FIGURE 3.5B....................................................................................................................61 Interface #1 (Projection Design) process running in an abnormal state FIGURE 4.1......................................................................................................................63 Relationship between three dimensions of simulated task environments (Ehret et al., 2000) FIGURE 4.2......................................................................................................................93 Detail Research Model with Hypotheses xv LIST OF TABLES TABLE 2.1........................................................................................................................45 SART Dimensions TABLE 4.1........................................................................................................................66 Demographic Variables TABLE 4.2........................................................................................................................67 Demographic Variables by Interface Experiment TABLE 4.3........................................................................................................................67 Colorblindness by Interface Experiment TABLE 4.4........................................................................................................................70 ANOVA for Demographic Variables based on SAGAT TABLE 4.5........................................................................................................................71 ANOVA for Demographic Variables based on PSAQ TABLE 4.6........................................................................................................................71 ANOVA for Demographic Variables based on Performance Measure TABLE 4.7........................................................................................................................74 ANOVA Results for Each Level of SA by Each Interface Experiment TABLE 4.8........................................................................................................................75 Multiple Comparisons SAGAT: SA1 at Normal Process Operating Conditions TABLE 4.9........................................................................................................................76 Multiple Comparisons SAGAT: SA1 at Abnormal Process Operating Conditions TABLE 4.10......................................................................................................................77 Multiple Comparisons SAGAT: SA1 at Both Normal and Normal Conditions TABLE 4.11......................................................................................................................78 Multiple Comparisons SAGAT: SA2 TABLE 4.12......................................................................................................................79 Multiple Comparisons SAGAT: SA3 xvi TABLE 4.13......................................................................................................................80 Multiple Comparisons SAGAT: SA Total TABLE 4.14......................................................................................................................82 ANOVA Results for Each PSAQ Question by Each Interface Experiment TABLE 4.15......................................................................................................................83 Multiple Comparisons PSAQ: How hard did you find the interface to use? TABLE 4.16......................................................................................................................84 Multiple Comparisons PSAQ: How well would you rate your performance on the experiment? TABLE 4.17......................................................................................................................85 Multiple Comparisons PSAQ: How aware were you of the evolving situation during the experiment? TABLE 4.18......................................................................................................................87 ANOVA Results for % Success Explained by the Interface Experiment TABLE 4.19......................................................................................................................88 Multiple Comparisons Performance Measure: Success Rate of Preventing the Pipe Rupture TABLE 4.20......................................................................................................................94 Summary of Hypotheses TABLE 5.1......................................................................................................................106 Design Guidelines 1 I. INTRODUCTION Automation has been an increasing trend in organizations throughout the 20th century. Automation systems are introduced by organizations expecting to improve productivity by speeding up work cycles, reducing the number of repetitive tasks, improving product quality and increasing employee flexibility (Chawat, 1992; Burmester et al., 2000). In most implementations, these desired benefits have been achieved. In turn, automation has drastically changed the role of the operator. The operator no longer performs manual labor in the field. They have often been moved to a control room far away from the physical process, and now their role is more of a monitor or supervisor of the automation system. In addition, technological advances have enabled organizations to gather and process data faster and less expensively. Today, automation systems are capable of producing and displaying a huge amount of data on both the status of its internal components and its external environment. As technology improves and moves towards a pervasive and ubiquitous environment, automation equipment is able to pass more and more information to the operator. Thus, operators are required to handle more data and more responsibility. For instance, in the 1970s, a typical operator manually controlled approximately 45 control valves in one process unit. Today an operator controls on average 175 control valves through an automation system interface (Labs, 2 2005). More specifically, the number of observable process variables in the power distribution sector grew from 200,000 to 700,000 between the years 1990 and 2000 (Burmester et al., 2000). Although experienced users tend to filter through the overabundance of data to generate information and acquire good situation awareness (SA), even the most expert operator can become swamped by the excessive amount of data provided by new technologies. Hence, in the presence of all this data, operators are finding that they are even less aware than ever before about the situations they are controlling. This has led to a huge gap between the massive amount of data produced and disseminated and the operator?s ability to effectively assimilate the data needed and to make a timely, accurate decision (Endsley & Garland, 2000b). SA can be described as knowing and understanding what is going on around you and predicting how things will change (Moray, 2004). The problem of poor operator SA continues to worsen as technology advances whether the operator is a pilot, a manufacturing operator, or a manager, and it can be seen through automation-facilitated accidents throughout the world. For example, in the March 23, 2005, Texas City, TX BP Amoco Refinery explosion, 15 workers were killed and 170 injured when a column was overfilled, overheated, and over pressurized on startup. A key problem identified in this catastrophic event was the difficulty experienced by the operator in maintaining an accurate awareness of the situation while monitoring a complex, fast moving environment (Labs, 2005). Several other accident studies throughout many industries have found that loss of or poor operator SA was related to accidents classified as human error. For instance, loss of SA has been associated with 88% of major air carrier 3 accidents that involved pilot errors and 58.6% of operational error in air traffic control operations (Endsley, 1995a). Due to the severity of the accidents that have occurred over the last 10 years, SA has become the focus of research that aims to understand operator performance in critical, dynamic environments (Endsley, 1988, 1999). Previous studies have identified the importance of human factors and cognition in interface design and that designing to support human capabilities could impact operator SA (Itoh & Inagaki, 2004). A wide range of design possibilities exists in this design space, which has to date only been explored at the edges (Kaber & Endsley, 2004). As more systems become automated in a wide variety of domains, better-defined guidelines are needed as to human factors, their effect on operator SA, and to clarify the role that the interface plays in the communication between the operator and the automation system in regards to SA (Itoh & Inagaki, 2004). This dissertation will examine the relationship between interface design, human factors, and operator SA. Problem Statement "What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it." (as quoted by Varian 1995) The enhancement of operator SA has become a major design goal for the development of automation interfaces in a wide variety of fields, such as the military, aviation, power plants, and advanced manufacturing systems (Endsley & Garland, 2000a; Sandom, 2001). Users of automated systems often have to monitor large volumes of information from a wide range of sources in order to acquire SA (Sandom, 2001), and 4 new technology has introduced many challenges that have aided in the encouragement of this goal particularly in regards to human factors (Guille & French, 2004). Incorporating human factors during the system development process exposes a very challenging set of problems in interface development (Sandom, 2001). Moreover, industrial process plants by nature are unique. This uniqueness presents additional difficulties when attempting to create operator interfaces for automation systems that will support operator SA. Before automation, operators had to manually monitor and control every piece of equipment in the plant. This led to high physical workloads and unsafe work environments in order for the operators to stay aware of the state of the process. Currently in day-to-day operations at many industrial manufacturing facilities, automation interfaces provide operators with information from the field that is needed to control the process. For instance, consider the task of an operator in a modern paper plant. Most of his normal activity is routine. Large complicated pieces of equipment, such as boilers, are relatively easy to operate. For the most part, the boiler is stable and responsive. The automation system monitors and eases the physical workload of the operator. In many situations, the automation system compensates for changes occurring in the boiler while usually providing little feedback to the operator. The sheer size of the boiler means that the operator cannot know everything that is occurring. Seymour et al (2000) suggested that an interface containing all the relevant variables from just one power boiler could lead to information overload because of too much data and not enough information. 5 Furthermore, Zuboff (1989) described the control room of a modern paper mill: Where once the operators roamed the floor, smelling, hearing, and feeling the processes, they are now poised above the floor, isolated from sound, in an air-conditioned, glass control room. Manufacturing operators do not get the same information about the state of the process from meters and displays as they did from physically being present on the floor. While this separation contributes to a safer and healthier work environment when equipment fails or when a problem in the field occurs, the operator?s physical isolation can dramatically increase the difficulties and the magnitude of correcting any problem that may occur. In other words, operators are physically isolated from process problems that may be occurring in the field while at the same time mentally overwhelmed by the overabundance of data being displayed through the interface. Diagnosing process problems, knowing when to take action, and determining the appropriate course of action in this environment can be a daunting task. This physical isolation could be acceptable if the Human Computer Interaction (HCI) kept the operator fully aware of the situation at hand as if they were physically located in the field and presented the information needed in a format that supports human capabilities and cognition. Previous studies have indicated that the key for effective operator performance in such systems is maintaining awareness of the situation (Endsley, 2000). SA is a phenomenon that can be profoundly affected by the design of HCI, particularly when an operator is controlling a complex, dynamic environment (Sandom, 2001). It is plausible that integrating human factor principles into the design of the HCI specifically to support 6 operator awareness of the process they control could help to overcome the problems associated with physical isolation and the magnitude of the data being displayed. Theoretical Underpinnings This dissertation will review SA, human factors, HCI, and automation. These key domains drive this research project. The following sections will explore each briefly. Situation Awareness SA is a term used originally in the aircraft industry, and achieving good SA is perhaps the most difficult aspect of an operator?s job (Kardos, 2004). There are several different definitions of SA. Endsley (1988) defines it as the perception of elements within a volume of space and time, the understanding of their meaning, and the projection of their status in the near future. Dominquez (1994) puts more stress on the impact of awareness on cue extraction and directing of attention. In other words, Dominquez (1994) sees SA as a continuous extraction of information and the integration of information to form a coherent mental model and the use of that mental model in directing perceptions and foreseeing future events. SA has been divided into three hierarchical levels linked together by cognitive processes. The three levels are as follows: 1 Level 1 - Perceiving critical factors in the environment 2 Level 2 ? Understanding what those factors mean, particularly when integrating with the relation to the person?s goals 3 Level 3 ? An understanding of what will happen with the system in the near future 7 SA is not an easy thing to create, and errors in SA can be related to various factors, which could be introduced through the design of an automation system?s interface. Endsley (1995b; 1999) classified three categories of SA errors. They are as follows: 1 Level 1 ? Failure to correctly perceive the situation 2 Level 2 ? Failure to comprehend the situation 3 Level 3 - Failure to project the situation into the future The highest percentage of reported SA error is reported as failure of the operator to monitor or observe data, which occurs in the level 1 SA category. Even though SA is not a completely linear process, it is hierarchical in nature meaning that level 2 and 3 are usually not be reached without level 1 being realized (Endsley, 1996; Kardos, 2004; Endsley, 2004). Models of SA have been developed (Endsley, 1995b), and several methods of measuring SA have been proposed to evaluate the degree to which new automation systems or displays actually impact SA (Adams et al., 1995; Endsley, 1995a). The SA conceptual framework, Figure 1.1, includes key elements that can be classified by human factors such as information processing, memory, and experience along with external factors such as interface design, automation, and workload. 8 Figure 1.1 Model of SA in Dynamic DM (from Endsley, 1995b) To achieve good SA and make decisions operators draw upon available information, sensory inputs, their working memory, and long-term memory stores. Fundamentally, SA is an informational concept. Some have argued that SA can eventually be reduced to information in an operator?s working memory (Bell& Lyon, 2000). However, SA is much more than that and designers need to take into account how operators perceive, structure, and use knowledge gained from their changing environment through the system?s interface. 9 Human Factors and HCI Human Factors is an umbrella term that includes the following research areas human performance, technology design, and HCI. Human factors research is a field that focuses on cognitive and perceptual factors related to how users interact with products, tools, computers, and procedures. Interests of human factors researchers include user interface design, attention, memory, visualization of data, human error and SA. Many attempts have been made to define HCI (Shneiderman 1998; Dix et al. 2004; Preece et al., 2002). HCI has been gaining attention in MIS research and has become a focal area in the MIS discipline (Zhang et al. 2002). More recently, Olson and Olson (2003) defined HCI as the study of how people interact with computer systems and technology, and they identified two important aspects of HCI design: design of computer systems and evaluation of computer systems. The basic goal of HCI is to improve interaction between computer systems and users. The introduction of inexpensive computers and computer interface technologies brings about a vast set of HCI design possibilities and challenges. Over the past decade, organizations have been under tremendous economic pressure to rapidly implement these relatively inexpensive complex computer technologies to enhance system performance and improve employee safety while reducing human error. Traditionally, the development of interfaces has been driven by technology not the user. Automation interfaces have typically displayed all of the data that is available, and at its lowest state. They have not focused on the user or the tasks for which the system is being used. These interfaces require the user to request and seek information through explicit user input (Narayanan et al., 2004). This problem has lead to many serious concerns regarding 10 human capabilities and limitations. Even though designing interfaces based on human factors is complex and usually more than half of an automation system?s code is associated with the interface, interface design and development is traditionally left until late in the development process of the system (Hall et al., 2001). Human factor guidelines when implemented properly are expected to improve the user?s interaction with the system they are controlling. Designing around human factors is done to accomplish the following objectives (Sanders & McCormick, 1993): 1. Enhance the effectiveness and efficiency of human activities, often with focus on work. 2. Enhance desirable human values, including improved safety, reduced fatigue and stress, increased comfort, greater user acceptance, increased job satisfaction and improved quality of life. Currently, thousands of pages of information about human factor guidelines exist. Most of these documents come from governmental agencies or groups such as the FAA, military groups, or governmental task forces. These guidelines are usually general and not operational from a design point of view. Design clarification and evaluation is needed. Automation The objective of an industrial automation system is to assess, adapt, and coordinate process manufacturing and maintenance activities to meet the organization?s production plan and product specifications. In a modern, manufacturing environment thousands of process inputs exist and the number of monitored data points continues to increase drastically. In most facilities, the number of process variables bombards the user 11 and can lead to cognitive overload. This bombardment of the user is represented in Figure 1.2, and it depicts what has been termed ?the information gap? (Endsley, 2000) and the ?gulf of evaluation? (Norman, 1990). PI DCS Eng details MSDS other Data mines data collection, storage Applications End User prospector break anakyzer ISO TRAINING Pro ces s da ta Figure 1.2 Adopted from Hodge 2002 The data collection and storage section represents thousands of data points that are collected throughout the manufacturing process. The application area represents software that pulls automatically from the data collection sources, which also supplies information to the users. These two areas coupled together can lead to complete bombardment of the system?s users. The information gap characterizes the difficulty an operator experiences when using an interface not designed to help them perceive the process data, to make a 12 situation assessment, and to perform a needed system change. Therefore, the information gap illustrates attributes of HCI design that affect the operator?s cognitive workload. It describes the conceptual workload of an operator receiving the data, making a decision, and taking action; therefore, the bigger the gap the less desirable the interface (Norman, 1990). In control rooms where thousands of process data points are potentially available, identifying information, the useful part of data, is a serious issue. Since displays may be used for many different purposes, data are often presented at the lowest level of detail possible, typically the sensor level. This type of design is usually not beneficial in the development of a good SA of the process for the operator to control. Display characteristics such as poor screen layout, data displayed at too low a level or situations that require the operator to access several display pages sequentially, while manually extracting and integrating data along the way, are likely to increase the operator?s mental work load. This can lead to poor operator SA and accidents occurring that are referred to as human error. For instance, an operator can get lost in the interface due to the vast number of display pages that are available, or the operator could be cognitively strained in trying to perceive and comprehend the needed information. There is rarely any effort to analyze the information and control display needs for specific operational activities, or to use that knowledge to design interfaces that support operator goals given the current state of the process. In addition, process control systems are typically semiautonomous. This relationship can be represented by Sheridan?s (1992, 2002) model of human supervisory control, Figure 1.3. 13 Figure 1.3: Sheridan?s Model of Human Supervisory Control Sheridan introduced this model in 1960. The term supervisory control depicts the change in the operator?s role from manual controller to a monitor or supervising role of the automation systems (Sheridan, 1976). Operators monitor automated actions in order to detect problems in the process being controlled. When necessary, the operator is expected to intervene and to know what action to take. This shift to supervisory control of automation systems heightens human performance issues. Transforming the operator?s role to a predominantly passive monitor who occasional interventions with the process system is likely to strain human capabilities in areas where humans are known to be weak (Wickens, 1992). Research has shown that it is difficult to design automation interfaces so an operator is sufficiently aware to know when and how they should override the automation system (Smith et al., 1994). To date, there are no proven methods a designer can use to specify displays, feedback guidelines, or warning systems that enable operators to monitor automation systems effectively, to remain fully aware of their environment, and to intervene when necessary. The theory of human-centered automation has been recognized widely as a promising approach to integrating humans and computers (Billing & Woods, 1994). 14 Human-centered automation believes that operators should bear the final responsibility for safety of an automated system. In order for this to be feasible given today?s technology, the operator must maintain correct SA of the process he is controlling. In the development of human-centered automation interfaces, the emphasis is on information presentation based on the task being performed and cognitive processes, such as attention, memory, the development of mental models, and system learnability. Designing an interface is a difficult undertaking because humans possess a wide range of experience, needs, abilities, and expectations. The designer ?stands with one foot in technology and one foot in the domain of human concerns, and these two worlds are not easily commensurable? (Winograd, 1997, p.158). Research Objectives and Plan To efficiently assess available data and to make effective decisions in a control room, operators need to be aware of the situations they are controlling as if they were physically located in the field. The desired goal is to develop an interface design framework that will support efficient and effective operator SA. To do so, the model in Figure 1.4 is presented. Figure 1.4: General Research Model Situation Awareness Projections of Future Status Level 3 Comprehension of Current Situation Level 2 Perception of Elements in Current Situation Level 1 Interface Design 15 This model captures the impact of the interface design at each level of SA and the performance of the operator. The human factor design guidelines implemented for each level of SA will be explored in chapter 2. From this model the following research questions emerge: 1. How does the interface design impact the effectiveness with which the operator can perceive the needed information? 2. How does the interface design impact the effectiveness with which the operator can comprehend the needed information? 3. How does the interface design impact the effectiveness with which the operator can project the future state of the process? 4. How does the interface design impact the effectiveness with which the operator performs the necessary process actions? The answers to these research questions will contribute to our knowledge of how people interact with automation systems and how interface design impacts SA and operator performance. These results could be informative not only for the improvement of current automation systems, but may also be helpful in developing interfaces for emerging technology such as adaptive automation systems. Research Approach This study focuses on the relationship between HCI, human factor guidelines, and operator SA. To examine these research questions as close as possible to a real world environment, a microworld (MW) lab experiment to be performed. MW experiments are dynamic computer generated environments that subjects interact with in a laboratory that 16 simulates real world conditions encountered in the field. The characteristics of MW experiments include improved accuracy and efficiency of data collection procedures and precise levels of experimental control (Brehmer, 1992; Omodei & Wearing, 1995). It has been proposed that these benefits are achieved with concurrent gains in internal validity, afforded from high levels of experimental realism, and external validity afforded by the replication of the real world interactive nature of most field phenomena (DiFonzo et al., 1998). The review of literature will provide the foundation for each human factor design chosen and the level of SA that its design is expected to impact. Employees from a large Southeastern paper mill will perform the experiments as operators. Data will be collected through SA instruments and questionnaires designed to measure each of the three levels of SA along with operator performance data. MW experiments preserve certain functional relationships of a complex task environment while pairing away other. The functional relationships preserved are defined by the questions of interest to the researcher. Different worlds of the same task may preserve and pare away different functional relationships. In this study, I will attempt to preserve the functional relationship inherent in the approach that operators use of automation interfaces to control a complex, dynamic environment while paring away other aspects of their task environment. Microworld experiments attempt to maintain the realism inherent in the preserved functional relationship while being tractable for the researcher and engaging to the participant. 17 Limitations and Assumptions Going into this research, potential limitations of the study were considered. As with most research the ability to generalize the results are always a major point of discussion. This is definitely the case given the topic. HCI research has proven hard to generalize and difficult to test, especially in the field of automation (Gray, 2001). For example, only 25% of the papers presented at the Intelligent User Interface conference contained an empirical evaluation (Gill & Leake, 2002). By investigating each design concept individually by SA level, it is plausible that the results from this dissertation could be generalized across industries. Expected Contributions HCI knowledge needs to be made accessible and applicable to designers in order to achieve its intended effect. Therefore the results will be presented in a format, which supports the task of design. This means it will be translated into an accessible design framework and efficient methods that other researchers and developers could design by or further test for other domains. In addition, the method used could provide HCI designers with a usable format for obtaining rapid user feedback and suggestions for improvement is also a worthwhile activity. The expected contributions in this study include a better understanding of the variables that impact the operator at each level of SA. According to tests conducted through the ASM Consortium, human-centered interfaces enabled operators to complete task scenarios 41% faster than traditionally designed interfaces and enabled operators to improve their dealings with system failures by 26%. In addition, a human-centered 18 design approach could lead to a reduction in human error that accounts for 80% of industrial accidents (Labs, 2005). Summary Developing and maintaining a high level of SA is the most difficult part of most jobs, and it can be one of the most critical and challenging tasks in many domains today (Endsley, 2000). All of the incoming data from different systems, the outside environment, fellow crewmembers, and other resources must all be brought together into an integrated whole. This integrated picture is the organizing feature from which all decisions are made and actions are taken. The key to enabling human operators to mange in the 21st century is in the development of system interfaces that efficiently and effectively support operator SA. Presenting an abundance of data will only be successful when it can be transmitted, absorbed and assimilated in a timely manner by the operator. Therefore, system success will be found through interfaces designed by developers who understand how to combine and present vast amounts of data that support all three levels of operator SA. 19 II. LITERATURE REVIEW The enhancement of operator situation awareness (SA) has become a major design goal for automation systems in a wide variety of fields, including aviation, aerospace, nuclear plants, and advanced manufacturing systems. One can easily grasp that SA has always been needed in order for people to perform tasks effectively and efficiently. For many years, having good SA was largely a matter of learning what to watch and listen for in the field and learning what cues to recognize in the physical process, in other words, training and field experience. Today, operators rely on the principles and design of HCI to observe and comprehend the overwhelming amount of process data that can vary rapidly. It is widely accepted that more data does not equate to more information. In many cases, instead of assisting in this issue, automation has only worsened the problem (Endsley & Kiris, 1995; Sarter & Woods, 1995). Human factors research involves discovering and applying information about human behavior, abilities, limitations, and other characteristics to the design of machines, systems, and environments for productive, safe, comfortable, and effective human use (Chapanis, 1985). This research paradigm seeks to determine relationships between stimuli presented to humans and the action they take. To apply human factors in HCI design, one must investigate the system?s users, how they think, use technology, and interpret their environment. The design of the interface will determine 20 the extent, to which SA and information processing are optimized, and ultimately, the efficiency and effectiveness of an operator and automation system. Related Research This chapter discusses work from the literature that provides the foundation for an automation interface design that could support operator SA. Specifically, this chapter discusses issues with traditionally designed automation systems and describes previous research related to SA, information processing, perception, memory, mental models, and feedback. Lastly, this chapter presents evaluation methods for SA. Automation Automation systems use technology such as computers, sensors, and interfaces to control business and industrial processes. Automation systems have undergone significant changes since the 1970?s due to substantial advances in technology. Today, these systems are fed an enormous amount of raw data, which are displayed, through a HCI. The number of observable process variables is increasing at a drastic rate due to the development of ubiquitous and pervasive computing technologies, and in addition, economic pressures are pushing facilities to operate more efficiently. These pressures have increased the need for automation systems. One of the most significant challenges in the field of automation is the creation of human-centered interfaces. HCI designers are working hard to develop usable interface standards for these new technologies (Bohn et al., 2004; Davis, 2002). Interfaces that are human-centered refer to HCI?s that go beyond the traditional design to improve human cognition and interaction with technology by making it more intuitive and efficient. They enable the operators to know far more about a situation than traditional interfaces. 21 Human-centered interfaces could create a proactive computing environment rather than one that relies solely on the user to actively seek out information. Figure 2.1 contrasts a traditional automation interface and a human-centered automation interface. Traditional automation systems capture and display raw data independently of the operator and operating goals. This interface design simply displays every available piece of process data. This design can be cognitively very demanding. The operator is taxed with manually having to search for and monitor hundreds and even thousands of process data points. A poorly designed HCI can result in frustrated operators, poor decision- making, operator errors, and additional operating costs. In addition, the development of new technology has greatly changed the way operators interact with the process. For instance, operators who were once closely tied to the process spent the majority of their time working out in the field have now been Traditional Automation Interface Human-Centered Automation Interface Passive Data Active Information Raw Data Raw Data Repetitive Searches to Discover Process Status and to Determine Action Required Human-Centered System Cues the Operator to Facilitate Awareness and Action Required of Figure 2.1: Process Control Interfaces 22 physically removed from the process and placed in a control room. This has affected the operator in positive and negative ways. In a positive way, the operator is not physically overworked and is able to operate in a safe controlled environment. However, on the other hand, the operator is not physically close to the sights, smells, and sounds of the process. The automation system?s HCI must help to overcome this change in operator function. Human-centered interfaces are theorized as a way to focus on reducing unnecessary cognitive load. To accomplish this goal, information must also be presented in a way that supports operator SA. Situational Awareness SA is knowing what has happened, what is happening, and what is about to happen. Operator SA is defined in terms of tasks and goals for a particular job. For example, a power boiler operator does not have to know everything about controlling a power plant, but he does have to know everything related to his goal of safely operating a power boiler. Although the components of SA vary between domains, the nature of SA and the methods used for attaining it can be described in very general terms. A general definition of SA that has been applied across a variety of domains depicts SA as ?the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future? (Endsley, 1988). Figure 2.2, displays this definition and helps to illustrate the concept. 23 Figure 2.2 Model of SA in Dynamic DM (Endsley, 1995b) SA is divided into three levels. Detail descriptions of the three levels follow. Level 1 SA - Perception Perception of cues, level 1 SA, is vital. Without a basic awareness of the key information, the likelihood of developing a correct picture of the situation decreases considerably. Jones and Endsley (1996) found that 76% of SA errors in pilots could be linked to system shortcomings due to problems with perception of needed information and thus lead to problems with human cognitive processes. 24 Level 2 SA - Comprehension As a construct, SA goes beyond perception. It incorporates how humans combine, interpret, store, and retain information. Thus it includes the integration of information and the understanding of its importance to the current goal, level 2 SA. The difference here is that the operator is actually comprehending the significance and deriving meaning from the data perceived from level 1 SA. Twenty percent of SA errors were related to problems from operators in achieving comprehension (Jones & Endsley, 1996). Level 3 SA - Projection The highest level of SA includes the ability to predict future events, level 3 SA, from the data that has been perceived and comprehended. This ability to project allows the operator to make decisions in a proactive manner rather than reactive. Endsley (2000) noted that operator experience seemed to be a key difference between those who were able to achieve level 3 SA and those who were not able. Both the perception of time and the sequential dynamics associated with each event plays an important role in the formulation of SA. Therefore, time has appeared as an important component of SA in many domains (Endsley, 2000). A critical time factor in achieving SA is often understanding how much time is available until an event occurs or an action must be taken. The rate at which information is changing is a part of SA regarding the current situation, which also allows for projection of future situations (Endsley, 1988). Understanding the operator?s goals plays a big role in the development of SA. Goals can be thought of as the ideal process state that the operator desires to achieve. In what has been termed a top-down decision-making process, the operator's goals and tasks 25 will direct which aspects of the environment are attended to in the development of SA (Casson, 1983). However, decision-making is also a bottom-up process. The operator might identify patterns in the environment, which indicate that a different path is necessary to meet their goal or that another goal should be pursued. Alternating between top-down and bottom-up processing is an important part in achieving good operator SA. Interfaces must be designed to support this dual process. Initially, decision-making is a top-down, goal-driven process in which goals actively guide information selection and processing. While simultaneously, it?s a bottom-up process that occurs as information is perceived and processed to form SA. The interface design must be able to support both functions. For instance, a power boiler operator whose goal is to generate and distribute energy could hear an emergency alarm that would trigger a new goal should be sought. Without understanding operator goals, the information in the environment has no meaning. The SA construct allows researchers and designers to address the issue of meaning, something that has been lacking in previous interaction research (Flach, 1996). Information Processing and SA Describing SA does little in conveying the intricate complexities of how people seek information, compile it, and learn its meaning in an ever-changing environment. Several researchers have put forth theoretical formulations for depicting the role of numerous cognitive processes and constructs on SA (Adams, et al., 1995; Endsley, 1988; Taylor, 1990). Endsley?s (1995b) SA model is based on Wicken?s (1992) information processing theory. This model will be used as a basis for discussing the cognitive 26 theories and human factor research relevant to SA. The cognitive procedures that are important for the development of SA are shown in Figure 2.3. Figure 2.3: Wicken?s Information Processing Model, 1992 The goal of this dissertation is to translate knowledge of human information processing into design techniques for HCI to support operator SA. As can be seen from Wicken?s (1992) model, sensory processing, perception, attention, working memory, and long term memory are the basic foundation that SA is built. Sensory information includes an individual?s perceptions and attention span. Short-term memory is generally referred to as working memory, and long-term memory involves declarative knowledge and facts. Operator SA relies on information in regard to its context and a state of knowledge obtained built on that information. From an information-processing viewpoint, operator SA appears to fall along a spectrum. Operator SA can be a specific instance of perceived information. This is similar to Gibson?s (1979) ideas of direct perception, and, therefore, would indeed have a direct relation to the physical cues 27 observed in the environment. On the other end of the spectrum, knowledge can arise from the complex interaction of various mental processes that are not directly related to observable physical cues. Thus, operator SA can be created by the simple perception of elements in the environment (Level 1 SA; Endsley, 1995b) as well as comprehending what those elements mean (Level 2 SA; Endsley, 1995b). Because all perceptual experience is hypothetical, an operator must develop an accurate mental model of the process to help him conceptualize how the individual elements work together, and how to project future process states (Level 3 SA; Endsley, 1995b). The design of a user interface will determine the extent that these information-processing components are required for the operator to develop and maintain good SA, and will ultimately influence the efficiency and effectiveness of the system?s performance. Working Memory, Attention, and Perception Working memory and attention are limited resources especially in a complex environment with multiple competing cues. Once the information is perceived, it must be integrated with previous information, compared to goals and projected into the future. Each of these activities entails heavy demands from an operator?s working memory. Previous research has shown the importance of the role attention plays in SA. How attention is directed has a fundamental impact on which information is incorporated for SA. Many factors can direct attention, such as, the use of color, interface cues, operating goals, expectations, and other information already processed. Endsley and Smith (1996) showed that fighter pilots? attention to targets on a tactical situation display was directly related to the tactical importance of those targets. Gugerty (1997) found that drivers gave more attention to cars in front of and near to them than to those behind or 28 farther away. Both of these examples illustrate the distribution of attention based on perceived operational importance of information and the deployment of attention consistent with operational goals. Thus, attention is typically prioritized based on the perceived importance of the information, but even experienced operators can misinterpret information importance and neglect to attend to the most critical information. Adams et al (1995) described the challenges aircrews experience when dealing with dynamically changing information and competing tasks and the role attention plays in managing this challenge. Jones and Endsley (1996) found that the most frequent cause related to SA errors involved situations where all the necessary information was present, but it was not attended to by the operator. In a manufacturing environment this could be too much data being display, poor use of color, or the over use of alarms in the system. Working memory constraints also create a limitation on SA (Endsley, 1988). Novice operators or those in novel situations must merge information, interpret it and make projections, all within the working memory process. Jones and Endsley (1996) found that working memory failure, where information was initially perceived and then forgotten, equated to 8.4% of SA failures. In addition, Gugerty and Tirre (1997) showed strong evidence of the impact of working memory in the discrimination between people with lower and higher levels of SA, and Adams et al (1995) argued that even experienced operators can also be faced with so much information that attention and working memory limitations could still be an issue. 29 Long-term Memory, Mental Models and SA While long-term memory stores can take any formation, the notion of mental models has gathered much support. Long-term memory stores as mental models are hypothesized to play a key role in assisting with the limitations of working memory (Endsley, 1988; 1995b). As operators gain experience, they develop internal models of the process they control. These internally developed models aid in efficiently directing limited attention. They provide a way to integrate information without overloading working memory. The use of mental models in achieving SA is believed to be dependent on the individual?s ability to pattern match critical cues in the environment with elements in their mental model, and being able to incorporate the use of these models into SA can provide the operator with quick retrieval of actions from memory (Klein, 1989). The term mental model is a general concept. It represents a theory that operators control process based on patterns or relationships they have observed in the environment. These relationships or mental models by which experienced operators actually control the process might not look anything like the physical layout in the field, yet it empowers the operator to monitor and control the process in an accurate and efficient manner. Designing based on a mental model concept could be helpful in that it offers a method for directing attention to important aspects of the situation and promoting understanding of the relationships within the process. Other researchers have posited a strong relationship between SA and mental models. Sarter and Woods (1991) declared that adequate mental models are a requirement for achieving SA. Mogford (1997) argued that a mental model is the basis or the fundamental knowledge for SA. 30 People form mental models when they need to make a decision or prediction in a particular situation (Preece et al., 2002). In the context of automation systems, an operator?s mental model will be greatly influenced by the interface design being employed especially now since they are physically removed from the process. The visible aspects of the display, the actions that seem approachable and prior experience of the operator together form the mental model of how the process works. The degree to which the operator?s mental model accurately reflects how the process truly does work has a significant effect on the operator?s ability to use the automation system (Norman, 1990). Processes are difficult to control and interfaces are difficult to use when designers fail to present a coherent mental model in the interface design. Operators are forced to develop their own mental model of the process, which is likely to be deficient in some ways, leading to incorrect actions, confusion over results, and frustration in using the interface. The lack of an accurate mental model can cause the operator a lack of understanding, in turn making the automation system harder to use. This increases the cognitive effort required to accomplish a task or to project future events. With experience, an operator might be able to overcome the effects of an interface with a poorly designed mental model (Marchionini, 1995). However, one could posit that the more accurate the interface displays the process operating model, the easier it could be for experienced and novice operators to form and maintain accurate mental models of the process from which to operator. While a mental model strategy can be effective, it has also been related to human errors in manufacturing (Carmino et al., 1988) and medicine (Klein, 1993). Fracker 31 (1988) states that while mental models may be helpful in facilitating SA by reducing working memory demands; they can also lead to major problems from biasing the selection and interpretation of information that cause human errors to occur. In addition, Jones and Endsley (1996) found that approximately 7% of SA errors could be linked to poor mental models and that 6.5% of SA errors included the use of incorrect mental models to process information. Another 4.6% of SA errors involved over-reliance on default values in the mental model. Together these three issues from the use of poor mental models accounted for approximately 18% of SA errors, most of the 20.3% of the cases comprising comprehension or level 2 SA errors. Projection and SA Some of the more interesting and recent SA research has been aimed at increasing our understanding of the highest level of SA (Endsley, 2004). Projection is a very interesting concept, and it could allow for proactive operator decision-making rather than reactive. Jones et al. (2003) found that pattern-matching to similar situations could yield acceptable levels of situation awareness if specific cues were focused on the proper patterns, but this was not the case when the cues needed had to be inferred from other cues. From this study, incorrect projections tended to occur when the focus was on cues that were not related to the situation or when the operator had developed an incorrect mental model. Operators make decisions that alter the process they are controlling. Cues are needed in the form of feedback to support the operator. Feedback could facilitate operator understanding, learning, and aid in revisions to the mental model they are employing to operating. To achieve level 3 SA, time is also a critical factor. Therefore 32 the operator needs to receive feedback in a manner that will allow them to understand the dynamic state of the process, and the criticality of the events occurring. This feedback should not only give the operator understanding of the current state of the process, but it should also allow them to project what is about to occur. This form of cueing could help to achieve level 3 SA, and allow operators to be able to predict what will happen next in the process and thus what actions they need to take to bring them closer to their goal. Feedback is not a new idea. The need for complete feedback is one of the major points of Norman (1990). Without appropriate feedback, operators will not know if their requests have been received or carried out, or if abnormal events are occurring and a disaster is about to ensue. Feedback is also essential for learning how the process responds to a wide variety of situations and circumstances. Feedback is an essential part of automation systems; however, adequate feedback to the operator has typically been left out of interface design (Endsley, 2000). A problem with automation is that it is designed to run under normal conditions. When situations exceed the capabilities of the automation system, inadequate feedback leads to difficulties for the operators to know the current state of the process, to be able to take control from the system, and to act in a proactive manner. In addition, when an operator makes a decision based on projection, it is really more of an informed guess based on their experience. These decisions are usually only valid when the operator is knowledgeable in his field and is employing accurate data and sound logic. Operators need to be able to learn through interface feedback so they can alter their behavior as needed and can formulate current and future process situations. 33 Current display design guidelines are insufficient to provide the continual time appropriate feedback that occurs naturally among operators (Endsley, 2000). To solve this problem and to reduce human error, appropriate design considerations on how to provide feedback and how to address the time considerations need to be addressed. In addition, feedback allows for learning reinforcement, and being able to harness learning and projection as a process through the use of automation HCI could be a key in the future development of automation systems. Creating Automation Interfaces to Support SA Because much of interface design entails working with trade offs between cognitive and perceptual processes, effective interfaces are those that support operator strengths while reducing memory-intensive activities (Marchionini, 1995). A vital factor in facilitating operator SA is to develop and maintain system standards to facilitate consistency and clarity throughout the automation system?s interface. A consistent and clear interface that is used throughout the automation system is dependent on interface standards, cueing, and feedback that help the operator build and maintain an accurate mental model of a process (Preece et al., 2000). Shneiderman (1998) points out that designing ?an environment in which tasks are carried out almost effortlessly and operators are ?in the flow? requires a great deal of hard work from the designer? (p. 10). The designer must assess the types and amount of information accessible, and then group the information in an interface design that will support the operator?s goals and tasks. All of this must be performed within constraints of available technology, while keeping in mind considerations such as total display size. 34 Interfaces should be designed to support human limitations such as attention and working memory. For instance, in traditional interface design the amount of attention required by the operator for task management processes is frequently high and very demanding. This is to some degree a result of system designers using poor mental models and ineffectively designed views with little or no feedback. This results in interfaces in which operators fail to achieve good SA due to imposing high demands on working memory and forcing the operator to constantly divide attention between primary goals and task management processes. It is plausible that progress in creating interfaces to support operator SA might be made if system designers considered alternative design models that impose less of a task-management burden on the operator and enables more attention to be given to perceiving, problem solving, and decision-making. While difficult, the effort required to create more intuitive interfaces that support operator SA could increase operating efficiencies, reduce downtime, and increase production capacity. The evolving trend from the traditional, technology-centered automation system to human-centered automation clearly has the potential to deliver systems that provide interfaces that are more intuitive, effective, and produce higher levels of SA. Systems that are designed based on these principles could promote a higher level of awareness in which the operator could solve real world problems more efficiently. Reducing cognitive load and reliance on memory is a fundamental aspect of creating effective interfaces to support SA, since operators: ?want to achieve their goals with a minimum of cognitive load and a maximum of enjoyment. They do not want to be distracted from their real task or divert scarce cognitive resources to the retrieval tasks. Moreover, humans tend to seek the path of least cognitive resistance? (Marchionini, 1992, p. 156). 35 Perception: Structuring Information and Providing Visual Cues Although the quantity of available information has drastically increased since Herbert Simon (quoted by Varian, 1995) spoke about the problem caused by its overabundance, current automation interfaces rarely help operators allocate their attention efficiently. Previous research demonstrates that the way information is arranged and presented affects how quickly tasks referring to that information can be completed (Preece et al., 2000). Yet despite the increasing quantity of process information that automation systems are accessing, this information is rarely organized and structured in a manner that aids the operator in optimally allocating attention, accomplishing his goal, and making decisions. From a physiological stance the task of visually seeking information is a complex process involving a number of interacting factors. A convenient metaphor for understanding how operators scan and process information is that of a spotlight (Pirolli et al., 2001). The spotlight identifies the area that is further processed by the brain. The spotlight can be either wide or narrow. When it is wide, the operator?s range encompasses a relatively broad physical area but usually at a low resolution. When it is narrow, the operator is more detailed focused on a specific area. The range of the spotlight can be dependent upon task, interface density in regards to information, and visual cues. Research in cognition and perception shows that when visually searching for information, operators automatically apply systematic techniques to distinguish target information from information they are not interested in, initially scanning the display area with a broad spotlight before focusing on the items of interest (Pirolli et al., 2001). By 36 taking a broad view of an area operators can detect various forms of models in the visible information and use those models to guide their actions (Rabbit, 1984), including the process of further recognition of information within each model. This initial detection can occur very rapidly, such that operators can sometimes recognize the overall model of a display before fully resolving all the details within the display. Traditionally automation interfaces have been designed with a black background, and color has been used for every piece of process data (Labs, 2005). This has led to the overuse of color making it very difficult for an operator to develop an awareness of critical events as they occur. Some industry professionals have referred to this design as ?the Christmas Tree? since every piece of process data and equipment is in color or even flashing at times. Even though the operator is being cued in a consistent manner, awareness of critical events is hard to develop in this traditional display design. According to Reising (2002), there is a movement to move away from the black screen to a light background with a gray scale color theme and to only use color to cue critical changing process data. One could posit that this could enhance the operator?s ability to perceive the critical process situations occurring. However, this design could negatively impact the operator?s ability to achieve level 1 SA of the entire process if the operator is only alerted to the critical events. On the other hand, one could posit that the traditionally designed interface will result in a higher level of operator SA of the entire process. This study will investigate the impact of each interface design has on the operator?s ability to achieve and maintain level 1 SA. H1a: The operator?s ability to perceive information when the process is in an abnormal state will be significantly greater when operating with the interface 37 design based on consistent visual cueing for abnormal events only versus cueing for all process events. H1b: The operator?s ability to perceive information when the process is in a normal state will be significantly greater when operating with the traditionally designed interface versus the interface designed to cue only on the abnormal process events. H1c: The operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed based on consistent visual cueing for abnormal events only versus cueing for all process events. Comprehension: Development of Effective Mental Models A substantial body of literature suggests that when people interact with machines, devices, computers, and even other people, they rely heavily on prior knowledge to develop mental models that help them understand the interaction and predict its behavior (Norman, 1990). These mental models, while not identical to the physical system, represent the components of the system in ?an analogical manner that parallels the structure of the state of objects in the world? (Preece et al., 2000). The creation of mental models is an intuitive strategy for reconciling observation with expectation (Kuhn, 1993) influenced by both our previous knowledge and the nature of the interaction itself (Norman, 1990). Given the natural creation of mental models, the challenge in creating an effective automation interface that supports SA is to design the interface such that the operator is easily able to develop an accurate mental model of the process (Norman, 1990). In 38 attempts to make interfaces understandable, a common approach used by system designers is to ?ground operator interface actions, tasks, and goals in a familiar framework of concepts that are already understood? (Neale & Carroll, 1997, p. 441). This approach has traditionally meant that interfaces were designed based on the physical structure and layout of the equipment in the field. Designing based on the physical layout of the process has been shown to be a benefit of graphical displays as compared to textual displays (Shneiderman, 1998), and traditionally automation interface designers have used structural drawings such as piping and instrumentation diagrams as their basis for display design. However, the physical layout of the equipment in the field may not be the best representative to help the operator develop a good mental model of the process. It is plausible that designing interface displays based on the process? actually operating model could positively impact the operator?s ability to achieve level 2 SA. This study will investigate the impact of process model used in interface design on the operator?s level of SA achieved and maintained. H2a: The operator?s ability to comprehend process information will be significantly greater when operating with the mental model interface design versus the physical model design. H2b: The operator?s ability to perform necessary process actions will be significantly greater when operating with the mental model interface design versus the physical model design. 39 Projection: Feedback Level 3 SA centers on an operator being able to make predictions about the state of the process he is controlling. A prediction is knowing that a certain event is about to occur. Learning to predict is a process of acquiring knowledge, skills, or attitudes through training and experience. For example, when a child touches fire for the first time, he quickly learns to not touch it again by receiving feedback in the form of pain. So then, the next time a child sees a burning fire he can predict that, ?if I touch it I will hurt myself?. Operators develop their ability to predict in a similar fashion, but their feedback comes in the form of information displayed on their interface. Learning is a feedback process (Sterman, 1989). So when operators or automation systems perform actions, feedback is essential for the operator to allow for the detection and correction of errors, and to be able to predict the future state of the process (Norman, 1990). If the operator is supplied with proper time critical feedback, he is more likely to learn accurate mental models, and to be able to make accurate predictions of future events. Automation systems typically have a desired or normal state, a means for adjusting the process towards that desired state, and an internal feedback loop in which the actual and desired states are compared. This comparison is needed so that further adjustments can be performed if the two states do not match. This process is called the control loop. When an operator manually controls the process, the operator is an essential part of the control loop. Before automation, problems such as high physical workloads existed along with an over reliance on the operator to always be alert, accurate, and experienced, but with 40 this also came the advantage of the operator fully being a part of the control loop. In automation systems, lower level controls are usually taken care of by the system, and the operator can easily be left out of the loop (Norman & Orlady, 1989). Being left out of the control loop can result in operating problems and safety issues. Previous research has shown that continual interaction with an automation system serves to keep the operator attentive and informed, and helps with continual training and learning, and thus allows the operator to be an active member of the control loop (Norman, 1990). As previously stated, operators do construct mental models of process they are controlling. These models are constructed entirely from the information available to them through the automation system?s HCI. Feedback should be included in the information presented to the operator. Presenting feedback in an appropriate, timely critical way is not easy to do. Currently, interface design guidelines do not exist on how to display this level of feedback to an operator; however, there are several examples of how not to, such as the overuse of alarms. A continual feedback process performed in an intuitive manner could promote operator learning, and thus allow operators to be able to predict the future state of the process. This means designing systems that are informative and promote learning without irrupting or irritating the operator. Feedback is essential because equipment does fail and because unexpected events do occur. Operators need to be able to not only cope with unexpected events, but they also need to be able to learn from process situations to be able to predict when these events are about to occur. For example, suppose that the automatic pilot could signal the crew that it was starting to compensate for fuel usage more than in normal conditions or at least more than an hour ago. This information could help to alter the operator?s current 41 SA allowing him to predict that an abnormal situation was either occurring or about to occur. Technically this kind of information is always available to the operators through the data being displayed on the interface, but these changes are generally subtle cues that are not discovered until a dangerous situation is occurring. One can easily posit that when designing an automation interface, the use of consistent, time critical feedback in the interface design could help the operator learn and better project the near future state of the process, thus impacting the operator?s ability to achieve level 3 SA. This study will investigate the impact of interface feedback on the operator?s ability to develop level 3 SA. H3a: The operator?s ability to project the future state of the process will be significantly greater when operating with the interface designed with time based feedback versus the traditionally designed interface. H3b: The operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed with time-based feedback versus the traditionally designed interface. Research Model for Each Level of SA The review of automation and human factor literature has tackled some of the issues with traditionally designed automation interfaces, and some of the reasons why achieving good SA can be difficult for the operator to accomplish. To test each proposed interface design discussed in this chapter, the model in Figure 2.4 is presented. 42 Interface Design Using consistent visual cues in traditional vs. abnormal events Design to promote accurate mental model development Using consistent time-based process feedback Situation Awareness Projections of Future Status Level 3 Comprehension of Current Situation Level 2 Perception of Elements in Current Situation Level 1 H1a H1b H1c H2a H2b H3a H3b Figure 2.4: Research Model Evaluating Operator Interfaces There are a number of reasons to evaluate operator interfaces (Landauer, 1997; Shneiderman, 1998) 1. to establish other standards and guidelines, to test scientific theory 2. to determine how existing systems or features compare 3. to explore whether people understand how to use a new system Regardless of the specific reason, evaluation is a vital component of human-centered design. HCI includes a complex set of components; hardware, software, and human behavior that usually interact in unpredictable ways (Landauer, 1997). Evaluation presents empirical evidence to overcome the ?unreliability of intuition and the variability of behavior? (Landauer, 1997, p. 206) inherent in humans and enables judgments to be 43 made about whether an interface can successfully be used by an operator in the designed situation (Preece et al., 2002). Most reasons are focused on at least one of two main objectives: To determine the effectiveness or potential effectiveness of the interface, or to provide a means for suggesting improvements in that interface. SA is a useful construct for evaluating HCI design, and it allows for better understanding of human factors such as cognition. In addition, the SA construct provides far greater diagnostic and sensitivity than is typically offered from traditional performance measures (Endsley, 2000). To determine the degree to which new technologies or design concepts actually impact operator SA, it is necessary to systematically evaluate them based on a the level of SA attained, thus providing a determination of which ideas have merit and which have unforeseen negative consequences (Endsley, 2000). Therefore, various methods have been developed to measure SA. Several of these measures are reviewed in the following sections. Objective Measures of SA Situation Awareness Global Assessment Technique Situation Awareness Global Assessment Technique (SAGAT) is a direct technique for measuring SA through questions, and it provides an objective evaluation of SA by comparing the real situation to the perceived situation (Endsley, 1988). SAGAT was originally developed for the aviation industry, but it has also been utilized in many other domains, such as, military, automotive, power plants, and medicine (French & Hutchinson, 2002; Gugerty, 1997). This method consists of a pool of questions that extract information from an operator in a simulated environment across all three levels of SA. 44 The simulation is frozen at random times while the operator answers a series of randomly selected questions about the current situation (Guille & French, 2004). Randomizing the questions is needed to counteract any possible learning effects. Once the simulation is complete, the answers are compared to what was actually happening during that point of the simulation. This method has its strengths and weaknesses. Strengths include that it is a direct, objective, and unbiased measure of SA pertaining to the operational environment. In addition it holds a high degree of construct validity (Endsley, 1996). Disadvantages include the intrusiveness of freezing the simulation in order to collect data, and that the method might not provide a true reflection of the operator?s SA as it relies to on some degree the operator?s memory (Endsley, 2000). Behavioral Measures for Inferring SA Operators are expected to act in a certain way based on their SA. Therefore, information about SA may be inferred from examining operator behavior on specific tasks. Such behavioral indices might include reaction time to make a response, time to complete a scenario, and decision-making, whereby a particular conclusion is used to infer the SA that underlies the decision (Endsley, 2000). The strength of this method includes it being objective, observable, and non- intrusive. However, this method assumes an appropriate behavior given the SA level. A disadvantage is that these assumptions may not always be correct. An operator may not act in the predicted or preferred way even if they had acquired perfect SA. 45 Subjective Measures of SA Situation Awareness Rating Technique Situation Awareness Rating Technique (SART) is a subjective measure of SA that utilizes operator self-ratings to assess perceived SA (Taylor, 1990). Originally designed for the aviation industry, SART uses a self-rating instrument to measure subjective SA on either 3 or 10-dimensions. The 3-dimensions chart is used when the 10-dimensions chart would be too intrusive or time consuming. SART reflects generic SA constructs, and since the constructs are general in nature, they can be applied to other domains (Guille & French, 2004). The dimensions are listed in Table 2.1. 10 Dimensions 3 Dimensions Instability of Situation Variability of Situation Complexity of Situation Attentional Demand Arousal Spare Mental Capacity Concentration Division of Attention Attentional Supply Information Quantity Information Quality Familiarity Understanding Table 2.1: SART Dimensions A SART analysis begins with the creation of scenarios that feature the situation of interest. During the experiment, the operator is provided with either a 3-Dimensions or 10-Dimensions SART chart to record their perceptions of SA at a given point in time. 46 The scores from these charts are then statistically analyzed to determine how different aspects affect SA. An advantage of this method is that it provides a level of diagnostic ability (Jones, 2000). A disadvantage is that this method will add to the workload of the operator, which could muddy the measure of SA. Post Trial Participant Subjective SA Questionnaire The Post Trial Participant Subjective SA Questionnaire (PSAQ) is a three-item instrument designed to assess a subject?s perceived level of SA, workload and quality of performance (Guille & French, 2004). Advantages include this instrument being administered at the end of a simulation. Each item is evaluated on a five-point scale. Operators are asked to rate themselves on how hard they were working, how well they performed, and how aware they were of the evolving situation during the simulation. Operators are also given the opportunity to elaborate further on all three of the items. A disadvantage is that the PSAQ is fairly new. Therefore, its validity has not been fully tested (Guille & French, 2004). Summary As automation technology continues to evolve and digital information becomes more accessible to operators, it becomes equally critical to improve the way operators interact with the available information. Operators no longer use automation simply to accomplish a handful of special-purpose tasks; instead they use the automation system to monitor and control almost every part of the process. To successfully integrate the overabundance of data and to overcome the physical separation, interaction design must evolve from simply seeing the process variables to seeing how the operators use the 47 information provided to control the process (Winograd, 1997). The work presented in this dissertation is intended to contribute to this evolution of HCI design and operator SA. 48 III. RESEARCH METHODOLOGY The introduction and literature review chapters presented theory on operator situational awareness (SA) and the relationship between human factors and interface design. This chapter describes the methodology that will be used to evaluate the operator interfaces designed to support SA. The designs will be evaluated by conducting a microworld (MW) field experiment that will assess each interface design according to the ideas proposed in the literature review. An overview of the research methodology is first provided. Next, the MW experiment that will be used for the proposed study is explained along with a description of the interfaces, and the specific tasks study participants will be asked to complete. Research Design Experimental testing in the field of human-computer interaction typically involves studying users in a controlled environment, such as a laboratory experiment. An experiment usually has propositions or hypotheses bases on theory being tested with an appropriate experimental design. This is done by manipulating an independent variable and collecting data associated with dependent variables. The data are then analyzed using statistical tests to draw conclusions about the viability of the hypotheses. Since the number of factors that can be practically manipulated is limited, experimental testing is most often used to investigate very specific elements of a system or interface to make general statements about particular interface design principles. Data 49 can be collected through a variety of techniques, such as observation, task activity logging, and user perception of the interface. A well-designed experiment can produce sound evidence to be used as support for conclusions about user performance, user preference, and interface design. There are disadvantages to experimental testing. A significant amount of preparation and planning is required to develop an appropriate experimental procedure; a lot of time and resources are necessary for conducting the experiment and for analyzing and interpreting the results. In addition, care must be taken to ensure that only the independent variables are manipulated or varied, and the potential problems, such as order effects, should be controlled through methods such as randomization, counter- balancing, and sampling. Microworld Experiments and Other Simulation Environments In field research, there is often too much complexity to allow for any more definite conclusions, and in laboratory research, there is usually to little complexity to allow for any interesting conclusions (Brehmer, 1992). MW experiments are one reaction to this complexity. Beginning with a complex task environment a MW seeks to preserve certain functional relationships, while paring away others. There can be multiple scaled worlds of the same task environment that differ as to which functional relationships are preserved and which are pared away. The nature of the research questions determines what is kept and what is removed. Other simulated lab environments include high-fidelity simulations, synthetic environments and scaled worlds (Brehmer, 1992). These environments differ from each other, but they all require the 50 following three dimensions (Figure 3.1): tractability, realism, and engagement. All three must be addressed in the experiment. Figure 3.1: Relationship between three dimensions of simulated task environments (Ehret et al., 2000) Tractability Tractability is the issue of ?complexity? referred to by Brehmer and D?rner (1993). The following issues must be addressed. The researcher must determine whether the simulated environment can successfully be used to pursue the question of interest. Tractability includes concerns such as collecting the right data, at the right level of detail, with an accurate timestamp. In addition, it addresses whether the participants can learn the simulated task in an acceptable amount of time and the usability of the simulated task environment. 51 Tractability is a relative dimension that is characterized by the research questions. Therefore, what might be a tractable simulation for one set of research questions may not be tractable for another. As in this study, the questions focus on the flow of information to and from the operator. One would expect a high-fidelity simulation of a complex system to be almost as intractable as the real-world they simulate. In contrast, there are few constraints on how MW experiments are constructed. These simulation environments can be built to the researcher?s specifications. Therefore, if these are not tractable, they have been built incorrectly. This will addressed using subject matter experts. Realism The next dimension is realism. The simulated task environment is realistic to the situation to the extent that experiences encountered in the simulated environment occur in the real task environment (DiFonzo et al., 1998). A MW experiment should be more realistic than a lab experiment designed to investigate the same functional relationships, but less realistic than a high-fidelity simulation. MW experiments are specifically focused on preserving certain functional relationships from the real-world environment. Maintaining these functional relationships maintains a type of realism. In general, MW experiments may try to maintain the realism of other aspects of the task environment unless such realism interferes with the tractability of the research questions of interest. This aspect will be addressed using subject matter experts. Engagement The third dimension is engagement. Engagement explains something about the participant?s motivation for performing the experiment. Participants may be engaged 52 because we give them extra credit in classes, because they are paid for their time, or because they see a benefit to them in the future through their participation. Alternatively, they may be engaged because they have deep knowledge of the real task environment and believe that it is interesting and important (Ehret et al., 2000). In this situation, engagement will come from their knowledge base and from the possibility that their input could possibly impact the interface designs from which they operate on a daily basis. Experiment Overview The SA interface designs that have been described in Chapter 2 will be evaluated by conducting a MW field experiment in which the study participants will perform tasks related to operating an automated water heating plant. The four proposed interface designs will be operationalized by developing four experiments with Flash 8, Macromedia, and ColdFusion. The tasks the participants will be asked to do during the study are intended to produce data on each interface design, while also providing information that can be used to evaluate the operator?s level of SA achieved and his performance. Each participant will operate with one of the four different interface designs. Each interface design will be used as a treatment to enable comparisons to be made between the proposed designed to support SA and task performance. The tasks in this phase are focused tasks; the purpose of the tasks is to observe how quickly and accurately the participants can perceive, comprehend, and project using the interface design in solving realistic process automation problems and to give the participants a basis for providing opinions, ideas, and preferences related to each interface design. 53 Quantitative data will be collected from the performance-based tasks to determine whether there are differences in user performance based on the interface used, the amount of effort required by the operator per interface used and whether there are differences in the level of SA achieved by each operator. SA will be measured objectively and subjectively through the use of two methods. The methods are as follows: 1) Situation Awareness Global Assessment Technique (SAGAT) 2) Post Trial Participant Subjective SA Questionnaire (PSAQ). The data collected from this study will then be analyzed and used to answer the following research questions about interface design and operator SA. To efficiently assess available data and to make effective decisions in a control room, operators need to be aware of the situations they are controlling as if they were physically located in the field, 1. How does the interface design impact the effectiveness with which the operator can perceive the needed information? 2. How does the interface design impact the effectiveness with which the operator can comprehend the needed information? 3. How does the interface design impact the effectiveness with which the operator can project the future state of the process? 4. How does the interface design impact the effectiveness with which the operator performs the necessary process actions? The experimental data and research questions will be used to produce three outcomes as suggested by Shneiderman: ?(1) specific recommendations for the practical problem, (2) refinements of theory of human performance, and (3) guidance to future 54 experimenters? (1998, p. 32). These outcomes form the basis of the discussions in Chapter 4 and Chapter 5. Study Participants Participants for the study will be recruited from MeadWestvaco Mahrt?s Mill. The participants will consist of operators, operation managers, and electrical and instrumentation maintenance employees. No special user characteristics will be sought in the recruitment of study participants; it is expected that the participants who agree to take part in the study will represent a reasonable variation in gender, personality, and computer experience. Study Procedure The study will be conducted in sessions with each individual study participant at a time, with a session expected to last approximately 20 minutes. A total of 120 participants will complete the study. Four interface variations will be used for the experiment (Figure 3.2 ? Figure 3.5). The interfaces used by each participant will be assigned randomly, and each participant will experiment with using two of the four interfaces during for the study. A training session on the purpose and operation of the plant will be given to each participant along with performance task instructions. As the experiment runs, the participant will be given SA questions to answer two times during the experiment. The experiment will be followed by a SA post-test interview. The experiment questionnaires are located in the Appendix. Plant Description The plant consists of four tanks, three pumps, three valves, and two heaters. The purpose of the plant is to heat water to 80? Fahrenheit for as long as possible without 55 incurring a pipe rupture. In order for the plant to be operating at its desired level, the following process state requirements must be met. Temperature requirement The temperature of the product fluid leaving the first heater should be between 70? and 75? Fahrenheit. The temperature of the product fluid sent to hot tank must be kept between 85? and 95? Fahrenheit. Flow requirement All flow rates must be within the range of 50 to 60 gallons per minute (gpm). Failure: Pipe Rupture Pipe rupture will occur after the flow to the heater as dropped 75% from its target level of 50 gpm for more than 1 minute. Once the pipe ruptures the operator must shutdown the entire plant. Alarm States If a process variable rises above or drops below it?s desired operating level within a plus or minus 15% to 35% level, then a low level alarm occurs which will turn the variable blue. If the increase or decrease is between 35% and 75%, then a medium level alarm will occur which will turn the variable yellow. If the increase or decrease is 75% or greater, then a high level alarm will occur which will turn the variable red. The process must reach steady state first. Then the alarm states will begin being displayed. Feedback (Only provided with Interface #4) The arrow displayed indicates feedback for the process variable it is pointing to. If the previous 5 samples taken on that variable show an increasing or decreasing trend, 56 the arrow will point either up or down respectively. If the variable has not changed, the arrow will point straight to the variable. Plan of Analysis To analyze the results of the experiments, Analysis of Variance (ANOVA) will be used. Specifically, the differences between the levels of SA achieved and the performance based on each interface will be examined using ANOVA. After each ANOVA analysis, a power and sample size analysis will be performed to determine the power level for each ANOVA test. The appropriate F percentile is used to construct the decision rule. SPSS is used to calculate the F-test significance and the associated p-values to evaluate if significant differences exist at the appropriate alpha level. The alpha used in this study will be 0.05. As the sample means of each group could differ in either a positive or negative direction, post hoc comparisons, such as Tukey or Games Howell, will be used depending on the results from the Test of Equal Variance. The conclusions drawn from these tests will ascertain the differences between the four groups: Interface #1 (Traditional), Interface #2 (Color for Alarms Only), Interface #3 (Mental Model Layout), and Interface #4 (Feedback). ANOVA will also be used to establish the validity of the randomization procedure used and to verify that any differences seen are based on interface and not demographic differences. Summary Each hypothesis test will indicate whether or not the individual proposed level of SA increased, whether or not the total level of SA increased, or whether or not the success rate increased based on the interface design used. The results of these tests will 57 be presented in chapter IV. The implications of the obtained results will be presented in chapter V. 58 0 gp m St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 gp m Mi ll W ate r To M ill 0 gp m 0 gp m 0? F 58 gp m Su pp ly Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 45 gp m 50 gp m 80 ?F Ru nn ing He ate r 1 He ate r 2 Ru nn ing Fig ur e 3 .2a : In ter fac e # 1 ( Tr ad itio na l D esi gn ) p ro cess no t r un nin g Fig ur e 3 .2b : In ter fac e # 1 ( Tr ad itio na l D esi gn ) p ro cess ru nn ing in an ab no rm al sta te 58 59 0 g pm Sto pp ed Sto pp ed Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pro du ctio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 58 gp m Su pp ly Ho t Co ld Pro du ctio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 45 gp m 50 gp m 80 ?F Ru nn ing He ate r 1 He ate r 2 Ru nn ing 0 g pm Sto pp ed Sto pp ed Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pro du ctio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 0 g pm Sto pp ed Sto pp ed Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pro du ctio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 58 gp m Su pp ly Ho t Co ld Pro du ctio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 45 gp m 50 gp m 80 ?F Ru nn ing He ate r 1 He ate r 2 Ru nn ing 58 gp m Su pp ly Ho t Co ld Pro du ctio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 45 gp m 50 gp m 80 ?F Ru nn ing Ru nn ing He ate r 1 He ate r 2 Ru nn ing Fig ur e 3 .3a : In ter fac e # 2 ( Co lor fo r A lar ms On ly) pr oc ess no t ru nn ing Fig ur e 3 .3b : In ter fac e # 2 ( Co lor fo r A lar ms On ly) pr oc ess ru nn ing in an ab no rm al sta te 59 60 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m Fig ur e 3 .4a : In ter fac e # 3 ( M en tal M od el La yo ut) pr oc ess no t ru nn ing Fig ur e 3 .4b : In ter fac e # 3 ( M en tal M od el La yo ut) pr oc ess ru nn ing in an ab no rm al sta te 60 61 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 0 g pm St op pe d St op pe d Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 0? F 0? F 0 g pm Mi ll W ate r To M ill 0 g pm 0 g pm 0? F 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m 45 gp m Ru nn ing Ru nn ing Su pp ly He ate r 1 He ate r 2 Ho t Co ld Pr od uc tio n 72 ?F 92 ?F 55 gp m Mi ll W ate r To M ill 80 ?F 58 gp m 50 gp m Fig ur e 3 .5a : In ter fac e # 4 ( Fe ed ba ck ) p ro cess no t r un nin g Fig ur e 3 .5b : In ter fac e # 4 ( Fe ed ba ck ) p ro cess ru nn ing in an ab no rm al sta te 61 62 CHAPTER 4: ANALYSES AND RESULTS This chapter presents the results of implementing the microworld (MW) field experiment as outlined in Chapter III of this dissertation. First, the chapter presents the experimental dimensions required to assure validity when conducting a MW experiment. Second, the procedure and participants in this study are discussed. Next, the Analysis of Variance (ANOVA) results for the demographic variables and for each experiment are reported, and finally, each hypothesis is evaluated and summarized. Microworld Experiment Dimensions MW experiments begin with a complex task environment, and then they seek to preserve certain functional relationships, while paring away others. The nature of the research questions determines what is kept and what is removed. Each MW environment differs, but they all require the following three dimensions (Figure 4.1): tractability, realism, and engagement. All three dimensions must be addressed. 63 Figure 4.1: Relationship between three dimensions of simulated task environments (Ehret et al., 2000) Tractability Tractability is the issue of ?complexity? (Brehmer & D?rner,1993). To address ?complexity?, the following questions must be answered. 1) Can the MW environment successfully be used to pursue the question of interest? 2) Is the MW experiment collecting the right data, at the right level of detail, with an accurate timestamp? 3) Can the participants learn the simulated MW task in an acceptable amount of time and the usability of the simulated task environment? To address the first two questions, three engineers who are experts in the automation industry reviewed the four designed experiments. The engineers evaluated each 64 experiment to verify that interface design operationalized its proposed human factor guideline. They also evaluated the questions to confirm that the appropriate data was being collected. To address the third question, six operation managers from MeadWestvaco?s Mahrt Mill piloted the four experiments. The experiments were piloted to ensure that the operators would have enough time and enough information to accurately run and evaluate each experiment. Realism The MW task environment has to be realistic to the situation to the extent that experiences encountered in the simulated environment occur in the real task environment (DiFonzo et al., 1998). MW experiments aim to maintain the realism of the task environment while being able to focus on the research questions of interest. The operations managers also addressed this issue of realism. During the piloting of the experiments, the managers where asked to comment on how realistic they felt each experiment was. This aspect was also addressed by using subject matter experts to take the experiments. Engagement Engagement explains something about the participant?s motivation for performing the experiment. Participants were engaged in this study because have deep knowledge of the real task environment, and since this is their job, they believe that it is interesting and important (Ehret et al., 2000). The participants were also engaged because their input in these experiments could possibly impact the interfaces from which they operate in the future. 65 Experiment Testing Participants and Randomization Participation in the experiment was voluntary. The participants? backgrounds indicated a heterogeneous group of operating areas, age, years of experience, and job type. Given the work environment of a paper mill, it was assumed going into the experiment that the participants would be heavily weighted in gender to male. Alternating the experiments randomized the participant population. By doing so, four groups were formed with 114 usable observations (28:Traditional Design; 28: Color for Alarms Only; 30: Mental Model Layout; 28: Feedback). The demographic variables across the four groups seemed similar and are presented in Table 4.1 and Table 4.2. Each participant was tested for colorblindness after they completed the experiment and the follow up questionnaires. Out of the 120 participants tested, six tested positive for red-green colorblindness. This resulted in 5% of the sample tested as being colorblind as shown in Table 4.3. Which is equivalent to the estimated 5% of colorblind males and females in the United States population (http://waynesword.palomar.edu/colorbl1.htm). It is important to point out that being red-green colorblind does not mean the individual cannot see red or green separately. In most cases, it means that the person lacks the receptors to be able to distinguish between red and green when those two colors are together. Red-green colorblindness is the most common colorblindness that exists (Montgomery, 2007). 66 Demographics Count Percent of Total Paper Machine 39 34.21% Pulp Mill 18 15.79% Recovery 24 21.05% Recycle 17 14.91% Operator Area Electrical & Instrumentation Maintenance 16 14.04% Male 103 90.35% Gender Female 11 9.65% 25-29 6 5.26% 30-34 10 8.77% 35-39 18 15.79% 40-44 19 16.67% 45-49 22 19.40% Age 50+ 39 34.21% 1-5 10 8.77% 6-10 18 15.79% 11-15 23 20.18% 16-20 36 31.58% Years of Experience 20+ 27 23.68% Lab 3 2.63% Inside 36 31.58% Outside 46 40.35% Job Type Both 29 25.44% Table 4.1: Demographic Variables 67 Demographics by Experiment (Count) Traditional Color for Alarm Only Mental Model Layout Feedback Paper Machine 8 13 10 8 Pulp Mill 5 2 6 5 Recovery 8 6 5 5 Recycle 2 5 5 5 Operator Area Electrical & Instrumentation Maintenance 5 4 2 5 Male 24 27 25 27 Gender Female 4 3 3 1 25-29 1 2 3 0 30-34 3 2 3 2 35-39 3 8 4 3 40-44 5 2 7 5 45-49 8 4 3 7 Age 50+ 8 12 8 11 1-5 1 4 5 0 6-10 3 6 3 6 11-15 6 4 5 8 16-20 12 7 9 8 Years of Experience 20+ 6 9 6 6 Lab 0 0 1 2 Inside 7 11 9 9 Outside 13 11 10 12 Job Type Both 8 8 8 5 Table 4.2: Demographic Variables by Interface Experiment Experiment Traditional Color for Alarm Only Mental Model Layout Feedback Total Yes 2 0 2 2 6 Color Blind No 28 30 28 28 114 Total Tested 30 30 30 30 120 % Color Blind 6.67% 0 6.67% 6.67% 5.00% Table 4.3: Colorblindness by Interface Experiment 68 Data Analysis ANOVA is used to analyze the effects of the explanatory variable under investigation on the response variable. The logic of ANOVA is fairly straightforward. It is used to determine the probability that differences exist in means across several groups. The ANOVA model assumes that: 1. Each probability distribution is normal. 2. Each probability distribution has the same variance. 3. The responses for each factor level are random selections for the corresponding probability distribution and are independent of the responses for any other factor level (Hair et al., 1995). In this MW experiment study, the explanatory variable is the interface design used. The response variables include the SAGAT objective questions for each level of SA and for the total SA achieved, the PSAQ subjective questions, and the success rate for preventing the pipe from rupturing in the experiment. For each ANOVA test performed, these assumptions will be analyzed and reported. Post hoc comparisons of the differences will be reported as either Tukey or Games-Howell. Tukey will be reported if the assumption of equal variance is met, and Games-Howell will be reported if the equal variance assumption is not met. Demographic Variables ANOVA analyses were performed using each of the measured results based on the demographic variables to determine if any differences existed. These tests indicate that no differences exist based on operating area, gender, age, years of experience, or job type at an alpha value of 0.05. In addition, these results indicate that the randomization 69 procedure used in the study worked. This helps to support that any differences seen in the experiments are due to the interface design used and not the individuals being tested. Table 4.4 shows the ANOVA results for the SAGAT measures for each level of SA and for the total SA achieved. Table 4.5 shows the ANOVA results for each PSAQ question. Table 4.6 shows the ANOVA results for the performance measure or the percent success in preventing the pipe from rupturing. 70 ANOVA of Demographics based on Situation Awareness (SA) Level: SAGAT p-value Equal Variance Assumption Met Residual Assumptions Met Operator Area 0.124 Yes Yes Gender 0.352 Yes Yes Age 0.606 Yes Yes Years of Experience 0.303 Yes Yes SA 1 during Normal Operating Conditions Job Type 0.713 Yes Yes Operator Area 0.227 Yes Yes Gender 0.482 Yes Yes Age 0.415 Yes Yes Years of Experience 0.403 Yes Yes SA 1 during Abnormal Operating Conditions Job Type 0.752 Yes Yes Operator Area 0.166 Yes Yes Gender 0.317 Yes Yes Age 0.543 Yes Yes Years of Experience 0.446 Yes Yes SA 1 Total Job Type 0.770 Yes Yes Operator Area 0.608 Yes Yes Gender 0.079 Yes Yes Age 0.962 Yes Yes Years of Experience 0.983 Yes Yes SA 2 Job Type 0.124 Yes Yes Operator Area 0.137 Yes Yes Gender 0.731 Yes Yes Age 0.137 Yes Yes Years of Experience 0.193 Yes Yes SA 3 Job Type 0.795 Yes Yes Operator Area 0.208 Yes Yes Gender 0.200 Yes Yes Age 0.560 Yes Yes Years of Experience 0.299 Yes Yes SA Total Job Type 0.442 Yes Yes Legend: * The mean difference is significant at the .05 level. Table 4.4: ANOVA for Demographic Variables based on SAGAT 71 ANOVA of Demographics based on Perception: PSAQ p-value Equal Variance Assumption Met Residual Assumptions Met Operator Area 0.605 Yes Yes Gender 0.132 Yes Yes Age 0.374 Yes Yes Years of Experience 0.770 Yes Yes How hard did you find the interface to use? Job Type 0.559 Yes Yes Operator Area 0.104 Yes Yes Gender 0.121 Yes Yes Age 0.280 Yes Yes Years of Experience 0.114 Yes Yes How well would you rate your performance on the experiment? Job Type 0.811 Yes Yes Operator Area 0.663 Yes Yes Gender 0.406 Yes Yes Age 0.282 Yes Yes Years of Experience 0.464 Yes Yes How aware were you of the evolving situation during the experiment? Job Type 0.249 Yes Yes Legend: * The mean difference is significant at the .05 level. Table 4.5: ANOVA for Demographic Variables based on PSAQ ANOVA for Success Rate p-value Equal Variance Assumption Met Residual Assumptions Met Operator Area 0.301 Yes Yes Gender 0.493 Yes Yes Age 0.078 Yes Yes Years of Experience 0.338 Yes Yes % of successfully preventing the pipe from rupturing Job Type 0.899 Yes Yes Legend: * The mean difference is significant at the .05 level. Table 4.6: ANOVA for Demographic Variables based on Performance Measure 72 Experiments ANOVA analyses were performed using each of the measures dependent on the interface used to determine if any differences existed. These tests indicate that some differences do exist based on the interface used, Traditional, Color of Alarms Only, Mental Model Layout, or Feedback. Table 4.7 shows the ANOVA results for the SAGAT measures. Table 4.14 shows the ANOVA results for each PSAQ question. Table 4.18 shows the ANOVA results for the performance measure. Power and sample size analyses were conducted for each ANOVA test, and each test resulted in a power level of 0.9 or higher. SAGAT For each level of SA, the percent of correct questions answered was calculated. This is the percent of SA achieved for each level. ANOVA analyses were performed for each level of SA, SA1 at Normal Process Operating Conditions, SA1 at Abnormal Process Operating Conditions, SA1 at Both Conditions, SA2, SA3, and SA Total. The analysis for SA1 at Normal Process Operating Conditions showed that significant differences exist. This analysis measures the participant?s ability to perceive the process data when the process is operating at its desired operating levels. The differences showed that the interfaces that use Color for Alarms Only produced a higher level of SA level 1. The comparisons are shown is Table 4.8. The analysis for SA1 at Abnormal Process Operating Conditions showed that significant differences exist. This analysis measures the participant?s ability to perceive the process data when the process is operating at abnormal conditions. Again, the 73 differences showed that the interfaces that use Color for Alarms Only produced a higher level of SA level 1. The comparisons are shown is Table 4.9. The analysis for SA1 at Both Normal and Abnormal Process Operating Conditions showed that significant differences exist. This analysis measures the participant?s ability to perceive the process data when the process is operating at normal and abnormal conditions. As previously seen, the differences showed that the interfaces that use Color for Alarms Only produced a higher level of SA level 1 given both normal and abnormal process operating conditions. The comparisons are shown is Table 4.10. The analysis for SA2 showed that significant differences exist. This analysis measures the participant?s ability to comprehend the process. The differences showed that the interfaces that were designed with a Mental Model Layout instead of a physical layout yielded a higher percentage of SA level 2. The comparisons are shown is Table 4.11. The analysis for SA3 showed that significant differences exist. This analysis measures the participant?s ability to project the future state of the process. The differences showed that the interface designed with Feedback resulted in a higher percentage of SA level 3. The comparisons are shown is Table 4.12. In addition, an ANOVA analysis was preformed to determine if the interface design impacted the total amount of SA achieved by the participants. The analysis for the total amount of SA achieved showed that significant differences exist base on the interface used. The differences showed that each proposed design improved the total level of SA achieved by the participants. The comparisons are shown is Table 4.13. 74 Tr ad itio na l Co lor fo r A lar ms O nly M en tal M od el La yo ut Fe ed ba ck SA GA T AN OV A p-v alu e Rs q(a d) M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n Sit ua tio n A wa ren ess Le ve l 1 at N or ma l Pr oc ess O pe ra tin g Co nd itio ns 0.0 00 *,* ** 28 .92 % 66 .95 % 19 .54 84 .82 % 15 .07 92 .50 % 8.8 0 94 .68 % 20 .36 Sit ua tio n A wa ren ess Le ve l 1 at Ab no rm al Pr oc ess Op era tin g Co nd itio ns 0.0 00 *,* *,* ** 20 .21 % 65 .18 % 20 .79 84 .82 % 14 .17 88 .33 % 18 .26 89 .73 % 20 .71 Sit ua tio n A wa ren ess Le ve l 1 at bo th No rm al an d Ab no rm al Pr oc ess Op era tin g Co nd itio ns 0.0 00 *,* *,* ** 37 .76 % 66 .06 % 9.6 7 84 .82 % 13 .72 90 .42 % 14 .85 92 .21 % 13 .09 Sit ua tio n A wa ren ess Le ve l 2 0.0 01 *,* ** 11 .70 % 82 .23 % 13 .57 85 .21 % 18 .31 95 .60 % 7.4 2 90 .57 % 10 .46 Sit ua tio n A wa ren ess Le ve l 3 0.0 00 *,* *,* ** 22 .20 % 61 .34 % 19 .88 65 .50 % 20 .32 73 .38 % 20 .42 88 .77 % 12 .81 Sit ua tio n A wa ren ess To tal 0.0 00 *,* *,* ** 38 .66 % 68 .57 % 8.2 6 78 .75 % 13 .72 87 .00 % 10 .39 90 .89 % 9.0 3 Le ge nd : * Th e m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el ** eq ua l v ar ian ce as su mp tio n m et at ???? = 0. 05 *** re sid ua l a ss um pt ion s m et (n or ma lit y, co ns tan t v ar ian ce , & in de pe nd en ce ) Ta ble 4. 7: AN OV A Re su lts for E ac h L ev el of SA by E ac h In ter fac e E xp eri me nt 74 75 95 % Co nf ide nc e I nt er va l Int er fac e De sig n (1 ) Int er fac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -17 .88 %* -32 .01 % -3. 74 % Me nta l M od el La yo ut -25 .55 %* -38 .13 % -12 .98 % Tr ad itio na l Fe ed ba ck -27 .73 %* -39 .01 % -16 .46 % Tr ad itio na l 17 .88 %* 3.7 4% 32 .01 % Me nta l M od el La yo ut -7. 68 % -19 .91 % 4.5 5% Co lor fo r Ala rm O nly Fe ed ba ck -9. 86 % -20 .75 % 1.0 3% Tr ad itio na l 25 .55 %* 12 .98 % 38 .13 % Co lor fo r A lar m On ly 7.6 8% -4. 55 % 19 .91 % Me nta l Mo de l La yo ut Fe ed ba ck -2. 18 % -10 .74 % 6.3 8% Tr ad itio na l -17 .88 %* 16 .46 % 39 .01 % Co lor fo r A lar m On ly -25 .55 % -1. 03 % 20 .75 % Ga me s-H ow ell Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut -27 .73 % -6. 38 % 10 .74 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 8: M ult ipl e C om pa riso ns SA GA T: SA 1 a t N or ma l P ro cess Op era tin g C on dit ion s 75 76 95 % Co nf ide nc e I nt er va l Int er fac e De sig n (1 ) Int er fac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -19 .64 %* -32 .66 % -6. 63 % Me nta l M od el La yo ut -23 .15 %* -35 .95 % -10 .36 % Tr ad itio na l Fe ed ba ck -24 .55 %* -37 .57 % -11 .54 % Tr ad itio na l 19 .64 %* 6.6 3% 32 .66 % Me nta l M od el La yo ut -3. 51 % -16 .31 % 9.2 9% Co lor fo r Ala rm O nly Fe ed ba ck -4. 91 % -17 .93 % 8.1 1% Tr ad itio na l 23 .15 %* 10 .36 % 35 .95 % Co lor fo r A lar m On ly 3.5 1% -9. 29 % 16 .31 % Me nta l Mo de l La yo ut Fe ed ba ck -1. 40 % -14 .20 % 11 .10 % Tr ad itio na l 24 .55 %* 11 .54 % 37 .57 % Co lor fo r A lar m On ly 4.9 1% -8. 11 % 17 .93 % Tu ke y Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut 1.4 0% -11 .40 % 14 .20 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 9: M ult ipl e C om pa riso ns SA GA T: SA 1 a t A bn or ma l P ro cess Op era tin g C on dit ion s 76 77 95 % Co nf ide nc e I nt er va l Int er fac e De sig n (1 ) Int er fac e D es ign (2) Me an Di ffe re nc e (1 -2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -18 .76 %* -27 .83 % -9. 68 % Me nta l M od el La yo ut -24 .35 %* -33 .28 % -15 .43 % Tr ad itio na l Fe ed ba ck -26 .14 %* -35 .22 % -17 .07 % Tr ad itio na l 18 .76 %* 9.6 8% 27 .83 % Me nta l M od el La yo ut -5. 60 % -14 .52 % 3.3 3% Co lor fo r Ala rm O nly Fe ed ba ck -7. 38 % -16 .46 % 1.6 9% Tr ad itio na l 24 .35 %* 15 .43 % 33 .28 % Co lor fo r A lar m On ly 5.6 0% -3. 33 % 14 .52 % Me nta l M od el La yo ut Fe ed ba ck -1. 79 % -10 .71 % 7.1 3% Tr ad itio na l 26 .14 %* 17 .07 % 35 .22 % Co lor fo r A lar m On ly 7.3 8% -1. 69 % 16 .46 % Tu ke y Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut 1.7 9% -7. 13 % 10 .71 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 10 : M ult ipl e C om pa riso ns SA GA T: SA 1 a t B oth N or ma l a nd A bn or ma l C on dit ion s 77 78 95 % Co nf ide nc e I nt er va l Int er fac e De sig n (1 ) Int er fac e De sig n (2) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -2. 98 % -14 .43 % 8.4 7% Me nta l M od el La yo ut -13 .37 %* -21 .13 % -5. 60 % Tr ad itio na l Fe ed ba ck -8. 34 % -16 .94 % 0.2 6% Tr ad itio na l 2.9 8% -8. 47 % 14 .43 % Me nta l M od el La yo ut -10 .39 %* -20 .41 % -0. 37 % Co lor fo r Ala rm O nly Fe ed ba ck -5. 36 % -16 .01 % 5.2 9% Tr ad itio na l 13 .37 %* 5.6 0% 21 .13 % Co lor fo r A lar m On ly 10 .39 %* 0.3 7% 20 .41 % Me nta l Mo de l La yo ut Fe ed ba ck 5.0 3% -1. 35 % 11 .41 % Tr ad itio na l 8.3 4% -0. 26 % 16 .94 % Co lor fo r A lar m On ly 5.3 6% -5. 29 % 16 .01 % Ga me s-H ow ell Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut -5. 03 % -11 .41 % 1.3 5% Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 11 : M ult ipl e C om pa riso ns SA GA T: SA 2 78 79 95 % Co nf ide nc e I nt er va l Int er fac e De sig n (1 ) Int er fac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -4. 16 % -17 .18 % 8.8 6% Me nta l M od el La yo ut -12 .04 % -24 .84 % 0.7 5% Tr ad itio na l Fe ed ba ck -27 .43 %* -40 .45 % -14 .41 % Tr ad itio na l 4.1 6% -8. 86 % 17 .18 % Me nta l M od el La yo ut -7. 88 % -20 .68 % 4.9 1% Co lor fo r Ala rm O nly Fe ed ba ck -23 .27 %* -36 .28 % -10 .25 % Tr ad itio na l 12 .04 % -0. 75 % 24 .84 % Co lor fo r A lar m On ly 7.8 8% -4. 91 % 20 .68 % Me nta l Mo de l La yo ut Fe ed ba ck -15 .38 %* -28 .18 % -2. 59 % Tr ad itio na l 27 .43 %* 14 .41 % 40 .45 % Co lor fo r A lar m On ly 23 .27 %* 10 .25 % 36 .28 % Tu ke y Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut 15 .38 %* 2.5 9% 28 .18 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 12 : M ult ipl e C om pa ris on s S AG AT : S A3 79 80 Ta ble 4. 13 : M ult ipl e C om pa ris on s S AG AT : S A To tal 95 % Co nf ide nc e Int er va l Int er fac e De sig n (1 ) Int er fac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r A lar m On ly -10 .18 %* -17 .54 % -2. 82 % Me nta l M od el La yo ut -18 .43 %* -25 .66 % -11 .19 % Tr ad itio na l Fe ed ba ck -22 .32 %* -29 .68 % -14 .96 % Tr ad itio na l 10 .18 %* 2.8 2% 17 .54 % Me nta l M od el La yo ut -8. 25 %* -15 .49 % -1. 01 % Co lor fo r Ala rm O nly Fe ed ba ck -12 .14 %* -19 .50 % -4. 78 % Tr ad itio na l 18 .43 %* 11 .19 % 25 .66 % Co lor fo r A lar m On ly 8.2 5% * 1.0 1% 15 .49 % Me nta l Mo de l La yo ut Fe ed ba ck -3. 89 % -11 .13 % 3.3 4% Tr ad itio na l 22 .32 %* 14 .96 % 29 .68 % Co lor fo r A lar m On ly 12 .14 %* 4.7 8% 19 .50 % Tu ke y Co mp ar iso ns Fe ed ba ck Me nta l M od el La yo ut 3.8 9% -3. 34 % 11 .13 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. 80 81 PSAQ PSAQ consists of three questions that measure the participant?s perception of how hard they found the interface to use, how well they think they performed on the experiments, and how aware of the state of the process they felt the interface kept them. For each question, the self reported averages were calculated and ANOVAs were performed to determine if the interfaces resulted in significant differences. The question related to how hard the interface was to use did not result in any significant differences. In other words, the participants didn?t feel that the interface design impacted how hard it was to perform the experiment. The comparisons are displayed in Table 4.15. The question related to how well the participant felt they performed on the experiment resulted in slightly significant differences with a p-value of 0.053. The only difference seen was between the Feedback design versus the Traditional design. The participants who conducted the Feedback design interface rated themselves as performing better than the participants who used the Traditional designed interfaces. The comparisons are displayed in Table 4.16. The question relating to the level of perceived awareness by the participant based on the interface yielded significant differences. Again, the only difference in this group was between the Feedback design versus the Traditional design. The participants who conducted the Feedback design interface rated themselves as having a higher level of awareness than the participants who used the Traditional designed interfaces. The comparisons are displayed in Table 4.17. 82 Tr ad itio na l Co lor fo r A lar ms O nly M en tal M od el La yo ut Fe ed ba ck PS AQ AN OV A p-v alu e Rs q(a d) M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n Ho w ha rd di d y ou fin d t he in ter fac e t o us e? 0.9 18 ** ,** * 0.0 0% 1.5 0.7 9 1.5 4 0.7 9 1.5 0.9 7 1.3 9 0.5 7 Ho w we ll w ou ld yo u ra te yo ur pe rfo rm an ce on th e ex pe rim en t? 0.0 53 ** ,** * 4.1 7% 2.6 8* 0.9 1 2.8 6 1.1 5 3.0 7 0.8 3 3.3 9 1.1 0 Ho w aw ar e w ere yo u o f t he ev olv ing sit ua tio n d ur ing th e ex pe rim en t? 0.0 29 *,* *,* ** 5.3 1% 3.3 6* 0.7 8 3.4 3 0.9 2 3.6 3 0.8 1 4.0 0 0.9 4 Le ge nd : * Th e m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el ** eq ua l v ar ian ce as su mp tio n m et at ???? = 0. 05 *** re sid ua l a ss um pt ion s m et (n or ma lit y, co ns tan t v ar ian ce , & in de pe nd en ce ) Ta ble 4. 14 : A NO VA R esu lts for E ac h P SA Q Qu est ion by E ac h In ter fac e E xp eri me nt 82 83 95 % Co nf ide nc e Int erv al Int erf ac e De sig n (1 ) Int erf ac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r Ala rm O nly -0. 04 -0. 59 0.5 2 Me nta l M od el La yo ut 0.0 0 -0. 55 0.5 5 Tr ad itio na l Fe ed ba ck 0.1 1 -0. 45 0.6 6 Tr ad itio na l 0.0 4 -0. 52 0.5 9 Me nta l M od el La yo ut 0.0 4 -0. 51 0.5 8 Co lor fo r Ala rm O nly Fe ed ba ck 0.1 4 -0. 41 0.7 0 Tr ad itio na l 0.0 0 -0. 55 0.5 5 Co lor fo r Ala rm O nly -0. 04 -0. 58 0.5 1 Me nta l Mo de l La yo ut Fe ed ba ck 0.1 1 -0. 44 0.6 5 Tr ad itio na l -.0 11 -0. 66 0.4 5 Co lor fo r Ala rm O nly -.0 14 -0. 70 0.4 1 Tu ke y Co mp ari so ns Fe ed ba ck Me nta l M od el La yo ut -.0 11 -0. 65 0.4 4 Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 15 : M ult ipl e C om pa riso ns PS AQ : H ow ha rd di d y ou fin d t he in ter fac e t o u se? 83 84 95 % Co nf ide nc e Int erv al Int erf ac e De sig n (1 ) Int erf ac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r Ala rm O nly -0. 18 -0. 88 0.5 2 Me nta l M od el La yo ut -0. 39 -1. 07 0.3 0 Tr ad itio na l Fe ed ba ck -0. 71 * -1. 41 -0. 02 Tr ad itio na l 0.1 8 -0. 52 0.8 8 Me nta l M od el La yo ut -0. 21 -0. 90 0.4 8 Co lor fo r Ala rm O nly Fe ed ba ck -0. 54 -1. 23 0.1 6 Tr ad itio na l 0.3 9 -0. 30 1.0 7 Co lor fo r Ala rm O nly 0.2 1 -0. 48 0.9 0 Me nta l Mo de l La yo ut Fe ed ba ck -0. 33 -1. 01 0.3 6 Tr ad itio na l 0.7 1* 0.0 2 1.4 1 Co lor fo r Ala rm O nly 0.5 4 -0. 16 1.2 3 Tu ke y Co mp ari so ns Fe ed ba ck Me nta l M od el La yo ut 0.3 3 -0. 36 1.0 1 Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 16 : M ult ipl e C om pa riso ns PS AQ : Ho w we ll w ou ld yo u r ate yo ur pe rfo rm an ce on th e e xp eri me nt? 84 85 95 % Co nf ide nc e Int erv al Int erf ac e De sig n (1 ) Int erf ac e De sig n (2 ) Me an Di ffe re nc e (1- 2) Lo we r Bo un d Up pe r Bo un d Co lor fo r Ala rm O nly -0. 07 -0. 67 0.5 3 Me nta l M od el La yo ut -0. 28 -0. 87 0.3 2 Tr ad itio na l Fe ed ba ck -0. 64 * -1. 25 -0. 04 Tr ad itio na l 0.0 7 -0. 53 0.6 7 Me nta l M od el La yo ut -0. 20 -0. 80 0.3 9 Co lor fo r Ala rm O nly Fe ed ba ck -0. 57 -1. 17 0.0 3 Tr ad itio na l 0.2 8 -0. 32 0.8 7 Co lor fo r Ala rm O nly 0.2 0 -0. 39 0.8 0 Me nta l Mo de l La yo ut Fe ed ba ck -0. 37 -0. 96 0.2 3 Tr ad itio na l 0.6 4* 0.0 4 1.2 5 Co lor fo r Ala rm O nly 0.5 7 -0. 03 1.1 7 Tu ke y Co mp ari so ns Fe ed ba ck Me nta l M od el La yo ut 0.3 7 -0. 23 0.9 6 Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 17 : M ult ipl e C om pa riso ns PS AQ : H ow aw ar e w ere yo u o f t he ev olv ing si tua tio n d ur ing th e ex pe rim en t? 85 86 Performance Measure The performance measure is determined by whether or not the participant was able to prevent the pipe from rupturing. A success percentage is calculated for each interface design. The ANOVA results showed that significant differences exist based on the interface used. The success rate showed significant improvements when the Mental Model Layout designed interface and the Feedback designed interface were used. The comparisons are displayed in Table 4.19. 87 Tr ad itio na l Co lor fo r A lar ms O nly M en tal M od el La yo ut Fe ed ba ck Pe rfo rm an ce M ea su re AN OV A p-v alu e Rs q(a d) M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n M ea n Sta nd ar d De via tio n % Su cce ss ra te of pr ev en tin g t he pi pe fro m ru ptu rin g 0.0 00 *,* ** 24 .03 % 35 .71 %* 0.4 88 0 42 .86 % 0.5 04 0 76 .67 % 0.4 30 2 96 .43 % 0.1 89 0 Le ge nd : * Th e m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el ** eq ua l v ar ian ce as su mp tio n m et at ???? = 0. 05 *** re sid ua l a ss um pt ion s m et (n or ma lit y, co ns tan t v ar ian ce , & in de pe nd en ce ) Ta ble 4. 18 : A NO VA R esu lts for % Su cce ss ex pla ine d b y t he In ter fac e E xp eri me nt 87 88 95 % Co nf ide nc e Int erv al Int erf ac e De sig n (1 ) Int erf ac e De sig n (2 ) Me an Di ffe re nc e (1 -2) Lo we r Bo un d Up pe r Bo un d Co lor fo r Ala rm O nly -7. 14 % -42 .29 % 28 .00 % Me nta l M od el La yo ut -40 .95 %* -73 .06 % -8. 84 % Tr ad itio na l Fe ed ba ck -60 .71 %* -87 .39 % -34 .04 % Tr ad itio na l 7.1 4% -28 .00 % 42 .29 % Me nta l M od el La yo ut -33 .81 %* -66 .55 % -01 .07 % Co lor fo r Ala rm O nly Fe ed ba ck -53 .57 %* -81 .02 % -26 .12 % Tr ad itio na l 40 .95 %* 8.8 4% 73 .06 % Co lor fo r Ala rm O nly 33 .81 %* 1.0 7% 66 .55 % Me nta l Mo de l La yo ut Fe ed ba ck -19 .76 % -42 .88 % 3.3 6% Tr ad itio na l 60 .71 %* 34 .04 % 87 .39 % Co lor fo r Ala rm O nly 53 .57 %* 26 .12 % 81 .02 % Ga me s-H ow ell Co mp ari so ns Fe ed ba ck Me nta l M od el La yo ut 19 .76 % -3. 36 % 42 .88 % Le ge nd : * T he m ea n d iff er en ce is si gn ific an t a t t he .0 5 l ev el. Ta ble 4. 19 : M ult ipl e C om pa riso ns Pe rfo rm an ce M ea su re: Su cce ss Ra te of Pr ev en tin g t he Pi pe R up tur e 88 89 Hypothesis Testing The results of the research model will be presented in sections by each hypothesis. Each section will include only the information that focuses on that particular hypothesis. A summary of the results from the study is located at the conclusion of this chapter. Hypothesis 1a Hypothesis 1a, which states that the operator?s ability to perceive information when the process is in an abnormal state will be significantly greater when operating with the interface design based on consistent visual cueing for abnormal events only versus cueing for all process events, is supported. A significant difference was shown to exist between the level of SA level 1 achieved when the interface with Color for Alarms Only was used versus the Traditional designed interface that uses color for process variables and alarms. The 95% confidence interval for the difference is (6.63%, 32.66%), and 20.21% of the change in SA level 1 achieved was as a result of the interface design used. Hypothesis 1b Hypothesis 1b, which states that the operator?s ability to perceive information when the process is in a normal state will be significantly greater when operating with the traditionally designed interface versus the interface designed to cue only on the abnormal process events, is not supported. The opposite was actually shown to exist. The interface designed with Color for Alarming Only resulted in a higher amount of SA level 1. The 95% confidence interval for the difference is (3.74%, 32.01%), and 28.91% of the change in SA level 1 achieved was a result of the interface design used. 90 Hypothesis 1c Hypothesis 1c, which states that The operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed based on consistent visual cueing for abnormal events only versus cueing for all process events, is partially supported. A significant difference in the performance measure was not seen at an alpha level of 0.05 when color was the only variable changed. However, a significant difference was seen in the total level of SA achieved. One can assume that given more time to operator the participant?s ability to achieve a higher amount of SA level 1 and a higher amount of SA total would result in better performance. The average difference in the success rate is 7.14%, and 24.03% of the change in performance was a result of the interface used. The 95% confidence interval for the difference in the total amount of SA achieved is (2.82%, 17.54%). Hypothesis 2a Hypothesis 2a, which states that the operator?s ability to comprehend process information will be significantly greater when operating with the mental model interface design versus the physical model design, is supported. A significant difference was shown to exist between the amount of SA level 2 achieved when the interface designed based on a Mental Model Layout versus a physical layout. The 95% confidence interval for the difference is (5.60%, 21.13%), and 11.70% of the change in SA level 2 achieved was a result of the interface used. Hypothesis 2b Hypothesis 2b, which states that the operator?s ability to perform necessary process actions will be significantly greater when operating with the mental model 91 interface design versus the physical model design, is supported. A significant difference is shown to exist in the participant?s performance between interfaces that used a Mental Model Layout versus a physical layout. The 95% confidence interval for the difference is (1.07%, 66.55%), and 24.03% of the change in performance was result of the interface design used. Hypothesis 3a Hypothesis 3a, which states that the operator?s ability to project the future state of the process will be significantly greater when operating with the interface designed with time based feedback versus the traditionally designed interface, is supported. A significant difference was seen in the amount of SA level 3 achieved when the interface was the Feedback design versus the interfaces with no feedback. The 95% confidence interval for the difference is (2.59%, 28.18%), and 22.20% of the change in the amount of SA level 3 obtained was a result of the interface used. Hypothesis 3b Hypothesis 3b, which states that the operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed with time-based feedback versus the traditionally designed interface, is partially supported. A significant difference was not seen when feedback was the only difference in the interface design at an alpha value of 0.05. However, when feedback was added in the interface design a higher level of self-reported awareness was seen, and a higher level of self-reported operator performance was reported. The 95% confidence interval for the difference for awareness is (0.04, 1.25), and the 95% confidence interval for the difference in self reported performance is (0.02, 1.41). 92 Summary of Hypothesis Testing In this chapter, the results of the underlying research model for this study were presented. The research model was analyzed using three different measures. Figure 4.2 displays the research model along with the hypotheses tested. First, the research model was analyzed by calculating the percent of SA obtained at each level and as a total. Secondly, the research model was analyzed based on self-reported levels of SA achieved. Thirdly, the research model was analyzed based on a performance measure. Lastly, post hoc comparisons were analyzed to determine where significant differences existed as a result of the interface used. Results of the analyses provided full or partial support for several of the hypotheses. A summary of the hypothesis testing results are provided in Table 4.20. These results and their implications will be discussed in the next chapter along with limitations and future research. 93 Legend: *The hypothesis is partially supported. ** The hypothesis is supported. Figure 4.2: Detail Research Model with Hypotheses Interface Design Using consistent visual cues in traditional vs. abnormal events Design to promote accurate mental model development Using consistent time-based process feedback Situation Awareness Projections of Future Status Level 3 Comprehension of Current Situation Level 2 Perception of Elements in Current Situation Level 1 H1a* * H1b H1c* H2a** H2b** H3a** H3b* 94 Hypothesis Conclusion H1a: The operator?s ability to perceive information when the process is in an abnormal state will be significantly greater when operating with the interface design based on consistent visual cueing for abnormal events only versus cueing for all process events. (SAGAT) Supported H1b: The operator?s ability to perceive information when the process is in a normal state will be significantly greater when operating with the traditionally designed interface versus the interface designed to cue only on the abnormal process events. (SAGAT) Not supported H1c: The operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed based on consistent visual cueing for abnormal events only versus cueing for all process events. (Pipe Rupture % success & SAGAT) Partially supported H2a: The operator?s ability to comprehend process information will be significantly greater when operating with the mental model interface design versus the physical model design. (SAGAT) Supported H2b: The operator?s ability to perform necessary process actions will be significantly greater when operating with the mental model interface design versus the physical model design. (Pipe Rupture % success) Supported H3a: The operator?s ability to project the future state of the process will be significantly greater when operating with the interface designed with time-based feedback versus no feedback. (SAGAT) Supported H3b: The operator?s ability to perform necessary process actions will be significantly greater when operating with the interface designed with time-based feedback versus no feedback. (Pipe Rupture % success & PSAQ) Partially supported Table 4.20: Summary of Hypotheses 95 CHAPTER 5: DISCUSSION AND IMPLICATIONS Situation awareness (SA) has become the focus of research that aims to understand operator performance in critical, dynamic environments (Parasuraman et al., 2000). Previous studies have identified the importance of human factors and cognition in interface design and have suggested that designing to support human capabilities could impact operator SA (Itoh & Inagaki, 2004). Given the need for operators to be able to achieve and maintain a high level of SA in the midst of a dynamic and uncertain operating environment, such as with automation systems, it is important to understand how human factors impact the operator?s ability to perceive, comprehend, and project the state of the process. Acquiring the knowledge and skills required to design human computer interfaces for automation systems should be driven and dictated by users. The primary objective of this study was to provide further insights into the relationships between human factors, human computer interaction (HCI) design, operator SA, and automation systems. Given that these human factor design guidelines have been vague and hard to operationalize, the second objective of this study was to develop specific design guidelines to improve each level of SA. This study, therefore, focused on three human factor design guidelines that were operationalized and tested ? color to improve perception, layout to improve comprehension, and feedback to improve projection. 96 Data for this study was collected from 120 participants from a large paper plant located in the Southeast United States. Four microworld (MW) interface experiments were developed and tested in the field by operations and maintenance employees. Three measures were used in this study - an objective SA measure, a subjective SA measure, and a performance measure. The paper mill employees were asked to volunteer to participant in this study to aid in the validly of conducting a MW experiment. The results of the data analysis and hypothesis testing were presented in Chapter 4. In this chapter, Chapter 5, the results are interpreted, the findings outlined in Chapter 4 are explained, and the implications of the findings are discussed. This chapter concludes with the study?s limitations and suggestions for future research. Findings The current study proposed that operators could achieve a higher level of SA and improve their operating performance in an automation system through the proper implementation and utilization of human factor guidelines in HCI design and development. The results of this study are interesting in that they provide a very detailed analysis of the nature of the relationships between the operator and specific interface designs. The following research questions were presented earlier in this study: 5. How does the interface design impact the effectiveness with which the operator can perceive the needed information? 6. How does the interface design impact the effectiveness with which the operator can comprehend the needed information? 7. How does the interface design impact the effectiveness with which the operator can project the future state of the process? 97 8. How does the interface design impact the effectiveness with which the operator performs the necessary process actions? The subsequent sections address the aforementioned research questions and discuss the findings related to each of these questions. The Operator?s Ability to Perceive and Interface Design The results of the present study indicate that using Color for Alarms Only positively impacts the operator?s ability to perceive the process data. Specifically, the analysis of the operator?s ability to perceive the process data indicated an increase during both normal and abnormal process operating conditions. An explanation for this result is that color being used for all process information causes the operators to become distracted and leads the operator to ignore colors as they appear on the interface. Operating from interfaces that use color only for alarms allows the operator to stay focused on the critical information needed to run the process. So when color is only being used to cue abnormal events in the automation interface, the operator?s ability to achieve and maintain high levels of SA level 1 increases. According to Jones and Endsley (1996) the most frequent cause of errors occurred when the data was present but not attended to by the operator. Furthermore, considering that SA is a hierarchal construct the operator being able to perceive the process data is critical in the operator being able to understand and project the current and future states of the process. My finding, as it pertains to the relationship between the use of color and SA, supports what others have implied in prior studies ? that the over use of color negatively impacts the operator?s ability to develop awareness of the process (Labs, 2005; Reising, 2002). For instance, Reising (2002) suggested that reducing the use of color was likely to 98 improve the operator?s ability to perceive the current state of the process. These findings and implications were supported by the participant?s comments and behavior while performing the experiments. Two comments stood out because they were repeated many times by participants who operated the Traditional Designed experiment. 1. I don?t like colors being used for both the alarms and the process. 2. It?s hard to notice the alarms because I?m so use to not seeing them. It was very interesting to observe the difference in the participants? behavior as they were operating the Traditional Design versus the Color for Alarms Only Design. With the Traditional experiment when process variables went into alarm states, the operators either didn?t notice them; or they completely ignored the alarms as if they didn?t believe a problem was occurring in the process. When testing the interface with Color for Alarms Only and the process variables went into alarm states, the operators became nervous and several participants restarted the process more than once to prevent the pipe from rupturing. Conventional wisdom suggests that if color is removed from the process variables then the operator might lose their ability to achieve SA when the process is operating in a normal state. In this case, however, the findings tend to defy conventional wisdom. One major surprise was that the use of color for only abnormal events actually increased the operator?s ability to perceive the process data during normal operations. Additionally, since the interface would be designed to use color for only alarms, the operator would not have to use their attention or working memory capacities to search for information that is critical at that point in the process. Therefore, reducing the use of color to only abnormal 99 process information could help reduce the human?s working memory constraints caused by the current condition of information overload. The Operator?s Ability to Comprehend and Interface Design The results of the present study indicate that, using a mental model layout increases the operator?s ability to comprehend the automation process and its relationships. This statement implies that designing automation interfaces based on accurate mental model development will help the operator in achieving an enhanced understanding of the process and will support the operator in developing a correct operating process model. Specifically, the analysis of the operator?s ability to comprehend the process information increases as compared to the traditional physical layouts, such as piping and instrumentation designs. An explanation for this result is that human?s naturally create mental models to perform actions. Therefore if the interface is designed to promote an accurate mental model of the process, the operator will not only be able to comprehend the process in a quicker fashion, but they will also be able to develop a more accurate mental model from which they control the process. In addition since SA is a hierarchal construct, the operator?s ability to comprehend the process data is critical to the operator?s ability to project the future state of the process. My finding, as it pertains to the relationship between the use of mental model layouts and SA, supports what others have implied in prior studies ? that the use of mental models could help the operator develop an understanding of the process (Norman, 1990). In addition, this study adds to the Shneiderman?s (1998) findings that using graphical displays showed a benefit over textual displays. The findings from this study indicate that graphical displays that promote an accurate mental model development will 100 led to a higher level of process understanding. These findings and implications were supported by the participant?s comments and performance. Participants who used the Mental Model Layout experiment gave the following comments. 1. Fairly simple, could be mastered quickly. 2. The experiment had a great layout. 3. Even though I?m not an inside operator, this was very doable. Finally, promoting the development of accurate mental models could also aid in reducing the load on the operator?s working memory. This study indicated that interfaces designed on this mental model concept were helpful in directing the operator?s attention and in increasing their ability to understand the relationships that exist in the automation system. The Operator?s Ability to Project and Interface Design The results of the present study indicate that, using time-based feedback increases the operator?s ability to project the future state of an automation process. This statement implies that designing automation interfaces with time-based feedback supports the operator?s ability to understand and respond in a proactive way. Specifically, the analysis of the operator?s ability to project the future state of the process increases as compared to the traditional design without feedback. An explanation for this result is that feedback is necessary for humans to learn and to be able to transfer working memory information into long-term memory stores. That is critical for the operator to not only be able to cope with abnormal events when they occur, but to also be able to recognize or to predict when abnormal situation are about to occur. This would allow the operator to proactively adjust or compensate to prevent a process failure from occurring. Therefore if the interface is designed with time-based feedback, the operator will be able to project the 101 future state of the process as well as being able to operate in an efficient and proactive manner. My finding, as it pertains to the relationship between the use of feedback and SA, supports what others have implied in prior studies ? that the use of feedback is essential for the operator to be able to predict the future (Norman, 1990; Endsley, 1988). Even though the concept of feedback has been around for almost 20 years, the least amount of research has been done in this area of SA. The findings from this study indicate the importance of time-based feedback for the operator to achieve and maintain a high level of SA. The operator must understand how much time is available until an event occurs or an action must be taken. These findings and implications were supported by the participant?s comments and performance. Participants who used the Feedback experiment gave the following comments. 1. Easy to use interface and easy to monitor. 2. Very understandable. 3. We could really use the arrows because when we pull up a screen we don?t know what has been happening. 4. Very helpful. 5. I really liked the arrows. 6. Designed very well for the brief time to be familiar with the process. Given the increasing amount of process automation data, it is critical to be able to structure it in a way that will allow the operator to know how to optimally allocate their attention. Time-base feedback is important so the operator will be able to understand how to accomplish his current goal and to make correct decisions. 102 Operator?s Performance and Interface Design Decision-making and performance measures are indirect measure of SA, since an operator could have perfect awareness but make a bad decision. On the other hand, an operator could make a correct decision while having a low level of SA. Previous research has suggested that there is a probabilistic link between SA, decision-making, and operator performance (Endsley, 2000). The results of the present study suggest that the link does exist. Improvements in operator performance were seen in the mental model layout experiment and in the time-based feedback experiment. I did not see an improvement in operator performance when there was only an improvement in operator perception. The findings from this study indicate that to improve operator performance the interface must use color for alarms but also needs to be designed based on a mental model layout. When time-based feedback was added to the interface design, operator performance increased by an even greater amount. The traditionally designed interface resulted in a failure rate of 64.29% and a mean total level of SA achieved of 68.57%. The time-based feedback experiment yielded a failure rate of 3.57% and a mean total level of SA achieved of 90.89%. These results were also supported with the self-reported performance measure from this study. Operators reported a belief that they performed better and were most aware when using the interface designed with time-based feedback. 103 Implications Research Implications Researchers investigating operator SA and HCI should contemplate evaluating their interface on each SA level along with a performance measure. The results of the present study indicate that in order to gain a full understanding of the impact the interface design has on the operator?s abilities, both direct and indirect measures need to be utilized. The results of the present study also suggest that there is a symbiotic relationship between the SA levels and performance. However, performance improvements were only seen once SA level 2 was increased. This shows that comprehension is critical for studies focusing on performance improvements. A general implication of the results is that reducing the operator?s attention and working memory load are important in all level of SA studies. Very few studies have actually looked at SA at each individual level. The findings of the present study indicate that demographic variables had no impact on the operator?s ability to ability to achieve or maintain SA or operator performance. Most importantly, novices and experts performed the same. It could quite possibly be that designing interfaces based on human factors could help to mask the differences that one would assume exists between novices and experts. Combining these two groups might provide researchers the opportunity to gain more insight into the SA construct at a higher level. Lastly, the effects of time-based feedback had a greater impact on the operator?s perception of how well they did and aware they were. This would imply that time-based feedback is a critical interface design issue. 104 Managerial Implications Organizations should not neglect the design of their interfaces but rather should devote just as much time and energy to the quality and detail as they devote to other corporate endeavors. This support includes the allocation of sufficient human and financial resources. Providing adequate resources is of great importance because the development of a human centered interface can be quite time consuming especially in a dynamic environment such as an automation system (Hall et al., 2001). Regardless of which level of SA is the focus, the interface design needs to reduce operator attention and working memory loads. By focusing the development of interfaces on these key design principles, problems caused by data overload and technological advance could be reduced. Additionally, the importance of colorblindness needs to be considered. Given that the population was 5% red-green colorblind and that red-green colorblindness is the most common, red and green should never be displayed next to each on the HCI. This design flaw could result in an operator missing an abnormal event that could cause a catastrophic failure. Furthermore, companies need to understand the importance of including the operator at the start of the development process. Especially since operator performance improvement was only seen once the screen layout promoted an accurate mental model. In order to develop interfaces with accurate mental models, operators should participate in an in-depth interview process with the designer, such as a cognitive process walk through. This is necessary so the designer will be able to understand how the operator truly uses the process data, operates the process, and where the data should be located on the interface. This could help prevent interfaces being designed that do not support an 105 accurate mental model for the operators. It is imperative that organizations understand and focus on time-based feedback, and it?s impact on operator performance. As articulated in previous research (Labs, 2005), a human centered design approach could lead to a reduction in human error that accounts for 80% of industrial accidents. Lastly, organizations need to understand all the benefits gained from designing interfaces that are human centered and that support operator SA. In the present study, no significant differences were found in the demographic variables. This is especially important when considering novice operators and the expert operators. This implicates that training time could also be reduced, and that as turn over occurs, the operator?s learning curve to be able to operator like an expert could be greatly reduced. 106 Design Guidelines SA level 1 ? Perception To improve perception, use color for alarms/abnormal events only. SA level 2 ? Comprehension To improve comprehension, design interface screens based on an accurate mental model of the process automation system. Operators should be included from the very beginning of the design process to ensure that an accurate operating model layout is being developed. SA level 3 ? Projection To improve projection, use time-based feedback in a way that the operator will be able to look at an item and quickly know if a problem is starting to occur. Table 5.1: Design Guidelines Limitations The current study has several limitations that should be considered when interpreting the results. The most significant limitation of the study is its relatively small sample size. Although the methodology chosen is able to accommodate smaller samples, larger samples would be more ideal. 107 Another limitation of the current study is that the entire sample came from a single mill. The mill chosen for this study was a large pulp and paper manufacturer located in the Southeast United States. Unfortunately, this restriction, could limit the generalizability of the study. However, since the study analyzed the results with several measures and at a very detailed level, the findings and implications could be applicable to other manufacturing facilities and to firms in other industries. Lastly, the present study used an experiment that had only one main operating screen with only seven process variables. A limitation to MW experiments is developing the proper balance between complexity and the research objectives. This type of operating design and process size is very simple compared to real life processes. For instance, some past benching that I have done in the power distribution showed operators were operating from 100 main interface displays that contained over 4,000 process variables. However, more complex MW experiments need to be developed. Future Research Suggestions Future research on SA and HCI should continue to examine factors that impact operator perception, comprehension, and projection. Future studies should also consider automation systems with a multiple screen design. I have shown using a mental model layout improves the operator?s ability to comprehend the process and will improve the operator?s performance. This study was designed with only one main operating screen. Future research is needed, with multiple screen processes to further conceptualizations the results of the present study, to get a clearer picture of the effect of a mental model layout and it?s impact on SA level 2 and operator performance. 108 It seems from the results of the present study that the operators did not perceive a difference based on interface design of how hard it was to perform the experiment. Since each operator performed only one experiment, they were not exposed to the other interfaces. It?s possible that the operators could have different perceptions of the interfaces if they operated with all four. This possibility certainly needs to be explored further. Different industries might also be analyzed in future studies. The present study focused on one manufacturing mill in the paper industry. Other researchers should study other types of firms in other sectors including healthcare, financial, and technology. The results of such studies could also be compared to the results of the present study to determine if, and where, similarities and differences exist. General theories could then be developed where appropriate and more specific or limited theories could be developed when differences are discovered. According to the SA construct, individual abilities impact the development of SA and performance. Therefore, psychometric measures such as computer self-efficacy and personality tests should be investigated. It would be interesting to see how these measures impact the operator?s ability to perceive, comprehend, project, and performance. Conclusion One of the purposes of this study was to demonstrate the relationship between human factor designs, SA, and automation systems. Another aim of this study was to develop specific interface design guidelines to support operator SA that could be operationalized easily. It has been shown in the present study that interface design does 109 impact the operator?s ability to perceive, comprehend, and project the process information in an automation system along with operator performance. Specifically, using color for alarms only positively improves the operator?s ability to perceive the current process data. Using a mental model layout design positively improves the operator?s ability to comprehend the process information. Using time-based feedback positively impacts the operator?s ability to project the future state of the process. In summary, the present study addressed the following research questions: 1. How does the interface design impact the effectiveness with which the operator can perceive the needed information? 2. How does the interface design impact the effectiveness with which the operator can comprehend the needed information? 3. How does the interface design impact the effectiveness with which the operator can project the future state of the process? 4. How does the interface design impact the effectiveness with which the operator performs the necessary process actions? The findings of the present study suggest that improvements in each level of SA must be addressed individually and as a whole. As the operator?s perception ability improves, his total level of SA achieved also increases. As the operator?s ability to comprehend increases, so does his ability to perform. Increasing the operator?s ability to project further increases the total level of SA achieved and operator performance. In addition, the findings indicate that the human factor design guidelines that are operationalized appropriately have a positive influence on operator awareness and performance. 110 Furthermore, the level of operator SA achieved can be related to decision-making and operator performance. The findings of the present study suggest that as the level of SA achieved increases operator performance increases which is critical to system success and firm performance. More specifically, the business case for interface development could directly influence the strategic impact of the process automation system. Thus, the success level achieved by the automation system is dependent on the HCI, and it?s human centered design. 111 BIBLIOGRPHY Adams, M.J., & Pew, R.W. (1990). Situational awareness in the commercial aircraft cockpit: A cognitive perspective. In Proceedings of the IEEE/JAIAA/NASA 9th digital avionics systems conference. Virginia Beach, VA, 519-524. Adams, M. J., Tenney, Y. J., & Pew, R. W. (1995). Situation awareness and the cognitive management of complex systems. Human Factors, 37, 85?104. Bainbridge, L. (1983). Ironies of automation. Automatica, 19, 775?779. Baskerville, R. L. and Myers, M. D. (2002). Information systems as a reference discipline. MIS Quarterly, 26(1), 1 ? 14. Baxter, G. D. and Bass, E. J. (1998). Human error revisited: Some lessons learned for situation awareness. In Proceedings of the Fourth Annual Symposium on Human Interaction with Complex Systems. Los Alamitos, CA. (pp. 81 . 87) IEEE Computer Society Press. Bell, H. & Lyon, D. (2000). Using observer ratings to assess situation awareness. In M. Endsley & D. Garland (Eds.), Situation Awareness Analysis and Measurement, (p. 129- 146). Mahwah, NJ: Erlbaum. Billings, C. E. (1997). Aviation Automation: The Search For A Human-Centered Approach. Hillsdale, N.J: Lawrence Erlbaum Associates. Billings, C. E. and Woods, D. D. (1994). Concerns about adaptive automation in aviation systems, in M. Mouloua and R. Parasuraman (eds), Human performance in automated systems: Current research and trends (Hillsdale, NJ: LEA), 24?29. Bohn, J., Coroama, V., Langheinrich, M., Mattern, F. and Rohs, M. (2004). "Social, Economic, and Ethical Implications of Ambient Intelligence and Ubiquitous Computing, Institute for Pervasive Computing". ETH Zurich, Switzerland. Bolstad, C. A., Riley, J. M., Jones, D. G., & Endsley, M. R. (2002). Using goal-directed task analysis with Army brigade officer teams. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 472?476). Santa Monica, CA: Human Factors and Ergonomics Society. 112 Brehmer, B. (1992). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81(3), 211?241. Brehmer, B., & D?rner, D. (1993). Experiments with computer simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9(2?3), 171?184 Burmester,M.(1997, April). Produktfamilie process control systemPCS7 V3.0?usability test (Rep.No. Siemens ZT IK 7; Siemens AUT E 111). Siemens internal project report, Munich, Germany. Burmester, M., & Beu, A. (1999, March). Results of focus groups 1 and 2?Projekt OVA (Rep. No. Siemens ZT IK 7; Siemens A&D AS V6). Siemens internal project report, Munich, Germany. Burmester, M., & Komischke, T. (1999). Nutzeranforderungen als Grundlage f?r die Entwicklung innovativer user interfaces in der Proze?f?hrung [User requirements as a basis for developing innovative user interfaces in industrial process control]. In U. Arend, Burmester, M., Komischke, T., & Wust, L. (2000). Innovative User Interfaces in Automation Engineering by Application of Usability Engineering Methods Shown by the Example of a Three Dimensional Plant Representation. International Journal of Human- Computer Interaction, 12(3&4), 359?373 Burns, C. M., & Hajdukiewicz, J. R. (2004). Ecological interface design. Boca Raton, FL: CRC Press. Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human-computer interaction. Mahwah, NJ: Erlbaum. Carey, J., Galletta, D., Kim, J., Te'eni, D., Wildemuth, B. and Zhang, P. (2004). The role of Human?Computer Interaction in the MIS curricula: a call to action. Communications of the Association for Information Systems, 12, in press. Carmino, A., Idee, E., Larchier Boulanger, J., & Morlat, G. (1988). Representational errors: Why some may be termed diabolical. In L. P. Goodstein, H. B. Anderson, & S. E. Olsen (Eds.), T asks, e rrors a nd m ental m odel s (pp. 240-250). London: Taylor and Francis. Carroll, J. M. (1997). Scenario-based design. In M. G. Helander, T. K. Landauer,&P. V. Prabhu (Eds.), Handbook of human?computer interaction (2nd ed., pp. 383?406). Amsterdam: Elsevier. Casson, R. W. (1983). Schema in cognitive anthropology. Annual Review of Anthropology, 12, 429-462. 113 Chapanis, A. (1985). Some reflections on progress. Proceedings of the Human Factors Society 29th Annual Meeting. Santa Monica, CA: Human Factors Society, pp 1-8. Charwat, H. J. (1992). Lexikon der Mensch-Maschine-Kommunikation [Lexicon of human?computer communication]. Munich, Germany: Oldenbourg Verlag. Christie, B., Scane, R., & Collyer, J. (1995). Evaluation of human?computer interaction at the user interface to advanced IT systems. In J. R. Wilson & E. N. Corlett (Eds.), Evaluation of human work (2nd ed., pp. 310?356). London: Taylor & Francis. Crandall, B., Klein, G., & Hoffman, R. R. (2006). Working minds: A practitioner?s guide to cognitive task analysis. Cambridge, MA: MIT Press. Davis, G.B. (2002). "Anytime/anyplace computing and the future of knowledge work". Communications of the ACM, Vol.12, No. 45, 67-73. de Kleer, J., & Brown, J. S. (1983). Assumptions and ambiguities in mechanistic mental models. In D. Gentner & A. L. Stevens (Eds.), Mental models (pp. 155?190). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. DiFonzo, N., Hantula, D. A., & Bordia, P. (1998). Microworlds for experimental research: Having your (control and collection) cake, and realism too. Behavior Research Methods, Instruments, & Computers, 30, 278-286. Dix, A., Finlay, J., Abowd, G. and Beale, R. 1993, 1998, 2004, Human?Computer Interaction (Harlow, UK: Prentice Hall). Dixon, S. R., & Wickens, C. D. (2003a). Control of Multiple-UAVs: A Workload Analysis. Paper presented at the 12th International Symposium on Aviation Psychology, Dayton, OH. Dixon, S. R., & Wickens, C. D. (2003b). Imperfect Automation in Unmanned Aerial Vehicle Flight Control (AFHD-03-17/MAAD-03-2). Savoy, IL: Institute of Aviation. DoD (2001). Network Centric Warfare: Department of Defense Report to Congress. Washington DC: Office of the Secretary of Defense. Dominguez, C. (1994). Can SA be defined? In Vidulich, M., Dominguez, C., Vogel, E., and McMillan, G. (Eds) Situation Awareness: Papers and Annotated Bibliograph(U) Volume AL/CF-TR-1994-0085. Armstrong Laboratory, Wright-Patterson Airforce Base, Ohio. 114 Dreyfus, S. E. (1981). Formal models vs. human situational understanding: Inherent limitations on the modeling of business expertise (No. ORC 81-3). Berkeley: Operations Research Center, University of California. Durso, F. T., & Gronlund, S. D. (1999). Situation awareness. In F. T. Durso, R. Nickerson, R. Schvaneveldt, S. Dumais, S. Lindsay, & M. Chi (Eds.), Handbook of applied cognition (pp. 284?314). New York: Wiley. Ehret, B. D., Gray, W. D., & Kirschenbaum, S. S. (2000). Contending with complexity: Developing and using a scaled world in applied cognitive research. Human Factors, 42(1), 8-23. Endsley, M. R. (1988). A construct and its measurement: The functioning and evaluation of pilot situation awareness (No. NOR DOC 88-30). Hawthorne, CA: Northrop Corporation. Endsley, M. R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65?84. Endsley, M. R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32?64. Endsley, M. R. (1996). Situation awareness measurement in test and evaluation. In T. G. O?Brien and S. G. Charlton (Eds.) (pp. 159-179). Handbook of Human Factors Testing and Evaluation. New Jersey: Lawrence Erlbaum Associates. Endsley, M. R. (1998). Automation and situation awareness. In Parasuraman, R. and Mouloua, M. (Eds) Automation and Human Performance: Theory and Applications. Lawrence Erlbaum and Associates: Mahwah, NJ. Endsley, M. R. (1999). Situation awareness in aviation systems. In Garland, D. J., Wise, J. A., and Hopkin, V. D. (Eds) Handbook of Aviation Human Factors. (pp. 257 . 276). London: Lawrence Erlbaum and Associates. Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical review. In Endsley, M. R. and Garland, D. J. (Eds). Situation Awareness analysis and Measurement. Mahwah, NJ : L. Erlbaum Associates. Endsley, E. W. (2004). ?Modular Finite State Machines for Logic Control: Theory, Verification and Applications to Reconfigurable Manufacturing Systems?. PhD thesis. University of Michigan. Endsley, M. R., Bolte, B., & Jones, D. G. (2003). Designing for situation awareness: An approach to human-centered design. London: Taylor & Francis. 115 Endsley, M. R., & Garland, D. G. (2000a). Pilot situation awareness training in general aviation. Proceedings of the 14th Triennial Congress of the International Ergonomics Association and the 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 2.357?2.360). Santa Monica, CA: Human Factors and Ergonomics Society. Endsley, M. R., & Garland, D. J. (Eds.). (2000b). Situation awareness analysis and measurement. Mahwah, NJ: Erlbaum. Endsley, M. R., & Kaber, D. B. (1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42, 462?492. Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37, 381?394. Endsley, M. R., & Robertson, M. M. (2000). Training for situation awareness in individuals and teams. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement. Mahwah, NJ: Erlbaum. Endsley, M. & Smith, R. P. (1996). Attention and decision making in tactical air combat. Human Factors, 38, 232-249. Flach, J. M. (1995). Situation awareness: Proceed with caution. Human Factors, 37(1), 149-157. Flach, J. M. (1996). Situation awareness: In search of meaning. Cseriac Gateway, VI(6), 1-4. Fracker, M. L. (1988). A theory of situation assessment: Implications for measuring situation awareness. In P r oceedings o f t he H u man F a ctors S o ciety 3 2nd A nnual Meeting (pp. 102-106). Santa Monica, Ca: Human Factors Society. French, H.T., Clarke, E., Pomeroy, D., Seymour, M., and Clark, C.R. Psychophysiological Measures of Situation Awareness (Summary Report). In Proceedings of Human Factors of Decision-making in Complex Systems Conference. 2003. Dunblane, UK. French, H. & Hutchinson, A., (2002). Measurement of Situation Awareness in a C4ISR Environment. 7th International Command and Control Research and Technology Symposium. Gaba, D.M., Howard, S.K. and Small, S.D., (1995). Situation awareness in anesthesiology. Human Factors, 37: pp. 20-31. 116 Gevins, A., Smith,M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R. and Rush, G. (1998). Monitoring working memory load during computer-based tasks with EEG pattern recognition methods, Human Factors, 40, 79?91. Gibson, J.J. (1950). The perception of the visual world. Boston: Houghton Mifflin. Gibson, J.J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin. Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin. Gill, Y, and Leake, D.B. (Eds.) (2002). IUI 02: proceedings of the 2002 International conference on intelligent user interfaces. San Francisco: ACM press Gray, P.H. ?A Problem-Solving Perspective on Knowledge Management Practices,? Decision Support Systems (31:1), 2001, pp. 87-102. Gray, W. D., & Fu, W.-T. (2001). Ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head: Implications of rational analysis for interface design. Proceedings of ACM CHI 2001: Conference on Human Factors in Computing Systems, New York: ACM. Gugerty, L.J., (1997). Situation awareness during driving: explicit and implicit knowledge in dynamic spatial memory. Journal of Experimental Psychology: Applied, 3, 42 ? 66. Driver situational awareness 177 HMSO, 1993, The Highway Code (London: HMSO). Gugerty, L., & Tirre, W. (1997). Situation awareness: a validation study and investigation of individual differences, Human Factors and Ergonomics Society 40th Annual Meeting (pp. 564-568). Santa Monica, CA. Gugerty, L. J., & Tirre, W. C. (2000). Individual differences in situation awareness. In M. R. Endsley & D. J. Garland (Eds.), Situational awareness analysis and measurement (pp. 249-276). NJ: Lawrence Erlbaum. Guerlain, S. A. (1995). Using the critiquing approach to cope with brittle expert systems. Paper presented at the Annual Meeting of the Human Factors and Ergonomics Society. Guerlain, S., & Bullemer, P. (1996). User-initiated notification: A concept for aiding the monitoring activities of process control operators. Paper presented at the Human Factors and Ergonomics Society 40th Annual Meeting, Santa Monica, CA. 117 Guille, M. and French, H.T. Toward a Methodology for Evaluating the Impact of Solider Combat System Technologies on Situation Awareness. in Land Warfare Conference 2003: Projected Forces, Protected Forces. 2003. Adelaide, South Australia: DSTO. Guille, M. and French, H.T., Toward A Methodology For Evaluating the Impact of Technologies on Infantry Situation Awareness. DSTO-TR-1628, 2004, DSTO: Edinburgh. Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C. Multivariate Data Analysis with Readings, 4th Edition, Prentice Hall, Englewood Cliffs, NJ, 1995. Hall AJ, Sivamohan MVK, Clark N, Taylor S, Bockett G. (2001). Why Research Partnerships Really Matter: Innovation Theory, Institutional Arrangements and Implications for the Developing New Technology for the Poor. World Development 29:5, pp. 783-797. Hancock, P. A., & Scallen, S. F. (1998). Allocating functions in human-machine systems. In R. Hoffman & M. Sherrick & J. Warm (Eds.), Viewing psychology as a whole: The integrative science of William N. Dember (pp. 509-539). Washington DC: American Psychological Association. Hart, S. G., & Sheridan, T. B. (1984). Pilot workload, performance, and aircraft control automation. In Human factors considerations in high performance aircraft (AGARD-CP- 371; pp. 18/11?18/12). Neuilly Sur Seine, France: NATO-AGARD. Hockey, G. R. J. (1986). Changes in operator efficiency as a function of environmental stress, fatigue, and circadian rhythms. In Boff, K., Kaufman, L., and Thomas, J. (Eds) Handbook of Perception and Psychophysics (2, pp. 44/1-44/49). Wiley: New York. Hoffman, R. R., & Lintern, G. (2006). Eliciting and representing the knowledge of experts. In K. A. Ericsson, N. Charness, P. Feltovich, & R. Hoffman, (Eds.). Cambridge handbook of expertise and expert performance (pp. 203?222). New York: Cambridge University Press Hollnagel, E., & Woods, D. D. (2005). Joint cognitive systems: Foundations of cognitive systems engineering. Boca Raton, FL: CRC Press. Hooey, B.L., & Foyle, D.C. (2001). A post-hoc analysis of navigation errors during surface operations: Identification of contributing factors and mitigating strategies. Proceedings of the 11th Symposium on Aviation Psychology. Columbus, OH: Ohio State University. Hooey, B. L., Foyle, D.C., Andre, A.D., & Parke, B. (2000). Integrating datalink and cockpit display technologies into current and future taxi operations. Proceedings of the AIAA/IEEE/SAE 19th Digital Avionics System Conference. 118 Hopkin, V. D. (1995). Human Factors in Air Traffic Control. London and Bristol, PA, Taylor & Francis. Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. G. Helander, T. K. Landauer, & P. V. Prabhu (Eds.), Handbook of human?computer interaction (2nd ed., pp. 367?381). Amsterdam: Elsevier. Howitt, S. L., & Richards, D. (2003). The Human Machine Interface for Airborne Control of UAVs. Paper presented at the 2nd AIAA Unmanned Systems, Technologies, and Operations, San Diego, CA. http://waynesword.palomar.edu/colorbl1.htm. Retained July 2007. Itoh, M., & Inagaki, T. (2004). A Microworld Approach to Identifying Issues of Human- Automation Systems Design for Supporting Operator?s Situation Awareness. International Journal of Human-Computer Interaction, 17(1), 3-24. Jones, D. G. (2000). Subjective measures of situation awareness. In M. R. Endsley & D. J. Garland (Eds.), Situation awareness analysis and measurement (pp. 113-128). NJ: Lawrence Erlbaum. Jones, D. G., & Endsley, M. R. (1996). Sources of situation awareness errors in aviation. Aviation, Space and Environmental Medicine, 67(6), 507-512. Jones et al., ?Measuring the readability of automatic speech-to-text transcripts?, Proc. Eurospeech, pp. 1585?1588, 2003. Kaber, D. B., & Endsley, M. R. (1997). Level of automation and adaptive automation effects on performance in a dynamic control task. Proceedings of the 13th Triennial Congress of the International Ergonomics Association (pp. 202?204). Helsinki: Finnish Institute of Occupational Health. Kaber, D. B., & Endsley, M. R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5(2), 113?153. Kaber, D. B., & Riley, J. (1999). Adaptive automation of a dynamic control task based on secondary task workload measurement. International Journal of Cognitive Ergonomics, 3(3), 169?187. Kaber, D. B., Riley, J. M., Lampton, D., & Endsley, M. R. (2005). Measuring situation awareness in a virtual urban environment for dismounted infantry training. Proceedings of the 11th International Conference on HCI. Mahwah, NJ: Erlbaum. 119 Kardos, M., (2004). Automation, Information Sharing and Shared Situation Awareness. Land Operations Division Systems Sciences Laboratory, DSTO-GD-0400. Kieras, D., & Meyer, D. E. (1997). An overview of the EPIC architecture for cognition and performance with application to human computer interaction. Human-Computer Interaction, 12,391?438. Kieras, D. E., & Wood, S. D. (2002). Modeling human error for experimentation, training and errortolerant design. Proceedings of the Interservice/Industry Training, Simulation & Education Conference. Orlando, FL: U.S. Army. Klein, G. A. (1989). Recognition-primed decisions. In W. B. Rouse (Ed.), Advances in man-machine systems research (Vol. 5, pp. 47?92). Greenwich, CT: JAI Press. Klein, G. A. (1993). A recognition-primed decision (RPD) model of rapid decision making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 138?147). Norwood, NJ: Ablex. Klein, G. (2000). Analysis of Situation Awareness from Critical Incident Reports. In M. Endsley & D. J. Garland (Eds.), Situation Awareness Analysis and Measurement (pp. 51-71). Mahwah, NJ: Lawrence Erlbaum Associates. Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire ground. Proceedings of the Human Factors Society 30th Annual Meeting (pp. 576?580). Santa Monica, CA: Human Factors and Ergonomics Society. Klein, G., Ross, K. G., Moon, B. M., Klein, D. E., & Hollnagel, E. (2003, May/June). Macrocognition. IEEE Intelligent Systems, 81?85. Klein, G. A., Zsambok, C. E., & Thordsen, M. L. (1993, April). Team decision training: Five myths and a model. Military Review, 36?42. Kirby, B., Seymour, R. S., and Flay, R. (2000). Battlespace visualisation. Paper presented at The Australian Battlespace Digitisation Symposium, DSTO Edinburgh, July 2000. Koopman, P., & Hoffman, R. R., (November/December 2003). Work-Arounds, Make- Work, and Kludges. IEEE: Intelligent Systems, 70?75. Kraiss, K. F. (1989). Human factors aspects of decision support systems. AGARD Conference Proceedings No. 453: Operational Decision Aids for Exploiting or Mitigating Electromagnetic Propagation Effects. Paper presented at the Electromagnetic Wave Propagation Symposium, San Diego, 120 Kuhn, W. (1993). Metaphors Create Theories for Users. In Spatial Information Theory. (Frank, A. U. & Campari, I., eds.), Lecture Notes in Computer Science, Vol. 716, Springer, pp. 366-376. Labs, W. (2005). User interfaces should empower operators. www.ControlGlobal.com. Landauer, C. "Conceptual Categories as Knowledge Struttin'es", pp. 44-49 in A.M. Meystel (ed.), Proceed- ings of ISAS'97: The 97. Laudon, K.C. and Laudon, J.P. (2004). Management Information Systems: Managing the Digital Firm, eighth edition (Prentice Hall: Upper Saddle River) Layton, C., Smith, P. J., & McCoy, E. (1994). Design of a cooperative problem-solving system for en-route flight planning: An empirical evaluation. Human Factors, 36, 94-119 Lee, J., & Moray, N. (1992). Trust, control strategies and allocation of function in human machine systems. Ergonomics, 35(10), 1243?1270. Levison, W., Elkind, J., & Ward, J. (1971). Studies of multivariable manual control systems: A model for task interference. NASA Contract report CR 1746. Washington, DC: NASA. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. Office of the Secretary of Defense. (2002). Unmanned Aerial Vehicles Roadmap 2002- 2027. Washington DC: Department of Defense. Mandel, T. (1997). The elements of user interface design. New York: Wiley. Marchionni, G. (1992). Interfaces for end-user information seeking. Journal of the American Society for Information Science, 43(2), 156-163. Marchionini, G. (1995). Information seeking in electronic environments. NY: Cambridge U. Press. McGraw, K. L., & Harbison, K. (1997). User-centered requirements: The scenario-based engineering process. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Meyer, D. E., & Kieras, D. E. (1997). A computational theory of executive control processes and human multiple task performance: Part 1: Basic mechanisms. Psychological Review, 104, 3?65. Mills, V. (2000). The impact of the Battlefield Command Support System on team performance in brigade and regimental headquarters during Exercise Predators Gallop 2000. DSTO Technical Report [DSTO-CR-0164]. 121 Mogford, R. H. (1997). Mental Models and SA in Air Traffic Control. The Internaltional Journal of Aviation Psychology, 6(4), 331-341. Molich, R. (1994). Preventing user interface disasters. Behaviour and Information Technology 13(1?2), 154?159. Montgomery, G. (2007). Breaking the Code of Color. Retreived from http://www.hhmi.org/senses/b130.html. Moray, N. (Ed.). (1979). Mental workload: Its theory and measurement. New York: Plenum. Moray, N., (2004). Ou? sont les neiges d0antan? In Human Performance, Situation Awareness and Automation; Current Research and Trends, D.A. Vincenzi, M. Mouloua and P.A. Hancock (Eds.), p. 4 (Mahwah, NJ: LEA). Moray, N., Inagaki, T., & Itoh, M. (2000). Adaptive Automation, Trust, and Self- Confidence in Fault Management of Time-Critical Tasks. Journal of Experimental Psychology: Applied, 6(1), 44-58. Mosier, K. L. and Chidester, T. R. (1991). Situation assessment and situational awareness in a team setting. In Proceedings of the 11th Congress of the International Ergonomics Association. International Ergonomics Association: Paris. Mosier, K. L., & Skitka, L. J. (1996). Human Decision Makers and Automated Decision Aids: Made for Each Other? In R. Parasuraman & M. Mouloua (Eds.), Automation and Human Performance: Theory and Applications (pp. 201-220). Mosier, K. L. and Skitka, L. J. (1998). Human decision makers and automated decision tools: Made for each other? In Parasuraman, R. and Mouloua, M (Eds) Automation and Human Performance: Theory and Applications. Lawrence Erlbaum and Associates: Mahwah, NJ. Mourant, R.R. and Rockwell, T.H., (1972). Strategies of visual search by novice and experienced drivers. Human Factors, 14, 325 ? 335. Muniz, E. J., Stout, R. J., Bowers, C. J., and Salas, E. (1998). A methodology for measuring team situational awareness: Situational Awareness Linked Indicators Adapted to Novel Tasks (SAIANT). In RTO HFM Symposium on Collaborative Crew Performance in Complex Operational Systems. Edinburgh, UK. 20 . 22 April, 1998. Narayanan, S., S. Ananthakrishnan, R. Belvin, E. Ettelaie, S. Gandhe, S. Ganjavi, P. G. Georgiou, C. M. Hein, S. Kadambe, K. Knight, D. Marcu, H. E. Neely, N. Srinivasamurthy, D. Traum, and D. Wang. (2004). ?The Transonics spoken dialogue translator: an aid for English-Persian doctor-patient interviews?. In Timothy Bickmore 122 (ed.) Dialogue Systems for Health Communication: Papers from the 2004 Fall Symposium, American Association for Artificial Intelligence, Menlo Park, California, pp. 97?103. Neale, D. C., & Carroll, J. M. (1997). The Role of Metaphors in User Interface Design. In T. K. Landauer, & P. Prabhu (Eds.), Handbook of Human-Computer Interaction, Second Edition (pp. 441-462). Amsterdam: Elsevier Science B. V Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press. Nielsen, J. (1995). Getting usability used. In K. Nordby, P. Helmersen, D. Gilmore, & S. A. Arnesen (Eds.), Human computer interaction?Interact ?95 (pp. 3?13). London: Chapman & Hall. Norman, D. A. (1981). Steps towards a cognitive engineering (Tech. Report). San Diego: University of California, Program in Cognitive Science. Norman, D. A. (1986). Cognitive engineering. In D. A. Norman & S. W. Draper (Eds.), Usercentered system design (pp. 31?61). Mahwah, NJ: Erlbaum. Norman, D.A., (1990). The ?problem? with automation: inappropriate feedback and interaction, not ?overautomation?. Philosophical Transactions of the Royal Society of London, B 327, 585 ? 593. Norman, S. D., & Orlady, H. W. 1989 (Eds.), Flight deck automation: Promises and realities. Moffett Field, California: National Aviation and Space Administration. Olson, G. M. and Olson, J. S. (2003). Human?Computer Interaction: psychological aspects of the human use of computing. Annual Review of Psychology, 54(1), 491 ? 516. Omodei M. M., McLennann J. and Wearing A. J., (2000). Microworlds and Field Investigation in the study of Decision Making: Implications for Decision Making in Natural Settings. Presented at the DSTO Human Factors SIG, March. Omodei, M. M., & Wearing, A. J. (1995). The Fire Chief microworld generating program: An illustration of computer-simulated microworlds as an experimental paradigm for studying complex decisionmaking behavior. Behavior Research methods, Instruments, & Computers, 27, 303?316. Parasuraman, R. (1990). Event related brain potentials and human factors research, in J. Rohrbaugh, R. Parasuraman and R. Johnson (eds), Event-related brain potentials: basic issues and applications (New York: Oxford University Press), 297?300. 123 Parasuraman, R. (1993). Effects of adaptive function allocation on human performance. In D. J. Garland & J. A. Wise (Eds.), Human factors and advanced aviation technologies (pp. 147?158). Daytona Beach, FL: Embry-Riddle Aeronautical University Press. Parasuraman, R. (2003). Neuroergonomics: brain and behavior at work, Theoretical Issues in Ergonomics Sciences, 4, 5?20. Parasuraman, R., Bahri, T., Deaton, J.,Morrison, J. and Barnes,M. (1990). Theory and design of adaptive automation in aviation systems, Technical Report No. CSL-N90-1, Cognitive Science Laboratory, Catholic University of America, Washington, DC. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse and abuse. Human Factors, 39, 230?253. Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation, IEEE Transactions on Systems, Man, and Cybernetics?Part A: Systems and Humans, 30, 286?297. Pew, R. W. (2000). The state of situation awareness measurement: Heading toward the next century. In Endsley, M. R. and Garland, D. J. (Eds) Situation awareness analysis and measurement. Mahwah, NJ : L. Erlbaum Associates. Pirolli, P. (1998). Exploring browser design trade-o!s using a dynamical model of optimal information foraging. Proceedings of the SIG-CHI on Human Factors in Computing Systems 1998, 33}40. New York, ACM. Pirolli, P. & Card, S. K. (1999). Information foraging. Psychological Review, 106, 643}675. Pirolli, P., Card, S. K. & Van der Wege, M. M. (2001). Visual information foraging in a focus #context visualization. In J. JACKO&A. SEARS, Eds. Proceedings of the SIG-CHI on Human Factors in Computing Systems 2001, 506}513. New York, ACM. Polson, P. G., Lewis, C., Rieman, J.,&Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man? Machine Preece, J., Rogers, Y. and Sharp, H. (2002). Interaction Design: Beyond Human ? Computer Interaction (New York: John Wiley & Sons). Prince, C., and Salas, E. (1997). Situation assessment for routine flight and decision- making. International Journal of Cognitive Ergonomics,1 (4), 315 . 324.. 124 Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J. and Pope, A. T. (1998). A closed loop system for examining psycho physiological measures for adaptive task allocation, International Journal of Aviation Psychology, 10, 393?410. Rabbitt, P. (1984). The Control of Attention in Visual Search. In R. Parasuraman & D. R. Davies (Ed.), Varieties of Attention (pp. 273-291). Orlando, FL: Academic Press. Raney, G. E. (1993). Monitoring changes in cognitive load during reading: an event- related brain potential and reaction time analysis, Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 51?69. Rasmussen, J. (1981). Models of mental strategies in process plant diagnosis. In J. Rasmussen & W. B. Rouse (Eds.), Human detection and diagnosis of system failures (pp. 241?258). New York: Plenum. Rasmussen, J., and L.P. Goodstein. (1988). Information technology and work. Pp. 175? 202 in Handbook of Human-Computer Interaction, M. Helander (ed.). New York: North- Holland. Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems engineering. New York: Wiley. Rasmussen, J., & Vicente, K. J. (1989). Coping with human errors through system design: Implications for ecological interface design. International Journal of Man- Machine Studies, 31, 517?534. Reising, J. (2002). Cognitive Engineering and Its Relationships to Future Aviation Systems. U.S. Air Force Research Laboratory. SOAR. Riley, V. (1994). A theory of operator reliance on automation. In M. Mouloua & R. Parasuraman (Eds.), Human performance in automated systems: Current research and trends (pp. 8?14). Mahwah, NJ: Erlbaum. Riley, J. M., Kaber, D. B., Hyatt, J., Sheik-Nainar, M., Reynolds, J., & Endsley, M. (2005). Measures for assessing situation awareness in virtual environment training of infantry squads (Final ReportNo. SATech-05-03). Marietta, GA: SA Technologies. Roth, E.M., K.B. Bennett, and D.D. Woods. (1987). Human interaction with an "intelligent" machine. International Journal of Man-Machine Studies 27:479?526. Ruff, H., Narayanan, S., & Draper, M. H. (2002). Human Interaction with Levels of Automation and Decision-Aid Fidelity in the Supervisory Control of Multiple Simulated Unmanned Air Vehicles. Presence, 11(4), 335-351. 125 Salvucci, D. D., & Lee, F. J. (2003). Simple cognitive modeling in a complex cognitive architecture. Proceedings of the Human Factors in Computing Systems: CHI 2003 (pp. 189?194). New York: ACM Press. Sanders, M.M. & McCormick, E.J. (1993). Human Factors in Engineering & Design 7th ed., McGraw-Hill, NY. Sanderson, P., Pipingas, A.,Danieli, F. and Silberstein, R. (2003). Process monitoring and configural display design: a neuroimaging study, Theoretical Issues in Ergonomics Science, 4, 151?174. Sandom, C. (2001). Situation Awareness. In J. Noyes, and M. Bransby (Eds). People in control: Human factors in control room design. Bodmin, UK: IEE. Santoro, T. P., & Amerson, T. L. (1998). Definition and Measurement of Situation Awareness in the Submarine Attack Center. Paper presented at the Command and Control Research and Technology Symposium. Sarter, N. B., & Schroeder, B. (2001). Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human Factors, 43, 573-583. Sarter, M., Berntson, G. G. and Cacioppo, J. T. (1996). Brain imaging and cognitive neuroscience, American Psychologist, 51, 13?21. Sarter, M., Givens, B. and Bruno, J. P. (2001). The cognitive neuroscience of sustained attention: where top-down meets bottom-up, Brain Research Reviews, 35, 146?160. Sarter, N. B. and Woods, D. D. (1991). Situation awareness?a critical but ill-defined phenomenon, International Journal of Aviation Psychology, 1, 45?57. Sarter, N. B., & Woods, D. D. (1995). ?How in the world did I ever get into that mode?: Mode error and awareness in supervisory control. Human Factors, 37, 5?19. Scerbo, M. W. (1996). Theoretical perspectives on adaptive automation. In R. Parasuraman & M. Mouloua (Eds.), Automation and human performance: Theory and application (pp. 37?63). Mahwah, NJ: Erlbaum. Schwartz, D. (1990). Training for Situational Awareness. Flight Safety International: Houston, Tx. Seymour, R. S., Grisogono, A. M., Unweisse, M., Johnson, W., Krieg, J. and Haub, J. (2000). An Overview of Situation Awareness Research in Land Operations Division. Paper presented at The Australian Battlespace Digitisation Symposium, DSTO Edinburgh, July 2000. 126 Sheridan, T.B. (1976). Toward a general model of supervisory control. Pp. 271?281 in Monitoring Behavior and Supervisory Control, T.B. Sheridan and G. Johannsen (eds.). New York: Plenum Press. Sheridan, T.B. (1992). Musings on telepresence and virtual presence. Presence, 1(1), 120- 125. Sheridan, T. B. (2002). Humans and Automation: System Design and Research Issues, John Wiley & Sons, Inc., Santa Monica, CA. Sheridan, T. B., & Verplank, W. (1978). Human and Computer Control of Undersea Teleoperators. Cambridge, MA: Man-Machine Systems Laboratory, Department of Mechanical Engineering, MIT. Shneiderman, B., Designing the User Interface: Strategies for Effective Human- ComputerInteraction, 3rd ed., Addison-Wesley, Reading, MA, 1998. Shneiderman, B. (1992). Designing the user interface: Strategies for effective human computer interaction (2nd ed.). Reading, MA: Addison-Wesley. Shneiderman, B. and Plaisant, C. (2004). Designing the User Interface ? Strategies for Effective Human ? Computer Interaction, 4th edn (Reading, MA: Addison-Wesley). Smallman, H. S., Schiller, E. and Mitchell, C. (1999). Designing a display for the area air defense commander: The role of 3-D perspective views and realistic track symbols in achieving rapid situation awareness. United States Navy Technical Report 1803. SSC San Diego, CA. Smith, S., and J. Mosier. (1988). Guidelines for Designing User Interface Software. ESD- TR-86-278. Washington, D.C.: U.S. Department of Defense. Sterman, J. D. (1989). Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Management Science, 35, 321-339. Strater, L. and Endsley, M. (2000) SAGAT: A Situation Awareness Tool Measurement for Commercial Airline Pilots. Proceedings of the Human Performance, Situation Awareness and Automation: User Centred Design for the New Millennium Conference. October 2000. Strater, L. D., Jones, D., & Endsley, M. R. (2003). Improving SA: Training challenges for infantry platoon leaders. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 2045?2049). Santa Monica, CA: Human Factors and Ergonomics Society. Taylor, R. M. (1990). Situation awareness rating technique (SART): The development of a tool for aircrew systems design. In Situational Awareness in Aerospace Operations (AGARD- CP- 478). Neilly Sur Seine, France. NATO-AGARD. 127 Thurman, D.A. (1995). Improving Operator Effectiveness in Monitoring Complex Systems: A Methodology for the Design of Interactive Monitoring and Control Interfaces. Master's thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta. Thurman, D.A., and C.M. Mitchell. (1994). A methodology for the design of interactive monitoring interfaces. Pp. 1738?1744 in Proceedings of the 1994 IEEE International Conference on Systems, Man, and Cybernetics, San Antonio, Texas, October 2?5, 1994. Piscataway, N.J.: IEEE. Thurman, D.A., and C.M. Mitchell. (1995). A design methodology for operator displays of highly automated supervisory control systems. In Proceedings of the 6th IFAC/IFIP/IFOR/SEA Symposium on Analysis, Design, and Evaluation of Man Machine Systems, Boston, Mass., June 27?29, 1995. Tarrytown, N.Y.: Pergamon. Turinsky, P. J., S. Baron, W.D. Burch, M.L. Corradini, G.E. Lucas, R.B. Matthews, and R.E. Uhrig. (1991). Western European Nuclear Power Generation Research and Development. FASAC Technical Assessment Report. McLean, Va.: Science Applications International Corporation. Uhrig, R.E., and R.J. Carter. (1993). Instrumentation, Control, and Safety Systems of Canadian Nuclear Facilities . JTEC/WTEC Monograph. Baltimore, Md.: World Technology Evaluation Center, Loyola College. July. USNRC (U.S. Nuclear Regulatory Commission). (1981). Guidelines for Control Room Design Reviews. NUREG-0700. Washington, D.C.: USNRC USNRC. (1984). USNRC Standard Review Plan. NUREG-0800, Rev. 1. Washington, D.C.: USNRC. USNRC. (1994). Human Factors Engineering Program Review Model. NUREG-0711. Washington, D.C.: USNRC. July. USNRC. (1995). Human-System Interface Design Review Guideline. NUREG-0700, Rev. 1. Washington, D.C.: USNRC. February. Vicente, K.J., K. Christofferson, and A. Pereklita. (1995). Supporting operator problem solving through ecological interface design. IEEE Transactions on Systems, Man, and Cybernetics 25(4):529?545. Varian, H. (1995). The information economy. Scientific American, 273(3), 200-202. Vicente, K.J., and J. Rasmussen. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man, and Cybernetics 22(4):589?606. 128 Wampler, J., Roth, E., Whitaker, R., Kendall, C., Stilson, M., Thomas-Meyers, G., and Scott, R. (2006). Using work-centered specifications to integrate cognitive requirements into software development. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 240?244). Santa Monica, CA: Human Factors and Ergonomics Society. White, J.D. (1994). Comparative Assessments of Nuclear Instrumentation and Controls in the United States, Canada, Japan, Western Europe, and the Former Soviet Union. JTEC/WTEC Annual Report and Program Summary 1993/94. Baltimore, Md.: World Technology Evaluation Center, Loyola College. Wickens, C.D. (1984). Engineering Psychology and Human Performance. Columbus, Ohio: Charles Merrill. Wickens, C. D. (1990). Applications of event-related potential research to problems in human factors, in J. Rohrbaugh, R. Parasuraman and R. Johnson (eds), Event-related brain potentials: basic issues and applications (New York: Oxford University Press), 301?309. Wickens, C. D. (1992). Engineering psychology and human performance (New York: Harper-Collins). Wickens, C. D. (1999). Automation in Air Traffic Control: The Human Performance Issues. In M. W. Scerbo & M. Mouloua (Eds.), Automation Technology and Human Performance: Current Research and Trends. Mahwah, NJ: Lawrence Erlbaum Associates. Wiener, E. L. (1988). Cockpit automation. In E. L. Wiener & D. C. Nagel (Eds.), Human factors in aviation (pp. 433?461). San Diego, CA: Academic. Wiener, E. L., & Curry, R. E. (1980). Flight deck automation: Promises and problems. Ergonomics,23, 995?1011. Wiener, E. (1989). Human Factors of Advanced Technology ("Glass Cockpit") Transport Aircraft. Tech. Rep. 117528. Moffett Field, Calif.: NASA Ames Research Center. Wilson, J. R. (1995).Aframework and a context for ergonomics methodology. In J. R. Wilson& E. N. Corlett (Eds.), Evaluation ofhumanwork (2nd ed., pp. 1?39). London: Taylor&Francis. Winograd, T. (1997). From Computing Machinery to Interaction Design. In: Denning, P. & Metcalfe, R. (Eds.): Beyond Calculation: The Next Fifty Years of Computing, pp. 149- 162. Springer-Verlag. 129 Woods, D.D. (1984). Visual momentum: A concept to improve the cognitive couple of person and computer. International Journal of Man-Machine Studies 21:229?244. Woods, D.D. (1992). Are guidelines on human-computer interaction a Faustian bargain? Computer Systems Technical Group Bulletin, August, p. 2. Woods, D.D. (1993). Cognitive systems in context. Pp. 2?9 in Cognitive Engineering in Aerospace Applications: Pilot Interaction with Cockpit Automation, N.B. Sarter and D.D. Woods (eds.). NASA Contractor Report 177617. Moffett Field, Calif.: NASA Ames Research Center. Woods, D.D., L.J. Johannesen, R.I. Cook, and N.B. Sarter. (1994). Behind Human Error: Cognitive Systems, Computers, and Hindsight . Wright-Patterson AFB, Ohio: Crew Systems Ergonomics Information Analysis Center. Woods, D. D., & Hollnagel, E. (2006). Joint Cognitive Systems: Patterns in Cognitive Systems. Boca Raton, FL: CRC Press. Woods, D. D., & Roth, E. M. (1988). Cognitive engineering: Human problem solving with tools. Human Factors, 30, 415?430. Zacharias, G., Miao, A., Illgen, C., Yara, J., & Siouris, G. (1996). SAMPLE: Situation awareness model for pilot-in-the-loop evaluation. Proceedings of the First Annual Conference on Situation Awareness in the Tactical Air Environment. Patuxent River: Naval Air Warfare Center. Zhang, P., Benbasat, I., Carey, J., Davis, F., Galletta, D. and Strong, D. (2002). Human? Computer Interaction research in the MIS discipline. Communications of the Association for Information Systems, 9(20), 334 ? 355. Zhang, P. and Dillon, A. (2003). HCI and MIS: shared concerns. International Journal of Human?Computer Studies, 59(4), 397 ? 402. Zinser, K., and R.L. Henneman. (1988). Development and evaluation of a model of human performance in a large-scale system. IEEE Transactions on Systems, Man, and Cybernetics 8(3):367?375. Zuboff, S. The Age of The Smart Machine: The Future of Work and Power. Basic Books. 1989. 130 APPENDICES 131 APPENDIX A: INSTRUMENTS 132 Age: 20-25 25-30 30-35 35-40 40-45 45-50 50+ Years of Experience: 1-3 3-5 5-7 7-10 10-15 15-20 20+ Job Type: Inside Outside Area:_________________________ Experiment:___________________ 133 First Set of SAGAT Questions: Asked at Normal Operating Conditions 1. How many flows are operating within their desired operating process targets? 1 2 3 4 2. Has the production tank reached its desired operating temperature? Yes No 3. How many pumps are running? 1 2 3 4 SA1 4. Are there any tanks in alarm states? Yes No 5. Is the process operating at its desired operating level? Yes No 6. Is the process close to pipe rupture? Yes No SA2 7. Is there a relationship between the temperatures leaving the two heaters? Yes No 8. Did you observe any possible process failures? Yes No 9. What is most likely going to be your immediate course of action? Do Nothing Restart the Process SA3 10. How likely is pipe rupture? 1 2 3 4 5 Not Likely Likely Very Likely Continue with the experiment 134 Second Set of SAGAT Questions: Asked at Abnormal Operating Conditions 1. How many valves are open? 1 2 3 4 2. Are there any temperature readings that are currently in an alarm state? Yes No 3. How many heaters are operating at their desired process state? 1 2 3 4 SA1 4. Are there any flow readings that are currently in an alarms state? Yes No 5. Is there a relationship between the flow rates to the production tank and the levels in the hot and cold tanks? Yes No 6. Is the process operating at its desired target levels? Yes No SA2 7. Is there a relationship between the flow rate into the heater and the temperature of the product leaving the heater? Yes No 8. Are there any process variables that are operating at a rate that could cause the process to be restarted? Yes No 9. Which process variable might first cause a pipe failure? a. Flow Rate to the First Heater b. Flow Rate to the Production Tank c. The Temperature Leaving Heater #2 d. The Level of the Hot Tank SA3 10. How likely is it that you will have to restart the process in the immediate future? 1 2 3 4 5 Not Likely Likely Very Likely Continue with the experiment 135 Third Set of Questions: PSAQ 1. How hard did you find the interface to use? 1 2 3 4 5 Not Hard Hard Very Hard 2. How well would you rate your performance on the experiment? 1 2 3 4 5 Not Well Well Very Well 3. How aware were you of the evolving situation during the experiment? 1 2 3 4 5 Not Aware Aware Very Aware The following CSE questions ask you to indicate whether you could use this unfamiliar software package under a variety of conditions. For each of the conditions indicate whether you think you would be able to complete the job using the software package. Then for each condition that you answer ?yes,? please rate your confidence about your judgment, by circling a number from 1 to 10, where 1 indicates ?Not at all Confident,? 5 indicates ?Moderately Confident,? and 10 indicates ?Totally Confident.? I could complete the job using the software package? 1. if there was no one around to tell me what to do as I go. Yes?1 2 3 4 5 6 7 8 9 10 No 2. if I had never used a package like it before. Yes?1 2 3 4 5 6 7 8 9 10 No 3. if I had only the software manuals for reference. Yes?1 2 3 4 5 6 7 8 9 10 No Not at all Confident Totally Confident 136 4. if I had seen someone else using it before trying it myself. Yes?1 2 3 4 5 6 7 8 9 10 No 5. if I could call someone for help if I got stuck. Yes?1 2 3 4 5 6 7 8 9 10 No 6. if someone else had helped me get started. Yes?1 2 3 4 5 6 7 8 9 10 No 7. if I had a lot of time to complete the job for which the software was provided. Yes?1 2 3 4 5 6 7 8 9 10 No 8. if I had just a built-in help facility for assistance. Yes?1 2 3 4 5 6 7 8 9 10 No 9. if someone showed me how to do it first. Yes?1 2 3 4 5 6 7 8 9 10 No 10. if I had used similar packages before this one to do the same job. Yes?1 2 3 4 5 6 7 8 9 10 No Additional Comments about the Experiment or the Interface: ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 137 Colorblindness Test What numbers do you see ? 138 Personality Test Select whichever position best reflects where you exist between each pair of words... 1) logical emotional 2) actual hypothetical 3) group oriented loner 4) talkative quiet 5) facts speculation 6) concrete abstract 7) tangible conceptual 8) analytical passionate 9) engaging withdrawn 10) fairness kindness 11) linear nonlinear 12) disorderly orderly 13) play work 14) spontaneous planner 15) messy clean 16) rational sentimental 17) improvise prepare 18) open private 139 19) insensitive sensitive 20) thinking feeling 21) many friends few friends 22) real surreal 23) late punctual 24) outgoing reserved 140 APPENDIX B: EXPERIMENT OVERVIEWS 141 Plant Description (From Interfaces Traditional, Color for Alarms Only, & Mental Model Layout) The plant consists of four tanks, three pumps, three valves, and two heaters. The purpose of the plant is to heat water to 80? Fahrenheit for as long as possible without incurring a pipe rupture. For the plant to be operating at its desired level, the following process state requirements must be met. Temperature requirement The temperature of the product fluid leaving the first heater should be between 70? and 75? Fahrenheit. The temperature of the product fluid sent to hot tank must be kept between 85? and 95? Fahrenheit. Flow requirement All flow rates must be within the range of 50 to 60 gallons per minute. Failure: Pipe Rupture Pipe rupture will occur after the flow to the heater as dropped 75% from its target level of 50gpm for more than 1 minute. Once the pipe ruptures the operator must shutdown the entire plant. Alarm States If a process variable rises above or drops below it?s desired operating level within a plus or minus 15% to 35% level then a low level alarm occurs which will turn the variable blue. If the increase or decrease is between 35% and 75%, then a medium level 142 alarm will occur which will turn the variable yellow. If the increase or decrease is 75% or greater, then a high level alarm will occur which will turn the variable red. The process must reach steady state first. Then the alarm states will begin being displayed. Plant Description (For Interface Feedback) The plant consists of four tanks, three pumps, three valves, and two heaters. The purpose of the plant is to heat water to 80? Fahrenheit for as long as possible without incurring a pipe rupture. For the plant to be operating at its desired level, the following process state requirements must be met. Temperature requirement The temperature of the product fluid leaving the first heater should be between 70? and 75? Fahrenheit. The temperature of the product fluid sent to hot tank must be kept between 85? and 95? Fahrenheit. Flow requirement All flow rates must be within the range of 50 to 60 gallons per minute. Failure: Pipe Rupture Pipe rupture will occur after the flow to the heater as dropped 75% from its target level of 50gpm for more than 1 minute. Once the pipe ruptures the operator must shutdown the entire plant. 143 Alarm States If a process variable rises above or drops below it?s desired operating level within a plus or minus 15% to 35% level then a low level alarm occurs which will turn the variable blue. If the increase or decrease is between 35% and 75%, then a medium level alarm will occur which will turn the variable yellow. If the increase or decrease is 75% or greater, then a high level alarm will occur which will turn the variable red. The process must reach steady state first. Then the alarm states will begin being displayed. Feedback The arrows displayed show feedback for the process variable it is pointing to. If the previous 5 samples taken on that variable show an increasing or decreasing trend, the arrow will point either up of down respectively. If the variable has not changed, the arrow will point straight to the variable. 144 APPENDIX C: INFORMATION SHEET 145