HUMAN COORDINATION OF ROBOT TEAMS: AN EMPIRICAL STUDY OF MULTIMODAL INTERFACE DESIGN Except where reference is made to the work of others, the work described in this dissertation is my own or was done in collaboration with my advisory committee. This dissertation does not include proprietary or classified information. ________________________________________________ E. Vincent Cross II Certificate of Approval: Richard Chapman Associate Professor Computer Science and Software Engineering Juan E. Gilbert, Chair Professor Computer Science and Software Engineering Cheryl Seals Associate Professor Computer Science and Software Engineering George T. Flowers Dean Graduate School HUMAN COORDINATION OF ROBOT TEAMS: AN EMPIRICAL STUDY OF MULTIMODAL INTERFACE DESIGN E. Vincent Cross II A Dissertation Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 10, 2009 iii HUMAN COORDINATION OF ROBOT TEAMS: AN EMPIRICAL STUDY OF MULTIMODAL INTERFACE DESIGN E. Vincent Cross II Permission is granted to Auburn University to make copies of this dissertation at its discretion, upon the request of individuals or institutions and their expense. The author reserves all publication rights. Signature of Author Date of Graduation iv DISSERTATION ABSTRACT HUMAN COORDINATION OF ROBOT TEAMS: AN EMPIRICAL STUDY OF MULTIMODAL INTERFACE DESIGN E. Vincent Cross II Doctor of Philosophy August 10, 2009 (Master of Software Engineering, Auburn University, 2006) (Bachelor of Science, Miami University, 2002) 131 Typed Pages Directed By Juan Gilbert Human Robot Interaction is an emerging area of interdisciplinary research that requires contributions from the areas of robotics, human computer interaction, cognitive sciences, multimodal interfaces and others. Due to the interdisciplinary nature of human robot interaction, the research proposed here advances not only the body of knowledge within human robot interaction but also in user interface design, information fusion, visual display and multimodal interaction. This research will focus on a comparative study between multimodal interfaces. Each interface style will require information fusion and multimodal interaction to provide the user with v appropriate capabilities and awareness to manage their robot team. The study will determine which interface improves the user?s ability to efficiently manage a robot team. The research proposed here will build upon findings, solutions and results provided by existing literature. This research will focus on the following human performance characteristics: 1. Mental Workload 2. Task Completion Time 3. Task Completion Rate vi ACKNOWLEDGEMENTS I want to thank everyone that has supported and encouraged me during this process. I specifically would like to thank my thesis advisor Juan Gilbert for his patience in helping me to define the scope of this project, his mentorship and friendship. I want to also thank my dissertation committee Richard Chapman, and Cheryl Seals for their assistance in this experiment. I want to also give thanks to my fellow HCCL lab members for their words of encouragement. I also thank my family and friends for their constant love and support during this process. Most of all, I want to give special thanks to God, for making this all possible. vii Style manual or journal used Journal of SAMPE Computer software used Microsoft Word 2002 viii TABLE OF CONTENTS LIST OF FIGURES ........................................................................................................... xi LIST OF TABLES ........................................................................................................... xiii 1 INTRODUCTION AND BACKGROUND ............................................................... 1 1.1 Motivation ................................................................................................... 1 1.2 Problem Description ................................................................................... 3 1.3 Goals and Contributions ............................................................................. 7 1.4 Organization ................................................................................................ 8 2 LITERATURE REVIEW ........................................................................................... 9 2.1 Overview ..................................................................................................... 9 2.2 Single Robot User Interface ...................................................................... 12 2.3 Multi-Robot User Interfaces ..................................................................... 16 2.4 Video Game User Interfaces ..................................................................... 23 2.5 Security User Interface ............................................................................. 26 2.6 2D vs. 3D Displays ................................................................................... 28 3 SYSTEM DESIGN ................................................................................................... 30 3.1 User Interface ............................................................................................ 30 3.1.1 Smart Viewer Monitor View Interface.......................................... 31 3.1.2 Smart Viewer Focus View Interface ............................................. 34 ix 3.1.3 Single Window Interface ............................................................... 35 3.1.4 Multimodal Interaction .................................................................. 36 4 IMPLEMENTATION RESULTS ............................................................................ 40 4.1 Experiment Design.................................................................................... 41 4.2 Experimental Setting ................................................................................. 41 4.2.1 Materials ........................................................................................ 41 4.2.2 Tasks.............................................................................................. 42 4.2.3 Environment .................................................................................. 42 4.2.4 Participants and Procedure ............................................................ 44 4.2.5 Data Collection .............................................................................. 45 4.2.6 Pre-Experiment Questionnaire ...................................................... 46 4.2.7 Performance Data and User Observations .................................... 47 4.2.8 Post-Experiment Questionnaire..................................................... 47 4.3 Results and Discussion ............................................................................. 47 4.3.1 Performance and Data Findings .................................................... 48 5 SUMMARY AND CONCLUSION ......................................................................... 78 5.1 Summary ................................................................................................... 78 5.2 Conclusion ................................................................................................ 80 5.3 Contributions............................................................................................. 83 5.4 Directions for Future Research ................................................................. 84 BIBLIOGRAPHY ............................................................................................................. 86 x APPENDICES .................................................................................................................. 90 APPENDIX A. Pre Survey Questionnaire ................................................................ 90 APPENDIX B. Post Survey Questionnaire .............................................................. 92 APPENDIX C. NASA TLX Mental Workload Rankings ........................................ 95 APPENDIX D. Speech Grammar ............................................................................. 98 APPENDIX E. Database Schema ........................................................................... 105 xi LIST OF FIGURES Figure 1: Multi Window Approach..................................................................................... 2 Figure 2: Idaho National Laboratory interface ................................................................. 13 Figure 3: Ecological Interface Design .............................................................................. 15 Figure 4: Minimalist approach to robot user interface ...................................................... 16 Figure 5: Adams Multi-robot interface ............................................................................. 20 Figure 6: HMD Multi-robot Interface ............................................................................... 21 Figure 7: Playbook interface ............................................................................................. 23 Figure 8: RIDE RTS Multi Robot Interface ..................................................................... 25 Figure 9: Multi-robot interface based on RTS game ........................................................ 26 Figure 10: DOTS Security Interface ................................................................................. 28 Figure 11: Monitor Interface ............................................................................................. 32 Figure 12: Mario Cart ....................................................................................................... 33 Figure 13: Smart Viewer Focused Window interface....................................................... 35 Figure 14: Single Window Interface ................................................................................. 36 Figure 15: Training Environment ..................................................................................... 43 Figure 16: Experiment Environment ................................................................................ 43 Figure 17: Average Time Too Complete Task ................................................................. 52 Figure 18: Mean task completion time within each interface ........................................... 56 Figure 19: Single Window Scatterplot Correlation .......................................................... 59 xii Figure 20: Monitor Window Scatterplot Correlation........................................................ 60 Figure 21: Focus Window Scatterplot Correlation ........................................................... 62 Figure 22: Single Window Scatterplot Correlation between number of robots and total mental workload................................................................................................................ 68 Figure 23: Monitor Window Scatterplot Correlation between number of robots and total mental workload................................................................................................................ 70 Figure 24: Focus Window Scatterplot Correlation between number of robots and total mental workload................................................................................................................ 71 Figure 25: Focus Window Scatterplot Correlation between time and total mental workload ........................................................................................................................... 73 Figure 25: Monitor Window Scatterplot Correlation between time and total mental workload ........................................................................................................................... 75 Figure 26: Single Window Scatterplot Correlation between time and total mental workload ........................................................................................................................... 76 xiii LIST OF TABLES Table 1: Speech commands .............................................................................................. 38 Table 2: Manual Control ................................................................................................... 39 Table 3: Demographic information ................................................................................... 48 Table 4: Years of computer usage .................................................................................... 48 Table 5: Average daily computer use ............................................................................... 48 Table 6: Overall Task Completion rate per interface ........................................................ 49 Table 7: Task completion rate with 4 robots per interface ............................................... 50 Table 8: Task completion rate with 5 robots per interface ............................................... 50 Table 9: Task completion rate with 6 robots per interface ............................................... 50 Table 10: Between interfaces average time in seconds to complete task ......................... 51 Table 11: Mean task completion time in seconds between interface comparison when using 4 robots .................................................................................................................... 52 Table 12: Mean task completion time in seconds between interface comparison when using 5 robots .................................................................................................................... 53 Table 13: Mean task completion time in seconds for between interface comparison when using 6 robots .................................................................................................................... 54 Table 14: Within interfaces mean task completion time in seconds to find all 3 bombs . 55 Table 15: Within Focused Window Comparison.............................................................. 57 Table 16: Within Monitor Window Comparison .............................................................. 57 xiv Table 17: Within Single Window Comparison ................................................................. 58 Table 18: Pearson Correlation of Single Window Interface ............................................. 60 Table 19: Pearson Correlation of Monitor Window Interface .......................................... 61 Table 20: Pearson Correlation of Focus Window Interface.............................................. 63 Table 21: Focus Window Regression Analysis ................................................................ 63 Table 22: 3 bombs found mean TLX Focus Window descriptive statistics .................... 66 Table 23: 3 bombs found mean TLX Monitor Window descriptive statistics ................. 66 Table 24: 3 bombs found mean TLX Single Window descriptive statistics .................... 67 Table 25: Single Window Pearson?s correlation between number of robots and total mental workload................................................................................................................ 69 Table 26: Monitor Window Pearson?s correlation between number of robots and total mental workload................................................................................................................ 70 Table 27: Focus Window Pearson?s correlation between number of robots and total mental workload................................................................................................................ 72 Table 28: Focus Window Pearson?s correlation between number of robots and total mental workload................................................................................................................ 74 Table 29: Monitor Window Pearson?s correlation between time and total mental workload ........................................................................................................................... 75 Table 30: Single Window Pearson?s correlation between time and total mental workload ........................................................................................................................................... 77 1 1 INTRODUCTION AND BACKGROUND 1.1 Motivation Various studies have shown the good, the bad and the ugly of current robot user interfaces. The majority of this research has focused on single robot interfaces in urban search and rescue environments. Urban Search and Rescue (USAR) has become the premier human-robot interaction (HRI) problem (Nourbakhsh, Sycara et al. 2005). It presents an unknown environment full of obstacles requiring the operator and robot to work as an effective team to explore efficiently. This has proven to be a daunting task for the best of systems (Yanco, Drury et al. 2004). One of the reasons this task has proven to be difficult has been deficiencies in user interface design. Yanco studied the interfaces of four teams competing at the Association for the Advancement of Artificial Intelligence (AAAI) USAR competition. She noted that there were issues with the system experts and domain experts in having appropriate area awareness of the robots (Yanco, Drury et al. 2004). She surmised that part of the difficulty was the operators using multiple windows to control one robot. Another study done by Casper and Murphy noted that when multiple windows were available robots were primarily operated via the video camera window. Although participants preferred to use the video camera, researchers noted that operators had difficulty doing the navigation and exploration of the environment when using only video and not any of the other sensors (Casper and Murphy 2003). 2 Early robot interfaces have attempted to provide the user with all the information in various windows. Researchers have begun to note that robot interfaces are providing too much information. (Casper and Murphy 2003; Trouvain, Schlick et al. 2003; Burke, Murphy et al. 2004; Scholtz, Antonishek et al. 2004; Yanco, Drury et al. 2004; Bruemmer, Few et al. 2005; Nielsen and Goodrich 2006). The majority of the robot user interfaces provide a map, a camera view or both as noted in Yanco?s (2004) AAAI study. In addition, the operator is often times presented with various sensory data such as sonar readings, camera views, robot health and others. This information is usually presented in separate windows requiring the operator to mentally fuse this information see Figure 1 (Baker, Casey et al. 2004). Figure 1: Multi Window Approach 3 Recent research has begun to remove the multiple window syndrome pictured in Figure 1 to a smaller set of windows (Baker, Casey et al. 2004). Additional research has focused on augmenting the camera view with a live map and user added markers (Nielsen, Bruemmer et al. 2006; Nielsen and Goodrich 2006). As shown the design of user interfaces vary greatly. The common theme of these interfaces is improving single robot control. There is a lack of focused research in the development of a multi-modal multi-robot user interface. Single robot solutions could prove to be precarious when trying to complete a mission. A single robot could potentially malfunction leaving the mission in jeopardy or require additional robots in completing the task. Additionally, cognitive overload is common issue with robot user interfaces. There is a continued need for the design of novel interfaces that allow an operator the means to control a robot team to complete a set of tasks. The following section will briefly introduce the problem, approach taken to solve the problem and contributions of this research. 1.2 Problem Description One of the goals of the HRI community is the ability for an operator to control multiple robots efficiently and effectively. Control of a single robot has been proven to be difficult (Trouvain, Schlick et al. 2003; Yanco, Drury et al. 2004). Although there is a continued need for the study and design of single robot interfaces it is also necessary to explore the design of multi-robot user interfaces. As with single robot user interfaces multi-robot user interfaces vary widely as there is no standard framework. Despite the growing interest in robot user interfaces the majority of research has been concerned with expanding the capabilities of the robots. This is seen in the early 4 AAAI robotic competition. Recently, researchers have begun to look at the user interface of robot systems (Scholtz; Fong, Thorpe et al. 2001; Adams 2002; Burke, Murphy et al. 2004; Yanco, Drury et al. 2004; Chen, Haas et al. 2006; Wagner, Endo et al. 2006). These researchers have all focused on the design of user interfaces. However, there is a lack of research that seriously examines the development of single window multimodal multi-robot user interfaces. Many of the robot user interfaces are designed to present information in different windows. In Yanco?s (2004) study on user interfaces at a major robotics competition she notes that necessary information is often in separate windows which places the information in separate areas of the interface, requiring the user to mentally fuse the information (Yanco, Drury et al. 2004). This multi window approach does not scale when adding additional robots. The designer will quickly run out of screen space and the user can easily become overwhelmed. Nielson?s study showed that a 3D map and video interface performed the highest when tele-operating a single robot. However, this study only accounts for a single robot in which navigation is tele-operated. There are additional unique challenges associated with multi-robot systems. One such consideration is the effect context switching has on the operator. A context switch occurs when an operator must switch their current cognitive focus from one robot to another. The cost of a context switch is based on the complexity of the environment and task that the robot was performing. There is also a time cost involved in a context switch which is directly related to the amount of time necessary for an operator to regain situational awareness (SA). This is a problem in current single window interface which require the operator to perform a context switch 5 whenever the operator needs to supervise a robot that is not in the current live video feed. Another challenge as briefly mentioned is updating and maintaining SA. Situational Awareness ?involves being aware of what is happening around you to understand how information, events, and your own actions will impact your goals and objectives, both now and in the near future?. When working with a multi-robot interface the operator has to maintain SA for each robot. Depending on the complexity of the environment, the task being accomplished and the number of robots it may not be possible for the operator to maintain SA for all robots. Another issue that must be managed differently when working with multiple robots is mental workload. Mental Workload is the amount of mental processing required to perform a task. In multi-robot systems the operator has the objective of supervising multiple robots tasks, the sensory data each robot is reporting, the actuators and their effect on the environment has that environmental changes affects the task of the other robots as such it is easy for an operator?s mental capacities. When creating a user interface for human robot interaction a number of considerations are necessary. One such consideration is how to display information such as the robot sensors. These include but are not limited to sonar, laser and infrared sensors. Another consideration is the internal state of the robot i.e., battery power, odometer readings, and self diagnostic checks. Navigation and control information has to be considered; will there be a map, and/or a camera. Questions about the interaction between the operator and robot(s) need to also be addressed. Will the robot be autonomous, semi autonomous, tele-operated or work on a sliding autonomy scale? In addition, the interface may need to be highly customized to the situation in which the operator will be using the robot. All of the above are design issues that must be 6 considered when developing a user interface. It has been difficult for experts to create interfaces that are capable of managing the above information especially when, in many cases, access to actual users such as, army personal, or search and rescue operators, is limited. In summary the problems to be addressed are the following. Current interfaces are designed to show only a single video which causes the operator to completely rely on the sensors and intelligence of the robots that are not in the live video. The operator finds out about a problem after it has occurred. For example if a robot runs into a problem navigating difficult terrain the operator has to react to an issue that could have been foreseen and pre-planned. This puts the operator in a reactive and not proactive decision making process. The single window interface also has a contextual switch time delay cost associated with changing the operator?s supervisory focus from one robot to another. Situational awareness and mental workload are another problem with current interface design which has garnered a lot of attention but has not been solved. With the single window interface design the operator has complete SA for the current robot that is in the main window but has an incomplete SA for all other robots that are not in the main window. When the operator wants to regain complete SA for a different robot there is a time delay before the operator is able to understand the environment the robot is working in, the actions that the robot has taken, what affect those actions had on the environment and whether or not the robot completed the task or will be able to complete the task. Additionally, in a single window interface it is not always clear what affect a robot has on another robot. For example if two robots are working together to build a sand castle if robot A delivers sand for robot B to use the operator cannot be sure of the affect this 7 environment change has had on the robot B until he/she switches to that robot. Mental workload is another problem. It has the affect of decreasing the ability of the operator to control multiple robots as such is must be managed in such a way that the operators mental load can be maintained at constant level or increase in small increments as the number of robots increase. Finally, there is a need for interfaces to be able to scale and work as efficiently with a varying number of robots. For interfaces to scale the SA and mental workload problem have to be resolved. 1.3 Goals and Contributions The major research goal is to evaluate human performance when using a novel multimodal multi-robot user interface. The system is multimodal in that a user can use keyboard, mouse and their voice as means of interaction. The results of this study will help advance the continued design and evaluation of user interfaces to improve human performance when working with a robot team. This research makes the following contributions to the fields of Human Robot Interaction, Multimedia Display, User Interfaces, Situational Awareness, Information Interfaces and Presentation, and Human-Computer Interaction. Specifically, this research accomplishes the following: ? Introduces and evaluates a novel alternative design for user interaction when working with a robot team. ? Investigate human performance by having the interface assist the operator in managing information from the robot. ? Investigate the preferred user interface through usability studies. 8 ? Investigate the weaknesses associated with the use of multimodal multi-robot interfaces. 1.4 Organization In the chapters that follow a research agenda will be examined. Chapter 2 gives an overview of the areas of research that pertain to the interaction and design of robot systems. Such areas of research include: robot user interfaces and multi-robot user interfaces. Chapter 3 will discuss the system design and implementation. This will be followed by how the system described in chapter 3 performed in answering the research questions by analyzing the details of the experimental results in Chapter 4. Finally, Chapter 5 discusses the conclusions of the study, including limitations; summarizes main contributions that this work made and points to some issues for future work. 9 2 LITERATURE REVIEW This chapter provides a review of the major areas of user interface research within Human Robot Interaction and Human Computer Interaction. This highly focused literature review will concentrate on the aspects of these areas that directly impact this research. 2.1 Overview The study of the interaction between man and robot is a growing area that is devoted to researching the best method for each to interact with one another. It is necessary to note that HRI is related to Human Computer Interaction (HCI) but there are some critical differences. Schultz noted six dimensions in which HRI and HCI differ (Schultz and Parker 2002): 1. Different requirements based on interaction roles 2. Interaction of the platform in the physical world 3. Dynamic nature of the hardware 4. Environment in which the interactions occur 5. Number of platforms the user is interacting with 6. Autonomous behavior of the platform 10 Fong notes that HRI differs from HCI because HRI is concerned with complex, dynamic controls systems operating with autonomy in real world environments (Fong, Thorpe et al. 2001). Due to these differences it is necessary to produce new research into the development of usable interfaces that allow users the necessary interaction capabilities when working with robots. User interface design is essential to making interactions between man and robot simple and intuitive. It is the initial point of input into the robot and output of the response of the robot and its environment. The interface allows the user to assess the state of the system and determine the next set of actions, based on the information returned. The amount of effort a user must put forward in understanding the information received in the output is directly determined by the interface (Hix and Hartson 1993). As such, the importance of the usability of the user interface cannot be understated specifically when dealing with robots in an environment in which mistakes can be costly. Robot research has traditionally focused on expanding the capabilities of the human and robot interaction (Perzanowski, Schultz et al. 1999; Perzanowski, Schultz et al. 2001; Yanco 2001; Iba, Paredis et al. 2004; Yanco, Drury et al. 2004). Limited research has focused on the user interface and interaction between the robot and its operator. Many of the interfaces used to interact with robots have been designed as an afterthought with the developers believing that their system is intuitive. This can be a costly mistake; the effectiveness of the supervisory control of a mobile robot system is dependent on the quality of the user interface design. The need to develop usable user interfaces led to a study that examined the interfaces of four teams competing in the Association for the Advancement of Artificial 11 Intelligence (AAAI-2002) search and rescue robotics competition (Yanco, Drury et al. 2004). The four teams? interfaces were similar in that they all used multiple windows in an attempt to give the operator environmental information and status information. Each window contained information such as, sensory data, live video, robot status, and the robot generated map. In the AAAI study, the domain expert had issues fusing the information from the multiple windows into a workable mental model of the environment (Yanco, Drury et al. 2004). This caused issues with navigation, obstacle avoidance, and task completion. The many windows approach also limits the scalability of the interface. Adding new sensors and robots would require additional windows increasing the mental workload of the operator. These interfaces which put various sensory and status information in different windows have required the operator to mentally fuse this information. By forcing the operator to perform mental rotations and mentally fuse the information it results in increased workload and decreased situational awareness causing a decrease in performance (Endsley 1988; Lee, Caven et al. 2001). Researchers have begun to design interfaces that can handle the amount of information a single robot gives an operator. A number of suggestions have been set forth by the HRI community to improve user interfaces. Researchers have recommended that systems 1) use a map; 2) fuse sensory data; 3) minimize the number of windows and 4) increase the amount of spatial information (Yanco, Drury et al. 2004). The rest of this chapter will look specifically at single robot user interfaces, multi- robot user interfaces, video game interfaces, security guard interfaces and 2D vs. 3D interfaces. 12 2.2 Single Robot User Interface A classic single robot interface was designed at the Idaho National Laboratory. This early interface design displayed information in various windows, as depicted by Figure 2. The interface contains five windows each displaying a different set of information. The upper left window contains the live video feed from the robot. It also contains the controls to move the robot?s camera. To the right of the video feed window are status indicators and controls for the robot. Below this window is the robot autonomy selector (manual mode, safe mode, shared mode, and autonomous mode) and movement status indicators (Bruemmer, Marble et al. 2002; Bruemmer, Marble et al. 2003; Bruemmer, Boring et al. 2004). The last two windows show a dynamically generated map and operational status. 13 Figure 2: Idaho National Laboratory interface This interface provides more information than may be necessary in a scenario. An expert user may find all this information helpful however it can be overwhelming for non expert users (Baker, Casey et al. 2004). This interface requires the operator to take the data displayed in different windows and mentally fuse the information to provide an accurate understanding of the robot, the environment, the task being performed and the effect the robot has had on the environment. By requiring the operator to mentally fuse the data the cognitive load of the operator is increased, thereby making it difficult to Live video feed and controls Status indicators and controls for robot Operational status and dynamically generated map Robot autonomy selector 14 operate multiple robots. Finally as previously mentioned, operators choose to focus on the live video feed. In this interface the live video feed is not the primary focal point. This interface does not have a focal point and is instead divided into four sections. Another single robot interface is based on an ecological interface paradigm. It combines video, map and robot pose information into a 3D augmented virtual interface (Nielsen, Goodrich et al. 2007). This interface as seen in Figure 3 has decreased the number of windows by building a 3D virtual representation of the environment while also including a live video. This system uses sensors and map generation to build a virtual representation of the environment. The sensors are used to locate obstacles and open space which is rendered in the virtual environment. By using simultaneous location and mapping (SLAM) the designers are able to render the walls and obstacles. A 3D model of the robot is rendered in the environment and a 3rd person chase camera is placed above and behind the robot. The goal of this interface is to improve the operator?s situational awareness of obstacles in the environment in relation to the robot. Finally, a live video is also placed in the virtual environment slightly ahead of the virtual robot. This is an interesting take on controlling a single robot however, as it would not scale well when trying to control multiple robots. There is also a loss of information when converting the live information from the environment to the virtual world. Information such as color, shape, material, images etc. of the obstacles is lost in the translation. Additionally, map generation is not a perfected science and could prove problematic when used in various disaster scenarios. 15 Figure 3: Ecological Interface Design Another single interface design which has also decreased the number of windows is shown in Figure 4. This interface has on the left a live video surrounded by ranging information which informs the operator of the distance of potential obstacles. Above the live video is an image from the rear facing camera which informs the operator of obstacles behind the robot. To the right of the rear view image is status information of the robot. Below the robot status information is a map generated by the robot (Baker, Casey et al. 2004). This interface has decreased the number of windows by only showing the most important information. It has also made the live video the primary view. This approach is an improvement, however it does not scale well when attempting to control multiple robots. Finally although the live video feed is the primary focus it is off center 16 of the interface. Prior research has already stated that live video feed is the main attraction, and as such needs to be in the center of the interface. Figure 4: Minimalist approach to robot user interface Research in the area of robot user interfaces is still in its infancy; to date there has been limited research on how interfaces should be developed to facilitate effective interactions between human and robot. 2.3 Multi-Robot User Interfaces Much of the HRI research has focused on the design and development of single robot interfaces. As missions that require larger number of robots become increasingly Live video with range indicators Rear view video Status information Dynamically generated map 17 necessary, research is needed which explores how a limited number of operators can control multiple robots. Although multi-robot interfaces can share and reuse many elements from single robot user interfaces, special attention must be paid to two problems. First, each robot operates within an arbitrary complex context (task, environment, etc.) and the contexts are coordinated or independent. In order to effectively supervise the multi-robot system the user interface must support the operator in smooth switching between different contexts. Second, a complex set of concurrent tasks must be planned, monitored and modified in order to maintain the system?s operational effectiveness. Although the majority of the research has been focused in single window interfaces some research is underway into the design of interfaces to control multiple robots. Controlling a single robot is a difficult task for a single operator. When using multiple robots the difficulty is increased by the need for the operator to manage multiple sensors, and actuators while maintaining situational awareness for each of the robots that the operator is controlling. Recent research using a simulated environment with semi- autonomous generic robots was performed investigating the effects of human supervisory control when using map based interfaces vs. camera based interfaces while controlling 1, 2, and 4 robots (Trouvain, Schlick et al. 2003). The results of this research showed that it was not possible for an operator to effectively supervise more than one robot when using the map, camera or a combination of the two interfaces. Although the Trouvain et al. study showed that controlling multiple robots was not possible other researchers have shown the opposite (Adams 1995 Dissertation). This could be due to the effect interfaces have on the ability of the operator to complete tasks with multiple robots. The interface 18 shown in Figure 5 has two sections. The upper section displays the camera view of the currently selected robot. The bottom area shows the two robots, a compass for each robot and status information. Figure 5: Trouvain multi window interface Not only is it important to design an interface that can control a robot team it is also necessary to determine the scalability of the interface. Scalability is the number of robots that an interface is capable of handling on behalf of the operator. The design of a scalable interface has begun to become of interest to researchers as the need for interfaces that can control multiple robots has increased. Ideally, interfaces would be able to handle an increase in robots without increasing the cognitive load of the operator (Humphrey, Camera view of selected robot Robots, compass and status information 19 Henk et al. 2007). Scalability of a robotic interface is one of the guidelines set forth by (Scholtz, Antonishek et al. 2004). However there are currently no widely adopted methods for creating scalable interfaces. Adams designed a scalable interface that was evaluated to determine if it enhanced the situational awareness and workload of the operator Figure 6 (Humphrey, Henk et al. 2007). This interface has a single live video view, surrounded by a relational halo that shows the general direction of the other robots. The right section of the interface shows a list of the robots and status information. Color changes in the task list represent state changes such as bomb detected or collision occurred. A relation map is provided as seen in the bottom right and a number of visual controls are shown at the bottom. The goal of the study was to determine what effect adding robots had on the operator. 20 Figure 6: Adams Multi-robot interface In looking for the best way to control multiple robots researchers have explored using interfaces that augment reality with virtual information. Researchers exploring the control of a swarm of robots have explored using Head Mounted Displays (HMD?s) as a means of informing the operator of relevant information (Daily, Cho et al. 2003). As can be seen in Figure 7, three small scale robots have a virtual arrow displayed showing the operator which direction to travel. Although this is a novel approach much work remains. One major issue is that the operator is required to remain in visual range of the robots any robots not in visual range the operator is unable to receive pertinent information. Live video with relational halo List of robots and their status Robot controls Relational map 21 Figure 7: HMD Multi-robot Interface When developing a multi-robot interface researchers have also looked at a 2D top down approach. In the case of Kaminka and Elmaliach they designed a 2D top down interface that explored the ability to control multiple robots by keeping them in formation (Kaminka and Elmaliach 2006). Their interface had two aspects to it a window with the live video feed and a second window with a top down view of the robot. This interface requires the operator to look at 2 windows (Live Video Feed and a Top Down View) and combine the information which increases mental workload. The interface is also lacking in sensory and environmental information which is required in a number of scenarios. Another 2D interface designed by Envarli and Adams focused on the operator?s role as a supervisor (Envarli and Adams 2005). The interface has a main window which displays each of the robots in a top down view with the robots appearing as circular objects. In the main window the robots had two circular rings. The inner ring 22 represented the robots and their team and the outer ring represented the robots status. Bottom left of the main window is a bar chart used for accessing team information and above that is information relevant to the robot population. Information such as percentage of task completion, expected time remaining, time already committed task type and task priority are shown in separate windows. This interface show promise for allowing an operator to delegate tasks to robots but it makes a few assumptions that are not reliable. For the operator to know something is wrong they have to be informed by the robots. It may not always be the case that a robot recognizes that it is malfunctioning. Additionally, there is no live video. A live video feed is the primary source of information for operators (Nielsen and Goodrich 2006). Another system called Playbook is designed for multi-robot control by focusing on task level control (Parasuraman, Galster et al. 2003). This interface creates a two dimensional representation of the world where the only sensor visualization is a 2D representation of the robots field of view Figure 8. The drawbacks to this interface are similar to those mentioned in the previous interface. The operator is totally reliant on the algorithms and the sensor data. The operator does not have access to the visual information afforded by live video. 23 Figure 8: Playbook interface New interface designs that are capable of decreasing situational awareness and task workload will allow an operator the ability to control multiple robots. User interface designers at times look to video games for inspiration on how to manage the large amount of data available to them. It is necessary to take a look at video game interfaces and what aspects of these interfaces can be used in designing interfaces. 2.4 Video Game User Interfaces Researchers have also attempted to model interfaces based on video game design. Heckel et al. designed an interface based on Real Time Strategy (RTS) games to control 24 multiple robots. In a RTS game the player is often tasked with managing multiple units. A RTS game provides an isometric view of the world which allows you to see the environment, the robots, their location, and their effect on the environment. In this interface seen in Figure 9 there is a main window which shows the environment and the robots in the environment. To the right of the main window are elements that allow the operator to control the camera, assign task to selected robots and receive information on task being performed by the robots (Heckel, Blakely et al. 2006). There are slight issues with using RTS for robot interfaces. The RTS interfaces in video games show as much data about the environment as live video. Current capabilities of robotic interfaces do not provide for a means to take the same visual information prevalent in a live video and place it in an artificial world that the RTS interface creates. As such important information will be missed. Additionally, the design of the RTS view is based on sensors and the map building algorithm. If the sensors are being absorbed or do not notice a potential danger in the environment then it will not be shown in the RTS interface. For example, if there is a wall that is absorbing sensor data that wall will never be shown in the RTS interface. The operator and robot would presume that the way is clear causing the robot to run into the obstacle. 25 Figure 9: RIDE RTS Multi Robot Interface Another interface based on RTS is shown in Figure 10 . The interface shows a graphical representation of the environment and the robots. This interface allows an operator to select multiple robots or individual robots and give tasks to each robot. The benefit of this interface and all interfaces based on RTS video games, is a third person perspective which allows you to see much of the environment and the exact location of robots and objects is easily observable (Jones and Snyder 2001). At first glance this RTS interface appears ideal. However, the real world does not easily map to a RTS interface. There is a loss of information when attempting to model the environment. There is a total reliance on sensors to understand the environment. Each robot system may not have the same consistent view of the environment causing the creation of the environment to not flow. 26 Figure 10: Multi-robot interface based on RTS game 2.5 Security User Interface Video security surveillance often times requires a single or small number of security personal to monitor many live videos. Security surveillance interfaces are designed to accommodate the security personal in detecting events throughout the system. A key aspect of security interfaces is the ability to inform the operator of important information through the use of object detection. In the case of security interfaces the object is often a person. The interface has to not only recognize an event of importance it also has to recognize the person that is causing the event and inform the operator of which camera the event is occurring. As such interface design is paramount 27 to the ability of security guards to perform their duties. These issues are similar to many of the issues seen in robot interfaces A security interface that explores this issue is the one used with Dynamic Object Tracking System (DOTS). The interface for DOTS is designed to accommodate the large number of live video feeds to improve the security personnel?s ability to complete their task. As can be seen in Figure 11 the DOTS interface displays multiple streams of live video each representing a different camera. The interface includes a camera bank on the left hand side, a timeline on the bottom of the interface, a floor plan bottom right, and a main player area middle to upper right. DOTS facilitates security personnel?s ability to complete their task through automatic detection of events and tracking of people (Girgensohn, Kimber et al. 2007). This interface and other similar security interface designs relate to HRI in that there is a single operator controlling multiple live video feeds. The security interfaces are task with managing the mental workload and updating the SA of the operator such that targets of interest can be monitored as they move throughout the environment. 28 Figure 11: DOTS Security Interface 2.6 2D vs. 3D Displays It is important to look at the extensive amount of literature that compares 2D views to 3D perspective views (Ellis, McGreevy et al. 1987; Bemis, Reeds et al. 1988; Baumann, Blanksteen et al. 1997; Hickox and Wickens 1999; John, Cowen et al. 2001; Smallman, John et al. 2001; Savage, Wiebe et al. 2004). Two-dimensional images afford a planar view in which all objects can be seen. 2D images usually use specifically designed symbols to represent objects. In this research a 3D image is two-dimensional (2D), but the viewing angle provides a 3D perspective. For example, rather than displaying an environment from directly above (a planar or bird?s-eye view), perspective Camera bank Timeline Floor plan Main player area 29 view technologies generally display the environment from a 30? or 45? angle. This information is necessary in understanding the study planned in this proposal. In recent 2D vs. 3D literature researchers performed a study in which participants had to determine if an airplane was either ascending or descending. The results of their study concluded that participants performed faster and were more accurate when using the 2D display (John, Cowen et al. 2001; Smallman, John et al. 2001). The study showed that many of the benefits of 3D such as, their naturalistic look, rapid perception of objects and integration of all three axis into one view, are mitigated due to the 3D perspective images being compressed onto a flat display. This results in all dimensions being somewhat ambiguous, causing oblique viewpoints and distortion of distances when trying to determine characteristics of a 3D scene. They also noted that the availability of information plays a significant role in which display method performs best. As such, when all information displayed is consistent then the 2D interface performed faster (Smallman, John et al. 2001). The importance of the design of the interfaces is further illustrated in a study performed by St. John. The study showed that 2D performs better when determining relative positions of objects (John, Cowen et al. 2001). 3D does perform better when identifying shapes and layouts. For simple distinct objects three-dimensional displays are believed to perform better in recognition due to the ability of the 3D rendering to showcase multiple views of an object in one rendering. In a 3D display multiple views must be generated and then combined mentally requiring more effort and time. 30 3 SYSTEM DESIGN To explore potential solutions to the aforementioned issues an interface is needed which will affords an operator the ability to view all live videos so that the operator can maintain situational awareness and be proactive in dealing with problems. The interface has to work in a similar fashion as the number of robots increase or decrease. This needs to be accomplished while not overloading the mental workload of the operator. The interface needs to inform the operator of relevant information only when necessary. It should also take into account additional interaction methods such as speech. By looking outside of the human robot interaction field at video game and security interfaces it is possible to design an interface that will accommodate an operator in controlling a team of robots. The interfaces designed in this study augment the ability of the robots by adding the human operator?s ability to make informed decisions. 3.1 User Interface The interface is the primary focus of the system; it is what provides the operator with the relevant information necessary to complete a task. As such the interface needs to accommodate the mental abilities of the operator by not overloading the operator with too much data. For this research two interfaces have been designed that facilitate the operator?s ability to control multiple robots. 31 3.1.1 Smart Viewer Monitor View Interface Figure 12 shows the Smart Viewer: Monitor View Interface. The monitor view displays multiple live video feeds each of the same width and height. Each live video provides all the necessary information for that robot. This is similar to video games with multiple players in which each window contains all the necessary information for that player?s character. Figure 13 is an example video game with multiple players. For this study that information is limited to the robots name and its radar but can be expanded to include battery power and other sensory data. The name of each robot represents a color for easy recall and also to correspond to the colors in the radar. Located in the top right corner of each live video is the radar. A radar is used instead of a map since dynamic map generation is an imperfect science that is highly reliant on sensory information (Humphrey, Henk et al. 2007). In this interface the radar shows where each robot is relative to the other robots in the live video. Around each live video is a colored border that depicts the current state of the robot. Using color as a means of communication is not new and is used here in similar aspects as research in security systems (Girgensohn, Kimber et al. 2007). There are three color notifications, green, yellow and red. Green is the robots way of informing the operator that all systems are working correctly and that the robot is performing the user defined task. Yellow is a notification used by the interface when the robot has non critical information for the operator. For example, when an obstacle is in the way of the robot completing its task and it is attempting to go around the obstacle. Finally, there is the red notification which informs the operator of a robot with critical information which requires immediate 32 attention from the operator. An example would be when a robot has found a potential bomb or is attempting to disarm a bomb. Figure 12: Monitor Interface Speech recognition Live video from each robot Color notification for each robot Robots Radar 33 Figure 13: Mario Cart The color notification gives a high level understanding of what the message entails but does not inform the operator of the actual message. To get the actual message the operator can use the mouse to scroll over the live video feed which would make a popup appear with the actual message. For yellow and red notifications the operator also receives a voice prompt informing the operator of the information. Prior research has shown that people can process information on separate tracks (Trouvain and Schlick 2006; Trouvain and Schlick 2007). Using the voice prompt has the potential of improving the ability of the operator to multitask. The voice prompt along with the color change allows the operator to notice which robot has the pertinent information. At the top of the interface is a bar which shows what the system heard you say. This allows the operator to recognize if the command recognized is the same as the 34 command given. There is also a bomb status which informs the operator of the number of bombs remaining to be disarmed. Finally there is a start button which starts the simulation. Once the simulation has started the start button switches to a stop button allowing the participant to end the study at anytime. The system is multimodal in that it allows the operator to use keyboard, mouse and speech commands to control the robots these interactions are further discussed in the interaction section. 3.1.2 Smart Viewer Focus View Interface The Smart Viewer Focus View Interface as seen in Figure 14 is slightly different than the monitor view. The focus view is divided into two sections the main window and the slideshow. The focus view allows the operator to enlarge the viewing area of any live video by double clicking the left mouse button on that robot?s live video. This will move that robot?s live video to the main window. This allows a particular live video to have a larger portion of the interface but still permits the operator to view all the live videos. The concept of the larger viewing area is to give a robot with important information a larger area for the operator to focus. For example if a robot has entered a new area or has found an object of interest the operator can double click with the left mouse on the live video for that robot which would move it to the main window increasing its size. All other elements of the focused view are the same as those previously described for the monitor view. 35 Figure 14: Smart Viewer Focused Window interface 3.1.3 Single Window Interface To determine if an interface with multiple live videos is effective and not overwhelming to the operator a third interface was developed. This interface is similar to other multi-robot interfaces in that it only shows one live video. In the single window interface there are two areas the elevator shaft which can be seen on the left and the main window, Figure 15. To view the live video of a different robot than the one currently displayed the operator has to double click on that robots corresponding area. For example if the operator wanted to see the live video of the purple robot then the operator would double click on the purple robot?s elevator which would switch the live video to Main Window Slideshow 36 the purple robot. All other interface elements remain the same as those described previously in the Monitor View. Figure 15: Single Window Interface 3.1.4 Multimodal Interaction Not only is the design of the interface important to the ability of an operator to supervise one or more robots, but special consideration is also needed for how an operator will interact with the robots. This is especially true if the operator has to supervise multiple robots. Many of the current single and multi-robot interfaces require the operator to use the mouse to click a button on the screen to issue commands (Trouvain, Schlick et al. 2003; Scholtz, Antonishek et al. 2004; Drury, Keyes et al. 2007). Elevator shaft Main Window 37 Clicking on the screen to issue a command is not optimal specifically when coordinating multiple robots. The operator has to first determine the location of the mouse followed by the location of the button; the operator must then move to the button and finally click the button. This all takes time which may be an issue when dealing with time sensitive tasks or missions. The alternative used in the Smart Viewer system is to use a multimodal approach. Multimodal interaction is when the interface is designed for interactions for multiple modalities. The most common is usually a combination of keyboard, mouse and speech recognition and text to speech. Other modalities that are used include but are not limited to, haptics, gestures, controllers and pen inputs. There has been research into the use of multi-modalities within in robotics (Perzanowski, Schultz et al. 1998; Perzanowski, Schultz et al. 1999; Perzanowski, Schultz et al. 2001 ); however this research has focused on a paradigm where the robot and operator could see each other. In this work the operator is not in line of sight with the robots as such, the robots cannot directly see or hear the operator?s commands. The system has an automatic speech recognizer (ASR) and a text to speech (TTS) engine. The system uses a simple command and control grammar. A command and control grammar allows the system designer to specify the commands that the system can recognize which improves speech recognition. The commands the operator use were determined earlier in a Wizard of Oz Study. A subset of the speech commands is shown in Table 1. The operator gives a speech command by first saying the robots name followed by a command. The robots name corresponds to a color for easier recall and to also link the name to the color in the radar. The commands can be given to a single robot or all the robots at any given time. 38 Speech Command Action [Robot Name | All robots] start searching Blue Robot start searching Robot(s) moves forward periodically spinning 360 degrees [Robot Name | All robots] go forward Blue Robot go forward Similar to search but the robot(s) does not periodically spin 360 degrees only moves forward. [Robot Name | All robots] turn [direction] White Robot turn left Robot turns left or right 90 degrees [Robot Name | All robots] stop Pink robot stop Robot stops moving Table 1: Speech commands Quite often the keyboard and mouse controls are done as an afterthought. In other interfaces the operator has to use the mouse to issue commands. In this interface the interaction takes special consideration into the design of the keyboard and mouse controls. The keyboard controls are based on a video game control scheme called the inverted T. The inverted T control allows the operator to control the robot using the W, A, S, and D keys. The ?W? key is pressed and held down to move forward, ?A? turns left, ?D? turns right and ?S? moves the robot backward Table 2. The robot stops moving when the operator releases the key. This is a popular control scheme within video games. It allows the operator to use their left hand to hit the appropriate movement keys and still keep their right hand on the mouse. In the Smart Viewer the keys are used when the operator needs to take manual control of a robot. To take manual control of a robot the operator has to first select the robot to control. In the Monitor and Focused View this is 39 done by single left clicking on the robots live video. In the Single Window interface the operator can only manually control the robot that is in the main window. Key Action W Move Forward A Turn Left S Move Backward D Turn Right Table 2: Manual Control The next section discusses the design of the experiment and experiment results. The experiment explores how effective the aforementioned interfaces perform. The experiment is a comparative analysis between all three interfaces. 40 4 IMPLEMENTATION RESULTS In the absence of experimentation, Smart Viewer design is simply theory. This theory makes significant contributions to the fields of Human Robot Interaction, Intelligent User Interfaces and Sensor Processing. However, these contributions must be supported via experimentation. Based on the approaches outlined in the previous chapter and upon the completion of implementation of the Smart Viewer system, a formal experiment was conducted to validate the system. The objective of this evaluation focuses on the system performance, with respect to task completion, task completion rate, mental workload, and user satisfaction. Statistical methods such as ANOVA and repeated measures ANOVA were used to evaluate and analyze the experimental results. This chapter reports the experiment with Smart Viewer Focus and Monitor Window and the Single Window interface. Section 4.1 details the goals of this experiment including the benchmark for the success of the Smart Viewer interfaces. The experiment settings, participants and procedure are specified in Section 4.2. Section 4.3 describes the data collection methods. Finally, the experimental results are described and discussed in Section 4.4. 41 4.1 Experiment Design The goal of this experiment is to evaluate the two Smart Viewer interfaces with respect to their ability to assist an operator in controlling a robot team to complete a task. A positive outcome would be for the Smart Viewer interfaces to assist the operator in completing the task in less time with a higher accuracy when compared to the single window interface. This should be accomplished while managing the workload in such a way that it does not drastically increase when compared to the single window interface. The Smart Viewer interfaces should be able to scale without decreasing the operator?s ability to complete the assigned task. Finally the Smart Viewer interfaces should rank higher in user satisfaction than the single window interface. The main data points used for evaluation are intervention response time, task completion rate, NASA TLX (Human Performance Research Group NASA Ames Research Center) and the Holistic Usability Measurement. However before any experiment can be performed, the correct hardware and software must be in place to support the experiment. 4.2 Experimental Setting 4.2.1 Materials The study was conducted in two separate rooms in the Shelby building at Auburn University. In one room was a Dell Inspiron. In the other was a custom built system. Each system was equipped with a headset, mouse and keyboard. There was also a separate machine that was used to record the screen for future analysis. The following software was also installed on each experiment machine for this study. 42 ? .NET framework 3.5 ? Microsoft Robotics Development Studio (MRDS) ? SQL Server Express 4.2.2 Tasks Each participant performed three experiments with one of the three interfaces. The participants were assigned 4, 5 and 6 robots to use in each experiment. Using one of the three interfaces, Monitor, Focus or Single the participant was tasked with using their robot team to explore a simulated environment with the objective of finding and disarming all three bombs. Once a participant found a bomb they had to disarm the bomb. To disarm the bomb the participant needed two robots to be within close proximity of the bomb to disarm the bomb. To perform these tasks two simulated environments were created. 4.2.3 Environment Using MRDS two simulated environments were created, a training environment and an experiment environment. The training environment is smaller than the experiment environment. Additionally, the training environment only has three robots Figure 16. The experiment environment was designed to accommodate a larger set of robots. The experiment environment contained 4, 5 or 6 robots depending on the experiment Figure 17. 43 Figure 16: Training Environment Figure 17: Experiment Environment 44 4.2.4 Participants and Procedure There were 99 participants in this study. The participants did not need any special skill to participate in this study. All participants were Auburn University students. All materials for the study were provided to the participants. The participants did not need to bring anything to the study. To reduce the causal effects of other factors, the following controls were applied: 1. Each participant was randomly assigned to one of the three interfaces (Monitor, Focused or Single Window) 2. Within each interface group the participant was randomly assigned a robot set. There were three robot team sets (4 robots, 5 robots and 6 robots). Each participant performed the experiment with each robot set. The order in which each robot set was assigned to the participant was random. For example a participant would complete the tasks with 4 robots followed by 5 robots and finally 6 robots or with 5 robots, then 4 robots and finally 6 robots. 3. The participants were given the same training 4. The participants worked in the same simulated environments 5. The delay time for each participant before starting the survey was the same. The pre-experiment survey was started upon arrival into the experiment room. The NASA-TLX was administered after each experiment. The post-experiment survey was started immediately after the participants had finished their final experiment. The following procedure was followed for each of the experiments: 1. Each participant read and agreed to an Informed Consent form. 45 2. Participants completed a pre-questionnaire for demographic information. 3. Each participant was randomly assigned to one of three interface groups, Monitor, Focused or Single Window interface. 4. The participant completed a training simulation. In the training simulation the proctor would explain the interface components and the multimodal interactions. The proctor would then assist the participant in finding and disarming a bomb. While assisting the participant the proctor would reiterate through demonstration the interface components and the multimodal interactions. Once the participant found and disarmed the first bomb with the proctor?s assistance they were then tasked with finding and disarming a second bomb. Once they found and disarmed the second bomb the training was completed. 5. Each participant completed the experiment three times using 4, 5 and 6 robots. The order of the robots used as previously explained was randomly assigned. 6. After each experiment the participant completed the NASA TLX 7. Upon finishing the last experiment the participant completed a post questionnaire. 4.2.5 Data Collection To achieve the objectives of the experiment, the following data was measured and collected. The following objective measurements were recorded by the system during the experiments. ? Intervention response time, defined as the elapsed time between a robot request for assistance and when the operator assigned a new task to the robot. 46 ? Active time, defined as the amount of time that a robot was moving. ? Manual time, defined as the amount of time the robot was manually controlled. This is when the operator uses the keyboard. ? Task completion rate, which is the number of bombs disarmed by the participant. ? Task completion time, which is the amount of time it took for the participant to find and diffuse all three bombs. The following are objective measure calculated after the study. ? Mental workload, measured using NASA-TLX which is a tool that uses subjective measures to determine the mental workload of the operator (Hart and L. 1988). ? User satisfaction, using the Holistic Usability Measure (HUM). Subjective measures were also collected through a pre and post questionnaire. 4.2.6 Pre-Experiment Questionnaire The pre-experiment questionnaire gathered general and demographic information about the participants. Data gathered included such general identifiers as age, gender and major. Additional questions ascertained the participant?s familiarity with computers and computer video games. It posed questions on how long they had used a computer, and how often. Computer video game experience such as have they played video games, how often and their experience with specific types of computer games. Details of the pre- experiment questionnaire can be found in Appendix A. 47 4.2.7 Performance Data and User Observations All performance data was collected via the system. The system recorded all objective data into a SQL Server database for later analysis. In addition to the performance data, informal and formal user observations were collected. 4.2.8 Post-Experiment Questionnaire The post-experiment questionnaire was designed to gather information about how the participants assessed the system. It gathered overall participant ratings using six bi- polar rating scales. The second part of the questionnaire included a series of Likert-type scales where participants rated their reactions to the system via statements concerning whether they found the medium easy to use, did they know what to do, etc. Details of the post-experiment questionnaire can be found in Appendix B. 4.3 Results and Discussion This section summarizes and discusses the results from the empirical comparison of the interfaces (Monitor, Focused and Single), including both quantitative and qualitative data. A summary of the participants? background obtained from the pre- experiment questionnaire will be presented first. This will be followed by the analyses of quantitative data collected during the experiment task with respect to task completion time, task completion rate and mental workload. A separate section will contain a comparison of participants? reactions to the three interfaces. 48 4.3.1 Performance and Data Findings The main purpose of this research is to show an alternative interface design that improves the operator?s ability to manage multiple robots. To validate this research, results from ability to complete the task (find and diffuse all 3 bombs), time to complete the task and the mental workload were explored. This section explores the statistical results. There were 99 participants for this study who performed a total of 287 experiments. Of the 99 participants 73% were male and 27% were female. At 97% almost all of the participants were between the ages of 19-25. Years of computer use and average hours of computer use were also recorded with 68% of the participants having used a computer for over 8 years and 75% of the participants averaging 1 ? 4 hours of computer use in a day. Table 3, Table 4, and Table 5 highlight the aforementioned information. Measurements Number of participants 99 Number of experiments 287 Male 73% Female 27% Age 19-25 97% English as native language 94% Table 3: Demographic information Number of years using a computer 1 year 2 - 4 years 5 - 7 years > 8 years Percent of participants 1% 8% 23% 68% Table 4: Years of computer usage Average hour of computer use in a day 1 hour 2 - 4 hours 5 - 7 hours > 8 hours Percent of participants 10% 65% 22% 3% Table 5: Average daily computer use 49 Table 6 shows the overall task completion rate for each interface. The Monitor Window interface had the highest overall task completion rate at 91% followed by the Focused Window interface and last was the Single Window interface. This suggests that overall regardless of the number of robots the participants that used the Smart Viewer interfaces were able to find all three bombs more often. Focused Monitor Single 3 Bombs Found 88 91 76 Total Runs 97 100 89 Task Completion 91% 91% 85% Table 6: Overall Task Completion rate per interface Table 7, Table 8 and Table 9 breaks down Table 6 and explores how well each interface performed per number of robots. When the participant was controlling 4 robots the Monitor Window interface had the highest task completion rate of 97%, followed by the Focused Window and finally the Single Window interface. The same occurred with 6 robots in which the Monitor Window was once again better at completing the task followed by the Focused Window and finally the Single Window. When using 5 robots the results are slightly different. The Single Window interface had the best task completion rate of 90% followed by the Smart Viewer interfaces. Overall all participants had a more difficult time completing the task regardless of interface when using 5 robots. Whereas the Monitor Window was the best with 4 and 6 robots when using 5 robots it performed poorly with the Single Window being better followed by the Focused 50 Window. Although the Monitor Window was not as efficient at completing the task with 5 robots overall it had the highest completion rate. In addition to understanding which interface helped the operator to complete the task of finding and diffusing all three bombs it is also important to know which interfaces helped the operator to complete the task in the least amount of time. 4 Robots Focused Monitor Single 3 Bombs Found 30 33 26 Total Runs 33 34 30 Task Completion 91% 97% 87% Table 7: Task completion rate with 4 robots per interface 5 Robots Focused Monitor Single 3 Bombs Found 28 26 26 Total Runs 32 33 29 Task Completion 88% 79% 90% Table 8: Task completion rate with 5 robots per interface 6 Robots Focused Monitor Single 3 Bombs Found 30 32 24 Total Runs 32 33 30 Task Completion 93.75% 97% 80.00% Table 9: Task completion rate with 6 robots per interface Table 10 shows the descriptive statistics for the amount of time to complete the task between interfaces when the operator used 4, 5 and 6 robots. As can be seen in 51 Table 10 the Smart Viewer interfaces had the best average time in completing the task. The Monitor Window had the best average time when using 5 and 6 robots and second best time when using 4 robots. The Focused Window interface had the best average time when using 4 robots and the second best average when using 6 robots; however it had the slowest average time to complete the task when using 5 robots. The Single Window interface had the second best average when using 5 robots but had the slowest task completion time average with 4 and 6 robots. A graphical representation of Table 10 is shown in Figure 18. The next set of tables further explores how each interface compares to one another when using a specific number of robots. Number Of Robots Interface Mean Std. Dev Max 95% Confidence Interval Min Lower Bound Upper Bound 4 Focused 512.911 161.558 265.84 1002.41 452.584 573.238 Monitor 531.602 180.285 293.59 1042.42 467.722 595.574 Single 547.542 155.663 337.55 944.31 484.669 610.416 5 Focused 666.911 218.840 192.92 1101.84 582.054 751.769 Monitor 640.654 224.497 323.25 1140.66 549.977 731.330 Single 648.027 188.442 341.80 1035.64 571.994 724.059 6 Focused 614.466 183.512 320.92 1057.30 545.941 682.990 Monitor 608.976 190.568 269.56 1064.88 540.269 677.683 Single 639.803 223.055 238.80 1069.39 577.500 661.488 Table 10: Between interfaces average time in seconds to complete task 52 Figure 18: Average Time Too Complete Task Table 11 looks at the average time to complete the task when using 4 robots for each interface. When the operator used 4 robots the Smart Viewer Focus Window had the best mean task completion time followed by Monitor Window. However there was no statistical significance between the times when using a one way ANOVA [F (2,89) = .034, p = 967]. Interface Mean Std. Deviation N Focused 512.911 161.558 30 Monitor 531.602 180.285 33 Single 547.542 155.663 26 Total 89 Table 11: Mean task completion time in seconds between interface comparison when using 4 robots 53 To investigating the performance of each interface using 5 robots it is necessary to see if there was any statistical significance in the difference between the three interfaces. Table 12 highlights the mean task completion time between interfaces in comparison when the participant used 5 robots. Interface Mean Std. Deviation N Focused 666.911 218.840 28 Monitor 640.654 224.497 26 Single 648.027 188.442 26 Total 80 Table 12: Mean task completion time in seconds between interface comparison when using 5 robots When the operator used 5 robots the Smart Viewer Monitor Window had the best mean task completion time followed by Single Window than Focused Window. However there was no statistical significance between the times when using a one way ANOVA [F (2,77) = .112, p = .894]. Table 13 highlights the mean task completion time between interfaces comparison with 6 robots. When the operator used 6 robots the Monitor Window had the best mean task completion time followed by Single Window than Focused Window. However there was no statistical significance between the times when using a one way ANOVA [F (2,82) = .511, p = .894]. 54 Interface Mean Std. Deviation N Focused 614.466 183.512 30 Monitor 608.976 190.568 32 Single 639.803 223.055 24 Total 86 Table 13: Mean task completion time in seconds for between interface comparison when using 6 robots The hypothesis stated was that the Smart Viewer interfaces would be significantly faster than the single window interface, specifically the monitor window would on average have the best time for 4, 5 and 6 robots. As can be seen in the tables the Smart Viewer interfaces were faster with the Monitor interface being fastest with 5 and 6 robots and the Focused interface being faster with 4 robots. An interesting fact is that overall 4 robots had the fastest time followed by 6 robots then 5 robots. This is interesting in that you would expect that with more robots the times would be faster however this is not exactly the case which can imply that the number of robots is having an adverse affect on the operator. Another interesting observation is that 6 robots had a lower average task completion time when compared to 5 robots. This could be due to workload having an adverse affect on the operator when using 5 and 6 robots but with 6 robots more area is covered and bombs are found faster. Workload for each interface is explored later in the chapter. The Smart Viewer interfaces were faster in completing the task, however, the difference was not statistically significantly. This means that the null hypothesis which states the difference in the times is due to chance cannot be discounted. Although the null hypotheses cannot be discounted it does not mean that the results are due to chance; 55 only that it is one possibility. As such what is required is further research and analysis of the data. Additionally, the Smart Viewer interfaces had the best mean task completion times for each set of robots although there was not a statistically significant difference. This does show however, that the interfaces that display multiple live video are a viable alternative to the Single Window interface. Additionally, it shows that the operator was able to manage the robots even with all the live videos being shown. The multiple live videos did not hamper the operator?s ability to complete the task. Between interfaces this research suggests the Monitor Window and Focused Window interfaces are viable alternatives. It is also necessary to explore within each interface how well they performed in helping the operator control a varying number of robots. This information allows for understanding in how the interfaces scale. Ideally the interfaces would accommodate the operator in such a way that there is not a significant difference in the amount of time it takes to complete the task regardless of the number of robots. Table 14 shows the descriptive statistics for task completion time within each interface. Figure 19 is a graphical representation of Table 14. Interface Number Of Robots Mean Std. Dev 95% Confidence Interval Lower Bound Upper Bound Focused 4 512.911 161.558 452.584 573.238 5 666.911 218.84 582.054 751.769 6 614.466 183.512 545.941 682.99 Monitor 4 531.602 180.285 467.722 595.574 5 640.654 224.497 549.977 731.33 6 608.976 190.568 540.269 677.683 Single 4 547.542 155.663 484.669 610.416 5 648.027 188.442 571.994 724.059 6 639.803 223.055 577.5 661.488 Table 14: Within interfaces mean task completion time in seconds to find all 3 bombs 56 Figure 19: Mean task completion time within each interface The following tables further analyze each interface and how well they performed in assisting the operator in controlling 4, 5 and 6 robots. Table 15 shows the mean task completion time in seconds between 4, 5 and 6 robots when the operator used the Smart Viewer Focused Window interface. 57 Mean Std. Deviation N Focused 4 512.911 161.558 30 Focused 5 666.911 218.84 28 Focused 6 614.466 183.512 30 Table 15: Within Focused Window Comparison As already stated when using the Focused Window the operator was able to complete the task on average faster with 4 robots, followed by 6 robots and finally 5 robots. Using a repeated measures ANOVA the difference in the mean times is not statistically significant [F (2, 54) = 2.169, p = .124 ] Table 16 shows the mean time comparisons between 4, 5 and 6 robots when using the Monitor Window interface. Mean Std. Deviation N Monitor 4 531.602 180.285 33 Monitor 5 640.654 224.497 26 Monitor 6 608.976 190.568 32 Table 16: Within Monitor Window Comparison When using the Monitor Window the operator was able to complete the task on average faster with 4 robots, followed by 6 robots and finally 5 robots. Using a repeated measure ANOVA the difference in the mean times was not statistically significant [F (2, 50) = 1.192, p = .312]. Table 17 shows the mean time comparisons between 4, 5 and 6 robots when using the Single Window interface. When using the Single Window the operator was able to 58 complete the task on average faster with 4 robots, followed by 6 robots and finally 5 robots. Using a repeated measure ANOVA the difference in the mean times is not statistically significant [F (2, 46) = 2.25, p = .121]. Mean Std. Deviation N Single 4 547.542 155.663 26 Single 5 648.027 188.442 26 Single 6 639.803 223.055 24 Table 17: Within Single Window Comparison The previously stated hypothesis was that the Smart Viewer interfaces would improve the operator?s ability to control an increasing number of robots. As such as the number of robots increase the amount of time required to complete the task would decrease. As can be seen there was no statistical significance within the interfaces between task completion times. Although the participants completed the task faster on average when using 4 robots they did not do it statistically faster. This indicates that the Smart Viewer interfaces are fairly stable at handling 4, 5 and 6 robots because as the number of robots change the time to complete the task did not have a statically significant change. This is a positive note for the ability of the Smart Viewer interfaces to handle increasing number of robots. Of the Smart Viewer Interfaces, the data suggests that the Monitor Window interface is the most stable. For the sake of completion the data was further explored to see if there was any potential correlation between the number of robots and the task completion time. This will show if these two variables have an effect on each other. A scatterplot is used to determine graphically if there is a correlation between the number of robots and the time to find all three bombs. Figure 20 shows that there is a 59 slight positive correlation between the number of robots and the task completion rate when using the Single Window interface. To determine if it is a statistically significant correlation a Pearson?s Correlation test was used Table 18. Figure 20: Single Window Scatterplot Correlation Table 18 shows that we cannot support the hypothesis that task completion time correlates significantly with the number of robots [r(76) = .198, p = .086]. Because there is not a statistically significant correlation between number of robots and time it is not necessary to perform a regression analysis. Scatterplots and Pearson?s correlation were also performed on Focus Window and Monitor Window to see if there was a significant correlation between number of robots and time. 60 Correlations NumRobots Time NumRobots Pearson Correlation 1 .198 Sig. (2-tailed) .086 N 76 76 Time Pearson Correlation .198 1 Sig. (2-tailed) .086 N 76 76 Table 18: Pearson Correlation of Single Window Interface Figure 21 shows that there is a slight positive correlation between the number of robots and the task completion rate when using the Monitor Window interface. To determine if it is a statistically significant correlation a Pearson?s Correlation test was performed. Figure 21: Monitor Window Scatterplot Correlation 61 Table 19 shows that we cannot support the hypothesis of task completion time correlates significantly with number of robots [r(91) = .165, p = .118]. Once again this is not a statistically significant correlation between number of robots and time. It is not necessary to perform a regression analysis. Correlations NumRobots Time NumRobots Pearson Correlation 1 .165 Sig. (2-tailed) .118 N 91 91 Time Pearson Correlation .165 1 Sig. (2-tailed) .118 N 91 91 Table 19: Pearson Correlation of Monitor Window Interface Figure 22 shows that there is a slight positive correlation between the number of robots and the task completion rate when using the Focus Window interface. To determine if it is a statistically significant correlation a Pearson?s correlation test was performed. 62 Figure 22: Focus Window Scatterplot Correlation Table 20 shows that we can support the hypothesis of Task Completion Time correlates significantly with Number of Robots r(88) = .214, p = 0.045. Pearson?s correlation shows that there is a statistically significant correlation between number of robots and task completion time as such we can reject the null hypothesis that the correlation is due to chance. A regression analysis was performed to determine the equation for predicting the task completion time as the number of robots increased Table 21. 63 Correlations NumRobots Time NumRobots Pearson Correlation 1 .214* Sig. (2-tailed) .045 N 88 88 Time Pearson Correlation .214* 1 Sig. (2-tailed) .045 N 88 88 *. Correlation is significant at the 0.05 level (2-tailed). Table 20: Pearson Correlation of Focus Window Interface Model Summary Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .214a .046 .035 193.80022 .046 4.119 1 86 .045 a. Predictors: (Constant), NumRobots Coefficientsa Model Unstandardized Coefficients Standardized Coefficients t Sig. B Std. Error Beta 1 (Constant) 342.645 126.792 2.702 .008 NumRobots 50.777 25.020 .214 2.030 .045 a. Dependent Variable: Time Table 21: Focus Window Regression Analysis The regression analysis gives the following equation for predicting the task completion time with the number robots where x = number of robots and y = task completion rate. 64 Y=50.777(x) + 342.645 The above statistical analysis has shown that there is no correlation between number of robots and task completion time when using the Monitor and Single Window interface. The lack of correlation highlights that these interfaces are fairly stable and that as the number of robots changes it did not affect the task completion time. However, the Focus Window showed a statistically significant weak positive correlation between number of robots and task completion time. A regression analysis was performed to determine the equation that would allow for the estimation of time based on the number of robots. Because this is a fairly weak correlation r = .246 with only a p =.045 significance it does not have a large affect on the ability of the interface to scale. It does however allow for the prediction of task completion time based on the number of robots. Finally this research explores what affects each interface has on the mental workload of the operator, if it is possible to predict the mental workload of the operator and if there is any correlation between workload and task completion rate. Mental workload is the mental cost required to complete a task or set of tasks. The mental workload of the participants was measured using the NASA TLX which is a subjective workload assessment tool that allows users to perform subjective workload assessments on participants working with various human-machine systems. It is a multi- dimensional rating procedure that derives an overall workload score based on a weighted average of ratings on six subscales. These subscales are Mental Demands, Physical Demands, Temporal Demands, Own Performance, Effort and Frustration. Mental Demand answers the question how mentally demanding was the task(s). Physical 65 Demand answers the question how physically demanding was the task. Temporal Demand answers the question how hurried or rushed was the pace. Performance answers the question how successful were you in completing the task. Effort answers the question how hard did you have to work to accomplish your level of performance. Frustration answers the questions how insecure, discouraged, irritated, stressed and annoyed were you. The TLX has been used to assess workload in various human-machine environments such as aircraft cockpits, command, control, and communication (C3) workstations. The mental workload was measured within each interface to determine if the interfaces helped in minimizing the mental workload of the operator as the number of robots changed. The following hypothesis was applied. The Smart Viewer interfaces would not have a statistically significant increase in mental workload when increasing the number of robots. The following tables summarize the 6 subscale factors and the total workload on the operator within each interface when the operator controlled 4, 5 and 6 robots. Table 22 shows the descriptive statistics for the operator?s mental workload when using the Focused Window interface. The total workload was lowest when the operator used 5 robots with an average of 34.01 followed by 4 robots and finally 6 robots. The difference between the total workloads was not statistically significant when using a repeated measure ANOVA, [F(2, 54) = .762, p > .05] as such we cannot support the hypothesis that an increase in the number of robots being controlled has a significant increase in workload. Because of this, we can conclude that once again the Focus Window interface is fairly stable in allowing the operator to control an increase in the number of robots without a significant increase in perceived workload when using the interface. 66 Focus Window 4 Robots 5 Robots 6 Robots Measurement Mean Std. Dev Mean Std. Dev Mean Std. Dev Mental Demand 41.17 19.9 38.93 24.51 44.83 20.23 Physical Demand 22.33 23.63 22.5 21.02 28.33 24.79 Temporal Demand 35.67 24.49 31.79 21.4 40.17 22.3 Performance 20.5 20.44 26.96 24.09 28 22.92 Effort 40 22.24 38 24.58 41.83 21.74 Frustration 28.83 21.44 29.29 27.38 29.5 21.43 Total Workload 34.23 18.11 34.01 21.08 38.16 19.09 Table 22: 3 bombs found mean TLX Focus Window descriptive statistics Looking at the Monitor Window the total workload was lowest for 4 robots followed by 5 then 6 robots Table 23. The difference between the total workloads was not statistically significant when using a repeated measure ANOVA, [F(2, 62) = 1.140, p > .05]. As such we cannot support the hypothesis that an increase in the number of robots being controlled has a significant increase in workload. Once again this shows the Monitor Window interface as being fairly stable in allowing the operator to control an increased number of robots without a statistically significant increase in mental workload when using the Monitor Window interface. Monitor Window 4 Robots 5 Robots 6 Robots Measurement Mean Std. Dev Mean Std. Dev Mean Std. Dev Mental Demand 46.36 22.37 47.2 21.27 54.84 17.00 Physical Demand 20.15 20.37 28.8 23.77 22.58 22.32 Temporal Demand 34.84 21.67 39.00 21.6 36.77 21.04 Performance 23.03 21.79 27.2 22.96 26.29 22.88 Effort 34.09 18.81 39.6 20.61 43.54 17.18 Frustration 31.36 23.56 27.8 18.03 35.81 24.22 Total Workload 37.44 15.14 38.8 17.91 43.36 12.81 Table 23: 3 bombs found mean TLX Monitor Window descriptive statistics 67 Finally, we have added the results from the Single Window interface Table 24. In the case of the Single Window interface the average total workload was lowest when using 4 robots followed by 5 robots then 6 robots. However once again the difference between the total workloads was not statistically significant when using a repeated measure ANOVA, [F(2, 58) = .793, p > .05], as such we cannot support the hypothesis that an increase in the number of robots being controlled has a significant increase in workload when using the Single Window interface. Single Window 4 Robots 5 Robots 6 Robots Measurement Mean Std. Dev Mean Std. Dev Mean Std. Dev Mental Demand 41.00 19.9 44.4 18.84 45.45 21.43 Physical Demand 18.00 16.44 20.8 14.38 24.1 22.18 Temporal Demand 28.6 18.11 34.8 15.38 34.09 21.08 Performance 24.2 20.85 35.2 25.6 21.36 18.91 Effort 32.00 15.41 34.8 19.76 32.73 18.81 Frustration 29.6 19.57 34.8 18.73 34.32 20.2 Total Workload 46.36 22.37 47.20 21.27 54.84 17.00 Table 24: 3 bombs found mean TLX Single Window descriptive statistics The workload measurements have shown that the Smart Viewer interfaces are helpful in managing the mental workload of the operator. This is supported by the lack of statistically significant change in the mental workload of the operator as the number of robots change. As the number of robots increase then the operator is able to manage the increased mental workload through the use of the Smart Viewer interfaces. As part of the research questions this work also explores any potential correlation between total mental workload and the number of robots and workload and time to complete the task. Figure 23 shows a potentially weak positive correlation between the number of robots and the total mental workload when the participant used the Single Window 68 interface. Pearson?s correlation test shows that this correlation is not statistically significant [r(72) = .138, p =.249] as such we cannot support the hypothesis of total mental workload correlates significantly with number of robots when using the Single Window interface Table 25. Figure 23: Single Window Scatterplot Correlation between number of robots and total mental workload 69 Correlations Number Robots Total Mental Workload Number Robots Pearson Correlation 1 .138 Sig. (2-tailed) .249 N 72 72 Total Mental Workload Pearson Correlation .138 1 Sig. (2-tailed) .249 N 72 72 Table 25: Single Window Pearson?s correlation between number of robots and total mental workload Figure 24 shows a potentially weak positive correlation between the number of robots and the total mental workload when the participant used the Monitor Window interface. Pearson?s correlation test shows that this correlation is not statistically significant [r(89) = .164, p = 0.124] as such we cannot support the hypothesis that there is a significant correlation between total mental workload and number of robots when using the Monitor Window interface Table 26. 70 Figure 24: Monitor Window Scatterplot Correlation between number of robots and total mental workload Correlations Number Robots Total Mental Workload Number Robots Pearson Correlation 1 .164 Sig. (2-tailed) .124 N 89 89 Total Mental Workload Pearson Correlation .164 1 Sig. (2-tailed) .124 N 89 89 Table 26: Monitor Window Pearson?s correlation between number of robots and total mental workload Figure 25 shows a potential weak correlation between the number of robots and the total mental workload when the participant used the Focus Window interface. 71 Pearson?s correlation test shows that this correlation is not statistically significant [r(88) = .085, p = .433] as such we cannot support the hypothesis that there is a significant correlation between task completion time and number of robots when using the Focus Window interface Table 27 Figure 25: Focus Window Scatterplot Correlation between number of robots and total mental workload 72 Correlations Number Robots Total Mental Workload Number Robots Pearson Correlation 1 .085 Sig. (2-tailed) .433 N 88 88 Total Mental Workload Pearson Correlation .085 1 Sig. (2-tailed) .433 N 88 88 Table 27: Focus Window Pearson?s correlation between number of robots and total mental workload This data has shown that there is not a statistically significant correlation between the number of robots and total mental workload for the Smart Viewer interfaces. This shows that the Smart Viewer interfaces are assisting the operator in managing their mental workload allowing them to work with varying number of robots without being overly taxed mentally. This research also explored the potential of a correlation between time to complete the task and mental workload. This is done to see if there was a positive, negative or no effect between the two. 73 Figure 26: Focus Window Scatterplot Correlation between time and total mental workload Figure 26 shows a potentially weak positive correlation between the task completion time and total mental workload when the participant used the Focus Window interface. Pearson?s correlation test shows that this correlation is not statistically significant [r(88) = .204, p = .056] as such we cannot support the hypothesis that there is a significant correlation between time and total mental workload Table 28. 74 Correlations Time Total Mental Workload Time Pearson Correlation 1 .204 Sig. (2-tailed) .056 N 88 88 Total Mental Workload Pearson Correlation .204 1 Sig. (2-tailed) .056 N 88 88 Table 28: Focus Window Pearson?s correlation between number of robots and total mental workload Figure 27 shows a potentially weak positive correlation between the time and the total mental workload when the participant used the Monitor Window interface. Pearson?s correlation test shows that this correlation is not statistically significant [r(89) = .199, p = .062] as such we cannot support the hypothesis that there is a significant correlation between time and total mental workload Table 29. 75 Figure 27: Monitor Window Scatterplot Correlation between time and total mental workload Correlations Time TotalMentalWorkload Time Pearson Correlation 1 .199 Sig. (2-tailed) .062 N 89 89 Total Mental Workload Pearson Correlation .199 1 Sig. (2-tailed) .062 N 89 89 Table 29: Monitor Window Pearson?s correlation between time and total mental workload Figure 28 shows a potential weak negative correlation between the time and the total mental workload when the participant used the Single Window interface. Pearson?s 76 correlation test shows that this correlation is not statistically significant [r(72) = -.014, p = .905] as such we cannot support the hypothesis that there is a significant correlation between time and total mental workload Table 30. Figure 28: Single Window Scatterplot Correlation between time and total mental workload 77 Correlations Time Total Mental Workload Time Pearson Correlation 1 -.014 Sig. (2-tailed) .905 N 72 72 Total Mental Workload Pearson Correlation -.014 1 Sig. (2-tailed) .905 N 72 72 Table 30: Single Window Pearson?s correlation between time and total mental workload This data has shown that there is not a statistically significant correlation between task completion time and total mental workload for the Smart Viewer interfaces. As the mental workload of the operator changes the time to complete the task is not affected in any of the interfaces. 78 5 SUMMARY AND CONCLUSION 5.1 Summary The Smart Viewer interface uses a new methodology that allows an operator to view multiple live videos and permits the operator to control robots through keyboard, mouse and speech. The main goal is to provide users with an interface that permits an operator to control multiple robots without overwhelming the operator?s mental workload. Through experimentation the Smart Viewer interfaces have shown that they are a viable alternative. The Smart Viewer interfaces had a higher completion rate (91%) in comparison to the Single Window interface (85%). When looking at each group of robots 4, 5, and 6 it must be noted that the Monitor Window had the lowest task completion rate when using 5 robots however it had the best task completion time with 5 robots. As previously stated there was not a statistically significant difference between the number of robots controlled within each interface. When comparing the Smart Viewer interfaces to the Single Window interface in task completion time the Smart Viewer interfaces had the better task completion time when using 4, 5 and 6 robots specifically when using 4 robots the Focus Window was best and when using 5 or 6 robots the Monitor Window performed best. Although these results were not statistically significant the fact that the Smart Viewer interfaces performed better in all experiments shows that the interface has potential and that further research is needed. Additionally, it was noted that on average the task completion time 79 was best with 4 robots followed by 6 and finally 5 robots regardless of interface. This is interesting in that with 5 robots it took longer on average for the participant to complete the task. As stated previously this may have been due to an increase in mental workload having an adverse affect on the operator when using 5 and 6 robots but with 6 robots more area is covered and bombs are found faster by chance. When looking at the mental workload for each interface for 4, 5 and 6 robots there was not a statistically significant change in the workload. As such this does not account for the difference in task completion time. The change is likely due to the fact that the 6 robots covered the area faster and participants found the bombs faster by accident. To determine how each interface handled an increase in the number of robots used by the operator a within interface analysis was performed. Both Smart Viewer interfaces, Monitor and Focus, performed well in enabling the operator to complete the task regardless of the number of robots. Regardless of the number of robots there was not a statistically significant change in the task completion time within each interface. This shows that the Smart Viewer interfaces are fairly stable and assists the operator in controlling their robot team. This data also shows that the Smart Viewer interfaces were fairly stable in assisting the operator in controlling multiple robots. An increase in the number of robots did not have a statistically significant change in task completion time. This is confirmed with Pearson?s correlation which showed that there was not a statistically significant correlation between the number of robots and task completion time when using the Monitor Window. With the Focus Window there was a statistically significant weak correlation [r(86) = .214]. Even though there was a correlation it is fairly weak as such the Focus Window is still a stable interface and is assisting the 80 operator in managing multiple robots. The data shows that the Monitor Window is able to manage multiple robots slightly better than the Focus Window. Mental Workload was also explored as part of this research. It is crucial for a user interface to assist in managing the mental workload of the operator if the operator needs to control multiple robots. As the number of robots increases so does the mental workload which is seen in the above data. The Smart Viewer interfaces were able to assist the operator in managing their mental workload. When the operator used the Monitor Window or Focus Window interface to control 4, 5, and 6 robots the analysis of the data showed that there was not a statistically significant change in the mental workload. This also shows the stability of the Smart Viewer interfaces and their ability to assist the operator in such a way that the workload did not change significantly when using differing number of robots. When exploring the correlations between the number of robots and the mental workload for the Smart Viewer interfaces showed no relation. The number of robots did not have a statistically significant effect on the mental workload and vice versa. 5.2 Conclusion This research suggests that the Smart Viewer interfaces are a viable alternative to the Single Window interface. The Smart Viewer interfaces had a better task completion rate and better task completion times when using 4, 5 and 6 robots. In addition, the Smart Viewer interfaces were able to manage the increase in number of robots such that there was not a statistically significant difference in the task completion time within each interface when using 4, 5 and 6 robots. Additionally, the Smart Viewer interfaces were 81 able to manage the mental workload in such a way that regardless of the number of robots controlled there was not a statistically significant change. This is in contrast with research done by Trouvain and Adams. In their research they have shown a statistically significant detriment in the ability of the operator to complete the task as the number of robots increase (Trouvain, Schlick et al. 2003; Humphrey, Henk et al. 2007). The reason these interface results contradict earlier research is most likely due to the Smart Viewer interfaces and Single View interface able to use multiple mental resources. Wickens (1988) put forth work that shows that the brain can process disassociate information concurrently. In this seminal work, Wickens states that information is processed through three dichotomies of separate resources. Perceptual and cognitive task use separate resources than selection and detection. This means that while thinking on one topic you can perform an action on a separate topic. For example, you can think about what will be for dinner while manually controlling a robot. Wickens notes that spatial tasks use different resources than verbal/linguistic tasks. For example, you can move a robot from one room to another while carrying on a conversation. Finally Wickens notes that auditory perception uses separate resources than visual perception (Wickens 1980; Wickens and Liu 1988). For example you can hear a set of instructions while doing object recognition. In more recent research, Wickens adds a fourth dimension, visual channel. With visual channels resources are allocated separately for focal vision and ambient vision. Focal vision deals primarily with object recognition, reading text and recognizing symbols, whereas ambient vision focuses on perception (Wickens 2002; Horrey, Wickens et al. 2006). An example would be the ability to recognize the image of a skunk while at the same time noticing movement off to the side. Each of these 82 dichotomies relate to the interface design for the Smart Viewer interface and the single window interface. The following states how each of the aforementioned resource allocations affect the three interfaces designed in this work. ? Perception and Cognition vs. Selection and Execution ? the Smart Viewer interfaces and Single window interface are designed such that the operator can be thinking about the location of the nearest robot while giving commands to a separate robot. ? Speech vs. Control ? the Smart Viewer interfaces and Single window interface allow for the operator to receive information and perform tasks using speech while at the same time taking manual control of the robots. These two tasks can be accomplished by the brain using separate resources. ? Auditory vs. Visual perception - the Smart Viewer interfaces and Single window interface are designed to allow the operator to receive information using both visual and auditory resources simultaneously. ? Focal vs. Ambient vision - this information supports the ability of the operator to focus on one robot while noticing movement and orientations in the other robots. This also supports the ability of the operator to focus on a robot while noticing the visual color changes signaling that another robot has information for the operator. Further research has focused exclusively on auditory and visual processing which further decomposes the auditory and visual into sub processes of spatial positional, spatial quantitative, auditory linguistic, and auditory emotional resources (Boles and Law 1998; Boles 2002; Boles, Bursk et al. 2007). In all there are 14 separate perceptual resources (Wickens and Christopher 2008). 83 The Monitor, Focus and Single View interfaces fully utilize the multiple resources available to the operator. This could be the reason why the between interface results are not statistically significant. The ability of the three interfaces to allocate resources onto different mental tracks may have a greater effect on ability to complete the task than having interfaces that display multiple live video feeds. Another reason the between interface results are not statistically significant could be from the actual design of the simulated environment. An issue of this study was the simplicity of the simulated environment. The environment did not contain any hazards; difficult terrain etc. as such the potential benefits of the Smart Viewer interface was minimized. A more complex environment would allow for more of a mental load on the operators. More research is needed to confirm the true potential of the Smart Viewer interfaces. 5.3 Contributions This research has made the following contributions to the field of Human Robot Interaction: ? Identified and addressed the limitations of current multi-robot interfaces. o Research into the improvement of multi-robot interfaces is ongoing. Advancements in technology are being recognized as potential solutions to the current problems. However, attention must be directed towards making these interfaces in such a way that they leverage the strengths of robots and people. In addition more focus is needed on improving the interactions between man and robot. 84 ? Introduced new interface designs for controlling multiple robots that improves on existing interface designs. o The Smart Viewer interfaces provide a multimodal interactive interface that allows an operator to view multiple live videos enhancing their ability coordinate and control multiple robots. 5.4 Directions for Future Research There are a number of questions remaining that warrant continued research. 1. This study was done in a simulated environment. A simulated environment has many more controls and does not adequately resemble a real environment. As such it is necessary to perform the same study using real robots in a real environment to determine if the results are the same. 2. For this system to truly be of use further research is needed in the area of Situational Awareness. The current system does not provide much assistance to the operator in improving situational awareness. One area that is deficit is in the ability of the operator to know where each robot has searched. In the current state the operator often searches areas that have already been explored. To improve the system it will be necessary to add additional virtual elements. In this case it would be a virtual bread crumb trail that would augment the live video feed. The virtual bread crumb trail would show the operator what areas have been searched and what areas have not been searched. 3. The simulated environment employed here was fairly simple in that there were no difficult areas or traps that the operator and robots had to be aware of. Future 85 research would have complex environments to further stress test the mental workload of the operator. 4. Further research is needed to explore how much of an affect multimodality has on the operators ability to control multiple robots (Multiple Resource Theory). 5. Additional research is also needed in exploring the interfaces ability with larger number of robots. 86 BIBLIOGRAPHY Adams, J. A. (2002). Critical Considerations for Human-Robot Interface Development. AAAI Fall Symposium: Human Robot Interaction Technical Report: 1-8. Baker, M., R. Casey, et al. (2004). Improved Interfaces for Human-Robot Interaction in Urban Search and Rescue. Proceedings of the IEEE Conference on Systems, Man and Cybernetics. Baker, M., R. Casey, et al. (2004). Improved Interfaces for Human-Robot Interaction in Urban Search and Rescue. In 2004 IEEE International Conference on Systems, Man and Cybernetics. Baumann, J. M., S. I. Blanksteen, et al. (1997). Recognition of Descending Aircraft in a Perspective Naval Combat Display. Virtual Environments. Bemis, S. V., J. L. Reeds, et al. (1988). "Operator Performance as a Function of Type of Display: Conventional Versus Perspective " Human Factors 30(2): 163-169. Boles, D. B. (2002). "Lateralized spatial processes and their lexical implications." Neuropsychologia 40(2125): 2135. Boles, D. B., J. H. Bursk, et al. (2007). "Predicting dual-task performance with the multiple resource questionnaire." Human Factors 49(32-45). Boles, D. B. and M. B. Law (1998). "Asimultaneous task comparison of differentiated and undifferentiated hemispheric resource theories." Journal of Experimental Psychology: Human Perception and Performance 24: 204-215. Bruemmer, D., D. Few, et al. (2005). ??Turn Off the Television!? Robotic Exploration Experiments with a 3D Abstracted Map Interface. In Proceedings of the 38th Hawaii International Conference on the System Sciences, Waikoloa Village, Hawaii. Bruemmer, D. J., R. L. Boring, et al. (2004). "I call shotgun!": an evaluation of mixed- initiative control for novice users of a search and rescue robot. IEEE International Conference on Systems, Man and Cybernetics. Bruemmer, D. J., J. L. Marble, et al. (2002). Mutual initiative in human-machine teams IEEE Conference on Human Factors and Power Plants. Bruemmer, D. J., J. L. Marble, et al. (2003). Mixed-Initiative Control for Remote Characterization of Hazardous Environments. Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Hawaii. Burke, J. L., R. R. Murphy, et al. (2004). "Moonlight in Miami: A Field Study of Human?Robot Interaction in the Context of an Urban Search and Rescue Disaster Response Training Exercise." Human Computer Interaction: 85-116. Casper, J. and R. Murphy (2003). "Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center." IEEE Transactions on Systems, Man, and Cybernetics Part B, 33(3): 367-385. 87 Chen, J. Y. C., E. C. Haas, et al. (2006). Human-Robot Interface: Issues in Operator Performance,Interface Design, and Technologies: 100. Daily, M., Y. Cho, et al. (2003). World embedded interfaces for human-robot interaction. In Proc. of the 36th Annual Hawaii International Conference on System Sciences, Hawaii. Drury, J. L., B. Keyes, et al. (2007). LASSOing HRI: Analyzing Situation Awareness in Map-Centric and Video-Centric Interfaces. Proceeding of the ACM/IEEE international conference on Human-robot interaction. Ellis, S. R., M. W. McGreevy, et al. (1987). "Perspective Traffic Display Format and Airline Pilot Traffic Avoidance " Human Factors 29(4): 371-382. Endsley, M. R. (1988). Design and evaluation for situation awareness enhancement. Proceedings of the Human Factors Society, Santa Monica, CA, Human Factors Society. Envarli, I. C. and J. A. Adams (2005). Task lists for human-multiple robot interaction. IEEE International Workshop on Robot and Human Interactive Communication, 2005. ROMAN. Fong, T., C. Thorpe, et al. (2001). Collaboration, Dialogue, and Human-Robot Interaction. 10th International Symposium of Robotics Interaction, Lorne, Victoria, Australia. Girgensohn, A., D. Kimber, et al. (2007). DOTS: Support for Effective Video Surveillance. In Proceedings of the 15th International Conference on Multimedia, Augsburg, Germany Hart, S. and S. L. (1988). "Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research." In: Hancock P, Meshkati N (Eds.). Human Mental Workload: 139-183. Heckel, F., i. Blakely, et al. (2006). The WURDE Robotics Middleware and RIDE Multi- Robot Tele-Operation Interface. In AAAI Mobile Robotics Workshop. Hickox, J. G. and C. D. Wickens (1999). "Effects of elevation angle disparity, complexity and feature type on relating out-of-cockpit field of view to an electronic cartographic map." Experimental Psychology: Applied: 284-301. Hix, D. and R. Hartson (1993). Developing User Interfaces, Wiley Professional Computing. Horrey, W. J., C. D. Wickens, et al. (2006). "Modeling drivers? visual attention allocation while interacting with in-vehicle technologies." Journal of Experimental Psychology: Applied 12(2): 67-86. Human Performance Research Group NASA Ames Research Center NASA TASK LOAD INDEX (TLX) v. 1.0. Moffett Field, California, NASA: 14. Humphrey, C. M., C. Henk, et al. (2007). Assessing the Scalability of a Multiple Robot Interface. International Conference on Human-Robotic Interaction. Washington D.C. Iba, S., C. J. J. Paredis, et al. (2004). Interactive Multi-Modal Robot Programming. 9th International Symposium on Experimental Robotics. John, M. S., M. B. Cowen, et al. (2001). "The Use of 2D and 3D Displays for Shape- Understanding versus Relative-Position Tasks." Human Factor 43(1): 79-98. 88 Jones, H. and M. Snyder (2001). Supervisory Control of Multiple Robots Based On a Real-Time Strategy Game Interaction Paradigm. In IEEE International Conference on Systems, Man, and Cybernetics, Tucson, AZ. Kaminka, G. A. and Y. Elmaliach (2006). Experiments with an ecological interface for monitoring tightly-coordinated robot teams. IEEE International Conference on Robotics and Automation. Lee, J. D., B. Caven, et al. (2001). "Speech-based interaction with invehicle computers: The effect of speech-based e-mail on drivers' attention to the roadway." Human Factors 43: 631-640. Nielsen, C., M. A. Goodrich, et al. (2007). "Ecological Interfaces for Improving Mobile Robot Teleoperation." IEEE Transactions on Robotics 23(5): 927-941. Nielsen, C. W., D. J. Bruemmer, et al. (2006). Mixed-Initiative Interactions for Mobile Robot Search. American Association for Artificial Intelligence Mobile Robot Workshop. Boston, Massachusetts. Nielsen, C. W. and M. A. Goodrich (2006). Comparing the Usefulness of Video and Map Information in Navigation Tasks. Proceeding of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, Salt Lake City, Utah. Nourbakhsh, I. R., K. Sycara, et al. (2005). "Human-Robot Teaming for Search and Rescue." IEEE Pervasive Computing: Mobile and Ubiquitous Systems: 72-78. Parasuraman, R., S. Galster, et al. (2003). Human Control Of Multiple Robots In the RoboFlag Simulation Environment. In IEEE International Conference on Systems, Man and Cybernatics. Perzanowski, D., A. C. Schultz, et al. (1998). Integrating Natural Language and Gesture in a Robotics Domain. In Procedures of IEEE International Symposium Intelligent Control, Piscataway, N.J. Perzanowski, D., A. C. Schultz, et al. (1999). Goal tracking in a natural language interface: towards achieving adjustable autonomy. IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA). Perzanowski, D., A. C. Schultz, et al. (2001). Building a Multimodal Human?Robot Interface. Computer Human Interaction (CHI), Seattle, Washington. Perzanowski, D., A. C. Schultz, et al. (2001 ). "Building a Multimodal Human-Robot Interface." IEEE Intelligent Systems 16(1): 16-21. Savage, D. M., E. N. Wiebe, et al. (2004). Performance of 2D versus 3D Topographic Representation for Different Task Types. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Santa Monica, CA. Scholtz, J. "Evaluation Methods for Human-System Performance of Intelligent Systems." Scholtz, J., B. Antonishek, et al. (2004). Evaluation of Human-robot Interaction in the NIST Reference Search and Rescue Test Arenas. Proceedings of PerMIS ?04 Workshop. Schultz, A. C. and L. E. Parker (2002). Multi-Robot Systems: From Swarms to Intelligent Automata. Smallman, H. S., M. S. John, et al. (2001). "Information Availability in 2D and 3D displays." IEEE Computer Graphics Applications 21(5): 51. Trouvain, B., C. Schlick, et al. (2003). Comparison of a Map- vs. Camera-based User Interface in a Multi-Robot Navigation Task. In Proceedings of the 2003 International Conference on Robotics and Automation. 89 Trouvain, B. and C. M. Schlick (2007). "A Comparative Study of Multimodal Displays for Multirobot Supervisory Control." Human Computer Interaction 13: 184-193. Trouvain, B. A. and C. M. Schlick (2006). Binaural Auditory Information Fusion for Effective Supervisory Control of Mobile Multirobot Systems. 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Heidelberg, Germany. Wagner, A. R., Y. Endo, et al. (2006). Multi-robot User Interface Modeling. In Proceedings of the 8th International Symposium on Distributed Autonomous Robotic Systems, Minneapolis, MN. Wickens, C. D. (1980). The structure of attentional resources, Hillsdale, NJ: Erlbaum. Wickens, C. D. (2002). "Multiple resources and performance prediction." Theoretical Issues in Ergonomics Science 3: 159-177. Wickens, C. D. and D. Christopher (2008). "Multiple Resources and Mental Workload." Human Factors 50: 449 - 455. Wickens, C. D. and Y. Liu (1988). "Codes and modalities in multiple resources: A success and a qualification." Human Factors 30: 599-616. Yanco, H. (2001). Designing Metrics for Comparing the Performance of Robotic Systems in Robot Competitions. Proceedings of the 2001 Performance Metrics for Intelligent Systems (PerMIS) Workshop, Mexico City. Yanco, H. A., J. L. Drury, et al. (2004). "Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition." Human Computer Interaction. Yanco, H. A., J. L. Drury, et al. (2004). "Beyond Usability Evaluation: Analysis of Human?Robot Interaction at a Major Robotics Competition." Human Computer Interaction 19: 117-149. 90 APPENDIX A Pre Survey Questionnaire ID: Label Age: 19-25 26-34 35 and older Gender : Male Female Highest degree obtained Diploma Or Equivalent BA BS MS PhD or equivalent Is Englsh your native or second language Native Language Second Language For approximately how many years have you been using a computer? less than 1 year 2 - 4 years 5 - 7 years 8 or more years On average, how many hours do you use a computer during the course of a day? less than 1 hour 2 - 4 hours 5 - 7 hours 8 or more hours Have you ever played a computer video game? yes no Approximately how many years have you been playing video games? less than 1 year 2 - 4 years 5 - 7 years 8 or more years 91 Approximately how many hours a day do you spend playing video games? less than 1 hour 2 - 4 hours 5 - 7 hours 8 or more hours Have you ever played a PC first person shooter or first person point of view video game such as but not limited to Doom, Quake, Call of Duty, World of Warcraft, Crysis or any similar game? Yes No Are you familiar with using "W", "A", "S", and "D" keys for movement? If you are not familiar with WASD than check "No"?: Yes No In the section below, choose the response that most accurately describes you.: I am computer literate?: Strongly Agree Agree Neutral Disagree Strongly Disagree I am good with computers?: Strongly Agree Agree Neutral Disagree Strongly Disagree I am good at computer video games?: Strongly Agree Agree Neutral Disagree Strongly Disagree I am good at first person shooter video games on the PC?: Strongly Agree Agree Neutral Disagree Strongly Disagree 92 APPENDIX B Post Survey Questionnaire ID: Overall Satisfaction: Terrible ....................................... Wonderful 1 2 3 4 5 Frustrating ....................................... Satisfying 1 2 3 4 5 Not Usable ................................................ Usable 1 2 3 4 5 Difficult ............................................... Easy 1 2 3 4 5 NOTIFICATION Use of color to notify me about information was? Poor ............................................... Excellent 1 2 3 4 5 Amount of information displayed in the notification? Inadequate ........................................... Adequate 1 2 3 4 5 The use of automatic speech to inform me relevant information? Inadequate ........................................... Adequate 1 2 3 4 5 Using the mouse to get information from each robot was? Inadequate ........................................... Adequate 1 2 3 4 5 93 INTERACTION Overall controlling the robot team was? Difficult ........................................... Easy 1 2 3 4 5 Giving tasks to each robot was? Difficult ........................................... Easy 1 2 3 4 5 How easy was the keyboard control for manually controlling a robot? Difficult ........................................... Easy 1 2 3 4 5 How easy was the speech control in controlling your robot team? Difficult ........................................... Easy 1 2 3 4 5 LEARNING Learning to use the system was ? Difficult ........................................... Easy 1 2 3 4 5 The tutorial was helpful? Disagree ........................................... Agree 1 2 3 4 5 How helpfull was your video game experience in helping you to control the robot team? Not Helpfull ........................................ Very Helpfull 1 2 3 4 5 How helpfull was the radar in navigating one robot to another? Not Helpfull ........................................ Very Helpfull 1 2 3 4 5 What is the max number of robots you believe you can control with this interface? ADDITONAL USER COMMENTS: How would you improve this interface to help you control more robots 94 Additional comments/suggestions Comment Comment 95 APPENDIX C NASA-TLX Mental Workload Rankings Definition of Task Demand Factor Mental demand How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving? Physical demand How much physical activity was required (e.g., pushing, pulling, turning, controlling, activating, etc.)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious? Temporal demand How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred? Was the pace slow and leisurely or rapid and frantic? Performance How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals? Frustration level How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Effort How hard did you have to work (mentally and physically) to accomplish your level of performance? 96 For each of the pairs listed below, circle the scale title that represents the more important contributor to workload in the display. Mental Demand or Physical Demand Mental Demand or Temporal Demand Mental Demand or Performance Mental Demand or Effort Mental Demand or Frustration Physical Demand or Temporal Demand Physical Demand or Performance Physical Demand or Effort Physical Demand or Frustration Temporal Demand or Performance Temporal Demand or Frustration Temporal Demand or Effort Performance or Frustration Performance or Effort Frustration or Effort 97 NASA-TLX Mental Workload Rating Scale Low High Mental Demand: How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, looking, searching, etc)? Was the mission easy or demanding, simple or complex, exacting or forgiving? Low High Physical Demand: How much physical activity was required (e.g., pushing, pulling, turning, controlling, activating, etc.)? Was the mission easy or demanding, slow or brisk, slack or strenuous, restful or laborious? Low High Temporal Demand: How much time pressure did you feel due to the rate or pace at which the mission occurred? Was the pace slow and leisurely or rapid and frantic? HighLow Performance: How successful do you think you were in accomplishing the goals of the mission? How satisfied were you with your performance in accomplishing these goals? Low High Effort: How hard did you have to work (mentally and physically) to accomplish your level of performance? Low High Frustration: How discouraged, stressed, irritated, and annoyed versus gratified, relaxed, content, and complacent did you feel during your mission? 98 APPENDIX D Speech Grammar all 99 robots stop robots stop searching robots stop moving robots move forward robots start searching stop move forward start searching 100 101 blue white purple brown pink cream robot 102 search searching start search start searching stop 103 forward straight backward backup turn go 104 left right 105 APPENDIX E Database Schema USE [Dissertation] GO /****** Object: Table [dbo].[TimeStats] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[TimeStats]( [pid] [varchar](50) NULL, [roboName] [int] NULL, [totalInterventionTime] [varchar](50) NULL, [numberOfInterventions] [int] NULL, [minIntervention] [varchar](50) NULL, 106 [maxIntervention] [varchar](50) NULL, [totalActiveTime] [varchar](50) NULL, [totalManualTime] [varchar](50) NULL, [numberOfTimesManualControl] [int] NULL, [minTimeManualControl] [varchar](50) NULL, [maxTimeManualControl] [varchar](50) NULL, [totalBetweenCommandTime] [varchar](50) NULL, [numberOfBetweenCommands] [int] NULL, [minBetweenCommands] [varchar](50) NULL, [maxBetweenCommands] [varchar](50) NULL, [interfaceType] [varchar](50) NULL, [totalTimeAsMainWindow] [varchar](50) NULL ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[TaskLoadScores] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON 107 GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[TaskLoadScores]( [PID] [varchar](10) NOT NULL, [Sub] [int] NULL, [Cond] [int] NULL, [Trial] [int] NULL, [MD] [int] NULL, [PD] [int] NULL, [TD] [int] NULL, [PF] [int] NULL, [EF] [int] NULL, [FR] [int] NULL, [wMD] [int] NULL, [wPD] [int] NULL, [wTD] [int] NULL, 108 [wPF] [int] NULL, [wEF] [int] NULL, [wFR] [int] NULL, [Score] [decimal](3, 1) NULL, CONSTRAINT [PK_TaskLoadScores] PRIMARY KEY CLUSTERED ( [PID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[Statistics] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON 109 GO CREATE TABLE [dbo].[Statistics]( [pid] [varchar](50) NOT NULL, [totalTime] [varchar](50) NULL, [bombsFound] [int] NULL, [totalNumberOfBombs] [int] NULL, [numberOfRobots] [int] NULL, [interfaceType] [varchar](50) NULL, [numberOfTimesWindowChanged] [int] NULL, [numberOfSpokenCommands] [int] NULL, [timeStamp] [varchar](50) NULL, CONSTRAINT [PK_Statistics] PRIMARY KEY CLUSTERED ( [pid] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF 110 GO /****** Object: Table [dbo].[SpeechRecognized] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[SpeechRecognized]( [PID] [varchar](50) NULL, [Recognized] [varchar](50) NULL, [TimeRecorded] [varchar](50) NULL ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[PreSurvey] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON 111 GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[PreSurvey]( [PID] [varchar](50) NOT NULL, [Age] [varchar](50) NULL, [Gender] [char](2) NULL, [HighestDegree] [varchar](50) NULL, [EnglishYourNativeLanguage] [varchar](10) NULL, [YearsOfComputerUse] [varchar](15) NULL, [AverageHoursOfComputerUseInAWeek] [varchar](15) NULL, [EverPlayedAComputerGame] [char](2) NULL, [YearsOfPlayingGames] [varchar](15) NULL, [AverageHoursGamePlayAWeek] [varchar](15) NULL, [FirstPersonShooter] [char](2) NULL, [ComfortableWASD] [char](2) NULL, [ComputerLiterate] [varchar](50) NULL, 112 [GoodWithComputers] [varchar](50) NULL, [GoodAtVideoGames] [varchar](50) NULL, [GoodAtFirstPersonShooter] [varchar](50) NULL, [timeStamp] [varchar](50) NULL, CONSTRAINT [PK_PreSurvey] PRIMARY KEY CLUSTERED ( [PID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[PostSurvey] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON 113 GO CREATE TABLE [dbo].[PostSurvey]( [PID] [varchar](50) NOT NULL, [OverAllSatisficationWonderfulTerrible] [int] NULL, [OverAllSatisficationSatisfyingFrustrating] [int] NULL, [OverAllSatisficationUsableNotUsable] [int] NULL, [OverAllSatisficationDificultEasy] [int] NULL, [NotificationUseOfColorPoorExcellent] [int] NULL, [NotificationInformationDisplayed] [int] NULL, [UseOfSpeechNotification] [int] NULL, [UseOfMouseToGetNotification] [int] NULL, [ControllingTheRobotTeam] [int] NULL, [AssigningTaskToEachRobot] [int] NULL, [EaseOfKeyboardControl] [int] NULL, [EaseOfSpeechControl] [int] NULL, [LearningToUseTheSystem] [int] NULL, [TutorialHelpful] [int] NULL, [UsefulVideoGameExperience] [int] NULL, [AbilityOfRadar] [int] NULL, 114 [MaxNumberYouCanControl] [varchar](3) NULL, [ImproveTheInterface] [varchar](max) NULL, [AdditionalCommentsSuggestions] [varchar](max) NULL, [timeStamp] [varchar](50) NULL, CONSTRAINT [PK_PostSurvey] PRIMARY KEY CLUSTERED ( [PID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[ParticipantRobots] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO 115 SET ANSI_PADDING ON GO CREATE TABLE [dbo].[ParticipantRobots]( [NumRobots] [int] NULL, [pid] [varchar](5) NULL ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO /****** Object: Table [dbo].[LoadValues] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[LoadValues]( [PID] [varchar](5) NOT NULL, [Type] [int] NOT NULL, 116 [Sub] [int] NULL, [Cond] [int] NULL, [Trial] [int] NULL, [MD] [int] NULL, [PD] [int] NULL, [TD] [int] NULL, [PF] [int] NULL, [EF] [int] NULL, [FR] [int] NULL, CONSTRAINT [PK_LoadValues] PRIMARY KEY CLUSTERED ( [PID] ASC, [Type] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO 117 /****** Object: Table [dbo].[InterfaceChoice] Script Date: 04/09/2009 03:03:06 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[InterfaceChoice]( [InterfaceNumber] [int] NULL, [pid] [varchar](5) NULL ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO