AN EXPLANATORY STUDY TO MEASURE THE EFFECT OF AN EXPANDED TARGET INTERFACE FOR A HANDHELD MEDICATION ORDER ENTRY TASK Except where reference is made to the work of others, the work described in this thesis is my own or was done in collaboration with my advisory committee. This thesis does not include proprietary or classified information. _____________________________ Marc Young Certificate of Approval: _________________________ _______________________ William A. Villaume, Co-Chair Bill G. Felkey, Co-Chair Associate Professor Professor Pharmacy Care Systems Pharmacy Care Systems _________________________ _______________________ Salisa Westrick Cheryl D. Seals Assistant Professor Assistant Professor Pharmacy Care Systems Computer Science & Software Engineering ______________________ Stephen L. McFarland Dean Graduate School AN EXPLANATORY STUDY TO MEASURE THE EFFECT OF AN EXPANDED TARGET INTERFACE FOR A HANDHELD MEDICATION ORDER ENTRY TASK Marc Young A Thesis Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment of the Requirements for the Degree of Master of Science Auburn, Alabama December 15, 2006 iii AN EXPLANATORY STUDY TO MEASURE THE EFFECT OF AN EXPANDED TARGET INTERFACE FOR A HANDHELD MEDICATION ORDER ENTRY TASK Marc Young Permission is granted to Auburn University to make copies of this thesis at its discretion, upon the request of individuals or institutions at their expense. The author reserves all publication rights. __________________ Signature of Author __________________ Date of Graduation iv THESIS ABSTRACT AN EXPLANATORY STUDY TO MEASURE THE EFFECT OF AN EXPANDED TARGET INTERFACE FOR A HANDHELD MEDICATION ORDER ENTRY TASK Marc Young Master of Science, December 15, 2006 (B.S. in Pharmacy, The University of Texas, 1995) (PharmD, Idaho State University, 2001) 179 Typed pages Directed by Bill G. Felkey and William A. Villaume This research experiment intended to determine whether an expanded target interface could improve a medication order task entered on a Personal Digital Assistant (PDA) compared to a standard interface. Subjects in this experiment used each of two simulated PDA medication order entry programs. This experiment required subjects to find items on two types of PDA interfaces in order to enter and complete a medication order. A repeated measures crossover design allowed users to experience both interfaces to assess individual responses to each interface. This study measured two primary v objective measures (errors and time) and one subjective measure (workload) within the two interface versions. The expanded target interface performed exactly as the non-expanded interface except for when a subject tapped an item. The tapping and subsequent highlighting of an item onscreen in the expanded interface caused the item and any adjacent items to increase in font size and separate more from surrounding items. The expectation was that the expanded target interface would provide improved performance through reduced error rates and favorable user opinions and would differ from the non-expanded interface in time of order entry. Data collection was divided into two phases: 1. A sixty minute PDA data entry of typed medication orders consisting of five components (patient name, medication name, strength, dose amount and frequency) on one of five matching PDA screens. 2. Completion of a subjective survey for each of the two interfaces and a demographic questionnaire. A total of 113 subjects (43 males, 62 females and 8 unidentified) participated in five experimental data collection sessions. The expanded target interface did not reduce error rate significantly compared to the non-expanded interface. Further, the expanded target with its higher workload ratings compared to the non-expanded interface did not reduce the effort as expected for subjects in order entry tasks. A significant difference in time for order entry and workload ratings between interfaces in general existed, however it was the expanded interface which had a slower order entry time and higher workload ratings versus the non-expanded interface. ACKNOWLEDGEMENTS First, I would like to thank Professor Bill Felkey for his agreeing to accept me as a graduate student on such a short notice without ever meeting me in person. Although I was not sure of what I wanted to study, he constantly presented me with new possibilities and always thought of my future success both in and out of the Navy when opportunities arose. I would also like to thank Dr William Villaume for agreeing to co-chair my committee and provide me guidance in preparing my thesis. With a rigid time schedule set by the Navy, he always tried to keep me focused and relevant. I would also like to thank Dr Cheryl Seals and Dr Salisa Westrick for their contributions to my graduate studies. A large amount of my thought processes and actual production for my research would have not been possible without the always insightful Bradford Barker. As a fellow graduate student, Brad always could look at my ideas with a new direction. In addition, the software programming he provided was essential to both my study and future endeavors for which I am truly grateful. I would have not been able to pursue my study without the undying support of my best friend and wife, Joni. She is always a breath of fresh air and I could not have done any of this without her support and love. To my two wonderful children, Hannah and vi vii Parker, I would also like to say thank you for allowing daddy to stay at school some days when you wanted me to be home to play. To my parents Earl and Betty I would like to thank you for giving me every opportunity that gave me an appreciation for life. You always gave me hope and sacrificed so I could succeed for which I could never thank you enough. viii Style Manual or journal used: Publication Manual of the American Psychological; Association, Fifth Edition. Computer Software used: Microsoft Office Word? XP, Excel? XP and SPSS? 12.0. ix TABLE OF CONTENTS LIST OF FIGURES ........................................................................................................... xi I. INTRODUCTION ...................................................................................................... 1 General Problem............................................................................................................. 4 II. REVIEW OF THE LITERATURE ............................................................................ 9 Medication Errors........................................................................................................... 9 Information Processing............................................................................................. 12 Information Technology in Healthcare......................................................................... 14 Computerized Provider Order Entry ........................................................................ 16 Mobile Provider Order Entry ................................................................................... 21 Human Computer Interaction....................................................................................... 24 Vigilance and Attention................................................................................................. 25 Fitts Law ................................................................................................................... 27 Visual Search and Signal Detection ............................................................................. 31 Significance................................................................................................................... 34 Concepts........................................................................................................................ 36 Problem Statement........................................................................................................ 37 Operational Definitions ............................................................................................ 38 Research Hypotheses ................................................................................................ 40 III. METHODOLOGY ............................................................................................... 43 Research Design ........................................................................................................... 43 Pilot Study..................................................................................................................... 44 Main Study .................................................................................................................... 47 Subjects and Sign-Up................................................................................................ 47 Procedure.................................................................................................................. 48 Application Development.......................................................................................... 49 Data Collection............................................................................................................. 55 IV. RESULTS................................................................................................................ 63 Statistical Analyses ....................................................................................................... 64 Hypotheses for Error Rates ...................................................................................... 65 Hypotheses for Time of Order Entry ........................................................................ 67 Hypotheses for NASA-TLX workload ratings ......................................................... 69 Post-Hoc Analyses .................................................................................................... 70 V. DISCUSSION AND IMPLICATIONS ................................................................... 74 General Findings .......................................................................................................... 74 Limitations ................................................................................................................ 84 Future Research........................................................................................................ 89 x REFERENCES ............................................................................................................. 94 APPENDICES ............................................................................................................ 100 Appendix 1 Pilot Study Room?????????????????.....100 Appendix 2 Pilot Study Information Sheet????????????......102 Appendix 2 Tapping Device??????????????????....105 Appendix 4 Study Consent Document??????????????....109 Appendix 5A NASA-TLX Non-expanded?????????????.....112 Appendix 5B NASA-TLX Expanded????????????????.116 Appendix 6 SNEQ Questionnaire?????????????????120 Appendix 7 Example of Medication Order Sheet???????????125 Appendix 8 CAPP Lab for Main Study????????????...........127 Appendix 9 Pre-recorded Scripted Presentation???????????..129 Appendix 10 Pop-up Questions????????????????........142 Appendix 11 Patient Names Used on Order Sheets??????????..148 Appendix 12 Medication Names Used??????????????......156 Appendix 13 Data Sheets for Subjects ??????????.?????160 Appendix 14 CAPP Lab Postures ??...????????????..........164 Appendix 15 Example of Study Data Record ?????????..??.....166 xi LIST OF FIGURES Figure 1. Diagram of the Medication-Use-Process????????????.10 Figure 2A. Example of PDA screen non-expanded version?????????...53 Figure 2B. Example of PDA screen expanded version???????????..53 Figure 3A. Select Frequency screen expanded version before selection????...54 Figure 3B. Select Frequency screen expanded version after selection?????..54 Figure 4A. Select Patient screen non-expanded version???????????55 Figure 4B. Select Patient screen non-expanded version after downward scroll?...56 Figure 4C. Select Patient screen non-expanded version after selection????...56 Figure 5. Example of PDA data record for one subject PDA entry?????...57 Figure 6. Alert screen for mode switch????????????????..60 Figure 7. Select Medication screen expanded version?..?????????..61 Figure 8. Initial Select Medication screen non-expanded and expanded version?61 Figure 9. Initial Select Amount screen non-expanded and expanded version??.62 Figure 10. Initial Select Strength screen non-expanded and expanded version??.62 Figure 11. Initial Select Frequency screen non-expanded and expanded version?..63 Figure 12A. Select Patient screen expanded version before tap?????????78 Figure 12B. Select Patient screen expanded version after tap?????????..78 1 I. INTRODUCTION In the late 1970?s and early 1980?s pharmacists were heralding the introduction of computers into pharmacy practice in order to improve business efficiency. Software programs created for early computers such as spreadsheets found widespread use in business and were consequently applied to the business operation of pharmacies. Computers facilitated operations such as bookkeeping, insurance claims submission and inventory management. In addition, computers automated the creation of pharmacy labels and established a consistent method for maintaining an accurate list of patients, doctors and medications. The initial use of computers in pharmacy served financial more than clinical needs (Fulford, 1987; Higby, 2002). Later in the 1980?s new technology advances and programming innovations such as database management further expanded the use of computer systems in pharmacies. Computers started to serve clinical in addition to financial roles. These newer pharmacy systems contained medication reference databases and the ability to check for potential drug interactions and medication therapy duplications. By the 1990?s the amount of data storage and methods for displaying data also improved as the new technology for these tasks became more affordable. New and more efficient clinical information systems (CIS) provided 2 more time for direct patient care and ultimately expanded the scope of clinical pharmacy practice (Chaffee & Bonasso, 2004). The later part of the 1990?s saw the rapid growth of communications technologies such as the World Wide Web (WWW). The development of the WWW provided a mechanism to access vast numbers of resources from multiple information providers all from one location. The resources of the WWW enhanced patient care with real time data access that could now complement the abilities of proprietary CIS. Recently, the miniaturization of computers into handheld devices (PDAs) and the development of wireless communications now provide healthcare professionals with mobile access to the WWW and CIS. This mobility should improve safety as critical information is now available earlier in the medication-use-process at the point-of-care. In 2004, President Bush and The Department of Health and Human Services (HHS) established a ten year plan for having all of the U.S. health records electronically stored and transmitted by the year 2014 (Retrieved October 15, 2005 from http://aspe.hhs.gov/sp/nhii/news/PresidentsHealthITPlan4-26- 2004.pdf). The transition to electronic health records will require adjustments for users as the method of data entry and retrieval changes from paper to digital. Electronic health records depend on the utilization of a technology such as Computerized Provider Order Entry (CPOE) for data entry. CPOE is a computer application that allows a medical provider to order diagnostic and treatment services, such as medications or laboratory tests, electronically instead of recording them on an order sheet or a prescription pad. CPOE technology provides a new order entry interface and changes the medication-use process (see Figure 1) dramatically, particularly at the prescribing and transcription stages. Figure 1. Medication-Use-Process as defined by United States Pharmacopeia (USP), 2004 ? (Retrieved October 15, 2005, from http://www.usp.org/pdf/EN/patientSafety/medicationUseProcess.pdf) Data entry was and still remains one of the key issues for designers of healthcare systems. For years many medical professionals such as physicians have relied upon a pen and paper method of documentation and data entry. Switching to a new technology and format of data entry is not simply a matter of changing the method of input but also presents issues of changing workflow and culture of users as well. Incorporating technology into healthcare environments can often require additional work from its users and often results in the emergence of new unpredicted types of interactions and behaviors (Ash et al., 2004; Tang & Patel, 1994). For example, the implementation of a new CPOE system was stopped at Cedars Sinai Hospital in Los Angeles in 2003 due to difficulties with the user interface and concerns for patient safety ("Cedars-sinai's CPOE system had benefits, despite complaints", 2003). The new reliance on computer interfaces for entry of healthcare data further emphasizes the need to verify that the technology used to improve safety does not magnify or create safety or other issues at the user interface. 3 4 General Problem Early computers found use only by the more technically oriented adopters. This was often due to their poor interface design in addition to other factors such as cost. For individuals, the user interface can either provide a pleasant productive experience or frustrate and disrupt their intended work (Schneiderman & Plaisant, 2005). Today, computers and related technology find a wide array of users from financial to medical due again, in large part, to vastly improved user interfaces. However, some technology advances in common use in industries such as banking and manufacturing have presently not yet found widespread use in healthcare. For example currently 35 million Americans access their accounts and pay bills through online banking thereby replacing the use of written checks and bank statements. Further, with the Check 21 legislation which took effect in 2004, even the users of paper checks will now have access to digitally stored copies of the paper checks versus receiving the original paper check for their records. The banking industry expects the number of online banking users to double by 2007 (Retrieved November 7, 2005, from http://www.checkfreecorp.com/cda/corp/L5.jsp?layoutId=50043&contentId=5011 2). Of particular note for healthcare is the amount of paper documentation that healthcare practitioners still use in comparison to other information critical industries such as banking. In contrast, healthcare transactions often take place with a mixture of paper and technology based interactions, often with wasteful, 5 inefficient, and duplicated efforts (Kropf, 2005). The historical method of paper documentation in our healthcare system creates a fragmented network with silos of information that makes sharing information and prevention of medical errors difficult (Roberts & Peel, 1997). In order to improve healthcare delivery and reduce medical errors, many organizations want to implement various technology solutions to integrate the locations where medical information currently resides. The user interface of these technologies represents the keystone that will determine the type experience and outcomes from the integration and implementation of technology. Technology can, when appropriately implemented, improve workflow efficiency and remove repetitive work done by humans. The efficiencies gained with technology result from the increased accuracy rates and ability to perform repetitive, complex actions with calculated precision. More and more healthcare organizations are looking to technology to handle increasingly complex work environments, often with shortages of personnel. One example of the use of technology in pharmacy to deal with personnel shortages is automated dispensing technology. Large mail order pharmacy operations depend on robotic dispensing machines to fill millions of prescriptions twenty four hours a day that in some cases has no human final check before being dispensed to a patient. The user interface of these technologies is very critical to their performance. In order for an accurate prescription to process from the robotic filling, a person using a computer interface must first enter it into the computer. Any inaccuracies or 6 errors produced at the interface could potentially reach a patient after robotic filling. A particular design type of technology that could address poor utilization of systems is fault tolerant technology. Fault tolerant technology provides good feedback and recovery from errors. Research has shown that the more errors people make and are aware of, the more their ability to compensate and correct these errors improves (Bates et al., 2001, Galvendy, 1997). So if the technology used by humans has fault tolerance, this should presumably create safer and improved environments. With a healthcare user interface, fault tolerance would allow for a user to recognize and recover from any user-generated errors and prevent them from potentially harming a patient such as prescribing a medication for the wrong patient. However, the current offering of vendor derived and homegrown computer systems that could provide these environments may or may not have undergone research to test whether an aspect such as the user interface is fault tolerant or potentially could cause errors. Moreover, current standards do not exist for guiding software development to assure better design based on empirical research. Much of the literature on healthcare technology performance contains subjective measures such as surveys or objectively compares a legacy paper based system to a new computerized system. An example of this comparison is when a hospital or organization switches from a traditional paper-based order system to computerized provider order entry (CPOE) technology. Organizations that implemented CPOE to replace paper based order entry have found significant 7 safety improvements in the form of medication error reduction and/or decreases in adverse drug events (Bates et al., 2001; Bates et al., 1998; Bates et al., 1999; King et al., 2003; Teich et al., 2000). However, a survey conducted in 2005 found only four percent of U.S. hospitals used CPOE ("Survey: CPOE adoption slowly growing", 2005). This low adoption rate and limited amount of data for those that have adopted CPOE leave room for more exploration. The environment in which healthcare professionals work has many complex social and technical interactions. Healthcare professionals rely on the technology and individuals with whom they work to make critical healthcare decisions. Using CPOE requires a certain level of trust in the technology as direct human involvement in the medication-use process decreases. The potential consequences of a poorly designed technology interface that produces errors or masks their occurrence in a healthcare setting are unacceptable as they have the potential to decrease the quality of life or even cause the death of patients. Concerns over poorly designed or error producing user interfaces with healthcare technology such as CPOE have begun to appear in the literature (Ash et al., 2004; Bates et al., 2001; Koppel et al., 2005). Given that all software user interfaces are the critical focal point between applications and end-users, research on the measurable impact of different user interfaces on outcomes should provide organizations and developers with valuable information to assess their operational processes. The use of handheld computers such as PDAs is growing amongst healthcare providers. Many healthcare technology vendors now provide mobile 8 versions of their applications. PDA uses of applications such as electronic prescribing where prescribers could enter a prescription for a patient while walking to their car at a shopping mall are expected to grow. Although these devices provide valuable resources and connectivity to their users, the research involving these devices is even less than their larger, more stationary computing predecessors. Creating a handheld version of technology like CPOE is more than simply shrinking the interface and form factors of these devices. The limited methods of data entry and viewing area for handheld devices combined with a mobile work environment require careful attention to adequately balance the form with the intended function of these mobile technologies to provide fault tolerant interfaces. Human Computer Interaction (HCI) research is another emerging field of study that has begun to explore methods to improve a computers? user interface through manipulations of onscreen elements. Methods such as expanding targets or onscreen zooming features have improved accuracy of users in simple target pointing tasks. This type of modification provides a more fault tolerant, efficient interface. Potentially, similar modifications to a healthcare technologies interface could also enhance user interactions especially for handheld devices. For example, a user of PDA performing electronic prescribing with such an interface might find greater ease of use and more fault tolerance while entering a prescription for a patient. 9 II. REVIEW OF THE LITERATURE Medication Errors The Institute of Medicine (IOM) reported in 1999 on the magnitude of safety problems and medical errors within the U.S. healthcare system. This IOM report drew international attention. The report estimated that as many as 98,000 people died each year from medical errors with 7,000 of them attributable to medication errors. The IOM report cited the poor performance of humans, the over-reliance on memory and underutilization of technology as key contributors to medication errors (To err is human: Building a safer health system, 1999). One key recommendation for reducing medication errors in all phases of the medication-use process was to implement computerized direct order entry technology. This method of ordering prevents medication errors from the misinterpretation of hand-written orders and the manual transfer of information from one system to another. The IOM report cautioned designers of order entry technology to create user interfaces that minimize introduction of new errors from the use of this technology (To err is human: Building a safer health system, 1999). The study of medication errors is not a new area of research. Since the 1960?s and the introduction of the unit-dose medication system, researchers have sought to improve the medication-use process. The dispensing and administration 10 steps in this process have been one of the main areas of study for error detection and reduction. Many organizations and researchers have used the self-report method and chart review method to detect and report medication errors (Bates et al., 1995b; Bates et al., 1998; Leape et al., 1995). The self-report method relies on the person that committed an error to have awareness of the error and then report the error. Chart review on the other hand requires a skilled practitioner to detect errors based on the trigger events in a patients medication chart that they determine resulted from a medication error (Flynn et al., 2002). Barker and colleagues developed and used an observation method to detect errors in the administration of medications to patients. This direct observation technique uses the observations of medication preparation and administration to determine medication errors. Their observation method defines ?any deviation from the physician medication order written in the patients chart? as a medication error. This method has the advantage of detecting more errors than other common methods such as self-report or chart reviews as well as having higher objectivity. However, this method costs more than the other two methods and requires an objective, trained observer to detect any errors (Allan & Barker, 1990). In one study of a comparison of these three methods in thirty six health facilities, the observation method had an accuracy rate of 82% for the true number of errors. Observation detected twenty two times more errors than chart review and 373 times more errors than self-report (Flynn et al., 2002). The observation method assumes that the original order written by the provider is not an error in itself and does not attempt to measure ordering errors (Allan & Barker, 1990). 11 Errors made in diagnostic, prescriptive selection, or transcription might go undetected by this methodology. If, for example, a provider ordered an incorrect strength of medication for a particular patient, this method only measures the accuracy in the administration of these orders. Further, if a provider used a direct method of order entry and mistakenly selected a medication, this method also would not account for that error. In contrast to the researchers who study medication errors from the administration perspective, others researchers such as Bates and colleagues at the Harvard Medical School define medication errors in broader terms. From this research group?s perspective, medication errors can occur at any stage of the medication-use process. Additionally, they also looked at consequences of medication errors such as whether an adverse drug event (ADE) resulted from the medication error. Not all ADEs result from medication errors and some ADEs could result from reactions to the appropriate use of medications. Further, some medication errors fall into a category of a potential ADE when no actual injury occurred. Unlike observation researchers, this group relies on incident reports and other practice related data such as chart reviews to determine when a potential ADE, medication error or ADE has occurred (Bates et al., 1995a; Kaushal et al., 2001; Morimoto et al., 2004). In several of their studies measuring the occurrence of medication errors and ADEs several key points emerged. First, the physician or other ordering providers account for many of the medication errors, particularly at the ordering stage of the medication-use process. One study found that seventy nine percent of 12 the potential ADEs occurred at the ordering stage (Kaushal et al., 2001). Upon evaluating such ordering errors, the researchers felt that direct computer order entry technology could reduce between eighty one percent of the ordering errors and ninety three percent of the ADEs (Bates et al., 1995a; Kaushal et al., 2001). Researchers who study general error occurrence, define an error as ?a planned sequence of mental or physical activities that fails to achieve its intended outcome? (Reason, 1990). Each step in any planned sequence has the potential to result in error. The basic classification of what caused an error is either the failure of actions as intended (slip) or the failure of judgment of means to achieve goals (mistakes). Researchers distinguish slips as errors in action with a failure in the execution stage of an action regardless of the plan. Mistakes result from a deviation from the plan or a failure in a judgmental process in the specification of the plan itself (Reason, 1990). Medical errors result from the failure in the planned execution of procedural actions or using an incorrect plan. Information Processing The processes of perceiving information and determining appropriate actions require time and could possibly result in error. Information processing relies on focused attention and aspects such as discriminating features of stimuli to help detect when an error has occurred (Galvendy, 1997). The ability to discriminate signals for information processing could decrease in an environment that has random visual and auditory distractions or interruptions. Distractions and interruptions result in poor quality or ?noisy? inputs responsible for poor sensory information processing. When affected by distractions and interruptions, people 13 can become confused due to similarity of information and take shortcuts (Grasha, 2000; Grasha & K, 2001). Pharmacists often work in fast paced environments filled with distractions, both auditory and visual, and are susceptible to producing errors. In the dispensing of a prescription a pharmacist interacts with both technology and people, often at the same time. Historically, one of the measures used to determine quality pharmaceutical care has been to measure dispensing error rates. Previous studies have demonstrated the relationship of interruptions, excessive workload and distractions in a pharmacy to dispensing errors (Flynn et al., 1999; Flynn et al., 1996). In 2003, twenty percent of the errors reported in the MEDMARX? medication error database involved automation or computerization. Computer entry errors, resulting from incorrect or incomplete information, were the fourth leading cause of reported errors. The most frequent cause of computer entry errors was performance deficit with distractions the leading contributor (United States Pharmacopeia (USP), 2004). Grasha has developed a cognitive systems model for healthcare that explains why the performance of a pharmacist suffers and results in error because of failed information processing. This model cites poor information presentation from sound-alike and look-alike names as well as environmental factors such as distractions contributing to pharmacy errors (Grasha, 2000; Grasha & K, 2001). For example, if neighboring words or phrases in a field of words are semantically similar, it can take longer to recognize the target word and it will be more likely 14 for a person to select the incorrect word. The presence of sound-alike and look- alike medication or patient names in pharmacy tasks and the errors that have occurred from them demonstrates the limitations of human information processing. It has been estimated that up to twenty five percent of medication errors might result from confusing pairs of medication names (Lambert, 1997). Thus, it might be possible to improve information processing and reduce some of these errors through techniques that provide better discrimination. Information Technology in Healthcare For many years many high risk industries, like aviation, have incorporated system changes to reduce error by relying less on people solely to detect and correct errors often caused by system flaws (Galvendy, 1997). However, in the mid 1990?s when the Leape et al. study analyzed system generated adverse drug events, the basic procedures in the medication-use-process in healthcare showed little change from the previous decade. Technology use was sporadic and uncoordinated with very limited connectivity or integration of systems. The focus on errors that occurred in healthcare was on the individuals that committed the errors, and not the system of medication use (Leape et al., 1995). More recently, investigations into medical incidents have begun to indicate that a human error often resulted from a poorly designed system (Ash et al., 2004; Bell et al., 2004). Challenges to the design and implementation of technology for healthcare environments stem from the complex, multi-dynamic nature of the work done by healthcare professionals who are a unique set of workers that have an expectation 15 for information delivery, both in terms of time and format. Users of healthcare technology expect these systems to provide technical level improvements such as new tools and new forms of presenting information while at the same being easy to use and guaranteeing safety for all involved (Gremy & Degoulet, 1993). Unlike other isolated single task computer users, many healthcare professionals do not work in a single area throughout the day but have multiple work areas and frequent interactions with other professionals. Communication with other practitioners and patients and the need to multi-task while negotiating interruptions and distractions is commonplace for many. In the healthcare work environment, professionals rely on individuals and systems to make critical decisions. Trusting one?s counterparts, whether human or machine is paramount (Bell et al., 2004). The lack of attention to the different requirements for different healthcare professionals plagued early computer systems and lowered their acceptance. The misunderstanding of what users needed and how to interface with them caused many users to think of technology as a hindrance not a benefit (Ash et al., 2004). Human performance when interacting with a technology interface relies on information processing whereby a person perceives, interprets and takes action on information perceived. The concern of producing unexpected and unnoticed errors, such as substitution errors where two similar items placed next to each other on a screen are confused, presents challenges to designers of healthcare technology. The user interface of any healthcare technology must adapt to the various users? cognitive and physical limitations while at the same time meeting 16 users? input/output needs. In essence, because healthcare practitioners perform critical work in distracting environments, the user interface technology must have greater fault tolerance. The potential for a user to make a slip when using an interface, such as a menu, and select look-alike names or simply make a slip when doing complex work is only one of many considerations for designing a healthcare technology such as an electronic order entry system (Ash et al., 2004; Bell et al., 2004). Safer systems with greater fault tolerant designs should prevent errors, make them easier to detect when they occur and reduce adverse outcomes if they do occur. With the knowledge that poorly designed technology could increase or create new errors, a focus on user-centered design should have priority for designers. However, one of the dilemmas that technology potentially creates is that users start to believe the technology is incapable of producing an error. Since humans learn to deal with errors by experiencing them, this might create a false sense of security (Bates et al., 2001; Nolan, 2000). Thus, users of healthcare technology will need a new kind of vigilance for interacting with new systems to detect and prevent user generated errors (Ash et al., 2004; Bell et al., 2004). Computerized Provider Order Entry Although new technology continues to provide advanced tools and organization, it still relies upon human interaction for much of the data entry and retrieval. In most cases today, medication order entry into a pharmacy system still starts with the pharmacy staff transcribing and entering handwritten prescription orders into a computer system. This process resulted in 11 percent of 17 the errors for preventable adverse events in one hospital study and drew widespread attention in the medical community. Together with ordering errors, transcription errors accounted for more than half of the total errors in the study (Leape et al., 1995). One of the key benefits of CPOE is the ability to eliminate the transcription process and the accompanying difficulty pharmacists sometimes face while deciphering illegible handwritten prescription orders for order entry. CPOE is a technology that dramatically changes the user?s perspective to create a medical order. This technology provides a new user interface and interaction technique in the ordering phase of medication use which allows the ordering provider to directly enter an order. When CPOE systems have eliminated the transcription process in medication order entry, they have demonstrated potential to reduce errors by 41-81% compared to legacy paper-based systems (Bates et al., 1998; Bates et al., 1999; King et al., 2003). The ability of CPOE technology to potentially reduce adverse drug events (ADE) and patient harm by allowing physicians to directly enter their orders into a computer became apparent as significant reductions in medication errors resulted from the use of order entry technology (Bates et al., 1998; Bates et al., 1999). The implementation of technologies, such as CPOE, to improve patient safety is now part of a national effort to improve medical care. The implementation of CPOE systems and other electronic resources in healthcare is a revolution that takes healthcare workers out of the paper age into the digital age overnight. However, five years after the seminal report by the Institute of 18 Medicine (IOM) recommended systems changes like CPOE, many institutions still lack this technology (Leape & Berwick, 2005). A recent survey of hospitals in 2004 that focused on prescribing and transcribing improvements in the medication-use-process found a less than admirable implementation of CPOE. Only 4.2 percent of the 493 hospitals that responded had CPOE installed. Twenty five percent of these CPOE systems required pharmacy re-entry of medication orders since ordering and medication management systems used by the pharmacy were non-integrated (Pedersen et al., 2005). In a fully integrated CPOE system the ordering providers communicate their order electronically to other parties involved in medication use, such as pharmacy and nursing staff, and no further order entry is required. Non- integrated systems eliminate the guesswork of the bad handwriting in the traditional transcription process, but allow for additional errors when pharmacy staff transcribes the order into the pharmacy system. In one study of a CPOE system installation that required pharmacy re- entry of physician orders, hardware or software problems contributed to seventeen percent of the CPOE errors. For example, one of the problems faced by the designers was when to require allergy information when entering an order. All allergy related CPOE errors occurred when the system didn?t require allergy information prior to accepting a medication order. Subsequent modifications to the interface provided guidance to collect allergy information when an order is initiated (Spencer et al., 2005). Another study concluded that the non-integrated CPOE system interface design facilitated substitution selection errors by 19 physicians, which were difficult for a pharmacist to identify when they re-entered these system generated orders (Jones, 2004). The transcription phase error potential was improved, but because of a lack of integration, a duplication of effort and potential for selection error by someone could still exist (Bates et al., 1999; Jones, 2004; Oren et al., 2003). Designing a healthcare technology such as CPOE to prevent errors such as order entry errors entails more than standard software and hardware functional development. To create a CPOE system and user interface that adequately supports the users? intended goals, designers should employ user-centered design. Principles of user-centered interface design for computers focus on analyses such as task analysis, functional analysis, environmental analysis, representational analysis and user analysis to determine their design (Johnson et al., 2005). User analysis is important in the CPOE application development since more than one type of user could interact with the system. Some organizations have different levels of providers (i.e. nurse, pharmacists) using an order entry system. One study comparing physicians to non-physicians found a significantly higher non-physician CPOE error rate between two hospitals using the same order entry system (George & Austin-Bishop, 2003). These results coincide with another principle of interface design, task analysis. Understanding the goals and particular tasks of different users might dictate an interface design that meets all the user?s needs. A nurse entering a medication order approaches the task differently than a physician. Role based interfaces might be appropriate to reduce unnecessary screen elements and increase efficiencies (Johnson et al., 2005). 20 An example of role based interfaces appeared in the experiment by Staggers and Kobus which demonstrated the ability of a new nursing user interface and input method task to provide an improved environment for its users through increased efficiency, reduced errors and higher acceptance. This study used a repeated measures research design for simulated nursing tasks using a familiar legacy-based text based interface and a new graphical user interface (GUI) interface. The new GUI interface radically changed the method of data entry and visual display for users. The GUI interface objectively performed significantly better making fewer errors and taking less time to enter orders. Additionally, the nurses subjectively rated the new GUI interface significantly higher than the legacy text based system (Staggers & Kobus, 2000). A study published in 2005 also questioned if a CPOE technology interface might facilitate or produce no impact on medication errors. This study found that the adoption of a CPOE system facilitated 22 new sources of medication errors, many of these resulting from user interface issues that allowed for user errors in performance. One of the issues mentioned by users was the ability to select wrong patients and medications due to the method of display and proximity of items on the screen. Although this study relied on data such as interviews with users and observations of staff, it demonstrated that from a user?s perspective the interface with a computer could potentially confuse and generate unintended errors (Koppel et al., 2005). 21 Mobile Provider Order Entry Another factor driving handheld computer adoption in healthcare is utility. In addition to mobility factors described previously, healthcare providers have found these handheld computers increasing able to do such tasks as carry evidence-based information, new lab results and organize other critical data at the point-of-care, as well as being able to serve as order entry devices. The ability to access resources with this technology and communicate with other caregivers is proving to be extremely useful. An early survey of physicians found a strong desire to carry a PDA if they had applications that provided information to improve patient care (Fischer et al., 2003). Literature reports on handheld computer use in healthcare have focused mainly on user acceptance, ways of use in a practice settings and reviews of different software. The potential to positively impact safety by reducing errors with handheld computers is often presumed, but has not been adequately studied (Clauson et al., 2004; Fischer et al., 2003; Lowry et al., 2003; Lynx et al., 2003; Young et al., 2001). Many CPOE implementations have been in inpatient healthcare settings or healthcare systems where this type of system first evolved thirty years ago (Bell et al., 2004). The majority of vendors of electronic medical applications such as electronic medical records (EMR) or CPOE systems have started including handheld versions of their programs or integrating with current handheld applications to extend a healthcare providers work area (Ying, 2003). The development of handheld electronic prescribing and other ordering capabilities is 22 an evolution of CPOE that now includes mobile applications for providers in and out of an inpatient setting (Fischer et al., 2003). Outpatient Mobile Provider Order Entry (MPOE) or electronic prescribing systems implementation currently lags behind their inpatient counterparts due to complexity of interfacing of multiple systems and the lack of tested standards such as the foundation standards from agencies such as The Center for Medicare and Medicaid Services (CMS) ( Retrieved November 15, 2005, from http://www.hhs.gov/news/press/2005pres/20051101.html). In addition, MPOE is not simply a scaled down version of the full sized counterpart. The limitations found in the shrinking of technology requires careful attention to human-computer issues such as ease of use and the user interface to provide adequate but limited devices (Ying, 2003). A handheld application should have a balance between functionality and form with only the most useful, productive functions appearing in the mobile version of an application (Johnson et al., 2005; Schneiderman & Plaisant, 2005; Ying, 2003). Direct manipulation of a full sized computer interface, with a mouse or stylus, carries a disadvantage for designers of a limited screen space. This problem is exacerbated in the even smaller screen environments of handheld computers that make normal screen designs used for desktop computers impractical (Johnson et al., 2005). Preventing errors while maintaining ease of use become more difficult the smaller the device and more mobile the user (Schneiderman & Plaisant, 2005). 23 Handheld computers may have a much lower resolution and smaller form factor as well as restricted methods for data entry such as keyboard tapping, thumb keyboards, handwriting recognition and stylus-based selection. These physical limitations of handheld computers can often make navigation of menus and lists difficult. This is especially true when many similar items, such as the names of medications or patients, are in close proximity (Koppel et al., 2005; Ying, 2003). If some users feel that a potential for new errors exist from factors such as proximity of medication names or patient names on full sized versions of a system like CPOE, then a mobile version might actually provide more opportunity for errors and therefore need special user interface design to prevent such errors. Environmental analysis pertains to the conditions in which a particular system is used. Where a device is used and under what conditions can vary with factors such as lighting and noise. Mobile device applications have to attract the attention of users operating in a limited space and account for the environmental factors as well as interruptions and distractions of users. With healthcare systems, more often the environment is harder to define with the availability of adequate communications and robust handheld computing devices in the hands of mobile healthcare professionals (Johnson et al., 2005; Schneiderman & Plaisant, 2005). Designers of mobile healthcare technology need to account for the relationship between usability issues within the device, the environment and the users to avoid contributing to or increasing error rates for associated tasks (Kushniruk et al., 2004). 24 Substantive research into handheld computers and other mobile devices is still in its infancy. These devices date back only a fraction of time compared to their larger counterparts but are following a similar pathway to their more stationary counterparts. Usability tests for different applications on portable devices have uncovered the limitations of human performance, primarily visual acuity and hand-eye coordination, which present ergonomic challenges to designers. As often happens with advancing technology, the understanding of how these devices impact the humans that interact with them lags behind their development. The desire to produce solutions for users can often supersede established design principles. With devices meant for multi-tasking, the trade-off of portability and limited input methods could lend a larger than expected opportunity for errors (Kjeldskov & Graham, 2003; Wichansky, 2000). Human Computer Interaction The study of Human Computer Interaction (HCI) is a multidisciplinary field encompassing areas such as psychology, sociology, human factors engineering and computer science. HCI research focuses on understanding the interaction of people with computer systems and how they affect one another. One of the main focus areas of HCI research is the computer interface (Schneiderman & Plaisant, 2005). Development of user interfaces, especially ones requiring direct manipulation, must acknowledge human factors such as the limits of a user?s physical ability, perception and information processing. Motor activities have automaticity and often can occur with little or no attention, often resulting from repetition or practice. Such automaticity of motor 25 activities allows a user slip if distracted or when performing a familiar routine. Slips, such as typing errors and menu selection errors, can result from a user?s lack of attention to a task or from a simple physical miscue when interacting with a poorly designed computer interface. In the case of a human interacting with a computer, slips could also result from a mismatch between user, task and computer (Arnold & Roe, 1987). An everyday example of this mismatch is receiving cash from an automatic teller machine. These machines often rely on both a user to align elements on a computer screen with external buttons to input choices. The user?s height, vision and how aligned the external buttons are with the corresponding choices determine how well this user interface performs for a simple transaction. Vigilance and Attention Another key component of a user?s interaction with an interface is vigilance. Vigilance is part of the human attention system and its ability to detect, orient and respond to sensory events. Vigilance requires concentration and focus to determine when a signal is present. Since the introduction of modern interface technology, researchers have sought to determine the effect of a user?s attention in tasks requiring sustained vigilance (Szalma et al., 2004). The Mackworth Clock Test in the post World War II era and subsequent vigilance studies have shown that a person?s vigilance is directly related to their ability to interpret elements in an interface. Additionally, the amount and accuracy of vigilance a person exhibits is also adversely affected by how long they maintain a vigilant state (Lichstein KL, 2000; Mackworth, 1961). 26 The introduction of computer technology and the repetitive tasks of their users have heightened concern for vigilance and possible performance detriment. Repetitive tasks such as data entry are monotonous and if arousal diminishes during these tasks, the potential for a distraction, interruption or similar looking items to cause an undetected slip increases. Both psychological and physical strains are involved in repetitive tasks such as data-entry. One study of data entry produced results that showed deterioration in performance similar to the Mackworth experiments. After thirty minutes of data entry work, a sharp decline in performance occurred over the next thirty minutes (Floru et al., 1985). Vigilance tasks produce significant stress on the person performing the tasks. The concern for the impact of these tasks on participants prompted the creation of subjective scales designed to measure the impact of the workload on individuals. The NASA Task Load Index (NASA-TLX) is an instrument for measuring demands of a task in addition to characterizing aspects of how a user characterizes the workload of a task requiring sustained attention. This index is a multi-dimensional workload assessment tool for measuring subjective ratings over the six dimensions of mental demand, physical demand, temporal demand, performance, effort and frustration level (Hart & Staveland, 1988). Factors that degrade vigilance increase workload and produce less desirable ratings with the NASA-TLX (Szalma et al., 2004). Another user specific internal factor that determines the level of attention is field independence. Persons that show field independence can overcome distracting background elements and thus are less distracted. In contrast, a field 27 dependent person is more susceptible to distractions (Dembo, 1977). Research suggests a correlation with the performance of activities requiring high levels of attention and distractibility. For example, distracted field dependent persons performing pharmacy tasks have demonstrated a higher error rate than field independent persons (Flynn et al., 1999; Grasha & K, 2001). Health care practitioners work in low error tolerant environments that require multiple interactions, with both technology and people, many of which are critical. Many of these interactions can occur in a non-sequential manner, with distractions or interruptions that can misdirect a user?s attention. With this in mind, a good CPOE user interface should provide efficient characteristics for maintaining a user?s attention. With greater attention, a user should have greater discovery of and correction of errors from slips made while entering choices (Arnold & Roe, 1987; Grasha, 2000; Schneiderman & Plaisant, 2005). Additionally, user interface design should avoid potential distracters in the interface itself(i.e. low clutter) and enhance discrimination of items, which enhances selective and focused attention, without negatively impacting a user?s workflow (Galvendy, 1997). Fitts Law A user interface may have a design that captures a user?s attention, but the user?s physical interaction with the interface is also important. When a user interacts with a computer interface, this interaction is subject to not only psychological but physical characteristics of the user as well. Humans build a mental model of the correct physical interactive behavior to perform a particular 28 task. A person?s performance when interacting with a particular computer interface results from a combination of his or her model, motor skill and the interface design (Carroll, 1997). The development of GUI and the introduction of pointing to objects on a screen changed interaction and input methods in computing dramatically (Guadagno et al., 2004). GUI interfaces with elements such as icons, buttons and menus make up some of the most fundamental methods of human computer interaction for today?s computer user. All of these elements depend on pointing as the input method for the computer interface (Balakrishnan, 2004). Prior to the modern microcomputer and GUI interfaces, technology in work settings often required motor tasks with visual control. In a landmark study in 1954, a principle of Human Factors known as Fitts Law determined the important factors of a person performing manual pointing tasks. Fitts Law describes the time and accuracy to hit a specified target based on its distance and size. It also described the performance capacity of the human motor system associated with visual and other feedback mechanisms. The Fitts experiment used a stylus device where users pointed to objects of varying sizes and distances in the physical world (Fitts, 1992). Classical pointing tasks in Fitts Law studies vary sizes of target widths and typically result in error rates of four percent (Mackenzie, 1992). These studies use simple tapping or selection or objects where the selection of the object is the only goal. The research that produced Fitts Law demonstrated that there exists a direct relationship between the user?s performance in physical pointing in the real 29 world and the physical characteristics of the target. The use of pointing devices to translate a user?s request for accurate movement while navigating over GUI objects on computer screen space converts physical pointing into virtual pointing. Virtual pointing mimics physical pointing to the degree that Fitts Law applies to it as well. However, virtual pointing is not constrained by the same laws of the physical world and thus a computer interface design could actually make virtual pointing easier (Dix et al., 2004; Fitts, 1992). Through Fitts Law, modern designers of computer interfaces gained a method to understand how to balance the size of a target on a screen with the type of input for maximum performance without decreasing the information presentation on a particular screen. Gradually reports of experiments with changes to the target that could improve pointing at the interface based on Fitts Law have begun to appear in the literature (Balakrishnan, 2004; Cockburn & Firth, 2003; Fitts, 1992; Guiard & Beaudouin- Lafon, 2004; Gutwin, 2002; Mackenzie et al., 2001; McGuffin & Balakrishnan, 2002; Zhai et al., 2003). These experiments demonstrated that changes to target parameters (i.e. width) on a computer screen could improve a user?s target acquisition time and accuracy. Expansion of targets produced beneficial results because it improved the user?s attention by isolating, highlighting and magnifying the target over against the competing details of the surrounding interface. Thus, virtual pointing was not merely a computer equivalent of physical pointing but a distinguishable new version. Most users don?t have error free target selection when interacting with a computer interface through virtual pointing. The two most common measures of 30 performance for virtual pointing activities are accuracy and speed. Speed is often measured as movement time for the pointing activity. Accuracy results often appear as an error rate (Balakrishnan, 2004; Guiard & Beaudouin-Lafon, 2004; Mackenzie, 1992). Error likely situations with a user interface could result from inadequate work space and layout making accurate pointing to one of these elements difficult. Even with highly skilled workers, errors in performance often result from lack of spatial precision (i.e. physical coordination) or misinterpretation (Galvendy, 1997). Given that the size of many on screen interface elements of computer displays may be as small as 10 pixels (2mm on a common display), the need to improve target selection in the common GUI interface seemed logical to many HCI researchers (Zhai et al., 2003). In the case of PDAs, the screen interface may have a greater concentration of smaller elements that could make target selection more difficult for a user. In order to improve the accuracy of small target acquisition, techniques such as expanded targets, sticky icons, bubble cursors and thumbnail viewers have undergone evaluation. Some of these techniques rely on dynamically changing displays that expand the target region as a cursor approaches an onscreen element and then return to normal as the cursor moves away. These techniques attempt to reduce distance while increasing width, the two main components that determine difficulty of pointing via Fitts Law (Fedak, 2004; Grossman & Balakrishnan, 2005; McGuffin & Balakrishnan, 2002; Zhai et al., 2003). 31 These previous studies as well as others that used expansion techniques have limited generalizability to most user interfaces since they used well spaced targets or isolated targets (Cockburn & Firth, 2003; Grossman & Balakrishnan, 2005). An experiment in 2005 introduced a new variation on target expansion for closely packed targets, the bubble cursor. Bubble cursors expand target areas located within the bubble cursor?s bubble region when the cursor approaches a target. Bubble cursors differ from standard target expansion in that their areas for expansion of targets dynamic adjustments based on surrounding targets. This dynamic character allows the expansion based on the closeness of other targets. Items for selection in this experiment differed from those found on menus and toolbars of today?s? GUI environments. However, this new method of activation and expansion demonstrated once again that by modifying elements of Fitts Law, such as width, target acquisition times and error rates improved (Grossman & Balakrishnan, 2005). Visual Search and Signal Detection The task of searching for an item in a field of other items is visual search. Visual search is essentially composed of the techniques a person uses to distinguish different items from each other. For example, a friends face or a car in a parking lot, through its attributes. How effective a person?s search is depends on the item representation, his or her attention and abilities to integrate all the visual stimuli factors for the item (The gale encyclopedia of psychology, 1996; Palmer et al., 2000; Zeneger & Fahle, 1997). The use of larger sized objects can also increase the attention accuracy for individuals as well. If a 32 person can direct his or her attention to a particular location, this can result in a more rapid response (Posner & Petersen, 1990). So using an expanded target in a user interface like a PDA would be akin to having binoculars to find your car in a parking lot, focusing and clarifying the item in the visual search for the advantage of the user. The minimal amount by which two stimuli must differ in order to allow for discrimination is called the just noticeable difference (JND). The low threshold theory of visual search establishes a premise that a distracter that is not the target sought in a visual search can produce a false-positive target-detect state if a JND is able to pass the user?s low threshold. Visual search incorporates signal detection theory (SDT) to describe the decision process a person uses to identify stimuli. SDT focuses on the cognitive process, such as attention, a person uses for pattern recognition and to distinguish a particular stimulus present within a ?noisy? environment of other stimuli (The gale encyclopedia of psychology, 1996; Palmer et al., 2000; Zeneger & Fahle, 1997). SDT also describes the limits of a person?s attention in the presence of multiple simultaneous stimuli and how the noise of distracter stimuli in an environment leads to user error. For example, increasing the number of items in a field that a person is searching adds noise and can increase the difficulty in the decision process. Furthermore, increasing the similarity of items further increases the likelihood of an incorrect choice (Cousineau & Shiffrin, 2004; Eckstein, 1998; McElree & Carrasco, 1999). For example, the co-location of patient names or medication names in an alphabetical list in a CPOE user interface could help 33 contribute to a selection error by a user. In addition, the more similar stimuli are, the longer it can take to determine the difference. Healthcare professionals require a sustained alert status that if not maintained, could result in lowered attention. If a person is generally alert but not engaged in processing information (i.e. giving his or her attention), interference between the signals detected in the brain provide opportunity to allow for false signal detection (Posner & Petersen, 1990). For example, a physician using a handheld computer to prescribe a hydralazine hcl while walking in a hospital may erroneously select hydralazine/HCTZ due to distractions and interactions coupled with the co-location of these similar medication names on a screen. When end-users perform visual search and signal detection, their brain performs information processing through pattern recognition, reasoning and decision making in order to verify a signal. For near-threshold signals, this can be difficult, especially if there are distractions of other signals confronting the observer. When a user scans a large number of items to find a target, they scan the field looking at elements of the items to determine when they have found the target. In order to achieve a detect state the stimulus to the brain has to be larger than the background activity present in the brain normally. Ideally to make signal detection easier, the environment should contain a method to increase the signal to noise ratio to help a user have a greater opportunity for information processing and pattern recognition (Bourne et al., 1986; Cousineau & Shiffrin, 2004, The gale encyclopedia of psychology, 1996). 34 Significance In a survey of healthcare Chief Information Officers (CIO), the top goal for organizations in 2005 is patient safety. The implementation of CPOE and other systems such as PDAs are at the top of the list of technologies to provide this goal ("16th annual himss leadership survey sponsored by superior consultant company/acs healthcare solutions." 2005). Both of these technologies address ways to improve safety by reducing errors at the ordering and transcribing phases of the medication-use-process. However, there also exists a need to examine the relationship between the type of interface design and key variables such as time, accuracy and subjective workload that are important to create efficient and effective healthcare technology. This might even have more significance as the devices for users get smaller and the types of user interface more constrained. The progression and development of MPOE or electronic prescribing includes the PDA as a primary operating platform. Healthcare practitioner?s mobility and varied practice settings make this device an ideal platform for supporting electronic prescribing. One primary reason for the migration to the PDA platform is the sheer number of capable devices. Smartphones are handheld devices that serve both as a mobile phone and a PDA. These devices have the ability to provide their users with far reaching wireless connectivity to resources, often at speeds rivaling traditional wired connections. Smartphones were the fastest growing category for mobile communication devices in 2004, a trend that 35 is expected to continue (Retrieved November 10, 2005, from http://www.palminfocenter.com/view_story.asp?ID=7978). The development of features such as user interface for a handheld technology cannot solely revolve around one anticipated user, such as a healthcare practitioner. The methods of input and output on smartphones and other mobile devices often support lower risk business oriented functions better than an application such as electronic prescribing. The revenue generated from business users who use them as message centers, video conference devices or e-mail clients will continue to dictate much of the form and function of these multifunction devices. The high risk nature of an activity such as electronic prescribing should occur with as little error as possible. A user interface for a MPOE system should allow for the variability of device type, input/output methods, environment, and user characteristics to ensure the safety of those involved. The user interface of a MPOE system should: ? Provide as efficient user interaction as possible by : o Reducing the chance for slips and potential need for re-work in data entry o Providing clear item differentiation ? Provide a fault tolerant interface by: o Providing a good mechanism for error discovery and recovery o Not contributing and when possible protecting from user generated order entry errors 36 ? Provide users with a method of data entry and retrieval that: o Places as low a mental and physical demand on the user as possible This study sought to measure the impact on visual search, signal detection and user performance in a simulated medication order entry task measured through individual performance factors (error and time) and user ratings (workload) with two different types of user interfaces (expanded and non-expanded). Concepts Phases of Medication-Use-Process Prescribing- To designate a medication or other treatment for a particular diagnosis or indication. Transcription- The manual transfer information from one location to another, such as transferring a written order into a computer. Dispensing- To prepare and distribute medications. Administration- The act of giving a medication. Monitor- Monitoring provides an ongoing verification of progress toward achievement of objectives and goals. Medication error- any preventable event that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of the health care professional, patient, or consumer. Adverse drug event- any unexpected or dangerous reaction to a drug Handheld computer- very small, portable microcomputer, also known as a PDA, that fits in the palm of the hand. 37 Portable computer- microcomputer that is designed for use while being transported. Resolution- the amount of detail and clarity of an image on a computer, determined by the width and height of the display screen. Form factor- the size and physical shape of a particular computer device. Mental demand- the amount of mental and perceptual activity (i.e. thinking, deciding, looking, and remembering) required to perform a selection task. Physical demand- the amount of physical activity (i.e. controlling, activating, selecting) required to perform a selection task. Temporal demand- the amount of time pressure a user feels based on the rate or pace of tasks. Performance- how successful a user feels they are in performing the goals set by the experimenter. Effort- the amount of mental and physical work to accomplish a level of performance. Frustration level- the amount of discouragement, stress, annoyance versus relaxation, gratification and content a user feels during a selection task. Problem Statement The Problem is: What is the effect of an expanded target interface for field independent and field dependent subjects on the number of order entry errors, time to complete order entry and subjective workload for a handheld medication order entry task? 38 Operational Definitions Expanded Target- In the stimulus portion of this experiment, the expanded target interface is produced on the PDA screen the first time an item is tapped on each entry screen. This tapped item and any adjacent item above and below in the list now appear in Palm? regular font size 14, expanding from original Palm? font AFPalm size 11. NASA-Task Load Index (NASA-TLX)-- a subjective workload assessment tool for assessments of operator(s) across the dimensions of mental demand, physical demand, temporal demand, performance, effort and frustration level. Medication order entry task-- The process of selecting items from designated fields in the handheld order entry system to complete a medication order found on an order sheet for an individual patient. Order entry error-- the incorrect selection and entry, as indicated by the order sheet, a component of a medication order entry task from a field in the handheld order entry system. Time for order entry-- The total time of order entry for a single medication order entry task is measured in seconds from the time the initial patient entry component screen appears. An entry occurs when a subject tapped an onscreen object twice consecutively. For statistical calculation purposes, time for order entry is represented as the the number of orders components entered in a minute. The fewer number of orders entered per minute indicates a greater amount of time of order entry. 39 Field-independence? the extent to which a person perceives part of a field as discrete from the surrounding field as a whole rather than embedded in the field; individuals who can understand visual cues, break up an organized visual field and keep part of it separate. Field-dependence?individuals who have trouble understanding visual cues and are unable to separate figures from background Distractibility Score?the score a subject achieves on the group embedded figures test (GEFT) to measure the degree of field dependence or field independence. The score is the number of correct figures identified with the national norm on the GEFT of 11.4. Scores above the norm classifies a person as field independent and scores below the norm classify them as field dependent. Distraction- A distraction is produced centrally using a one second in duration alarm sound to alert users of a projected multiple choice question. Each of 15 questions will appear for only 15 seconds after alarm sounds. PDA action associated with a distraction- a PDA action is any event that is captured through PDA event logging as output. These actions are associated during a distraction if they occur within 15 seconds of initial alarm sounding or closely following a distraction if 15 seconds after distraction disappearing. Extra Taps- the minimum number of PDA stylus taps needed to enter any data point from a category on an order sheet correctly or incorrectly is two. When a user requires more than two taps to enter a correct or incorrect entry, they have produced extra taps. 40 Research Hypotheses Hypothesis 1: The expanded target interface will have a lower error rate for order entry than the non-expanded target interface. Null Hypothesis: The expanded target interface will not have a lower error rate for order entry than the non-expanded target interface. Hypothesis 2: Field independent subjects will have a lower error rate for order entry than field dependent subjects. Null Hypothesis: Field independent subjects will not have a lower error rate for order entry than field dependent subjects. Hypothesis3: The expanded target interface will reduce the error rate for order entry for field independent subjects less than it will for field dependent subjects. Null Hypothesis: The expanded target interface will not reduce the error rate order entry for field independent subjects less than it will for field dependent subjects. Hypothesis 4: The time for order entry will differ for the expanded target and non-expanded target interface. Null Hypothesis: The time for order entry will not differ for the expanded target interface and non-expanded target interface. Hypothesis 5: Field independent subjects will be quicker at order entry than field dependent subjects. 41 Null Hypothesis: Field independent subjects will not be quicker at order entry than field dependent subjects. Hypothesis 6: The relative time for order entry with expanded target and non-expanded target interfaces will differ for the field independent and field dependent subjects. Null Hypothesis: The relative time for order entry with expanded target and non-expanded target interfaces will not differ for field independent and field dependent subjects. Hypothesis 7: The subjective workload ratings will differ for the expanded target and non-expanded target interfaces. Null Hypothesis: The subjective workload ratings will not differ for the expanded target and non-expanded target interfaces. Hypothesis 8: The subjective workload ratings will differ for field independent and field dependent subjects. Null Hypothesis: The subjective workload ratings will not differ for field independent and field dependent subjects. Hypothesis 9: The relative subjective workload ratings for expanded target interface versus the non-expanded target interface will be lower for the field independent than the field dependent subjects. 42 Null Hypothesis: The relative subjective workload ratings for expanded target interface versus the non-expanded interface will not differ for the field independent and field dependent subjects. 43 III. METHODOLOGY Research Design The objective of this study was to determine whether performance differed between users of two different types of CPOE interfaces on a PDA. More specifically, the study sought to evaluate the effect of these two interfaces on time to complete order entry tasks, medication order entry error rates and subjective workload of users. The two simulated software systems created specifically for this study reflected current commercially available handheld e-prescribing software. The two systems functioned identically except for the method of display for order entry of items on the screen by the users. The stimulus version used an expanded target method for clarifying items on the PDA screen and provided an additional level of fault tolerance in an attempt to reduce order entry errors. The control version did employ the expanded target method. The study design consisted of a randomized crossover repeated measures design with field independence as a between subjects independent variable and expanded/non-expanded target interface as a within subjects independent variable. The three dependent variables consisted of order entry error rate, time to complete an order entry, and subjective workload ratings. In addition, a demographic questionnaire provided other data used in subsequent analyses. Prior to 44 conducting this study, the researcher received approval from the Auburn University Investigational Review Board (IRB). Pilot Study Before conducting a pilot experiment, the researcher conducted a pre- pilot exercise with five graduate students in the Pharmacy Care Systems Department at Auburn University. These individuals participated as intended study subjects would except, in addition, after the experiment the researcher also asked them to provide verbal feedback to the researcher. Also these subjects had knowledge of the study design and hypotheses prior to the pre-pilot. One of the main goals of the graduate student pre-pilot was to determine the feasibility of the experiment in the allotted time with main study subjects. Both the pilot study and pre-pilot exercise would also serve to identify any operational flaws with the two interfaces or usability concerns that the graduate students thought needed correction. The researcher recruited 50 pharmacy students who were not part of the study population for an additional pilot study. Nine of the eligible fifty pharmacy students participated in the pilot experiment. The pilot study was conducted in a different location than the main study. In this location, the pilot subjects sat in chairs 17.5 inches from the ground and had available desks 29 inches from the ground, 17.5 inches wide and 59.5 inches long. An International light INC IL 1350 radiometer/photometer verified that the light levels at each subject?s desk work area met standards that previously produced reductions in errors for pharmacy related tasks and were within Illuminating Engineering Society (IES) of North 45 America standards (Buchanan et al., 1991, Ies lighting ready reference, 1984). The light levels for all locations fell between the IES healthcare facilities recommended level, for areas such as nursing station desks and general pharmacy, of 50-100 footcandles (fL). The light levels on the diagram depicted (in Appendix 1) demonstrate desk level light meter readings for each subject work area. Upon entering the experiment room, all of the subjects removed and stored their watches, cell phones, handheld computers and any other pieces of technology. The subjects could not access these during the study. Additionally, no other source of time measurement was available to the subjects. The exact length of the experiment was unknown to the subjects. The researcher informed all subjects about the experiment using a prepared information letter (in Appendix 2) that described the experiment and its potential benefits. The subjects knew that the commitment for the study would not exceed two hours. All participants had prior instructions to bring any corrective eye wear in order to participate in the experiment. Any student who did not have his or her corrective eye wear or who could not use a PDA independently was excluded. An anonymous unique identification used throughout the experiment consisted of last four digits of the subject?s social security number and first three digits of his or her hometown zip code. This ID would allow subjects to access their results after the conclusion of the study but would not allow the researcher to identify the subjects. 46 As a result of the pre-pilot and pilot study, the researcher modified instruments used in the experiment and adjusted methods for collecting data in the following ways: From pre-pilot: ? Instructions given to subjects via a verbal scripted presentation received revisions to clarify navigation on the PDA, use of and activation of the onscreen scroll bar, the understanding of switch between the two versions used on the PDAs, and establishment as use of unique user ID. ? In the scripted presentation, increased the number of captured screen images for the various entry screens encountered. ? Increased the length of user demonstration, doubled from 80 seconds to 160 seconds, to increase familiarity of versions and the switch between versions. ? The time component of the PDA event logging did not perform as expected. Subsequent software modifications provided the appropriate time data in the pilot exercise. From pilot study: ? The need to keep a synchronized central master experiment time to relate distractions and event logged actions on the PDA. ? Creation of a single recorded scripted presentation for each session to reduce variability. During the pilot session, one subject experienced a near loss of battery power. As a result of this encounter, the researcher designed a procedure to 47 assess each PDA?s in situ battery life. The researcher conducted a test of each PDA?s battery through one hour of tapping within the software used in this study. An automated tapping device (in Appendix 3) constructed by the Auburn University steel maintenance shop using a J Engelsmann AG? automated piston device served as the tapping instrument. This apparatus provided a cradle to hold the PDA inverted over a piston holding a PDA stylus that could tap the PDA screen repeatedly. The researcher turned each PDA on and placed it in the apparatus for one hour to establish if it could maintain power for the duration of the PDA experiment. All PDA?s performed satisfactorily in the battery life test. Main Study Subjects and Sign-Up The sample for this study consisted of students from the spring semester of 2006 in their 1 st year in the Doctor of Pharmacy program at the Auburn University Harrison School of Pharmacy. The class was selected due to the large number of potential subjects and the ability of the researcher to arrange for inclusion of the experiment into the participants syllabus for the Contemporary Aspects of Pharmacy Practice (CAPP) course. All data were anonymous and none of the data served as an evaluation of a student during the course. Subjects could elect for their data not to be included in the research experiment and only data for subjects that agreed to participate in the study appear in the final analysis. The researcher had only a limited number of the same model PDAs, so five separate sessions were necessary. All of the sessions took place in the same location, four at 2 PM and one session, # 4, took place at 8 48 AM. The experiment period lasted from 04/17/2006 to 04/21/2006. No significant events occurred during the five data collection periods. All of the data collected for subjects occurred in the Contemporary Aspects of Pharmacy Practice (CAPP) laboratory due to its ease of access and ability to accommodate the subjects and equipment. The researcher informed all subjects about the experiment using a prepared consent form (in Appendix 4) that described the experiment and its potential benefits. The subjects knew that the commitment for the study would not exceed two hours. All participants had prior instructions to bring any corrective eye wear in order to participate in the experiment. Again, students who did not have their corrective eye wear or who could not use a PDA independently was excluded. As before in the pilot, an anonymous unique identification used throughout the experiment which would allow subjects to access their results after the conclusion of the study but would not allow the researcher to identify the subjects. Procedure The procedure for this experiment included the development of the simulation software that served as an instrument for measuring error rate for order entry and time to complete order entry tasks as well as a set of pharmacy-related questions for projected display to serve as a distraction. The administration of a previously validated and commercially available group embedded figures test (GEFT) (Mindgarden, Inc.) followed the use of the software programs. The subjects used the paper modified versions of the commercially available NASA- 49 TLX (in Appendices 5A & B) developed by Hart and Staveland (1988) to provide subjective workload ratings after conducting order. Finally, collection of demographic information through a modified version of Staggers Nursing Computer Experience Questionnaire (SNCEQ ? ) (Staggers, 1998) as found in Appendix 6. Application Development The researcher worked with an experienced software developer to create the PDA simulation versions used in this experiment. Prior to development, the researcher evaluated several current commercially available handheld e- prescribing systems to determine what common functionalities existed for users of these medication order entry systems. After reviewing these systems, the researcher created a common framework for the CPOE interface based on these commercial systems. One of the most common features found in these systems was the use of sequential or alphabetical lists for selection of patient names, medications, and so forth. Two versions of this PDA CPOE interface were created, one with an expanded target interface and one without and expanded target interface. Other than the presence this of this expansion the two interfaces were identical. In the control version, all items on the screen appeared in a sequential list at standard Palm? font AFPalm size 11(see figure 2A). Figures 2A. Example of PDA Figure 2B. Example of PDA non-expanded expanded The screen of the non-expanded version moved with scrolling by use of on-screen arrows. This scroll function moved the last name on a screen to the top of the next screen and displayed subsequent names sequentially for downward scroll (see Figures 4A & B) or moved the first name on top of screen to bottom of next screen in reverse fashion for upward scroll. Once the desired target item appeared on the screen, users selected the item on screen by tapping with their stylus which highlighted that selection (see figure 4C). A second tap of the item when highlighted entered the item and proceeded to next screen. If a user tapped and highlighted an incorrect item, a user simply tapped the correct item to move highlighted region over the correct choice. The researcher decided to use the expanded target method as a stimulus in this experiment. The expanded target area consisted of a designated item and any adjacent item above and below in the list which all appeared in Palm? regular font size 14 (see figure 2B). Otherwise the remaining items appeared at standard Palm? font AFPalm size 11 (72 points = one inch) used in the control version 50 interface. Only a limited set of fonts, for both expanded and non-expanded displays, were available in the Palm? operating system. Smaller font size determination for the remaining list in expanded version and entire list in non- expanded version relied on human factors engineering visibility standards for alphanumeric characters for visual acuity up to 20/200 at normal reading distance of 12-18 inches (Sanders & Mccormick, 1987). The expanded target area on the screen also provided target separation for the user interface by directing attention to the expanded areas. Figures 3A. Select frequency screen in expanded version before selection Figure 3B. Select frequency screen in expanded version after selection 51 Like the non-expanded version in the expanded target version the user could only perform order entry for the item highlighted, now in the expanded target region. In order to select a desired item not in this field, the use of scroll function or stylus tapping was necessary to move the desired target into the expanded target region for subsequent entry. When a user tapped an item on the screen, the expanded area appeared and adjusted to have this item centered and highlighted in the expanded target area at the aforementioned font sizes (see Figures 3A & B). Once an item was highlighted, it was tapped a second time for entry. Figure 4A. Select Patient Screen non-expanded 52 Figure 4B. Select Patient Screen non-expanded after downward scroll Figures 4C. Select Patient Screen non-expanded, after downward scroll and selection Verification for each unit also included the appropriate sequence of screens as well as interface presentation. After development, testing of the system included verifying that the PDA entry tasks performed in the experiment were adequately accomplishable on each PDA. A set of ten orders (in Appendix 7) entered on each PDA after installation of each respective software version confirmed that each PDA collected all of the data elements necessary for analysis for each order entered. In order to provide accurate and synchronized time data in the log file, a commercial clock synchronization program, AkClockSync? 53 54 (Akeysoft Group), was loaded on each PDA. This program synchronized the PDA time clock with a computer clock which was synchronized with an internet atomic clock at http://www.time.nist.gov. Additionally, an expert panel that included experienced HCI and Informatics researchers reviewed the two systems to ensure appropriate design. Each system collected data through event logging, storing all screen interactions and positions of a user in a database on the PDA. Synchronizing the PDA removed the database which then required conversion to a text file. The text file produced the data used to determine such as if an entered order contained an error and the time each order took to process. The text file data elements included: 1. Time- Time of entry accurate to seconds 2. Subject- Subject ID 3. Order- Order number 4. Action - When an item was tapped on the screen, the data file produced an ?S? if it was the initial tapping of this item or ?R? or ?W? if the second time it was tapped was right or wrong 5. Clicked -?S? for stimulus or ?C? for control 6. Screen- Screen type,?P? for patient, ?D? for drug, ?S? for strength, ?A? for dose amount, and ?F? for frequency Time Subject Order Action Clicked Stimulus Screen 5:32:08 PM 9027360 1 Reeves, Christine S C P Figure 5. Example of a PDA data record for one subject?s PDA entry 55 From the text file data, subsequent extrapolation in a spreadsheet program provided the base data (see Figure 5) used for all subsequent analyses that produced results found in the results and discussion sections. Data Collection Prior to the subjects arriving in the room used for this study, the researcher prepared the CAPP laboratory by arranging desks in rows. Each row had no more than four desks. The height of the workbench desks equaled 36 ? inches above the ground, 30 inches wide and 60 inches long. The CAPP laboratory does not contain chairs for the workbench desks for students to sit in during the experiments. A light meter verified that the light levels at the desk work areas met standards that previously produced reductions in errors for pharmacy related tasks and within Illuminating Engineering Society of North America (Buchanan et al., 1991, Ies lighting ready reference, 1984). The light levels for all locations fell between the IES healthcare facilities recommended level, for areas such as nursing station desks and general pharmacy, of 50-100 footcandles (fL). The light levels on the diagram (in Appendix 8) demonstrate desk level light meter readings for each subject work area with an International light INC. IL 1350 radiometer/photometer. Upon entering the CAPP laboratory, all of the study subjects removed and stored their watches, cell phones, handheld computers and any other pieces of technology. The subjects could not access these during the study. Additionally, no other source of time measurement was available to the subjects. The exact length of the experiment was unknown to the subjects. 56 The subjects received PDAs using the Palm? operating system (version 3.5.3) with the standard resolution of (160 x 160) with 8bit Active Matrix TFT color display. The menu of items for selection, arrows and numbers all were in black font on contrasted illuminated gray background. The only data appearing in color on the screen is the highlighted blue screen name at the top of the screen. The subjects then received instructions (in Appendix 9) on how to use the PDAS in a pre-recorded PowerPoint? (Microsoft, Redmond, WA.) presentation that included screen shots of the software. Part of the familiarization also included four 40 second demonstration blocks, identical to the experiment design, that accustomed the subject to stylus pointing, tapping and how to enter medication orders with both versions with a PDA. After this demonstration, a question and answer period immediately followed which allowed for any clarification on order entry. Questions for clarification focused on the entering of the user ID and switch between modes. In order to simulate the distractions found in healthcare settings and other environments, the researcher would direct at random intervals the study subjects attention toward a projection screen when alerted by the researcher with an alarm sound. The alarm was sounded when a question randomly appeared. This LCD projected screen contained multiple choice questions (in Appendix 10) from a past course, pharmaceutical calculations, as well as general knowledge questions. The subjects knew that the questions would only appear for an unknown brief period (15 seconds) and that the percentage of correct answers to these questions was part of the experiment?s results. The subjects who placed their anonymous 57 unique ID on a separate scantron answer form would then answer the questions using a # 2 pencil provided. The researcher noted the time on the master experiment time of the distraction for each study group for subsequent data analysis. At the beginning of the PDA order entry portion of the experiment, each subject received one of thirty-six unique sets of 300 typed orders similar to the sample set (in Appendix 7). After all of the subjects logged into their PDA systems, the experiment began with the subjects proceeding to enter as many of these 300 orders as they could until time expired (60 minutes). The order entry experiment took place in a crossover repeated measures design with four distinct block trials of fifteen minutes so each subject would use each version twice (i.e. A/B/A/B or B/A/B/A). Subjects used a stylus to select an item on the screen for entry in both versions. One-half of the PDAs for this study started with one of the versions for medication order entry, one-half with the other version both of which were in one program loaded on each PDA. The evenly numbered PDAs began with the control version first, odd number began with stimulus version first. The researcher randomly distributed the PDAs to subjects to achieve as close to an even distribution of starting versions per session as possible. An equal distribution of even and odd numbered PDAs was not possible in each session due to varying class size. In addition, some subjects restarted their PDAs during the experiment and dropped from the data pool which affected the evenness of PDA distribution in final results. Each PDA changed between versions after each block trial time period elapsed. An alert screen (see Figure 6) preceded the change and required the subject to tap a button to acknowledge the mode switch and proceed with order entry. Figure 6. Alert screen for mode switch After they logged on, each subject encountered a screen that required patient selection by stylus tapping (see Figure 4A). Patient names appeared in alphabetical order on the screen from top to bottom. The 300 patient names used in the experiment on the order sheets and PDAs (in Appendix 11) were 150 each of the most commonly used male and female names in the 1960 U.S. census. Each name was used only once in an order set. The entire list of patient 300 patient names served as a master list. The patient names used had no a priori determination for look-alike or sound-alike characteristics. Once a subject selected a patient?s name, they then proceeded to a medication look up screen that functioned exactly as the patient look up screen on both versions (see Figures 7 & 8). 58 9. Figure 8. Select Medication Screen non-expanded Figure 7. Select Medication Screen expanded Medication names for the order sheets (in Appendix 12) came from a list of 154 medication names previously identified by the Joint Commission for Accreditation of Healthcare Organizations (JCAHO) and The United States Pharmacopoeia (USP) as look-alike or sound-alike medication names. For example, Lamictal? and Lamisil? are two medication names previously identified as look-alike. An algorithm that required each medication be used at least once but no more than twice generated the 300 medication orders used in the experiment. A master medication list created originated from the USP and JCAHO lists plus the adjacent medication names found in the Food and Drug Administration (FDA) Orange Book. Additionally, the order sets were synchronized with the patient and medication master lists to a particular PDA. Each order set was matched so that the PDA patient and medication screen lists 59 showed the item on the order sheet and thirty two surrounding choices from a master list, which occupied only three screens at a time, and helped to minimize scrolling in the experiment. The researcher used this minimized scrolling feature to increase the number of events in PDA order entry. The location of the order set item (i.e. patient or medication name) on these three screens was random. After the medication was selected, a subject then proceeded to subsequent screens for providing the details for the medication such as dose amount, strength and frequency (see Figures 9-11). Figure 9. Initial Select Amount Screen non-expanded and expanded version Figure 10. Initial Select Strength Screen non-expanded and expanded version 60 Figure 11. Initial Select Frequency Screen non-expanded and expanded version These screens functioned identical to the previous screens in both expanded and non-expanded version of the software. To select an item from the menu required tapping with the stylus. All possible choices appeared in the menu list that otherwise functioned as the patient or medication lists in their respective version. An algorithm used to create the order sheet sets selected the choices for dose amount, strength and frequency from standard lists in the frequencies listed. At the conclusion of the sixty minute PDA order entry portion, the subjects proceeded to complete the paper instruments. The subjects received scripted and verbal instructions on how to complete the paper versions of each NASA-TLX survey using their unique user ID. The researcher also distributed definitions of the six sub-scales used the NASA-TLX survey for each PDA version. Participants then received instructions from the GEFT manual on how to complete the GEFT test. A practice period was part of the instructions in the GEFT manual. Subjects then completed a standardized paper and pencil GEFT 61 62 that would establish whether they had field dependence or independence. After all subjects completed all test instruments, they completed the SNCEQ demographic information form (in Appendix 6). An a priori ? level of .05 for each one-tailed t-test and ? level of .025 for each two-tailed t-test was established. 63 IV. RESULTS A total of 113 subjects (43 males, 62 females and 8 unidentified) participated in the five experimental data collection sessions. The mean age for all participants was 24 years of age and the median age was 23 years of age with the youngest subject aged 20 years and the oldest at 35 years of age. Scores on the GEFT test demonstrated a total of 64 (57%) of subjects as field independent and 49 (43%) of subjects as field dependent. Not all subjects had full data available for analysis and thus were not included in final analysis. Cases were dropped during data analysis due to missing data elements from PDAs, from missing NASA-TLX data or outlier PDA error data as follows: ? For hypotheses that used error rates and time (1-6), a total of 101 subjects had full, usable PDA data files or were not identified as outliers. Subjects that restarted their PDAs during the exercise were dropped from the data pool. Three subjects were dropped from the data pool because they were identified as outliers. One participant was dropped for an extremely high overall error rate of 83% across all trials. The highest overall error rate for all other subjects was 7.6% and the mean error rate for the study was 1.1%. The researcher dropped two participants for disproportionately high error rate in both second block trials. For example, one of these subjects had an error rate of 0 % in trial one for both the expanded and non- 64 expanded interfaces. In contrast, this subject had an error rate of 10% in trial two of the non-expanded interface and 21% in trial two of the expanded interface. The mean number of orders entered per minute for the non-expanded interface was 11.05 and for the expanded interface 10.3. For the remainder of the hypotheses, only 109 subjects had complete NASA-TLX data for both the expanded version and non-expanded version that were used in final analysis. In addition, the number data entry events including errors that took place during or after a distraction was very minimal and not included as a separate analysis. Statistical Analyses The researcher used SPSS version 12 for all analyses with the exception of power analysis. All non-significant independent t-test results had power analysis performed with an online calculator from DSS Research (Retrieved June 6, 2006, from, http://www.dssresearch.com/toolkit/spcalc/power_a2.asp). The error rate data for hypotheses 1-3 required a logit transformation because the data were not normally distributed and were right skewed toward zero. The logit transformation stretched the tail of the distribution as recommended for proportions by Cohen and Cohen (1983). The following steps occurred in the logit transformation: 1. For subjects with error rates of P = 0, compute an adjusted error rate with P = 1/ (2*number of entries). 2. For all subjects compute logit error rate values L = 1/2 ln [(P)/1-P]. 65 All analyses for error rates were performed on the transformed data. However, the original means and standard deviations are reported for ease of readability. For an example of what the data record for a subject in the study used in analyses looked like (see Appendix 16). In this data record for example, an error was identified on order number four when the patient ?Clay, Wilma? was selected and entered instead of ?Cross, Joyce.? Hypotheses for Error Rates H A 1: The expanded target interface will have a lower error rate for order entry than the non-expanded target interface. The mean total error rate for the non-expanded and expanded interfaces respectively were M = 0.010, SD = 0.021 and M = 0.011, SD = 0.018. The pairwise sample t-test for type of interface was not significant with respect to error rate, t (100) = -1.614, p < 0.945, one-tailed, since the result was in the opposite direction than predicted. In other words, the expanded target interface had a non-significantly higher error rate than the non-expanded interface. A pairwise t- test for the effect of each interface type on error rate in Trial 1 was non-significant (expanded Trial 1: M = 0.011, SD = 0.016; non-expanded Trial 1: M = 0.018, = 0.067; t (100) = -0.421, p < 0.674, power = .18). A pairwise t- test for the effect of each interface type on error rate in Trial 2 was non-significant (expanded Trial 2: M = 0.010, SD = 0.027; non-expanded Trial 2: M = 0.009, SD = 0.022; t (100) = -0.823, p < 0.412, power = .06). 66 A follow-up analysis on error rates to test for a learning effect from first trial to second trial for both interfaces was conducted. A pairwise t-test for the effect of trial on error rate for order entry was significant for the non-expanded interface (Trial 1: M = 0.018, SD = 0.067; Trial 2: M = 0.009, SD = 0.022; t (100) = 2.74, p < 0.007, two-tailed). A pairwise t-test for the effect of trial on error rate was significant for the expanded interface, (Trial 1: M = 0.011, SD = 0.016; Trial 2: M = 0.010, SD = 0.027; t (100) = 2.43, p < 0.017, two-tailed). H A 2: Field independent subjects will have a lower error rate for order entry than field dependent subjects. The independent t-test for the effect of field independence on error rate for order entry was non-significant (field independent subjects: n = 58, M = 0.003, SD = 0.004; field dependent subjects: n = 43, M = 0.007, SD = 0.013; t (99) = 1.432, p < 0.0778, one-tailed, power = .62). In order to test whether field independent and field dependent subjects differed in the learning effect between trial one and trial two for each of the interfaces a difference variable was computed and used as the dependent variable in an independent t-test. In the non-expanded interface an independent t-test was non-significant for a learning effect, (field dependent subjects: M diff = 0.020, SD diff = 0.096, field independent subjects: M diff = 0.001, SD diff = 0.002; t (99) = 1.272, p < 0.210, two-tailed, power = .25). In the expanded interface, an independent t-test was non-significant for a learning effect, (field dependent subjects: M diff = 0.001 SD diff = 0.003, field independent subjects: M diff = 0.001, SD diff = 0.023; t (99) = 0.173, p < 0.863, two-tailed, power = .05). 67 H A 3: The expanded target interface will reduce the error rate for order entry for field independent subjects less than it will for field dependent subjects In order to test whether field dependent subjects benefited more from the expanded target interface in reducing their order entry error rate, a difference variable was computed between the error rates for the two types of interfaces. This difference variable served as the dependent variable in an independent t-test. Results of the test yielded a non-significant difference in the total error rate between the two types of target interface for field independent and field dependent subjects, (field dependent subjects: M diff = 0.012, SD diff = 0.020, field independent subjects: M diff = 0.010, SD diff = 0.017; t (99) = 0.851, p < 0.199 one- tailed, power = .13). Hypotheses for Time of Order Entry H A 4: The time for order entry will differ for the expanded target and non-expanded target interface. To test this hypothesis, a paired samples t-test was performed using the number of medication order entry tasks per minute as the dependent variable and the interface versions as the independent variable. The type of interface had a significant effect on number of orders entered per minute, (expanded interface: M = 10.356, SD = 1.849, non-expanded interface: M = 11.052, SD = 2.110; t (100) = 4.499, p < 0.01, two-tailed). Further analysis with paired sample t-tests attempted to determine if a learning effect existed such that users had a different number of entries per minute 68 in the second trial versus the first trial for the interfaces. The expanded interface had a significant learning effect for the number of entries per minute, (Trial 1: M = 9.974, SD = 1.915; Trial 2: M = 10.809, SD = 2.272; t (100) = -4.465, p < 0.01, two-tailed). The non-expanded interface had a significant learning effect for the number of entries per minute, (Trial 1: M = 10.626, SD = 2.231; Trial 2: M = 11.979, SD = 6.525; t (100) = -2.087, p < 0.039, two-tailed). H A 5: Field independent subjects will be quicker at order entry than field dependent subjects. The independent t-test result for the effect of field independence on the number of orders entered per minute was non-significant, (field independent participants: n = 58, M = 10.779, SD = 1.691; field dependent participants: n = 43, M = 10.593, SD = 2.151; t (99) = 0.468, p < 0.641, one-tailed, power = .12). In the non-expanded interface an independent t-test was non-significant for a learning effect, (field dependent subjects: M diff = -1.079, SD diff = 2.293, field independent subjects: M diff = -1.556, SD diff = 8.395; t (99) = 0.362, p < 0.718, power = .07). In the expanded interface, an independent t-test was non-significant for a learning effect, (field dependent subjects: M diff = -0.8341, SD diff = 1.595, field independent subjects: M diff = -0.8356, SD diff = 1.960; t (99) = 0.004, p < 0.997, power = .05). H A 6: The relative time for order entry with expanded target and non-expanded target interfaces will differ for the field independent and field dependent subjects. 69 The mean number of orders entered per minute for field independent individuals for non-expanded and expanded interface versions respectively were M = 11.216, SD = 1.965 and M = 10.356, SD = 1.740 while for field dependent individuals it was M = 10.830, SD = 2.467 and M = 10.355, SD = 2.007. The independent t-test result for the effect of field independence on the difference in number of orders entered per minute for the non-expanded versus expanded interfaces was non-significant, (field independent: M diff = 0.8603, SD diff = 1.526; field dependent: M diff = 0.4814, SD diff = 1.564; t (99) = 1.221, p < 0.227, two- tailed, power = .23). Hypotheses for NASA-TLX workload ratings H A 7: The subjective workload ratings will differ for the expanded target and non-expanded target interfaces. In the NASA-TLX subjective survey, a higher score indicated a greater perceived workload for an interface version. A paired sample t-test of NASA- TLX score for type of interface was significant, (expanded interface: M = 57.85, SD = 16.624; non-expanded interface: M = 51.63, SD = 19.085; t (108) = -4.092, p < 0.01, two-tailed). H A 8: The subjective workload ratings will differ for field independent and field dependent subjects. Independent t-tests for the effect of field independence on the NASA-TLX index for each interface were non-significant. The results for expanded interface were (field independent subjects: M = 56.76, SD = 17.526; field dependent subjects: M = 59.64, SD = 15.327; t (108) = 0.898, p < 0.371, two-tailed, power = 70 .23). Results for the non-expanded interface were (field independent: n = 63, M = 49.11, SD = 20.118; field dependent subjects: n = 47, M = 55.43, SD = 17.158; t (108) = 1.732, p < 0.086, two-tailed, power = .55). H A 9: The relative subjective workload ratings for expanded target versus the non-expanded target interface will be lower for the field independent than the field dependent subjects. An independent t-test of the effect of field independence on a difference variable between the workload scores for the non-expanded and expanded interfaces yielded a non-significant effect on NASA-TLX score, (field dependent: M diff = -5.297, SD diff = 16.630; field independent: M diff = -7.298, SD diff = 15.503; t (107) = 1.103, p < 0.273, one-tailed, power = .16). Post-Hoc Analyses A post-hoc analysis testing the effect of certain demographic factors on the dependent variables of error rate, time of order entry, and NASA-TLX workload score yielded several significant results. The demographic factors and questions selected were: ? Biological sex ? Estimate of errors made ? Likert questions- PDA Use, PDA Experience, PDA Knowledge and PC Experience 71 Of the total study population reported, females made up sixty-two percent of the population which is common in pharmacy school classes today. In this experiment, there was no hypothesis proposed for a difference in performance for biological sex. However, the researcher used biological sex as an independent variable with the study variables in independent t-tests to assess if males and females differed with regard to the dependent variables. Biological sex produced significant results in the time for order entry across both interfaces and for order entry in the expanded interface. An independent t-test demonstrated that biological sex had a significant effect on the number of entries per minute overall, (men: n = 39, M = 10.161, SD = 1.733; women: n = 57, M = 10.970, SD = 1.870; t (94) = -2.143, p < 0.035, two-tailed). There was also a significant effect from an independent t-test for the number of entries per minute based on biological sex within the expanded interface, (men: M = 9.732, SD = 1.699; women: M = 10.703, SD = 1.828; t (94) = -2.263, p < 0.01, two-tailed). Biological sex had no significant effect in the number of entries per minute within the non-expanded interface, (men: M = 10.619, SD = 1.987; women: M = 11.254, SD = 2.247; t (94) -1.422, p < 0.158, power = .31). A Chi-square analysis demonstrated no relationship between biological sex and field independence (? 2 [1] = 0.011, p < 0.916). No significant correlation existed between the number of actual errors made and estimate of errors (r = 0.045, p < 0.669). Also, the same correlation was performed for males and females separately with no significant results (males: r = 0.003, p < 0.983; females: r = 0.067, p < 0.638). 72 The demographic questionnaire contained several questions that asked participants to rate their PDA Use, Knowledge and Experience as well as desktop/laptop/tablet PC Experience. The answers for PDA Use and Knowledge ranged on a Likert scale from 0 (none) to 4 (extensive) and for PDA and PC experience from 0 (novice) to 6 (expert). Descriptive statistics are as follows: (PDA Use: M = 0.44, SD = 0.89, skewness = 2.402, kurtosis = 5.715; PDA Knowledge: M = 0.38, SD = 0.793, skewness = 2.558, kurtosis = 6.862; PDA Experience: M = 1.91, SD = 1.419, skewness = 1.904, kurtosis = 3.209; PC Experience: M = 4.32, SD = 1.2=1.328, skewness = -0.617, kurtosis = 0.200). Kolmogorov-Smirnov test indicated that these four variables departed significantly from normality, (PDA Use: t (93) = 0.421, p < 0.01; PDA Knowledge: t (93) = 0.435, p < 0.01; PDA Experience: t (93) = 0.299, p < 0.01; PC Experience: t (93) = 0.211, p < 0.01). The answers for PDA Use, Knowledge, Experience and PC Experience all had a significant right skew in distribution. For example, the majority (72%) of subjects rated themselves as novices with scores of zero on PDA Use and Knowledge. The researcher performed post-hoc analyses for the effect of biological sex on the answers given for these factors. For PDA Use, an independent t-test established a significant difference between males and females, (males: M = 0.82, SD = 1.189, females: M = 0.22, SD = 0.503; t (88) = 2.979, p < 0.005, two-tailed). In addition, an independent t-test established that biological sex had a significant effect on PDA Knowledge, (males: M = 0.64, SD = 0.986, females: M = 0.24, SD = 0.619; t (88) = 2.252, p < 0.028, two-tailed). For PDA Experience, an 73 independent t-test established biological sex had a non-significant difference, (males: M = 2.29, SD = 1.675, females: M = 1.67, SD = 1.178; t (88) = 1.959, p < 0.054, two-tailed). In an independent t-test for PC Experience, biological sex did not show a significant difference, (males: M = 4.47, SD = 1.39, females: M = 4.25, SD = 1.278; t (88) = 0.769, p < 0.444, two-tailed). Given that the sample reported almost no experience with and knowledge of PDAs, there was no basis for dividing the sample into experienced versus novice PDA users. Consequently, no t-tests were run to investigate whether PDA experience and knowledge had an effect on the three dependent variables of error rate, time of order entry and workload rating. 74 V. DISCUSSION AND IMPLICATIONS This chapter will discuss the findings of this research experiment, their implications, their limitations and possible future directions of this study. General Findings This research experiment sought to determine whether an expanded target interface could improve the efficiency and effectiveness of a medication order task entered on a PDA when compared to a standard interface. In a repeated measures crossover design subjects entered medication orders using both interfaces in order to assess individual performance with each interface. This study measured performance using two primary objective measures (error rate and time of order entry) and one subjective measure (workload). In this experiment, an expanded target interface performed exactly as the non-expanded interface except for when a subject tapped an item in a list. The tapping and subsequent highlighting of an item onscreen in the expanded interface caused the item and any immediately adjacent items to increase in font size and separate more from surrounding items in a list. The expectation was that the expanded target interface would generally provide improved performance through reduced error rates and decreased perceived workload. In addition, the 75 experiment checked whether the time per order entry differed between the two interfaces. The ability of individuals to locate objects within a pattern or field of similar objects can vary from person to person. Someone who can easily perceive a desired object when it is located within a field of similar objects is field independent. Given their inclination for finding items or patterns, field independent subjects were expected to have lower overall error rates and times for order entry. In addition, field independent subjects were expected to have lower overall perceived workload. The expanded target was expected to provide a greater benefit for the field dependent participants than for the field independent participants. The relative benefit of the expanded interface over the non-expanded interface was expected to be significantly greater for the field dependent than for the field independent subjects. The results for type of interface did not support the general expectation that the expanded interface would improve the entry of medication orders with a PDA in comparison to the non-expanded interface. In fact, the results indicated the opposite because the non-expanded interface yielded significantly better performance than the expanded interface. First, the results did not indicate a significantly lower overall error rate for the expanded target interface versus the non-expanded target interface as H A 1 predicted. In fact, the non-expanded interface had a non-significantly lower overall error rate, which was the opposite of the prediction in this hypothesis. 76 There were two possible arguments with regard to the expected effect of the expanded target interface on the speed of order entry. The expanded target interface might improve the clarity of the target items, which could improve order entry. On the other hand, the extra perceptual adjustment to the expansion could slow down order entry. Given these circumstances and the lack of previous evidence, the researcher proposed H A 4 as a non-directional hypothesis. H A 4 predicted a difference in the time for order entry between the interface versions. This hypothesis was supported by significant results overall. The non-expanded interface had a significantly higher number of orders entered per minute than the expanded interface. For similar reasons stated previously in regard to H A 4, H A 7 was formulated as a non-directional hypothesis and asserted that NASA-TLX workload ratings would differ between the two interfaces. This hypothesis was supported by significant results in which the expanded interface received higher workload ratings compared to the non-expanded interface. In summary, the expanded interface did not significantly lower error rates compared to the non-expanded interface. A significant reduction in time for order entry did occur but not with the expanded target interface but rather with the non- expanded interface. Finally, a significant difference in workload ratings existed between the non-expanded and expanded interfaces with the lower workload ratings received by the non-expanded interface compared to the expanded interface. The expanded interface did not lower error rates for order entry on a PDA. However, it did slow down order entry and it did create the perception of a greater workload. 77 The performance results for subjects, based on their degree of field independence, also deviated from expectations and none of the corresponding hypotheses had significant results. Although the mean error rate for field independent subjects was lower than the mean error rate for field dependent subjects as H A 2 predicted, it was not significantly lower. Also, H A 5 predicted that field independent subjects would have quicker order entry times than field dependent subjects. Although field independent subjects had more entries per minute, this result was not significant. Finally, H A 8 had the expectation that a difference between field independent and field dependent subjects in workload ratings for the interfaces would take place. Although the mean workload ratings for field independent and field dependent subjects for both interfaces showed some variance, they were not significantly different. In the expanded interface mean workload ratings were, (field independent: M: 56.76, SD = 17.526; field dependent: M = 59.64, SD = 15.327; power = .23). For the non-expanded interface mean workload ratings were, (field independent: M = 49.11, SD = 20.118; field dependent: M = 55.43, SD = 17.158; power = .55). These non-significant results demonstrate that in both interfaces, field independent subjects had lower workload scores compared to field dependent subjects. The expanded target interface possibly might not provide any additional performance benefits for a user in general compared to a non-expanded interface. However, the expanded target could potentially provide a distinct advantage for field dependent subjects compared to field independent subjects. Specifically, H A 78 3 predicted that the expanded target interface will reduce the error rate for order entry for field dependent subjects more than it will for field independent subjects. But the non-significant results did not support the hypothesis that the expanded interface improved the performance of the field dependent subjects significantly more than the performance of the field independent subjects. H A 6 predicted that the relative time for order entry with expanded target versus non-expanded target interfaces will differ for the field independent and field dependent subjects. The prediction in this hypothesis did not have significant results. Finally, H A 9 predicted that the relative difference in subjective workload ratings for expanded target versus the non-expanded target interface will be lower for the field independent than the field dependent subjects. This prediction was not supported by significant results. The expanded target interface did not provide the expected benefits over the non-expanded interface. Although significant differences between them in time for order entry and NASA-TLX did occur, in both cases the non-expanded target interface was the superior version. As a whole, no improvement in error rate with the expanded interface occurred for the study subjects. For field independent and field dependent subjects, neither interface proved more beneficial for either error rate or time for order entry. Finally, the workload ratings of the expanded target versus the non-expanded target interface occurred in the opposite direction such that the expanded interface received higher workload scores from both field dependent and field independent participants. Put quite simply, the non-expanded interface generally had quicker time for order 79 entry, received significantly lower workload ratings from both field dependent and independent subjects than did the expanded interface, and proved no different from the expanded interface in regard to error rate. This study sought to determine if an expanded target interface generally was better than a non-expanded interface. The results of this study didn?t support the conclusions found in the study hypotheses which expected the expanded interface to perform better than the non-expanded interface. First, the expanded target interface did not reduce error rate significantly compared to the non- expanded interface. Further, the expanded target with its higher workload ratings compared to the non-expanded interface required more not less perceived workload from subjects in their order entry tasks. Finally, a significant difference in time for order entry in general existed, however it was the expanded interface which had a slower order entry time versus the non-expanded interface. One possible explanation for the poor performance of the expanded target centers on how the researcher operationalized the expanded target on the PDA. The method of creating an expanded target area used in this study was one of several ways in which the expansion was possible on a PDA. For example, in the expanded interface, prior to a subject tapping an item on a screen, the list of items appeared in the non-expanded presentation (Figure 12A). When an item was tapped it became highlighted and expanded. In addition, an expansion area was created where any adjacent items to the highlighted item also expanded (See Figure 12B). The process of creating the expansion area caused a small shift in item location for all items in the expansion area and any items below the expansion area. This shift was the result of the decision by the researcher not to reduce the number of items on the screen by one item when expansion occurred. If the software had reduced the number of items on the screen by one, then one item not in expansion area would have shifted off-screen as expansion occurred leaving the items found within the expanded area stationary while all other items moved slightly in their position onscreen. Instead, in the version used in the study, the expansion area and all other items below this area shifted downward slightly on the screen. The non-expanded version had no comparable shift in items. Figures 12A and 12B illustrate the performance of the expanded target area and the shift of items. Note that the target ?Wilkerson, Sue? shifted down by almost a full line in Figure 12B. If a subject intended to maintain focus on the target, the subject must refocus down by almost one line. This extra step requires both physical and psychological adjustment. 80 Figure 12A. Select Patient Screen Expanded version before tap Figure 12B. Select Patient Screen Expanded version after tap 81 Having to refocus to check the item highlighted apparently increased the subjective workload. Presumably a subject had to slightly shift their second tap in order to achieve entry which could increase the time of order entry. The researcher anticipated that the expansion of items on the PDA would allow easier and quicker location for all participants, and especially for field dependent subjects. This would ultimately result in more items entered in the expanded target interface. The results did not support this expectation and the non- expanded interface was more efficient for all subjects (i.e. more orders entered in the non-expanded interface). There is another possible way in which the target expansion may have also contributed to the slower entry time in the expanded interface. The PDAs used in this experiment had 8 Mb of Random Access Memory (RAM) and a 20 MHz processor. Elements such as RAM, memory and processor power strongly affect the performance of a computer, such as a PDA. The expansion of the onscreen elements in the expanded version required additional PDA computing resources which could have resulted in actual or perceived delays and resulted in a higher workload rating. These delays could have also decreased order entry time in the expanded interface compared to the non-expanded interface. As expected, the NASA-TLX workload scores did significantly differ between the two interfaces. However, the expanded target interface was supposed to provide a more pleasant user experience through its larger item size and separation which would translate into lower workload for users. Contrary to this expectation, the workload scores for the non-expanded interface were lower than 82 for the expanded interface. The fact that the expanded version had a significantly higher workload rating could stem from previously mentioned issues with the expansion area shifting and/or from the PDA performance during this operation. The researcher anticipated that field dependent subjects would prefer the expansion more than field independent subjects since it caused highlighted items to stand out from the general background field. One of the primary factors that makes someone field dependent is difficulty in separating items within a field, especially when they look similar. Field dependent subjects mean workload scores were not significantly different than field independent subjects for the either the non-expanded or expanded version. In fact, the mean workload score was non-significantly lower for the non-expanded version for both field dependent and field independent subjects. The lower workload ratings for the non-expanded interface might have been due to slower response times in the expanded target interface and the previously mentioned need for onscreen adjustment after expansion in the expanded interface. Even if the expansion technically provided safety and presentation benefits, the perception for all subjects was that the expanded interface had a greater workload. Since this experiment consisted of a repeated measures design, the researcher examined if a significant learning effect existed for either the expanded or non-expanded interface. With regard to error rate, a significant learning effect existed in both of the interfaces meaning that the second time a subject used the interface they had a lower error rate. In addition, a learning effect occurred for time of order entry. In both interfaces, subjects entered more orders per minute in 83 trial two compared to trial one. In addition, the field dependent and field independent subjects did not differ significantly in the size of these learning effects for error rate and speed of entry. The researcher used a demographic questionnaire to identify the participant?s biological sex. With this information, supplemental analyses with the three dependent measure of error rate, time for order entry and workload ratings based on subjects? biological sex were performed. A significant difference in mean time for order entry based on biological sex was unexpected. Women had more efficient order entry overall and in the expanded target interface than men. What this might indicate is that women had a higher degree of diligence for the medication order entry task compared to men. Given that the expanded target interface required a user to maintain a more consistent focus, a greater diligence in the task of order entry could result in quicker recognition of onscreen items, particularly when the expansion occurred. Women appeared to have paid more attention to the task at hand but their actual performance for selecting the correct item for order entry was no better since no significant difference in results for error rate based on biological sex occurred. Implications CPOE systems may have the ability to reduce errors in the medication-use process (Bates et al., 2001; Bates et al., 1998; Bates et al., 1999; Jones, 2004; Kaushal et al., 2003). However, most of the previous studies on the effect of CPOE systems have focused on full sized CPOE systems. Handheld CPOE systems are relatively new and have not been investigated as to their impact. The 84 reduced size of the user interfaces for CPOE systems may negatively impact the performance of the interface as the primary method for viewing and entering data. This research experiment wanted to determine whether an expanded target area within a handheld user interface could improve data entry in a handheld CPOE system for the ordering of a medication. The results of this study indicate that the expanded target interface functions did not provide any real improvement when compared to the non-expanded target interface for users of a handheld medication order entry system. The expanded interface lacked a significant reduction in error rate. At the same time it increased the order entry time and had a higher workload rating. If an expanded target interface was included in a CPOE system, it might be better to include it as an option for those users that specifically request it rather than as the default user interface. Limitations The results of this study failed to show that the expanded interface was significantly better than the non-expanded interface. In fact it was worse in two aspects, workload and time. However, these results do have some limitations. The type and use of expansion and characteristics for displaying expansion if changed or if used on a Palm? PDA besides the Palm? IIIc could produce different results. For example, a PDA with a faster processor could display the expanded target area and accept data entry faster than the study PDAs. The same PDA with a faster processor could also have a better resolution than the ones used in this experiment. This increase in resolution could also affect the presentation 85 items both in the expanded and non-expanded target which could produce different results. The method of expansion used in this study may also have limitations. The researcher operationalized the expansion area in this experiment in a unique way that shifted the target item selected by a user and other items on the screen during the expansion. Results from this study have generalizability to a PDA similar to the one used in this experiment with a CPOE system using this studies method of expansion. The use of a different method of expansion in a subsequent study might produce different results. Additionally, the researcher pre-tested all of the PDAs used in this experiment for factors such as screen response to stylus taps and battery life. However, other unique factors such as time between screen to screen responses with the PDAs used not found upon routine testing could have introduced unexpected variance. If there was a large systematic variance in PDA screen to screen response time and presentation, this might have altered the time for order or NASA-TLX workload scores. This study has limited generalizability to an operational healthcare setting. This study simulated a medication order entry task in a student population who only focused on data entry and could not access other information including the results of their entry. The subjects did not have to make any inferences or conclusions based on their data entry and thus served as data entry clerks who used only a lower level processing of the information. In practice however, CPOE systems can often be used by different levels of providers such as physicians, pharmacists, nurses and even unit clerks, each with a particular role 86 and scope of activity. A health care practitioner doing the same order entry task while engaged in patient care would use higher level of processing and have more of a personal stake in their data entry than the study participants. The study results would apply to a medication order entry task conducted by someone such as unit clerk who works at the lower level processing of the information and is not responsible for reviewing clinical data. The results could also apply to a clinician who is performing order entry but not engaged in higher level cognitive processing, such as after a making a treatment decision, where they simply use lower level processing to follow their plan. The subjects in this experiment were younger than most health care practitioners, with only five subjects aged 30 years or more. One factor that changes as someone ages is that they become more field dependent. When a person reaches his or her 25 th year, the process of increasing field dependence begins (Witkin et al., 2002). The mean and median GEFT scores for this study population of 12 and 13 respectively matched or exceeded college age values obtained in previous studies and indicated a predominance of field independent subjects (57% of total). The results of this study may not apply to a population with a different GEFT score distribution or average than the study population. In addition to increasing field dependency with increasing age, older subjects might have a greater impairment of visual acuity that could affect results. Data from previous studies indicate that as subjects increase in age from their twenties to their forties, the number of adults needing vision correction increased by 60 percent (from 54% to 90%) (Mutti & Zadnik, 2000). As someone ages, 87 their ability to focus on near objects gradually decreases, often requiring vision correction for reading. In some cases the use of bi-focal or tri-focal vision correction could require a person to look through a small area in their glasses to clarify objects. The results of this study are limited to younger users with higher degrees of field independence and visual acuity. Using older subjects could affect the results of the study based on their visual acuity and level of field independence. It is possible that the expanded interface might be a significant improvement for older users of a handheld CPOE device. Nevertheless, such results would only warrant having the expanded interface as an option for older users. Another limitation is related to the subjects? PDA skills (i.e. use, experience and knowledge). This group of subjects had very low ratings of their PDA skills, some of which produced significant post-hoc results. This low level of PDA skills in not unusual as many pharmacy students do not begin to acquire PDAs until their later years in pharmacy school when they approach their internship in pharmacy settings. However, these results as well as the NASA- TLX results stem from self-report on a questionnaire which in itself has limitations. Using a self-report method of data collection as part of an experiment can threaten the internal validity of results. Internal validity threats pertain to the design of an experiment. If a threat is strong enough, it could affect the interpretation of results since the measurement itself would no longer be considered valid. Some particular limitations of the self-report method of data collection that could affect the internal validity of the results: 88 ? Frame of reference: The subjects may have misunderstood the question based on a lack of frame of reference. ? Schedule: The main experiment took place in a classroom setting as part of required class participation. The demographic questionnaire was the last instrument given and it was the only thing that stood between the subjects leaving the lab period which could have affected their answers. ? Motivated respondent: Since this exercise took part in a classroom setting; a subject could have felt compelled to answer a question even when they did not have a firm opinion. Also, with rating scales such as Likert scales, subjects can tend to rate on extremes or in a central tendency. ? Language: The NASA-TLX survey had an accompanying set of definitions for the six workload factors. Even with these definitions, a subject might have misunderstood a survey question. History of subjects is another threat to internal validity from the study design that limits the interpretation of results. Since this study had five different collection periods with subjects in a class, subjects in sessions 2-5 could have performed differently based on information from their classmates in previous sessions. Finally a third threat to the internal validity of this research experiment is from the instrument itself, specifically the PDA interfaces. The software used in this study was a unique system created by the researcher. The researcher made conscious design decisions on interface functionality such as how to employ the expanded target. Most of the results obtained in this study did not achieve significance. The difference in performance results between the two interfaces 89 was very small. Power analyses demonstrated medium or very low power for most of the non-significant results. However, these results were in the anticipated direction. Modifications to the expansion presentation such as eliminating the need to re-focus on the target after item expansion could provide a greater difference in the two interface types. These modifications could also increase the power in a replication of this study. Another explanation for the lack of power in these results could stem from the population. An increase in the sample size would increase the possibility of finding significance from closely matched results. Either of these modifications could also help protect from a Type II error which could possibly exist with this study as currently designed. Future Research One of the limitations mentioned in this study was the PDA platform itself. Within the Palm? platform only certain methods of expansion or character representation exist. For instance, in order to select an item to provide expansion, it required physical tapping on the screen. Recently, the availability of magnetic styluses for Tablet PC?s and other handheld computers has allowed a different method of expansion and selection with a stylus. With a magnetic stylus, expansion of onscreen elements occurs when the stylus is in close proximity to elements on a screen. This method is similar to desktop zooming when a mouse cursor moves near an item on the screen and it expands. With this technique, users could scan a field of items with their stylus allowing each one to expand until they find the one they desire. A replication of this study using a magnetic stylus might have different results since users would not tap actually the 90 screen to initiate expansion and could use move the expanded area without touching the screen. The Palm? OS has been the dominant operating system for PDAs for many years. However, in recent years their market share has decreased. Also, many handheld users have migrated to smartphones with Pocket PC, Palm? and other operating systems. Use of other platforms such as the newer Palm? OS PDAs, Pocket PC, or various mobile phones could provide different results upon replication of this study One factor that could change future results with other platforms is the screen resolution. All of the PDAs used in this study had a standard resolution (160 X 160). When compared to platforms such as newer Palm? OS PDAs (whose resolutions extend up to 320 X 480) or Pocket PCs (whose resolutions range from 360 X 480 to 1024 X 1280), the PDAs used in this study had low resolution. This increase in resolution could affect the interaction with both the non-expanded and expanded target. Also, newer Palm? PDAs have faster processors and more RAM than the ones used in this study. PDAs with more RAM and/or a faster processor could provide increased performance and smoother transitions from of the expansion as well as a better PDA response time for items entered which could improve order entry efficiency and decrease the perception of a heavier workload. Future research should include other handheld operating systems to measure their performance with a task such as medication order entry. 91 The study population for this experiment included only first year pharmacy students. Although this group is familiar with many aspects of the medication-use process, they still have little practical experience as a health care provider. Further replication of this study should include a more diverse set of subjects who have various experiences as health care providers (i.e. pharmacist or physician) to assess their performance with both interfaces as well as their opinions as to the workload of each interface. Many of these subjects might already be using a handheld application for order entry and could compare the systems. Additionally, active healthcare practitioners would most likely have a broader age range and set of physical characteristics that might affect their performance results. Active practitioners could have broader range of visual acuity, many with bi-focal or tri-focal correction, which might affect how they view the differences between the expanded and non-expanded target interface. Also, physical characteristics such as arthritis which increase with age could affect their interaction with a PDA using a stylus. The main study took part in the CAPP laboratory designed to have students in a standing position during instruction. Postural work where tasks are performed with a constant posture and muscle effort can lead to fatigue. Although the posture of subjects and the amount of fatigue was not studied, it could have affected results in this study. The researcher noted that many subjects would lean on the desks during data entry (see Appendix 16) and shift positions regularly with audible groans throughout the experiment. However given that subjects significantly improved as they went from trial one to two, lowering their 92 error rate and increasing their time for entry, fatigue may have not had a significant affect on the results of the study. Future research of data entry with PDAs or other devices should consider the impact of fatigue from environmental factors on a subject to determine if a particular design can benefit more than another one can. Healthcare practitioners such as physicians often move from location to location when attending to their patients. This may require these individuals to leave their local work area and travel to see a patient. Electronic resources such as electronic medical records (EMR) and order entry systems used by these providers should work on multiple different devices and platforms such as Tablet PC, Desktop PC, and PDA to allow these practitioners to provide care in multiple locations. In some instances, a practitioner could migrate between platforms during one care event. This study only used one platform for medication order entry. Future research might compare the effectiveness of a system across multiple platforms particularly within a specific user which could also provide valuable insight for user interface design. Today, much of medication ordering today still requires using paper. However, with a national push to have a complete electronic prescribing environment, many users may soon experience a new electronic way of entering orders. In some instances users that dislike a user interface or system often create workarounds or actively seek to avoid using a system designed as a safety net. A classic example of a poorly evaluated user interface leading to failure was the failed CPOE system implementation at Cedars-Sinai Hospital, Los Angeles in 93 2003. This CPOE system was actually an improvement over the hospital's paper- based system with alerts and resulted in a change in 35 % of orders. During its use, no detection of increased error rates occurred but the physicians complained about the interface until the system was suspended ("Cedars-sinai's CPOE system had benefits, despite complaints", 2003). The suspension of this system demonstrates that the workload created by a user interface of a system providing improvements in safety The purpose of this study was to empirically compare expanded target and non-expanded target interface versions of a handheld medication order entry technology both objectively and subjectively. In order to create a truly safer and better healthcare system, the objective and subjective impact on users of a system?s user interface should be part of assessment criteria. The results from this experiment might aid future researchers in designing or testing potential user interfaces for healthcare systems such as a medication order entry systems. If an expanded target interface were included in a CPOE system, handheld or otherwise, it would be wise not to shift target items in the creation of the expansion as in this study. It might also serve users better to have an expanded target not as the default presentation but rather as an option for users who perceived the expansion as helpful and responsive to their needs. 94 REFERENCES 16th annual himss leadership survey sponsored by superior consultant company/acs healthcare solutions. (2005). Retrieved August 15, 2005, from http://www.himss.org/2005survey/index.asp Allan, E. L., & Barker, K. N. (1990). Fundamentals of medication error research. Am J Hosp Pharm, 47(3), 555-571. Arnold, B., & Roe, R. (1987). User errors in human-computer interaction. Amsterdam: Elsevier Science Publishers North-Holland. Ash, J. S., Berg, M., & Coiera, E. (2004). Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. J Am Med Inform Assoc, 11(2), 104- 112. Balakrishnan, R. (2004). "Beating" fitts' law: Virtual enhancements for pointing facilitation. International Journal of Human-Computer Studies, 61(6), 857. Bates, D. W., Boyle, D. L., Vander Vliet, M. B., Schneider, J., & Leape, L. (1995a). Relationship between medication errors and adverse drug events. J Gen Intern Med, 10(4), 199-205. Bates, D. W., Cohen, M., Leape, L. L., Overhage, J. M., Shabot, M. M., & Sheridan, T. (2001). Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc, 8(4), 299-308. Bates, D. W., Cullen, D. J., Laird, N., Petersen, L. A., Small, S. D., Servi, D., et al. (1995b). Incidence of adverse drug events and potential adverse drug events. Implications for prevention. Ade prevention study group. Jama, 274(1), 29-34. Bates, D. W., Leape, L. L., Cullen, D. J., Laird, N., Petersen, L. A., Teich, J. M., et al. (1998). Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Jama, 280(15), 1311-1316. Bates, D. W., Teich, J. M., Lee, J., Seger, D., Kuperman, G. J., Ma'Luf, N., et al. (1999). The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc, 6(4), 313-321. Bell, D. S., Cretin, S., Marken, R. S., & Landman, A. B. (2004). A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities. J Am Med Inform Assoc, 11(1), 60-70. Bourne, L., Dominowski, R., Loftus, E., & Healy, A. (1986). Cognitive processes (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall. 95 Buchanan, T. L., Barker, K. N., Gibson, J. T., Jiang, B. C., & Pearson, R. E. (1991). Illumination and errors in dispensing. Am J Hosp Pharm, 48(10), 2137-2145. Carroll, J. (1997). Human-computer interaction: Psychology as a science of design. Ann Rev Psych, 48, 61-83. Cedars-sinai's CPOE system had benefits, despite complaints. (2003). Retrieved May 23, 2006, from http://www.ihealthbeat.org/index.cfm?Action=dspItem&itemID=99093 Chaffee, B. W., & Bonasso, J. (2004). Strategies for pharmacy integration and pharmacy information system interfaces, part 1: History and pharmacy integration options. Am J Health Syst Pharm, 61(5), 502-506. Clauson, K. A., Seamon, M. J., Clauson, A. S., & Van, T. B. (2004). Evaluation of drug information databases for personal digital assistants. Am J Health Syst Pharm, 61(10), 1015-1024. Cockburn, A., & Firth, A. (2003). Improving the acqusition of small targets. Paper presented at the Proceedings in HCI 2003. Cousineau, D., & Shiffrin, R. M. (2004). Termination of a visual search with large display size effects. Spatial Vision, 17(4-5), 327-352. Dembo, M. (1977). Teaching for learning: Applying educational psychology in the classroom. Santa Monica, Ca: Goodyear. Dix, A., Finaly, J., Abowd, G., & Beale, R. (2004). Human-computer interaction (3rd ed.): Pearson Education. Eckstein, M. P. (1998). The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing. Psychological Science, 9(2), 111-118. Fedak, C. (2004). Interacting with big interfaces on small screens: A comparison of fisheye, zoom, and panning techniques. Paper presented at the Proceedings of the 2004 conference on Graphics interface, London, Ontario, Canada. Fischer, S., Stewart, T. E., Mehta, S., Wax, R., & Lapinsky, S. E. (2003). Handheld computing in medicine. J Am Med Inform Assoc, 10(2), 139- 149. Fitts, P. (1992). The information capacity of the human motor system in controlling the amplitude of movement. J Exp Psy, 121(3), 262-269. Floru, R., Cail, F., & Elias, R. (1985). Psychophysiological changes during a vdu repetitive task. Ergonomics, 28(10), 1455-1468. Flynn, E. A., Barker, K. N., Gibson, J. T., Pearson, R. E., Berger, B. A., & Smith, L. A. (1999). Impact of interruptions and distractions on dispensing errors in an ambulatory care pharmacy. Am J Health Syst Pharm, 56(13), 1319- 1325. Flynn, E. A., Barker, K. N., Gibson, J. T., Smith, L. A., & Berger, B. A. (1996). Relationships between ambient sounds and the accuracy of pharmacists' prescription-filling performance. AJHP, 38(4), 614-619. 96 Flynn, E. A., Barker, K. N., Pepper, G. A., Bates, D. W., & Mikeal, R. L. (2002). Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm, 59(5), 436-446. Fulford, M. (1987). Computer use in independent community pharmacies in alabama. Auburn University Harrsion School of Pharmacy, Auburn. The gale encyclopedia of psychology. (1996).). Detroit: Gale. Galvendy, S. (Ed.) (1997) Handbook of human factors and ergonomics. (2nd ed.). New York: Wiley-Interscience. George, D., & Austin-Bishop, N. (2003). Error rates for computerized order entry by physicians versus nonphysicians. Am J Health Syst Pharm, 60(21), 2250-2252. Grasha, A. F. (2000). Into the abyss: Seven principles for identifying the causes of and preventing human error in complex systems. Am J Health Syst Pharm, 57(6), 554-564. Grasha, A. F., & K, S. (2001). Psychosocial factors, workload, and human error in a simulated pharmacy dispensing task. Perceptual and Motor Skills, 92, 53-71. Gremy, F., & Degoulet, P. (1993). Assessment of health information technology: Which questions for which systems? Proposal for a taxonomy. Med Inform (Lond), 18(3), 185-193. Grossman, T., & Balakrishnan, R. (2005). The bubble cursor: Enhancing target acquisition by dynamic resizing of the cursor's activation. Paper presented at the Proceeding of the SIGCHI conference on Human factors in computing systems, Portland, Oregon, USA. Guadagno, L., VandeWeerd, C., Stevens, D., Abraham, I., Paveza, G. J., & Fulmer, T. (2004). Using pdas for data collection. Appl Nurs Res, 17(4), 283-291. Guiard, Y., & Beaudouin-Lafon, M. (2004). Fitts' law 50 years later: Applications and contributions from human-computer interaction. International Journal of Human-Computer Studies, 61(6), 747. Gutwin, C. (2002). Improving focus targeting in interactive fisheye menus. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, Minneapolis, Minnesota, USA. Hart, S., & Staveland, L. (1988). Development of nasa-tlx (task load index) results of empirical and theoretical research. In H. PA (Ed.), Human mental workload (pp. 139-183). Amsterdam: North-Holland. Higby, G. J. (2002). The continuing evolution of american pharmacy practice, 1952-2002. J Am Pharm Assoc (Wash), 42(1), 12-15. Ies lighting ready reference. (1984).). Ney York: Illuminating Engineering Society of North America. Johnson, C. M., Johnson, T. R., & Zhang, J. (2005). A user-centered framework for redesigning health care interfaces. Journal of Biomedical Informatics, 38(1), 75. 97 Jones, J. L. (2004). Implementing computerized prescriber order entry in a children's hospital. Am J Health Syst Pharm, 61(22), 2425-2429. Kaushal, R., Bates, D. W., Landrigan, C., McKenna, K. J., Clapp, M. D., Federico, F., et al. (2001). Medication errors and adverse drug events in pediatric inpatients. Jama, 285(16), 2114-2120. Kaushal, R., Shojania, K. G., & Bates, D. W. (2003). Effects of computerized physician order entry and clinical decision support systems on medication safety: A systematic review. Arch Intern Med, 163(12), 1409-1416. King, W. J., Paice, N., Rangrej, J., Forestell, G. J., & Swartz, R. (2003). The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics, 112(3 Pt 1), 506- 509. Kjeldskov, J., & Graham, C. (2003). Human-computer interaction with mobile devices and services. Berlin: Springer-Verlag. Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R., Kimmel, S. E., et al. (2005). Role of computerized physician order entry systems in facilitating medication errors. Jama, 293(10), 1197-1203. Kropf, R. (2005). Jonas and kovner's health care delivery in the united states. New York: Springer. Kushniruk, A., Triola, M., Stein, B., Borycki, E., & Kannry, J. (2004). The relationship of usability to medical error: An evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. Medinfo, 11(Pt 2), 1073-1076. Lambert, B. L. (1997). Predicting look-alike and sound-alike medication errors. Am J Health Syst Pharm, 54(10), 1161-1171. Leape, L. L., Bates, D. W., Cullen, D. J., Cooper, J., Demonaco, H. J., Gallivan, T., et al. (1995). Systems analysis of adverse drug events. Ade prevention study group. Jama, 274(1), 35-43. Leape, L. L., & Berwick, D. M. (2005). Five years after to err is human: What have we learned? Jama, 293(19), 2384-2390. Lichstein KL, R. B., Richman SL. (2000). The mackworth clock test: A computerized version. Journal of Psychology, 134(2), 153-161. Lowry, C. M., Kostka-Rokosz, M. D., & McCloskey, W. W. (2003). Evaluation of personal digital assistant drug information databases for the managed care pharmacist. J Manag Care Pharm, 9(5), 441-448. Lynx, D. H., Brockmiller, H. R., Connelly, R. T., & Crawford, S. Y. (2003). Use of a pda-based pharmacist intervention system. Am J Health Syst Pharm, 60(22), 2341-2344. Mackenzie, I. S. (1992). Fitts law as a research and design tool in human- computer interaction. Human-Computer Interaction, 7, 91-139. Mackenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001). Accuracy measures for evaluating computer pointing devices. CHI, 3(1). Mackworth, N. (1961). Researches on the measurement of human performance. New York: Dover Publications. 98 McElree, B., & Carrasco, M. (1999). The temporal dynamics of visual search: Evidence for parallel processing in feature and conjunction searches. Journal Of Experimental Psychology-Human Perception And Performance, 25(6), 1517-1539. McGuffin, M., & Balakrishnan, R. (2002, April 20-25). Acquisition of expanding targets. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, Minneapolis, Minnesota, USA. Morimoto, T., Gandhi, T. K., Seger, A. C., Hsieh, T. C., & Bates, D. W. (2004). Adverse drug events and medication errors: Detection and classification methods. Qual Saf Health Care, 13(4), 306-314. Mutti, D. O., & Zadnik, K. (2000). Age-related decreases in the prevalence of myopia: Longitudinal change or cohort effect? Invest Ophthalmol Vis Sci, 41(8), 2103-2107. Nolan, T. W. (2000). System changes to improve patient safety. Bmj, 320(7237), 771-773. Oren, E., Shaffer, E. R., & Guglielmo, B. J. (2003). Impact of emerging technologies on medication errors and adverse drug events. Am J Health Syst Pharm, 60(14), 1447-1458. Palmer, J., Verghese, P., & Pavel, M. (2000). The psychophysics of visual search. Vision Research, 40(10-12), 1227. Pedersen, C. A., Schneider, P. J., & Scheckelhoff, D. J. (2005). Ashp national survey of pharmacy practice in hospital settings: Prescribing and transcribing--2004. Am J Health Syst Pharm, 62(4), 378-390. Posner, M., & Petersen, S. (1990). The attention system of the human brain. Annual Review of Neuroscience, 13, 25-42. Reason, J. (1990). Human error. Cambridge: Cambridge University Press. Roberts, J., & Peel, V. (1997). Studies in health technology and informatics (Vol. 43 Pt B). Amsterdam: IOS Press. Sanders, M. S., & Mccormick, E. J. (1987). Human factors in engineering and design (Sixth ed.). New York: Mcgraw-Hill. Schneiderman, B., & Plaisant, C. (2005). Designing the user interface. Boston: Pearson. Spencer, D. C., Leininger, A., Daniels, R., Granko, R. P., & Coeytaux, R. R. (2005). Effect of a computerized prescriber-order-entry system on reported medication errors. Am J Health Syst Pharm, 62(4), 416-419. Staggers, N. (1998). Staggers nursing computer experience questionnaire. Staggers, N., & Kobus, D. (2000). Comparing response time, errors, and satisfaction between text-based and graphical user interfaces during nursing order tasks. J Am Med Inform Assoc, 7(2), 164-176. Survey: CPOE adoption slowly growing. (2005). Retrieved 18 Oct 2005, from http://www.ihealthbeat.org/index.cfm?Action=dspItem&itemID=111993 Szalma, J. L., Warm, J. S., Matthews, G., Dember, W. N., Weiler, E. M., Meier, A., et al. (2004). Effects of sensory modality and task duration on 99 performance, workload, and stress in sustained attention. Human Factors, 46(2), 219-233. Tang, P. C., & Patel, V. L. (1994). Major issues in user-interface design for health professional workstations - summary and recommendations. International Journal Of Bio-Medical Computing, 34(1-4), 139-148. Teich, J. M., Merchia, P. R., Schmiz, J. L., Kuperman, G. J., Spurr, C. D., & Bates, D. W. (2000). Effects of computerized physician order entry on prescribing practices. Arch Intern Med, 160(18), 2741-2747. To err is human: Building a safer health system. (1999).). Washington D.C.: Institute of Medicine. United States Pharmacopeia (USP). (2004). Computer entry a leading cause of medication errors in U.S. Health systems. Rockville, Md:. Wichansky, A. (2000). Usability testing in 2000 and beyond. Ergonomics, 43(7), 998-1006. Witkin, H., Oltman, P., Raskin, E., & Karp, S. (2002). Group embedded figures test manual: Mindgarden. Ying, A. (2003). Mobile physician order entry. J Healthc Inf Manag, 17(1), 58- 63. Young, P. M. C., Leung, R. M. W., Ho, L. M., & McGhee, S. M. (2001). An evaluation of the use of hand-held computers for bedside nursing care. International Journal of Medical Informatics, 62(2-3), 189. Zeneger, B., & Fahle, M. (1997). Missed target are more frequent than false alarms: A model for error rates in visual search. J Exp Psy, 23(6), 1783- 1791. Zhai, S., Beaudouin-Lafon, M., & Guiard, Y. (2003, April 5-10). Human on-line response to target expansion. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, Ft. Lauderdale, Florida, USA. 100 APPENDICES 101 Appendix 1 Pilot Study Room 102 103 Appendix 2 Pilot Study Information Sheet 104 INFORMATION SHEET for Research Study Entitled ---The effect of expanded target interface for a handheld medication order entry task--- You are invited to participate in a research study involving the medication-use process. The purpose of this research is to measure the effectiveness of new technology. This study is being conducted by (Marc Young, PharmD, Graduate Research Associate in Pharmacy Care Systems) under the supervision of Professor Bill Felkey and Associate Professor William Villuame PhD from the Auburn University Harrison School of Pharmacy, Pharmacy Care Systems department). We hope to learn the effectiveness of new technology in the medication use process. You were selected as a possible participant because you possess the specific skills, abilities, vision and knowledge required of the intended user of such technology. If you decide to participate, today you will use technology to perform pharmacy related tasks and provide opinions on the performance of technology as well take a standardized educational test to determine characteristics about yourself. If you have corrected vision, you will need to have your correction (glasses or contacts) with you in order to participate. Additionally, you must be able to use handheld computer (PDA) without assistance. All of these events should take approximately 90 minutes. After completion of this research and the data is analyzed, you will have access to all of your results including the educational test assessment through a user specific but anonymous identification on the following website: http://pharmacy.auburn.edu/youngmt/researchresults. All of the results collected in this study are collected anonymously using an anonymous identification that consists several of your unique identification items unknown to any of the researchers. The results of your performance with the technology as well as your individual characteristics might help you in the future when performing similar tasks or using similar technology in a pharmacy work setting. Additionally, this research may serve as reference for future research into the medication-use process. We (I) cannot promise you that you will receive any or all of the benefits described. Your participation today will be counted as a service project for Kappa Psi Pharmaceutical Fraternity. Information collected through your participation might fulfill an educational requirement, receive publication in a professional journal, and/or be presented at a professional meeting. You may withdraw from participation at any time, without penalty. After you have provided anonymous information you will be unable to withdraw their data after participation since there will be no way to identify individual information. 105 Your decision whether or not to participate will not jeopardize your future relations with Auburn University or the Pharmacy Care Systems department. If you have any questions we (I) invite you to ask them now. If you have questions later, Marc Young, PharmD @ 334-844-5153 or youngmt@auburn.edu will be happy to answer them. For more information regarding your rights as a research participant you may contact the Auburn University Office of Human Subjects Research or the Institutional Review Board by phone (334)-844-5966 or e-mail at hsubjec@auburn.edu or IRBChair@auburn.edu . . HAVING READ THE INFORMATION PROVIDED, YOU MUST DECIDE WHETHER TO PARTICIPATE IN THIS RESEARCH PROJECT. IF YOU DECIDE TO PARTICIPATE, THE DATA YOU PROVIDE WILL SERVE AS YOUR AGREEMENT TO DO SO. THIS LETTER IS YOURS TO KEEP. ___________________________________ Investigator's signature Date ___________________________________ Co-investigator's signature Date (if appropriate) 106 Appendix 3 Tapping Device 107 108 109 110 Appendix 4 Study Consent Document 111 CONSENT DOCUMENT for Research Study Entitled ---The effect of expanded target interface for a handheld medication order entry task--- You are invited to participate in a research study involving the medication-use process. The purpose of this research is to measure the effectiveness of new technology. This study is being conducted by Marc Young, PharmD, Graduate Research Associate in Pharmacy Care Systems under the supervision of Professor Bill Felkey and Associate Professor William Villuame PhD from the Pharmacy Care Systems department. We hope to learn the effectiveness of new technology in the medication use process. You were selected as a possible participant because you possess the specific skills, abilities, vision and knowledge required of the intended user of such technology. Today you will use technology to perform pharmacy related tasks and provide opinions on the performance of technology as well take a standardized educational test to determine characteristics about yourself. If you have corrected vision, you will need to have your correction (glasses or contacts) with you in order to participate. Additionally, you must be able to use a handheld computer (PDA) without assistance. All of these events should take approximately 90 minutes. All of the data collected in this study are collected anonymously using an anonymous identification that consists of several unique, personal identification items unknown to any of the researchers. After completion of this exercise and the data is analyzed, you will have access to all of your results including the educational test assessment through your user specific but anonymous identification on the following website: http://pharmacy.auburn.edu/youngmt/researchresults. This exercise is part of a course requirement in PYDI 5120 but allowing the use of your data for research is not. We are asking permission to use your data from this course assignment for research purposes. For your participation today, you will receive class credit. The results of your performance with the technology as well as your individual characteristics might help you in the future when performing similar tasks or using similar technology in a pharmacy work setting. 112 Additionally, the results of your performance might be used in a research project or may serve as reference for future research into the medication-use process. If you agree to allow your anonymous data from today?s exercise to be used for this research study, you must make this known today by signing this consent document. If you do not want your data to be used for research purposes, then simply do not sign this document. You will receive class credit for your participation regardless of whether you volunteer your anonymous data for research and no additional credit is given for participation in research. We (I) cannot promise you that you will receive any or all of the benefits described. Information collected through your participation will help fulfill an educational requirement and might receive publication in a professional journal, and/or be presented at a professional meeting. You may withdraw from participation at any time, without penalty; however, you will have to fulfill alternative requirements for equivalent class credit. After you have provided anonymous information you will be unable to withdraw their data after participation since there will be no way to identify individual information. Your decision whether or not to participate will not jeopardize your future relations with Auburn University or the Pharmacy Care Systems department. If you have any questions we (I) invite you to ask them now. If you have questions later, Marc Young, PharmD @ 334-844-5153 or youngmt@auburn.edu will be happy to answer them. For more information regarding your rights as a research participant you may contact the Auburn University Office of Human Subjects Research or the Institutional Review Board by phone (334)-844-5966 or e-mail at hsubjec@auburn.edu or IRBChair@auburn.edu . . HAVING READ THE INFORMATION PROVIDED, YOU MUST DECIDE WHETHER TO PARTICIPATE IN THIS RESEARCH PROJECT. IF YOU DECIDE TO PARTICIPATE, YOUR SIGNATURE WILL SERVE AS YOUR AGREEMENT TO DO SO. A COPY OF THIS FORM IS YOURS TO KEEP. ___________________________________ Investigator obtaining consent Date __________________________________ Participants signature Date __________________________________ Printed name 113 Appendix 5A NASA-TLX Non-expanded Version USER ID_____________ NASA-TLX for Non-expanded Version Circle the vertical line on the scale at the point that best indicates your experience of the task with the non-expanded interface as shown in the example. The non- expanded interface did not provide any enlargement of a small area on the screen after tapping with a stylus: Example: Low High 114 Mental Demand Low High Physical Demand Low High Temporal Demand Low High 115 Performance Poor Good Effort Low High Frustration Low High Circle the factor in each pair that represents the more important contributor to workload for the Expanded interface handheld order entry task. The Expanded interface provided an enlargement of a small area on the screen after tapping with a stylus: 1. Physical Demand or Temporal Demand 2. Performance or Mental Demand 3. Frustration or Mental Demand 116 4. Mental Demand or Effort 5. Temporal Demand or Frustration 6. Physical Demand or Performance 7. Temporal Demand or Effort 8. Physical Demand or Frustration 9. Effort or Performance 10. Mental Demand or Physical Demand 11. Effort or Physical Demand 12. Performance or Frustration 13. Mental Demand or Effort 14. Performance or Temporal Demand 15. Frustration or Effort 117 Appendix 5B NASA-TLX Expanded Version USER ID_____________ NASA-TLX for Expanded Version Circle the vertical line on the scale at the point that best indicates your experience of the task with the expanded interface as shown in the example. The expanded interface provided an enlargement of a small area on the screen after tapping with a stylus: Example: Low High 118 Mental Demand Low High Physical Demand Low High Temporal Demand Low High 119 Performance Poor Good Effort Low High Frustration Low High Circle the factor in each pair that represents the more important contributor to workload for the Expanded interface handheld order entry task. The Expanded interface provided an enlargement of a small area on the screen after tapping with a stylus: 1. Physical Demand or Temporal Demand 2. Performance or Mental Demand 3. Frustration or Mental Demand 120 4. Mental Demand or Effort 5. Temporal Demand or Frustration 6. Physical Demand or Performance 7. Temporal Demand or Effort 8. Physical Demand or Frustration 9. Effort or Performance 10. Mental Demand or Physical Demand 11. Effort or Physical Demand 12. Performance or Frustration 13. Mental Demand or Effort 14. Performance or Temporal Demand 15. Frustration or Effort 121 Appendix 6 SNEQ Questionnaire Computer Experience Questionnaire The purpose of this instrument is to determine your computer experience. Completing this questionnaire will take approximately 5-10 minutes. Please consider each item carefully and circle the number most closely corresponding to your use, knowledge of, or participation in computer-related activities. User ID _______________ Gender M F Age ____ years old Do you have corrected vision to 20/20? Y N How many total entry errors do you think you made? _______________ A. General Computer Applications Instructions: Each item should be rated in two ways using two sets of numbers. The first rating describes your past or present computer use. The second rating describes your knowledge level of the named computer function. Computer application Past or Present Computer Use Computer Knowledge Scale: 0 = none 4 = extensive Scale: 0 = none 4 = extensive 1. Writing reports, papers, documents, or other text (word processing) 0 1 2 3 4 0 1 2 3 4 2. Sending messages to others (electronic mail) 0 1 2 3 4 0 1 2 3 4 3. Data/file management such as employee licensing information (database management) 0 1 2 3 4 0 1 2 3 4 4. Research data analysis 0 1 2 3 4 0 1 2 3 4 5. Searching for books, articles, or other library information(bibliographic retrieval) 0 1 2 3 4 0 1 2 3 4 6. Creating pictures, slides, or overhead displays (computer graphics) 0 1 2 3 4 0 1 2 3 4 122 123 7. Managing projects (project management) 0 1 2 3 4 0 1 2 3 4 8. Use a handheld computer or PDA at home, work or school 0 1 2 3 4 0 1 2 3 4 9. Using educational tutorials (computer assisted instruction) 0 1 2 3 4 0 1 2 3 4 10. Calculating budget or other numerical data (spread sheets) 0 1 2 3 4 0 1 2 3 4 11. Communicating with other computer systems (communication software) 0 1 2 3 4 0 1 2 3 4 12. Copying, deleting, changing directories, and performing disk/ system functions (operating system) 0 1 2 3 4 0 1 2 3 4 13. Doing data recovery, finding files, or system performance indices (utility programs) 0 1 2 3 4 0 1 2 3 4 14. Writing computer programs (computer programming) 0 1 2 3 4 0 1 2 3 4 Computer application Past or Present Computer Use Computer Knowledge Scale: 0 = none 4 = extensive Scale: 0 = none 4 = extensive 15. Using filed expert information (expert systems/artificial intelligence) 0 1 2 3 4 0 1 2 3 4 16. Computer assisted software engineering (CASE) 0 1 2 3 4 0 1 2 3 4 17. Writing macros for spreadsheets or word processing packages 0 1 2 3 4 0 1 2 3 4 18. Authoring computer assisted instruction programs 0 1 2 3 4 0 1 2 3 4 19. Writing database management programs with text and 0 1 2 3 4 0 1 2 3 4 124 graphics (desktop publishing) 20. Using the world wide web (www access, interaction, retrieval) 0 1 2 3 4 0 1 2 3 4 Role Activity Past or Present Computer Use Role Knowledge Scale: 0 = none 4 = extensive Scale: 0 = none 4 = extensive 1. Identifying application requirements 0 1 2 3 4 0 1 2 3 4 2. Computer system design 0 1 2 3 4 0 1 2 3 4 3. Computer system selection 0 1 2 3 4 0 1 2 3 4 4. Computer system implementation 0 1 2 3 4 0 1 2 3 4 5. Computer system evaluation 0 1 2 3 4 0 1 2 3 4 6. Teaching computer classes 0 1 2 3 4 0 1 2 3 4 C. Formal Computer Knowledge Please circle: 1. The number of college-level computer science courses you have taken: 0 1 2 3 4 or more 2. The number of college-level information system management/informatics courses you have taken: 0 1 2 3 4 or more 3. The number of short courses (less than 1 week) on computer applications such as Word you have taken: 0 1 2 3 4 or more 4. How much you read computer magazines or books none extensively 0 1 2 3 4 or more 125 D. Computer Experience Rating Please rate your level of desktop/laptop/tablet PC computer experience: Check the series of dots that describe your level novice ? ? ? ? ? ? ? expert Please rate your level of handheld/PDA computer experience: Check the series of dots that describe your level novice ? ? ? ? ? ? ? expert 126 Appendix 7 Example of Medication Order Sheet 127 Order Shet Set ID Patient Drug Strength Amount Frequency ---------------------------------------------------------------------------------------------------------------------- -------- 1 Epperson, Steven CATAFLAM 140mg 1-2 Q2h 2 Woehrman, Benjamin ERYTHROCIN 30mg 1 Q1-h 3 Stevenson, Janet REVEX 15mg 3 BID 4 Cross, Joyce LEVSIN 20mg 4 QID 5 Keller, Arthur EQUAGESIC 25mg 1/2 Q4-6h 6 Harrell, Sherri ESTRATEST HS 40mg 3-4 QD 7 Rice, Marcus ADVICOR 60mg 2-3 Q8h 8 Watson, Wayne CARDIZEM 10mg 2-3 TID 9 Harris, Donald ZETIA 80mg 1 Q6-8h 10 Hubbard, Stephanie REVIA 110mg 3 Q4h 128 Appendix 8 CAPP Lab for Main Study 129 130 Appendix 9 Pre-recorded Scripted Presentation 131 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zBefore we begin this exercise, please remove and store your watches, cell phones, pagers, handheld computers or any other device with a clock. Please put all phones or pagers on silent. zYou will not be able to access these during this exercise. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe first part of this exercise requires the use of a handheld Personnel Digital Assistants (PDAs) like the one shown in this picture zYou will receive your PDA later to use as an order entry device PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zYou will also receive unique order sets containing five fields needed to enter orders on 5 separate matching screens types on the PDA. These fields and screen types are: ?patient, drug, strength, dose amount and frequency zIt is important to keep these order sets in the order you receive them to accurately complete the order entry during the study PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zTo minimize scrolling and searching within the 5 screen types each PDA is designed to scroll up or down only 3 screens of choices, such as patient names, relevant to that order. zYou will not need to search beyond these 3 screens for the item indicated on the order sheet 132 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe PDA will randomly start on one of the 3 choice screens, one of which contains the item listed on the order sheet that you should select by tapping. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation z For Example: In this case only three screens of patient names will be available, versus all possible patient name screens, one of which contains the patient name ?Tom Wiggins? zThe user would need to scroll up one screen to select ?Tom Wiggins? PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe item on the order sheet that you are looking for may not appear on the opening screen. In order to find an item, scroll up or down using the onscreen arrow button. For example, if you wanted to select ?Tom Wiggins? as a patient, you would need to tap the up scroll arrow to see his name. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zWhen no more screens with choices exist in that direction, an onscreen arrow is not highlighted and thus not functional. For example, if you are on the last or 3 rd screen, only the up scroll arrow will work and thus it is the only one highlighted as shown in this example. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNote: The onscreen arrows must be activated initially by tapping the up arrow on the very first ?Select Patient? screen 133 zOnce the up arrow is tapped, both arrows will now function on this and all subsequent screens PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe PDAs you will use in the exercise contain two versions of the same order entry program zThe sequence and function of the screens is identical in both versions. zEach PDA will start with one of the versions and alternate between the two versions at designated times. The switch between modes is not related to the number or orders you have entered. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe switch between modes is preceded by an alert zWhen you see this alert, simply tap ?OK? and continue entering orders, now in the other version PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf during the exercise you feel you made an incorrect entry, you will not be able to go back to a previous screen. zFor this experiment, your goal is to enter as many orders as accurately as you can during the exercise period. zThe length of the PDA order entry exercise will not be revealed to you, so continue to work until the experiment is over. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe order sets you will be given correspond to the PDA screens: ?patient name, medication name, strength, dose amount and frequency zNote that the strengths used for the medications are not intended to represent actual dosages for these products. 134 zFor the purposes of the exercise, enter exactly what is indicated on the order set for each order. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe order sets contain an ID number to the left of each individual order row that corresponds with a number in the lower right of your PDA as you progress. zUse this number to help stay on track. For example, only attempt to enter the patient on order #2 on this screen PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zID Patient Drug Strength Amount Frequency z----------------------------------------------------------------------------------- ----------------------------- z1 Epperson, Steven CATAFLAM 140mg 1-2 Q2h z 2 Wiggins, Tom ERYTHROCIN 30mg 1 Q1-2h z3 Stevenson, Janet REVEX 15mg 3 BID z4 Cross, Joyce LEVSIN 20mg 4 QID z5 Keller, Arthur EQUAGESIC 25mg 1/2 Q4-6h z 6 Harrell, Sherri ESTRATEST HS 40mg 3-4 QD 135 z7 Rice, Marcus ADVICOR 60mg 2-3 Q8h PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf you cannot find an item on one the 3 screens, verify the order number zRemember that based on the order number, only the relevant choices will appear on the 3 choice screens, so coordinating the order number of the PDA with the order sheet is important. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zWe will now have a small demonstration overview in this slide show. zAfter the overview, you will take a few minutes to practice using the software where you will experience both versions or modes used in the exercise zPlease do not turn on your PDAs and begin tapping until instructed. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe actual exercise and the demonstration work exactly alike. The exercise will simply be longer periods in which each version is used and switched between as in the demonstration z You will receive this notification on the PDA when they switch before it occurs zTap ?OK? to continue zYou also have a reference sheet that describes the basic navigation and method of entry needed in both versions 136 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIn order to select an item at any time on a screen, tap it once with your stylus found in the back of your PDA zIf you do not have a stylus on your PDA, raise your hand and one will be provided PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zWhen you turn on your PDA, you will see a menu screen similar to this one. zThe program we will use today, ?Handheld? has it?s icon highlighted in this picture. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zPlease do not attempt to adjust any settings on the PDAs zThe screen area and buttons at the bottom of the PDA will not be used today once order entry is started zTapping them once you start entering orders could result in disruptions or exiting the ?Handheld? program PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf for some reason you exit the program, simply tap the ?Handheld icon? to restart the program zNote that you will start back at order #1 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zAfter selecting ?Handheld?, on the next screen you will be asked to enter a seven digit ID zThis is the ID that you established when you arrived today and have on a card in front of you 137 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe unique user id is the last 4 digits of your social security number and then the first 3 digits of your hometown zip-code zIf you don?t have a hometown zip-code, use your local zip-code ?For example for Auburn it would be ?368? and a user id for Social Security last 4 of ?1234? would be ?1234368? PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThis ID used throughout the exercise provides anonymity & will allow you to access your results at a later time zTap in your ID and then tap start PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf you make an error while entering this ID, do not tap start. Instead tap the ?home? icon zThis will take you to the screen with the ?Handheld? icon zSelect ?Handheld? to return to user ID entry PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow a screen asking if this is this a demonstration will appear zFor the practice period after this slide show, when instructed tap the ?Yes? button PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow the ?Select Patient? screen appears zEach of the 5 screen types has an order number indicator on the lower right of the screen that coordinates with your order sets 138 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zRemember to verify the order number on each screen is the same as the order number for that row to stay coordinated with the order set and PDA. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zID Patient Drug Strength Amount Frequency z----------------------------------------------------------------------------------- ----------------------------- z1 Epperson, Steven CATAFLAM 140mg 1-2 Q2h z 2 Wiggins, Tom ERYTHROCIN 30mg 1 Q1-2h z3 Stevenson, Janet REVEX 15mg 3 BID z4 Cross, Joyce LEVSIN 20mg 4 QID z5 Keller, Arthur EQUAGESIC 25mg 1/2 Q4-6h z 6 Harrell, Sherri ESTRATEST HS 40mg 3-4 QD z7 Rice, Marcus ADVICOR 60mg 2-3 Q8h PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf during the exercise an item, such as a name, doesn?t appear on the screen, tap an up or down arrow to find it zRemember to activate the scroll arrows on the first ?Select Patient? screen by tapping the up arrow once 139 zOnce an item on the screen is tapped, it is highlighted PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zIf this is the desired item, tap the highlighted item once more to enter and proceed zIf an incorrect item is tapped and highlighted, simply find and tap the correct item on the current screen or by scrolling zAfter tapping that item will highlight and a second tap will enter this item and proceed PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zAfter the select patient screen is the ?Select Medication? screen zSimply find and select the medication that matches the particular order from the order set, in this case ?Cardura? PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow the ?Select Strength? appears zRemember strengths used in this exercise may or may not actually exist for these medications zSimply select the strength for the particular order as indicated by tapping as described earlier PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow the ?Select Amount? screen is next zSelect the amount as indicated by the particular order in the order set PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zFinally, the last screen is the ?Select Frequency? screen zSelect the indicated frequency from the order on the order set PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zWhen the demonstration and the actual exercise are completed, the PDA will display the screen shown zClick ?OK? and wait for further instructions 140 PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zThe demonstration will take several minutes and switch between modes based on a timer not number of entries zYou can select any item on each screen in the demonstration since this is not part of the experiment results zPractice scrolling and tapping on the screens zTake note of the speed at which taps are recognized and processed zIf a PDA seems ?stuck?, tap the item again using single taps PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow you will take a few minutes and practice in the demonstration mode zTurn on your PDAs: For the Palm IIIC the power button is a green button located in the lower left PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zSelect the ?Handheld? icon on the main screen zEnter your unique user id and tap start PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zSelect ?Yes? to begin the demonstration zAfter the demonstration, set down your PDA and wait for further instructions PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow we will begin the experiment and enter orders zDuring the time you will be entering orders, the researcher will randomly direct your attention using a sound 141 zEach time you hear this sound, direct your attention toward the projection screen where a series of multiple choice questions will be displayed PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zYou will have 15 seconds to read and answer the question before it disappears zThe answers to all of the questions are part of the results for the study and need to be answered on answer sheets provided PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zYou will now receive an answer sheet for these questions. Please put your user id on them both written and bubbled in using the #2 pencil provided zIf you do not know the answer, guess zOnce you mark your answer, continue entering orders PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zPlease ask any questions for this part of the experiment now PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zNow you will enter orders into the PDA from the medication order set sheets being passed out starting at order #1 zIf your PDA turned off, turn it on zSelect the ?Handheld? icon on your PDA for the experiment PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation zEnter your unique seven digit user id. Once this is entered, proceed to the message box that asks ?Is this a demonstration??, select the ?No? button. PDA-based Prescription Processing: A Simulated Interface Exercise and Evaluation 142 zProceed through the experiment entering as many as orders as you can as accurately as you can until you reach the conclusion screen telling you the experiment is over zClick ?OK? and wait for further instructions 143 Appendix 10 Pop-up Questions 144 Questions Question #1 ?Which of the following is a different way of representing 1 X 10 -5 ? ?A. 0.0001 ?B. 0.00001 ?C. 0.000001 ?D. 0. 001 Question #2 ? What is the result of the following expression? 0.002 X 1 X 10 3 ?A. 20 ?B. 0.2 ?C. 200 ?D. 2 ?E. 0.02 Question #3 ?Which of the following is the correct conversion for 1 pint? ?A. 500ml ?B. 400ml ?C. 473ml 145 ?D. 480ml ?E. 453ml Question #4 ?Which of the following is the correct conversion for 1 pound? ?A. 400g ?B. 498g ?C. 454g ?D. 500g ?E. 1000g Question #5 ?Which of the following is the correct conversion for 1 ounce? ?A. 32g ?B. 30g ?C. 26.4g ?D. 27g ?E. 28.4g Question #6 ?What is the metric measurement for a tablespoonful? ?A. 5 ml ?B. 30 ml 146 ? C. 25 ml ?D. 10 ml ?E. 15 ml Question #7 ?What is the metric measurement for a teaspoonful? ?A. 5 ml ?B. 15 ml ?C. 25 ml ?D. 10 ml ?E. 30 ml Question #8 ?What is the name of the Dean of Harrison School of Pharmacy? ?A. Lee Edwards ?B. Lee Evans ?C. Lee Jeans ?D. Lee Ewans Question #9 ?What is the correct Latin abbreviation for ointment? ?A. oin 147 ?B. oint ?C. ung ?D. unt Question #10 ?What is the correct Latin abbreviation for powder? ?A. powd ?B. pulv ?C. pow ?D. pod Question #11 ?Which of these is not the correct conversion? ?A. 1 pound =2.2 g ?B. 1 pound = 2.2 kg ?C. 1 dl= 100 ml ?D. 1L= 1000 ml Question #12 ?What does the Latin abbreviation ?qs? stand for? ?A. quality standard ?B. qualify staff ?C. quantity sufficient 148 ?D. quotient started Question #13 ?What county is Auburn University located in? ?A. Auburn ?B. Leigh ?C. Lee ?D. Tiger Question #14 ?Which of these is not a pre-requisite for entry into Harrison School of Pharmacy? ?A. Statistics ?B. Calculus ?C. Drug Information ?D. Physics Question #15 ?Who is the Auburn School of pharmacy named after? ?A. James I. Harrison ?B. Janet I. Harris ?C. James I. Harris ?D. Jamie I. Harrison 149 Appendix 11 Patient Names Used on Order Sheets 150 Patient Total Yates, Bonnie 1 Abbott, Cathy 1 Acosta, Willie 1 Adams, Raymond 1 Adkins, Norma 1 Allen, Timothy 1 Alvarado, Ethel 1 Andrews, Herman 1 Arnold, Tim 1 Baker, Gregory 1 Baker, Rufus 1 Ball, Sharon 1 Barber, Virginia 1 Barbour, Raymond 1 Barker, Sandra 1 Barlow, John 1 Barnett, Byron 1 Barrett, Willard 1 Beck, Freddie 1 Berry, Leo 1 Blair, Judy 1 Boone, Stacy 1 Booth, Kristina 1 Bowers, Glenn 1 Bowman, Kelly 1 Boyd, Edwin 1 Bradford, Marcia 1 Briggs, Hazel 1 Brock, Megan 1 Bryan, Brittany 1 Bryant, Philip 1 Buchanan, Leslie 1 Burton, Adrian 1 Byrd, Everett 1 Cain, Holly 1 Calhoun, Misty 1 Campbell, Henry 1 Carlson, Clayton 1 151 Patient Total Carmichael, Thomas 1 Castillo, Rafael 1 Cavender, Dennis 1 Chase, Lucy 1 Chavez, Lester 1 Christian, Gwendolyn 1 Clark, Harley 1 Clark, Ronald 1 Claycamp, Hugo 1 Cody, Robert 1 Cohen, Ashley 1 Cole, Nathan 1 Combs, Lydia 1 Conner, Phyllis 1 Cooper, Ralph 1 Copeland, Lucille 1 Cortez, Ana 1 Cox, John 1 Cox, Nicholas 1 Curtis, Bill 1 Daum, Peter 1 Davenport, Tonya 1 Davidson, Jordan 1 Deleon, Violet 1 Delgado, Jean 1 Dennis, Doris 1 Denny, Ralph 1 Diaz, Jimmy 1 Dickerson, Agnes 1 Downs, Jerry 1 Doyle, Sherry 1 Durbin, Jack 1 Dyer, Christy 1 Eckelman, Gerald 1 Eckelman, Herschel 1 Edwards, Ryan 1 Ellis, Allen 1 Epperson, Steven 1 Espinoza, Bessie 1 Estrada, Rosa 1 Fernandez, Roland 1 Figueroa, Dolores 1 Fischer, Maxine 1 Fisher, Norman 1 Fisher, William 1 Fletcher, Kent 1 152 Patient Total Flores, Jesse 1 Foster, Clarence 1 Francis, Anne 1 Frazier, Tyrone 1 Fuller, Neil 1 Gallagher, Stella 1 Garner, Diane 1 Garrett, Fernando 1 Gates, Stacey 1 George, Claude 1 Gibson, Vincent 1 Glover, Katherine 1 Gomez, Chad 1 Grant, Tom 1 Gray, Adam 1 Green, Andrew 1 Green, Delmar 1 Guzman, Donna 1 Haggard, Gordon 1 Hall, Earl 1 Hall, Ernest 1 Hall, Gary 1 Hampton, Marilyn 1 Hanson, Lonnie 1 Harmon, Judith 1 Harper, Vernon 1 Harrell, Sherri 1 Harrington, Marion 1 Harris, Bruce 1 Harris, Donald 1 Harrison, Donald 1 Harrison, Marvin 1 Haynes, Mary 1 Henderson, Carlos 1 Henderson, Russell 1 Hensley, Myrtle 1 Herrera, Bob 1 Hicks, Frederick 1 Hill, Eric 1 Hill, Wayne 1 Hodges, Lois 1 Holmes, Francisco 1 Hood, Ella 1 Horton, Morris 1 Houston, Emma 1 Howard, Roy 1 Hubbard, Stephanie 1 Hudson, Leon 1 153 Patient Total Ingram, Kathy 1 Jefferson, Erin 1 Jensen, Felix 1 Jesse, Arnold 1 Jimenez, Isaac 1 Johnson, John 1 Jones, Edgar 1 Jones, Michael 1 Jordan, Manuel 1 Judd, Damon 1 Keith, Olga 1 Keller, Michelle 1 Kelley, Tyler 1 Kemp, Bobbie 1 Kennedy, Ricky 1 King, Jeffrey 1 Kirk, Maureen 1 Kramer, Viola 1 Lamb, Samantha 1 Lang, Glenda 1 Larsen, Leah 1 Lewellen, Jeanne 1 Lind, Robert 1 Lindsey, Carmen 1 Lloyd, Beatrice 1 Long, Ernest 1 Love, Lisa 1 Lucas, Salvador 1 Lucas, William 1 Lynch, Jessie 1 Lyons, Jessica 1 Maddock, William 1 Maldonado, Wanda 1 Marsh, Darlene 1 Marshall, Travis 1 Massey, Erica 1 Mathews, Mattie 1 Maxwell, Rita 1 Mcbride, Pauline 1 Mcclain, Terry 1 Mcdonald, Glenn 1 Mckinney, Marshall 1 Medina, Kurt 1 Meyer, Franklin 1 Meyers, Brandy 1 Miles, Perry 1 Miller, Martin 1 Miller, Richard 1 154 Patient Total Mitchell, Walter 1 Monroe, Marvin 1 Monroe, Melinda 1 Morales, Eddie 1 Moran, Clara 1 Morrison, Ron 1 Mosley, Nina 1 Moss, Heather 1 Mueller, Becky 1 Munoz, Ruth 1 Murphy, Keith 1 Murray, Jacob 1 Myers, Danny 1 Nading, Paul 1 Nash, Elsie 1 Navarro, Ramona 1 Neal, Dave 1 Newton, Kelly 1 Nguyen, Cory 1 Nicholson, Delores 1 Norton, Lynn 1 Nysewander, Victor 1 Oneal, Marlene 1 Ortega, Andrea 1 Ortiz, Jeff 1 Osborne, Florence 1 Ostick, John 1 Palmer, Oscar 1 Parrish, Claire 1 Patel, Toni 1 Patrick, Lorraine 1 Paul, Tracy 1 Pena, Enrique 1 Perry, Victor 1 Potter, Irene 1 Powell, Martin 1 Ramirez, Harry 1 Ramos, Barry 1 Reedy, Homer 1 Reese, Theresa 1 Reeves, Christine 1 Regan, Richard 1 Reyes, Alexander 1 Richardson, Lawrence 1 Robert, James 1 Robertson, Floyd 1 Robertson, Max 1 155 Patient Total Robertson, Micheal 1 Rodgers, Rose 1 Rodriquez, Sergio 1 Romero, Clinton 1 Rose, Jon 1 Roy, Jamie 1 Ruiz, Maurice 1 Salazar, Carolyn 1 Sanchez, Juan 1 Sandoval, Cindy 1 Santos, Frances 1 Schneider, Angela 1 Scott, Stephen 1 Sharp, Amy 1 Shelton, Clifton 1 Shepherd, Russell 1 Shields, Tracey 1 Shoaf, Donald 1 Short, Jo 1 Silva, Hugh 1 Simon, Marjorie 1 Singleton, Vivian 1 Skinner, Georgia 1 Sparks, Joann 1 Stephens, Darrell 1 Sterns, John 1 Stevenson, Janet 1 Stewart, Joe 1 Stillabower, Charles 1 Stone, Ronnie 1 Stringer, James 1 Stuckey, Joseph 1 Tabor, Louis 1 Thompson, Kenneth 1 Thornburg, Maurice 1 Thornton, Teresa 1 Torres, Bruce 1 Townsend, Tammy 1 Trimpe, Ernest 1 Tucker, Bradley 1 Tyler, Wendy 1 Vasquez, Nathaniel 1 Vaughn, Elizabeth 1 Voiles, Loren 1 Wagner, Everett 1 Walsh, Deborah 1 Walters, Alfredo 1 Walton, Lori 1 156 Patient Total Washington, Craig 1 Waters, Carrie 1 Watson, Wayne 1 Weales, William 1 Weaver, Shane 1 Weber, Sarah 1 Weisner, Gregory 1 Wells, Melvin 1 Wheeler, Leslie 1 Whitaker, Tamara 1 Wilcox, Vicki 1 Williams, John 1 Williams, Robert 1 Williamson, Chester 1 Woehrman, Benjamin 1 Wong, Jill 1 Wood, Randy 1 Woodard, Velma 1 Woodruff, Alva 1 157 Appendix 12 Medication Names Used 158 Drug Total Zyrtec 2 Zyprexa 2 Accupril 2 Accutane 2 Acetazolamide 2 Acetohexamide 2 Advicor 2 Aggrastat 2 Aggrenox 2 Aldara 2 Alora 2 Altace 2 Altocor 2 Alupent 2 Artane 2 Atrovent 2 Cafergot 2 Calciferol 2 Calcitriol 2 Captopril 2 Carafate 2 Cardene 2 Cardene SR 2 Cardizem 2 Cardizem CD 2 Cardizem SR 2 Cardura 2 Carteolol 2 Carvedilol 2 Cataflam 2 Catapres 2 Celebrex 2 Celexa 2 Chlorpromazine 2 Chlorpropamide 2 Codeine 2 Diflucan 2 Diprivan 2 Edecrin 2 Efudex 2 Eldepryl 2 Eldopaque Forte 2 Eloquin Forte 2 159 Drug Total Enalapril 2 Equagesic 2 EquiGesic 2 Erythrocin 2 Eskalith 2 Estratab 2 Estratest 2 Estratest HS 2 Ethmozine 2 Etidronate 2 Etomidate 2 Etretinate 2 Zetia 2 Eurax 2 Glipizide 2 Glucophage 2 Glucotrol 2 Zebeta 2 Zantac 2 Glyburide 2 Guaifenesin 2 Gunfacine 2 Haloperidol 2 Halotestin 2 Heparin 2 Hespan 2 Hydralazine 2 Hydrocodone 2 Hydrocortisone 2 Hydroxyzine 2 Kogenate 2 Kogenate-2 2 Lamicel 2 Lamictal 2 Lamisil 2 Lamivudine 2 Lamotrigine 2 Wellbutrin XL 2 Lasix 2 Leucovorin 2 Leukeran 2 Leukine 2 Levaquin 2 Levoxine 2 Levsin 2 Levsin Sl 2 Lomotil 2 Lonox 2 160 Drug Total Ludiomil 2 Naprelan 2 Naprosyn 2 Nasalide 2 Nasarel 2 Navane 2 Nelfinavir 2 Neoral 2 Nephrox 2 Neumega 2 Neupogen 2 Nevirapine 2 Niacin 2 Niaspan 2 Nicardapine 2 Nicoderm 2 Nifedipine 2 Niferex 2 Nitroderm 2 Nizoral 2 Norvasc 2 Prilosec 2 Prozac 2 Quinidine 2 Quinine 2 Wellbutrin SR 2 Relafen 2 Remegel 2 Renagel 2 Reserpine 2 Retrovir 2 Revex 2 ReVia 2 Rezulin 2 Rifabutin 2 Rifampin 2 Risperdal 2 Ritonavir 2 Seroquel 2 Serzone 2 Tegretol 2 Topomax 2 Toprol 2 Toprol XL 2 Ultane 2 Ultram 2 Urised 2 Urocit-K 2 161 Drug Total Vancenase 2 Vanceril 2 Verelan 2 Vexol 2 Viracept 2 Viramune 2 Virilon 2 VoSol 2 Accolate 1 Refresh 1 Lanoxin 1 Glutofac 1 Glucotrol XL 1 Eulexin 1 162 Appendix 13 Data Sheets For Subjects 163 SUBJ Gender AGE ERRORS SUBJ Gender AGE ERRORS 101368 M 23 1 3858366 F 21 4 337366 M 23 5 3879352 F 22 2 400951 F 25 6 3951359 M 21 5 408750 F 24 10 4385357 F 20 3 411352 M 26 4398358 F 25 5 447325 F 23 4441354 M 22 0 976368 M 2 4493329 F 23 10 980352 M 24 0 4521355 F 22 1055354 F 25 10 4581366 M 22 1 1165368 M 29 7 4610323 F 23 3 1334724 F 23 1 4650352 F 22 8 1447352 F 22 10 4669356 M 24 3 1459359 M 22 0 4990900 F 22 4 1486338 F 23 10 5098352 0 1580363 F 22 10 5134358 F 22 5 1679350 F 21 5 5182368 M 34 7 1693368 M 25 4 5211365 F 22 2 1762352 F 23 3 5321355 1843351 F 23 1 5336392 F 24 10 1874320 F 22 1 5355356 F 22 3 1925330 F 28 1 5383356 F 21 1 2169368 M 35 3 5431360 F 24 10 2331368 F 28 4 5615356 F 29 2392390 M 24 5 5635351 F 22 5 2610364 M 28 5 5642368 F 23 3 2614351 M 23 10 5672352 F 22 8 2699354 F 24 10 5769368 M 23 3 2963368 F 24 10 5816367 F 21 7 2972327 F 21 5903384 F 23 5 3011305 F 23 2 6059368 M 23 10 3082366 6094356 M 21 4 3412427 M 23 2 6147368 M 24 22 3420850 M 24 4 6260356 M 22 2 3440368 23 6317365 F 22 115 3609363 F 23 1 6383362 M 32 10 3793325 F 21 20 6391354 F 22 5 6434393 F 29 0 6488354 M 25 0 6496302 F 22 5 6512463 F 25 4 6766368 M 33 3 6828561 M 22 2 164 SUBJ Gender AGE ERRORS 7263384 M 29 0 7271357 F 21 10 7321361 F 21 5 7479354 M 22 3 7523359 7601362 M 22 5 7644370 M 25 119 7659374 M 24 7702505 M 20 20 7721631 F 25 3 7812359 F 24 5 7838358 F 23 5 8013368 M 30 4 8016863 F 23 15 8062356 F 21 0 8222358 M 24 7 8340368 22 2 8485301 M 30 2 8535366 F 22 2 8674841 M 23 10 8731368 M 22 20 8759974 F 22 5 8800787 F 22 1 8977355 F 22 10 9016363 F 22 3 9049360 F 21 0 9251357 F 22 10 9334362 F 23 5 165 Appendix 14 CAPP Lab Postures 166 167 Appendix 15 Example of Study Data Record 168 2:30:19 PM 0 4 Clay, Wilma S S P 2:30:20 PM 0 4 Clay, Wilma W S P 2:30:30 PM 0 4 NAVELBINE S S D 2:30:32 PM 0 4 NAVELBINE W S D 2:30:42 PM 0 4 10mg S S S 2:30:43 PM 0 4 10mg W S S 2:30:52 PM 0 4 0-1 S S A 2:30:53 PM 0 4 0-1 W S A 2:31:02 PM 0 4 BID S S F 2:31:03 PM 0 4 BID W S F 2:31:44 PM 0 5 McNealy, Chester S S P 2:31:45 PM 0 5 McNealy, Chester R S P 2:31:52 PM 0 5 NICARDIPINE HCL S S D 2:31:53 PM 0 5 NICARDIPINE HCL R S D 2:31:57 PM 0 5 50mg S S S 2:31:58 PM 0 5 50mg R S S 2:32:03 PM 0 5 5-Apr S S A 2:32:04 PM 0 5 5-Apr R S A 2:32:18 PM 0 5 Q12h S S F 2:32:19 PM 0 5 Q12h R S F