Participatory Design of Warning Symbols Using Distributed Interactive 
Evolutionary Computation 
 
by 
 
Adam Kelly Piper 
 
 
 
 
A dissertation submitted to the Graduate Faculty of 
Auburn University 
in partial fulfillment of the 
requirements for the Degree of 
Doctor of Philosophy 
 
Auburn, Alabama 
May 14, 2010 
 
 
 
 
Keywords: warnings, symbols, genetic algorithms 
participatory design, risk perception 
 
Copyright 2010 by Adam Kelly Piper 
 
 
Approved by 
 
Jerry Davis, Chair, Associate Professor of Industrial and Systems Engineering 
Nathan Dorris, Affiliate Professor of Industrial and Systems Engineering 
Woojin Park, Affiliate Professor of Industrial and Systems Engineering 
 ii 
 
 
 
 
 
Abstract 
 
 
 Safety warnings play an important role in communicating risk via product labels 
and environmental signs.  With the diversification of cultures and languages in the United 
States, and with the increasing globalization of most industries, emphasis on the 
communication of this risk through symbols and other non-written forms has increased.  
Both ANSI and ISO have developed voluntary standards for the production and 
evaluation of warning symbols, but many symbols currently in use have been found 
deficient with respect to the comprehension and effectiveness guidelines found in these 
standards.  In other cases, commonly used symbols have not undergone effectiveness 
evaluation at all.  Thus, there remains a need to produce warning symbols shown to be 
effective in communicating risk to a multicultural, multilingual, global society. 
 Though the ANSI and ISO standards fail to specify a technique for developing 
symbol designs, three techniques were identified from the literature.  Of these three, the 
focus group method was claimed by its developers to be the most effective in producing 
high quality symbol designs because it involves realistic users of the symbols in more 
aspects of the design process than either of the other techniques.  The focus group method 
requires human participants to sort and filter many designs into a single proposed symbol.  
This type of search task is well suited to machine computation, and this research will 
model the focus group method of human design generation and consolidation as a 
distributed interactive genetic algorithm which will evaluate and generate designs using 
 iii 
simple simultaneous feedback from a group of human users.  The literature revealed a 
similar interactive evolutionary computation algorithm used to design safety symbols in a 
prior study, although that algorithm used a single participant and still required human 
designers to evaluate many symbols by hand to determine the best design.  The proposed 
distributed interactive genetic algorithm will remove the designer?s input at this stage of 
the design process by allowing the users and the algorithm to determine a final design for 
the group without designer interference. 
 First, a survey was administered to 145 university students and safety 
professionals to determine an ordered list of safety messages (or referents) sorted by their 
perceived difficulty to convert into symbols.  From this list, two referents were chosen for 
the study, one easy (?Hot Exhaust?) and one difficult (?Do Not Touch with Wet Hands?).  
Seventy American university students, 35 born in the U.S. and 35 born in India, were 
recruited to sketch symbol designs for each of the two referents.  These designs were 
evaluated by a panel of safety professionals to identify the graphical attributes contained 
in each drawing, and the presences or absence of each identified attribute in a given 
symbol created a binary attribute matrix for each referent.  These matrices were summed 
and clustered using a K-means clustering algorithm to determine the centroid values of 
each cluster of symbol drawings.  Thirty-five attributes were identified by the panel 
among the ?Hot Exhaust? drawings, and the clustering revealed that only three of them 
were present among the centroid values of each of the five identified clusters.  Likewise, 
28 attributes were identified for the ?Do Not Touch with Wet Hands? drawings, but only 
five were present in the centroid values of the four clusters identified for this referent. 
 iv 
 From these centroidal attributes, a version of the distributed interactive genetic 
algorithm was created for each referent.  Forty-six participants, divided into four groups 
of 10-12 by country of origin, designed symbols using the algorithm, and the symbol 
most representative of each group was compared by 401 participants from around the 
globe to symbols generated using a traditional method and to symbols in use currently.  
The results indicated that for the easier referent, ?Hot Exhaust?, the algorithm produced 
symbols that performed as well or better than symbols produced by other means, 
including the symbol currently in use.  However, for the more difficult referent, ?Do Not 
Touch with Wet Hands?, other symbols performed better than those produced by the 
algorithm.  Additionally, the algorithm generally converged in 20 generations or less, 
which falls within the recommended limitations of such algorithms within the literature.  
However, the algorithm converged faster for U.S. and multinational groups than for 
groups of participants from other single nations.   
 In summary, the distributed interactive genetic algorithm technique showed 
promise as a design tool for developing symbols that perform as well or better than 
current design methods.   Furthermore, the algorithm?s performance may vary depending 
on the difficulty level of the referent tested as well as on the composition of the 
participant groups used in the design process.  Further research is needed to confirm and 
characterize these relationships. 
 
 v 
 
 
 
 
 
Acknowledgments 
 
 
 I offer my most sincere appreciation for the contribution of my dissertation chair, 
Dr. Jerry Davis, of his patience, insights, encouragement and mentoring in the 
development of both a dissertation and an academic of which I can be proud.  Deepest 
appreciation I also owe Dr. Nathan Dorris, Dr. Woojin Park, and Dr. David Umphress, 
who each agreed, without hesitation, to support this research.  I would also like to thank 
Mr. Pemsith Mendis, who contributed invaluable programming expertise and research 
assistance.  To Dr. Kent Oestenstad, to the Deep South Center for Occupational Health 
and Safety and to the National Institute for Occupational Safety and Health, I thank you 
for the financial support that made my research and my education possible. 
 To my precious wife, Dr. Susan Piper, I cannot express in written words the value 
I place on the encouragement, the example, the feedback, and the support you have 
provided me though the many years it took to arrive at this moment.  To my parents, my 
sisters and my brothers-in-law, thank you for always asking about my progress and 
encouraging me to keep pressing towards the goal.  I would also like to express special 
gratitude to Dr. Rob Thomas, who, for more than a decade, has served as a mentor to me 
as both a professional and a godly man.  Without his encouragement to ?come back and 
finish?, I would never have made it here.  Finally, I thank God for his wisdom and grace 
and for Micah 6:8 to which I strive to live:  ?What does the LORD require of you but to 
do justice, and to love kindness, and to walk humbly with your God??
 vi 
 
 
 
 
Table of Contents 
 
 
Abstract............................................................................................................................... ii 
Acknowledgments ...............................................................................................................v 
List of Tables .................................................................................................................... vii 
List of Figures.................................................................................................................. viii 
Chapter 1.  Introduction  .....................................................................................................1 
Chapter 2.  A Review of the literature on the production and development of safety 
warning symbols ..................................................................................................................5 
Chapter 3.  Determining a ranked order of warning referents by ease of conversion from 
written referents to graphical symbols...............................................................................31 
Chapter 4.  Synthesis and clustering of symbol attribute matrices from hand-drawn safety 
symbols ..............................................................................................................................47 
Chapter 5.  Developing and testing a distributed interactive genetic algorithm to design 
safety warning symbols......................................................................................................68 
Chapter 6.  Conclusions .....................................................................................................98 
References  ......................................................................................................................105 
Appendices  .....................................................................................................................123 
 
 
 vii 
 
 
 
 
 
List of Tables 
 
 
Table 1. Referents used in the ease of conversion survey .................................................37 
Table 2. Post-hoc analysis results for All 145 participants ...............................................40 
Table 3. Post-hoc analysis results for 55 University Students ..........................................41 
Table 4. Post-hoc analysis results for 44 Uncertified Professionals..................................41 
Table 5. Post-hoc analysis results for 46 Certified Professionals......................................42 
Table 6. Ease of conversion rank order of nine referents by strata ...................................42 
Table 7. Kendall?s Tau-b concordance between strata for the final ranked order of 
referents..................................................................................................................43 
Table 8. Semantic annotation summary of two drawing sets by a three-member panel ...59 
Table 9. Primary attribute sets that describe the centroid vectors of each symbol cluster 64 
Table 10. Attribute subsets by stratum ..............................................................................65 
Table 11. Design Parameters of the Hot Exhaust genotype...............................................78 
Table 12. Design Parameters of the Do Not Touch with Wet Hands genotype ................78 
Table 13. Composite evaluation results of the best symbol by each DIGA design group 88 
Table 14. Convergence analysis of one-tail paired t-test statistics comparing the 
coefficient of variation of the first and final generations between subjects for each 
DIGA trial ..............................................................................................................90 
Table 15. Results of ANOVA and Tukey HSD analysis for ?Hot Exhaust? Symbol      
Evaluations.............................................................................................................92 
Table 16. Results of ANOVA and Tukey HSD analysis for ?Do Not Touch with Wet 
Hands? Symbol Evaluations ..................................................................................93 
 
 viii 
 
 
 
 
 
List of Figures 
 
 
Figure 1. ANSI Z535.4 format with three panels, horizontal and vertical versions..........12 
Figure 2. ISO 3846-2 format with multiple symbols, horizontal and vertical versions.....13 
Figure 3. A reciprocal table for a nine-treatment balanced incomplete block experiment34 
Figure 4. Final ranked order of warning referents by ease of conversion for all strata .....43 
Figure 5. The two warning referents selected for use in the current study and their relative       
rank ........................................................................................................................53 
Figure 6. Row vector representing the semantic attributes of a single symbol drawing ...55 
Figure 7. K-means cluster results from ?Hot Exhaust? attribute matrix with combined 
strata.......................................................................................................................60 
Figure 8. K-means cluster results from ?Do Not Touch with Wet Hands? attribute matrix 
with combined strata..............................................................................................61 
Figure 9. K-means clustering results for the U.S. stratification.........................................62 
Figure 10. K-means clustering results for the Indian stratification ...................................62 
Figure 11. IDEA pseudo code............................................................................................72 
Figure 12. Server-side DIGA Flow Chart..........................................................................73 
Figure 13. Client-side DIGA Flow Chart ..........................................................................73 
Figure 14. Pseudo code for DIGA symbol design algorithm.............................................74 
Figure 15. Illustration of two-point crossover ...................................................................75 
Figure 16. Illustration of single-point mutation.................................................................76 
Figure 17. Genotype encoding of both symbol phenotypes ..............................................77 
Figure 18. Client interface for Hot Exhaust operated by DIGA participants ....................81 
 ix 
Figure 19. Best ?Hot Exhaust? symbols from each DIGA group......................................87 
Figure 20. Best ?Do Not Touch with Wet Hands? symbols from each DIGA group........87 
Figure 21. Convergence of ?Hot Exhaust? algorithm based on number of times a 
participant selected a new best symbol per generation..........................................89 
Figure 22. Convergence of ?Do Not Touch with Wet Hands? algorithm based on number 
of times a participant selected a new best symbol per generation .........................90 
Figure 23. Draft and final focus group symbols for ?Hot Exhaust? ..................................91 
Figure 24. Draft and final focus group symbols for ?Do Not Touch with Wet Hands?....91 
Figure 25. Previously published manufacturer?s symbol for ?Hot Exhaust and ISO 
symbol for ?Do Not touch with Wet Hands?.........................................................92 
 
 
 
1 
 
 
 
CHAPTER 1 
INTRODUCTION 
 
 Hazard warnings communicate safety information, often in the vicinity of the 
hazard, through a variety of modalities.  Classical warnings, such as the light house and 
fog horn, have been used for centuries to aid those exposed to hazards to which they 
might have been unaware (Egilman & Bohme, 2006).  Common warnings in modern 
American society include flashing lights and bell alarms for railroad crossings, printed 
pharmaceutical information about potential side effects and interactions, and traffic signs 
alerting drivers to the risk of deer crossing the highway.   One of the most common forms 
of hazard warnings is the static visual warning, which may include written messages, 
graphical symbols, color coding schemes, or all of these (ANSI, 2007a, 2007b; ISO, 
2003, 2006). These kinds of warnings are found in a variety of locations, such as on 
product labels, in written manuals, on industrial signage and in places used by the public.  
Graphical symbols have been suggested to improve comprehension of visual 
warnings as well as to attract attention to them (Boersema & Zwaga, 1989; Davies, 
Haines, Norris, & Wilson, 1998).  Unfortunately, the process of producing effective 
graphical symbols for static visual warnings can be a tedious and time consuming task 
involving iteration after iteration of research participant input, designer evaluation and 
field trials (Goldsworthy & Kaplan, 2006; Green, 1993; Young & Wogalter, 2001).  For 
this reason, many designers borrow or modify existing designs rather than attempt to 
 2 
create their own from scratch (Edworthy & Adams, 1996).  However, using older 
symbols that were designed prior to the publication of current guidelines may mean that 
the borrowed symbols have not been evaluated for their effectiveness (Deppa, 2006) or 
that they were not designed with a global, diverse population in mind (Huer, 2000; 
Laughery, 2006).  While more and more symbols are being designed and tested, at least 
in part, to address the former concern, the latter concern continues to grow more 
problematic as global trade and immigration diversify the user populations of nearly 
every product or piece of equipment. 
The use of participatory design, or the development and evaluation of design 
concepts using potential users of the system or product, is believed to improve the quality 
of the final designs (Schuler & Namioka, 1993).  This technique has been used, to 
varying degrees, in the design of graphical warning symbols for years (Green, 1993; 
Macbeth, Moroney, & Biers, 2000; Pettendorfer & Mont'alvao, 2006).  Furthermore, 
recent advances in computational methods have led to the addition of technology to assist 
in the participatory design process (Carnahan & Dorris, 2004; Carnahan, Dorris, & 
Kuntz, 2005; Dorris, 2004; Dorris, Carnahan, Orsini, & Kuntz, 2004; Dozier, Carnahan, 
Seals, Kuntz, & Fu, 2005b; Parmee, Abraham, & Machwe, 2008).  This has proved to be 
a promising development in the attempt to involve more diverse participants in the design 
process because the technological innovation allows communication of design 
information with less interference from the barriers of language, culture and geography.   
 
 3 
Research Objectives   
 The literature, reported in Chapter 2 of this dissertation, reveals a gap in the 
incorporation of innovative computational technology in the participatory design process 
of producing graphical safety symbols.  While computational technology has been used 
to assist symbol designers before, it has yet to be incorporated into the methods that are 
the most participatory in nature (Macbeth et al., 2000; Pettendorfer & Mont'alvao, 2006). 
Furthermore, the literature clearly reports the missing component of cultural and first-
language diversity among the participants recruited to design and develop these symbols 
(Huer, 2000; Lesch, Rau, Zhao, & Liu, 2009; Russo & Boor, 1993).  Thus, the objective 
of this research is to bridge the gap between participatory design and computational 
technology by using advanced computational techniques to replace a traditional symbol 
design focus group with a group of design participants interacting through a computer 
network.  In this way, users of various cultures, language proficiencies and even 
geographic locations can interact and share ideas meaningfully and simultaneously in the 
symbol design process. 
 
Research and Dissertation Organization   
The chapters of this dissertation are organized according to the publication 
format.  The dissertation is comprised of six chapter manuscripts.  Chapter One is a 
traditional introduction, and Chapter Six is a traditional conclusion.  Chapter Two is a 
comprehensive literature review of the safety warning symbol development process and 
the use of interactive evolutionary computation to design risk communication.  Each of 
the remaining chapters is a stand-alone manuscript describing the purpose, methods, 
 4 
results and discussion of an experiment.  Because of the special arrangement of this 
format, a brief survey of the most relevant literature is provided in each of the remaining 
manuscripts. The experiment in Chapter Three surveys safety professionals to determine 
the expected difficulty of converting written warning messages to graphical symbols in 
order to sort those warning messages by difficulty.  Chapter Four reports on the 
production of two pools of symbol proto-designs, with each pool portraying a safety 
message from a significantly different difficulty level identified in Chapter Three.  Each 
pool of candidates was analyzed for their semantic attributes and grouped into clusters 
with similar design intent. Software using distributed interactive evolutionary 
computation was developed with the capability of producing symbols comprised of 
components from the median symbol of the clusters identified in Chapter Four, the users 
of which performed the symbol generation and refinement role traditionally performed by 
safety symbol design focus groups.  Chapter Five summarizes the development and 
performance of the algorithm and reports the results of a comparison of the newly 
produced symbols to previously published symbols and to symbols generated in an 
additional experiment following the focus group method of symbol development. The 
limitations of the study, the study recommendations, and the overall conclusions are 
discussed in Chapter Six.  The appendices contain details outlining the recruitment and 
participation of human subjects, the specific protocols used for each experiment, 
summaries of the collected data, and other information which support the results 
presented in the chapter manuscripts. 
 5 
 
 
 
 
CHAPTER 2 
A REVIEW OF THE LITERATURE ON THE PRODUCTION AND 
DEVELOPMENT OF SAFETY WARNING SYMBOLS 
 
Introduction to Warnings 
 Wogalter (2006) defines a warning as a safety communication ?? used to inform 
people about hazards??  While the last two decades have seen a dramatic increase in the 
amount of warnings research (Laughery, 2006), warnings have been used by people for 
millennia (Stanton, 1994).  For example, bells were once used to alert villagers of an 
advancing enemy force, and lighthouses have long warned mariners of reefs or rocky 
shores.  With the industrial age, new hazards arose, and many new warnings were 
developed.  Pedestrians and passengers were alerted to oncoming trains by lamps and 
whistles, while hand signals and signs used by railroad workers helped ensure that 
locomotives and people avoided undesirable interactions (Egilman & Bohme, 2006).  As 
industrial production and therefore industrial hazards increased, the development of 
warnings aimed at industrial workers also increased.  Due to both growing concerns for 
the safety of workers and the emergence of workplace injury litigation citing a ?failure to 
warn,? organizations such as the National Safety Council and even the U.S. Congress 
contributed to the development and use of warnings (Clark, Benysh, & Lehto, 2003; 
Egilman & Bohme, 2006).  Today, both voluntary standards and legal statutes exist that 
 6 
recommend, or in some cases require, the use of safety warnings (ANSI, 2007c; ISO, 
2003; OSHA, 1996).   
Laughery and Wogalter (2006) list three specific purposes and a fourth general 
purpose for warnings.  Warnings attempt to inform people about hazards, their 
consequences, and how to avoid them.  Warnings purport to influence behavior; 
specifically, they promote safe behavior.  Warnings serve as a reminder of previously 
learned information, including the nature of hazards and their consequences, how to 
avoid them, and where and when to be vigilant.  Finally, warnings? ultimate purpose is to 
make the world safer for its human occupants.  In this regard, they serve a public safety 
goal of protecting members of society, and therefore they have received considerable 
attention from government and standardization organizations.  Laughery and Wogalter 
(2006) present a brief but informative summary of the growth of regulatory interest in 
warnings in the U.S. during the 20th century.   
In practice, warnings may take a variety of forms.  Though not exhaustive, 
Hammer (1989) provides an informative list of warnings targeted to a variety of human 
senses.  Most people have experienced warnings that target the olfactory (odorant added 
to natural gas to detect leaks), tactile (rumble strips on a highway to warn of upcoming 
intersections) and gustatory (a bitter chemical added to poisonous products to keep 
children from consuming them) senses, though examples of these are relatively rare.  
Warning modalities that utilize the visual and auditory senses are more common (Cohen, 
Cohen, Mendat, & Wogalter, 2006; Laughery & Wogalter, 2006).  Auditory warnings for 
fire, severe weather and burglary are well known examples among the general public.  
Industrial safety warnings that use the auditory channel include backup alarms on 
 7 
vehicles, atmospheric contaminant alarms and the voice of an attendant guarding a 
confined space (Hammer, 1989).  Familiar static visual warnings include those which 
appear on product packaging and labels and those found on signs in the workplace and in 
public areas (Lesch, 2006; Rousseau & Wogalter, 2006).  A visual warning may also be 
dynamic, such as an animated hazard warning sign, an electronic scrolling traffic sign or 
even a set of hand signals to and from crane operators to those on the ground (Hammer, 
1989; Wogalter, Racicot, Kalsher, & Noel Simpson, 1994).   Some warnings may even 
involve more than one of these modalities.  Several studies have specifically explored the 
efficacy of various warning modalities, both within and across sensory channels 
(Campbell et al., 2004; A. H. S. Chan & Ng, 2009; Haas & Edworthy, 2006).  In fact, 
mixed modal warnings, especially those that utilize multiple sensory channels, have been 
shown to improve warning effectiveness in some contexts (Cohen et al., 2006).   
Warnings are passive in their protective function in that they require a response 
from each warning recipient in order to be effective.  Specifically, an effective warning 
must be noticed, understood and heeded (Miller & Lehto, 2001).  Other researchers have 
defined more detailed models of the warning process (Clark, 1988; Lehto & Papastavrou, 
1993; Rogers, Lamson, & Rousseau, 2000; Wogalter, Dejoy, & Laughery, 1999), and 
Lehto (2006) provides a good historical summary of this research.  However, it is not the 
aim of this research to explore these models further or to comment on their adequacy.  
Rather, the purpose of this literature review is to examine the process of designing the 
graphical symbols used in warnings and other safety communications. 
 
 8 
Warning Symbols 
Much of the warnings research from the last two decades has focused on 
evaluating the effectiveness of warnings as a communication system, and in a majority of 
circumstances, visual warnings were the primary modality of interest (Smith-Jackson & 
Wogalter, 2006).  Though they differ in their taxonomy, several researchers have 
reported that, regardless of modality, the warning process involves a series of stages 
which must all succeed in order for the warning to be effective at changing behavior 
(Lehto, 2006; Rogers et al., 2000; Wogalter et al., 1999).  Though a discussion of these 
individual stages are not salient to this research, Rogers et al. (2000) provided a thorough 
summary of the variables identified in empirical research that affect a visual warning 
system?s performance.  They identified more than 50 person-related or warning-related 
variables that affect warning effectiveness based on their effect on at least one stage of 
the warning process.  Laughery and Wogalter (2006) further contributed to this 
understanding by labeling some variables specifically as design variables.   
Though several of these design variables (e.g. color, message length, signal word) 
can be present in warnings without symbols, the use of symbols as an important design 
component has been noted in several studies, according to Laughery and Wogalter 
(2006). The effect of symbols (pictorials, icons, graphics, pictograms, etc.) on the 
warning process has been studied extensively.  In general, research has determined that 
symbols can aid warning performance by calling attention to the warning and enhancing 
the comprehension of the warning message (Wogalter, Silver, Leonard, & Zaikina, 2006).   
Specifically, Laughery, Young, Vaubel and Brelsford (1993) reported that symbols were 
useful in gaining attention for warnings, especially for those in which printed information 
 9 
is small or illegible (Kalsher, Wogalter, & Racicot, 1996), and Davies et al. (1998) found 
symbols to be especially valuable when space on the sign or label was restricted.  
Furthermore, Friedmann (1988) found that the presence of well-designed symbols 
increased the probability that salient information written in the warning would be read.  
Jaynes and Boles (1990) reported that pairing symbols with verbal warnings improved 
compliance over either component presented alone, while Lesch (2008a, 2008b) found 
that the pairing of accident scenarios and symbols increased comprehension and recall of 
prior knowledge more than did a pairing of symbols with verbal labels.  Interestingly, 
Kalsher et al. (1996) notes that warnings that contain graphical symbols are preferred by 
people over warnings that do not.   
Nevertheless there have also been empirical studies which found little or no 
benefit to the inclusion of symbols with warnings.  Both Otsubo (1988) and Friedman 
(1988) found that symbols generally had no effect on noticeability of or compliance with 
warnings, while at the same time noting that the most noticed warnings, including some 
with symbols, were also the most heeded.  More complex or abstract symbols were found 
to distract from the actual hazards by Mayer and Laux (1989), although the inclusion of 
simple and concrete symbols improved warning noticeability in their study.  Jaynes and 
Boles (1990) qualified the benefits they reported from pairing symbols with written 
warnings by also reporting that symbols alone were heeded less often than written 
warnings alone.  Though research has suggested that there are many benefits to the use of 
warning symbols, symbols that are designed poorly may actually be detrimental to 
warning effectiveness.  Therefore, this research concentrates on the design and evaluation 
of warning symbols rather than on other aspects of the warning process. 
 10 
Symbols as a Culture and Language Bridge 
An additional advantage of warning symbols over other warning components is 
that symbols have the potential to be understood by a greater number of people (Wogalter 
et al., 2006).  Research has reported warning symbols to be both language-independent 
(Liu, Hoelscher, & Gruchmann, 2005) and culture-neutral (Edworthy & Adams, 1996).  
Hodgkinson and Hughes (1982) found that pictorial instructions could circumvent 
language barriers among multi-national customers when unpacking and assembling IBM 
typewriters, though several design iterations were necessary to produce an adequate 
version.  Foster and Afzalnia (2005) tested symbol comprehension in the UK, Korea and 
Iran, and they argue that agreement among the results suggests that standardizing 
international symbols may be possible.  Kalsher et al. (1996) reported that well-designed 
pharmaceutical symbols may be critical in reaching patients who have low literacy or low 
language proficiency, though they caution that poorly designed symbols may actually 
decrease comprehension in these populations.  However, some research challenges the 
notion that symbols are culturally neutral (Smith-Jackson, 2006).  Huer (2000) reports on 
several studies that have found a dependency of symbolic communication on cultural 
experience, and she suggests that culture and language interact and cannot be easily 
separated in a communication context.  Russo and Boor (1993) reported that symbols, 
such as the ?X? (i.e. a cross) may have an opposite meaning in Egypt than in Western 
countries, and Dowse and Ehlers (2001) found an overwhelming preference among low-
literate South Africans for symbols designed locally rather than internationally.  
Unfortunately, the involvement of potential users in symbol design is very rare. Dorris 
(2004) and Huer (2000) suggest that individuals with limitations in language proficiency 
 11 
bear the greatest risk from poorly designed symbols, yet both authors report that the few 
research studies that make use of potential users in symbol design almost exclusively do 
so only in the symbol evaluation stage.  Thus, there remains a significant dearth in 
symbol design research that incorporates potential users in the design process.  It is the 
intention of this research to fully utilize culturally diverse research participants to both 
design and evaluate warning symbols. 
 
Designing Symbols 
The development and implementation of the graphic symbols which comprise a 
portion of, or in some cases the entirety of, a safety warning has proven to be a challenge 
to researchers. According to Dorris (2004), the procedure for producing a safety warning 
symbol involves three steps.  First, the symbol?s intended message must be determined.  
The message intent may be to prohibit certain actions (e.g., ?Do not touch.?), to prescribe 
or require certain behavior (e.g., ?Wear safety glasses.?), or to communicate information 
about a hazard (e.g., ?Danger. High Voltage.?) (ISO, 2006).  This message is known as 
the symbol?s referent.  Second, a pool of candidate symbols must be generated either 
from existing sources or by creating new symbols.  Finally, the candidates must be 
evaluated to determine the most appropriate symbol for the referent based on empirical 
determinations of communicative effectiveness (Dorris, 2004). 
Several voluntary standards exist, both American and international, which 
propose non-binding guidelines for the development of safety symbols for use on product 
labels, in product manuals, in industrial workplaces and in public areas (ANSI, 2007a, 
2007b, 2007c; ISO, 2002, 2004, 2006, 2007, 2008).  These guidelines set some 
 12 
presentation criteria for color, shape, font size, and component orientation, and they have 
grown more harmonious over the past two decades (Deppa, 2006).  However, differences 
remain between ANSI and ISO standards.  For example, ANSI Z535 encourages warning 
designers to include four hazard aspects: seriousness, hazard type, hazard consequences, 
and avoidance actions.  Because European warnings may be viewed by recipients 
speaking as many as 16 different languages, the ISO 3864 standard adopted a text 
optional convention (Deppa, 2006).  In most cases, only one of the four aspects of the 
hazard can be portrayed by a given symbol, which means that ISO style warnings may 
differ in both appearance and function from ANSI warnings.  ANSI Z535.4 (2007c) also 
specifies the use of either a two- or three-panel format with separate panels that include a 
signal word panel (e.g. ?Danger?), and either a message panel, a symbol panel or both.  
More recent ISO 3864.2 revisions have incorporated the use of optional message  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 1.  ANSI Z535.4 format with three panels, horizontal and vertical versions 
 13 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 2.  ISO 3846-2 format with multiple symbols, horizontal and vertical versions. 
 
 
and signal word panels to communicate more than one hazard aspect, although multiple 
symbols may also be used for this purpose.  Additional harmonization efforts have 
occurred between ANSI Z535.3 and ISO 3846-3 to provide synchronized guidance for 
symbol design criteria such as the use of representational rather than abstract symbols 
and solid graphical representations of the human body (ANSI, 2007a; ISO, 2006).   
Figures 1 and 2 provide an example of ANSI Z535.4 and ISO 3684-1 formats, 
respectively. 
 
 While the ANSI Z535 and ISO 3864 and 9186 families of standards offer 
guidance for the appearance and function of warning signs and labels including the use of 
symbols, there is little guidance provided on how to produce symbols for use in these 
warnings.  Although ANSI Z535.3 includes a flow chart for the design of a symbol, the 
only guidance regarding how to proceed from Step 1 ? Identify Need for Symbol to Step 
 14 
2-Select Candidate Symbols to Test states that it should involve ?Decisions based on 
graphic design principles and analysis of users? (ANSI, 2007a).  Unfortunately, this 
offers little advice to symbol designers. Therefore, the methodologies for the production 
of warning symbols have developed primarily outside of these standardization 
organizations.  Most researchers recognize two, and in some cases three, primary 
techniques for producing the graphical symbols used in safety warnings (Dorris, 2004; 
Green, 1993; Macbeth et al., 2000; Macbeth, Moroney, & Biers, 2006; Pettendorfer & 
Mont'alvao, 2006).  The most traditional, and still widely used, method of developing 
symbols is also the least complex.  In this method, a graphic artist interprets the 
verbalized wishes of the designers to create a set of symbol candidates.  Sometimes these 
symbol sets are tested for comprehension; sometimes they are put directly into practice 
without evaluating their communicative effectiveness (Ringseis & Caird, 1995; Roberts 
et al., 2009).  In order to improve the symbol design quality, features may be built 
gradually and tested at each stage (Dewar, 1999; Dorris, 2004).  Whether tested or not, 
this method is often iterative (Zwaga & Mijksenaar, 2000) with symbols passed between 
designers, artists and test subjects multiple times before a symbol is finalized 
(Wisniewski, Isaacson, & Hall, 2007).  In this dissertation, this method will be referred to 
as the Designer Method.  
Another method of developing symbols actually recruits the participation of 
potential users of the symbols in their design.  This method, pioneered in the automobile 
and defense industries for icon design, including safety symbols (Green, 1993; Howell & 
Fuchs, 1968; Karsh & Mudd, 1962; Mudd & Karsh, 1961), is known as the Production 
Method.  In the production method, a sample of participants develops simple sketches of 
 15 
symbols individually from scratch.  Rather than the symbol designers communicating 
their wishes and ideas to a graphic artist, the artist instead analyzes the drawings created 
by the participants.  It is the responsibility of the artist to consolidate the themes found 
among the symbol drawings to create a final symbol or symbols from those themes.  
Green (1993) presents a thorough review of the early users of the production method, 
including actual line drawings produced in previous studies (Green, 1979; J. R. Sayer & 
Green, 1988).  The production method has evolved over time to include many variants 
(Dorris et al., 2004; Goldsworthy & Kaplan, 2006; Green, 1993; Ringseis & Caird, 
1995), which offer innovate new ways to make use of the unique design contributions of 
potential warning recipients. This method utilizes participatory design, a design strategy 
that suggests the involvement of potential users of a product or system in its design will 
produce a product or system more suited to its intended user (Schuler & Namioka, 1993).  
Sloan and Eshelman (1981) empirically compared symbols produce by the production 
method to those produced by the designer method. They determined that the symbols 
produced under the production method performed better in every case, and that the use of 
participatory design in the development of warning symbols appeared to contribute 
significant benefit.   
While many methodological variants may fall under the production method 
(Green, 1993; Pettendorfer & Mont'alvao, 2006; Ringseis & Caird, 1995), some symbol 
designers have suggested that a distinct new method has emerged from the production 
method referred to as the Focus Group method.  In this method, rather than drawing 
symbols individually and passing them directly to a graphic artist, participants are 
organized into small focus groups where their drawing designs are revealed and discussed 
 16 
(Dorris, 2004; Goldsworthy & Kaplan, 2006; Macbeth & Moroney, 1994; Macbeth et al., 
2000, 2006; Mayhorn & Goldsworthy, 2007).  Based on this discussion, a consensus 
symbol design is produced within the focus group by the participants themselves.  In this 
way, the group synthesizes the themes of the various participants into a consensus 
drawing with real-time input from the original designers of the candidate symbols and 
without interference from designers.  In this paper, this variant of the production method 
is referred to separately as the Focus Group Method.   
The proponents of the focus group method suggest that it removes from the 
graphic artist the responsibility of interpreting the thematic desires of the participants, 
instead placing that responsibility with the participants themselves (Macbeth & Moroney, 
1994).  The graphic artist is called upon only to clean up and professionalize the drawings 
produced from the focus group (Dorris, 2004). Since human factors engineers and 
designers have found participatory design to produce better products, more suited to the 
needs and preferences of their potential users (Dewar, 1999), one might hypothesize that 
the focus group method may produce the most effective symbols since this method 
allows its participants the most input and control over the design process.   
Some empirical research supports this expectation.  Macbeth et al. (2000) report 
that the focus group method proved superior to the production method for developing 
aircraft maintenance symbols using active aircraft maintainers as participants.  They 
noted that the symbols designed in the focus groups were preferred by the evaluation 
participants and that the production process took significantly less real time using the 
focus group method.  However, Dorris (2004) observes than in actual person-hours, the 
focus group method took far greater number of hours than did the production method.  
 17 
Pettendorfer and Mont?alvao (2006) combined aspects of the focus group and production 
methods and reported qualitative improvements between the symbols produced under the 
production method with the consensus symbols designed in focus groups.  However, the 
authors made no direct comparison between comprehension, preference, or production 
time between the two methods.   
The focus group method of symbol production faces several challenges found in 
many focus groups which can impede the ability of the group to perform its task.  Some 
of these challenges, such as culture and language barriers, variations in prior experience 
and topic familiarity and conflicting personality traits, seem particularly relevant to the 
development of warning symbols because the consequences of suppressed or unilateral 
design ideas could lead to poorly designed symbols (Dorris, 2004; Easton, Easton, & 
Belch, 2003; Garmer, Ylven, & Karlsson, 2004; Huer, 2000; Klein, Tellefsen, & 
Herskovitz, 2007; Newby, Soutar, & Watson, 2003; Sweeney, Soutar, Hausknecht, 
Dallin, & Johnson, 1997).    The current study attempts to overcome these challenges by 
introducing a distributed interactive genetic algorithm for symbol development.   
 
Evaluating Symbols 
 The incorporation of high quality symbols into safety warnings has many benefits 
(Friedmann, 1988; Wogalter et al., 2006), while the utilization of poor quality symbols 
can be detrimental to the comprehension of and subsequent compliance with the warning 
(Dorris, 2004; Huer, 2000).  Though a large percentage of the warning symbol research 
has concentrated on the determination of adequate symbol performance and the 
characteristics that produce it, an unusually small percentage of this research involves 
 18 
real-world field studies (Dejoy, Cameron, & Della, 2006).  ANSI Z535.3 (2007a) and 
ISO 9186-1 (2007) each specify testing procedures and performance criteria which must 
be met in order to determine that a symbol performs well.  For example, ANSI proposes 
an 85% passing rate in open-ended comprehension testing from a test sample of at least 
50 participants well representative of the intended users.  ISO proposes a similar testing 
technique, but with a 67% score required to pass and 50 participants from each of three 
culturally diverse countries.  Both standards insist that symbols have less than 5% critical 
confusion from the open-ended testing.  Critical confusion occurs, according to Wogalter 
et al. (2006), when someone misinterprets the message of a symbol as encouraging an 
unsafe behavior that may lead to an injury or when the individual interprets the opposite 
of the intended meaning.  Common means of delivering open-ended comprehension tests 
include the presentation of the symbol in either written or pictorial context with two 
questions are asked of the participant:  ?Exactly what do you think this symbol means?? 
and ?What action would you take in response to this symbol.??.  ANSI (2007a) 
recommends binary judging criteria of correct or incorrect, while ISO proposes a 
weighted scale of correctness (2007). 
The open-ended comprehension test has been recommended as the gold standard 
for evaluating symbol designs (Hicks, Bell, & Wogalter, 2003).  However, due to its 
expense and difficulty, other means of evaluating symbols have been proposed.  To 
reduce the size of the symbol set for final testing, an intermediate step of comprehension 
estimation, or comprehensibility judgment, is described by both ANSI Z535.3 (2007a) 
and ISO 9186-1 (2007).  In this test, participants are provided both the symbol and its 
meaning and are asked to estimate the percentage of the population that would 
 19 
understand the symbol.  Once again, at least 50 well-representative participants are 
needed for the ANSI method, while 50 participants from each of three culturally diverse 
countries are needed for the ISO method.  Young and Wogalter (2001) report on several 
studies of this evaluative test, which they call population estimation, noting that its results 
were found to correlate highly to the results of open-ended comprehension testing.  
However, Wolff (1995) observes that another common evaluation test, the multiple 
choice test, has proven to depend heavily on the quality of the distracters in identifying 
symbols that were judged as poor by other methods.  Lesch (2005) notes that true 
comprehension is often underestimated by open-ended testing, creating a type I error, and 
overestimated by multiple choice, creating a type II error.  She introduces semantic 
relatedness testing as one that is highly correlated to other high performing evaluations, 
but that avoids some of the overestimation and underestimation common in other tests.  
This evaluation mode is similar to a true-false test in that a symbol is paired with a label 
that may or may not be representative of its meaning.  Users must determine whether or 
not it is accurately described by the label (Lesch, 2005).  This research will rely heavily 
upon comprehension estimation to identify the final symbol designs since many design 
candidates will be considered for the same referent.  Comprehension estimations can be 
made for multiple symbol variants from the same referent by the same participant, 
whereas open-ended comprehension testing cannot. 
 In addition to the manner of determining symbol effectiveness, several factors 
affecting warning comprehension and compliance have been identified by empirical 
research.  Along with 39 warning-related factors, Rogers et al. (2000) identified 19 
personal factors affecting warning efficacy.  However, this dissertation will consider only 
 20 
those aspects of symbol design which contribute to effectiveness.  Rogers et al. (2000) 
lump together most symbol-related factors into a single term they call symbology.  In a 
similar summary, Laughery and Wogalter (2006) also define a single pictorial factor to 
represent the effect of symbols on warning effectiveness.  However, other researchers 
have identified several symbol characteristics of interest to this discussion.  McDougall, 
Curry and de Bruijn (1999) identified and evaluated five symbol-related factors, 
normalizing and measuring each factor for a set of 239 symbols.  Concreteness, the 
degree to which a symbol pictorially matches a person, place or object, was found to 
positively influence usability for inexperienced users, but this effect waned over time as 
users gained experience (Isherwood, McDougall, J.P, & Curry, 2007; McDougall, de 
Bruijn, & Curry, 2000).  Visual complexity, the amount of intricacy or detail in the 
symbol, may affect the amount of time needed to identify and interpret a symbol, thereby 
reducing its effectiveness for short term exposures (McDougall et al., 2000).  Familiarity 
refers to both the frequency of exposure to the symbol as well as to the objects or 
situation it depicts (Isherwood et al., 2007).  Semantic distance, or the closeness of a 
symbol?s image to its intended function, has been recently proposed as a major 
contributor to effectiveness (McDougall et al., 1999), although more research is needed 
(Isherwood et al., 2007).  Hicks et al. (2003) propose an additional factor referred to as 
ease of visualization, which measures the ease in which the symbol?s message can be 
visualized.  This is an important concept in that it is the only factor on the list that is 
independent of the actual symbol design.  This is relevant to the current study because it 
affects the development of symbols, not just their evaluation.  The symbol design process 
begins with a message, or referent (Dorris, 2004), and it must be visualized before it can 
 21 
be converted into a symbol.  However, visualizing a referent and producing a symbol 
from it are not the same task, so the ease to which visualization is possible does not 
necessarily predict the ease of producing a symbol for the referent. 
 
Interactive Evolutionary Computation 
 The process of design has long been the domain of discipline experts who use 
experience and creativity to propose new products or systems (Dorris, 2004).  However, 
with advancements in computational power and artificial intelligence, technology can 
now play a significant role in the design process.  Conceptualizing any design problem as 
a search space with an optimal solution to known or unknown objective functions allows 
the usage of meta-heuristic search algorithms to assist human designers with especially 
difficult problems (Roy, Hinduja, & Teti, 2008). Evolutionary computation (EC) refers to 
a collection of meta-heuristics that solves complex optimization problems by utilizing 
principles of biological evolution to evolve problem solutions in large solution spaces 
(Dreo, Petrowsdki, Siarry, & Taillard, 2003; Rees & Koehler, 2006).  Takagi (2001) 
considers these meta-heuristics to be part of the EC family: Genetic algorithms (GA), 
Evolutionary Programming (EP), evolutionary strategies (ES) and genetic programming 
(GP).  However, other researchers may consider additional meta-heuristics, such as Ant 
Colony Search or Particle Swarm Optimization, to be evolutionary computation because 
of their analogy to biological systems. 
Recently, EC has been applied to human factors and safety problems such as 
avoiding pilot error (Chouraqui & Doniat, 2003), estimating chemical exposures 
(Johnston, Phillips, Esmen, & Hall, 2005; Nomen, Sempere, Pey, & Alvarez, 2003; 
 22 
Northage, 2005), detecting sensor faults (Klim?nek & ?ulc, 2004, 2005; Lo, Wong, & 
Rad, 2006), and predicting crowd dynamics (Garrett et al., 2006; Langston, Masling, & 
Asmar, 2006; Muhdi et al., 2006).  These design problems may involve single or multiple 
objective functions which are known or unknown, and Roy et al. (2008) discusses many 
of the current design challenges facing meta-heuristic optimization today.  In each of the 
cases above, the objective to be maximized or minimized could be defined 
mathematically.  However, some design problems depend largely, or even entirely, on the 
perception of humans (Dorris, 2004). 
Interactive evolutionary computation (IEC) allows machine and human to work 
together to optimize a problem or design a solution.  Parmee, Abraham and Machwe 
(2008) suggest that IEC is particularly suited to exploring open-ended concepts in design 
because the high level of human/machine interaction stimulates creativity and innovation. 
Takagi (2001) reports that IEC has been used to design music, hearing aids, clothing and 
animation, among others.  He notes the superiority of IEC, rather than formulae defined 
by statistical regression, to search designs for which human perception or understanding 
is valuable.  Carnahan and Dorris (2004) were the first to apply this technique to the 
design of safety warnings when they developed an IEC design tool to allow both English 
and Spanish-speaking sawmill workers to produce their own graphic symbols for two 
warning referents.  Interactive evolutionary computation, specifically an interactive 
genetic algorithm, was a good design addition to the symbol design process because of its 
iterative nature observed by Wolff (1995).  While the iterative nature of symbol design 
may improve the symbol quality (Zwaga & Mijksenaar, 2000), repetitive searches of the 
same search space are more well-suited to machines than to humans (Sanders & 
 23 
McCormick, 1993). While these users had no previous experience designing hazard 
communication, Dorris (2004) was able to demonstrate that their individually-created 
symbol designs were statistically equivalent in estimated comprehension to symbols 
currently in use in industry. 
Roy et al. (2008) states that many current design problems, such as complex 
mechanical systems, are complex enough that traditional EC algorithms cannot 
effectively solve them.  A technique known as distributed evolutionary computation, 
which makes use of multiple processors in parallel to evaluate solutions (Rupela & 
Dozier, 2002), has provided substantial improvement to some of these iterative and 
complex design problems.  This technique was applied to IEC by Dozier, Carnahan, 
Seals, Kuntz and Fu. (2005a; 2005b), which involved the evolution of design solutions 
using input from multiple participants simultaneously.  Their experiment allowed 14 
participants to design emoticons in parallel, comparing them to emoticons designed by 
individual users.  The process uses an interactive distributed evolutionary algorithm 
(IDEA) to evolve solutions of multiple clients (e.g. participants) by using the judgment of 
one participant to affect newly proposed solutions to other participants.  The IDEA is 
?distributed? because, rather than allowing only a series of individual participants to 
interact with the algorithm and evolve their own solution, many participants may interact 
in parallel, sharing information through the algorithm.  This allows the IDEA to converge 
to single solutions that have incorporated multiple participants? judgments (Dozier et al., 
2005a; Dozier et al., 2005b).   
Essentially, adding a distributed element to the previous IEC design of safety 
symbols so that participants could design symbols in parallel would be analogous to 
 24 
Macbeth and Moroney (1994) adding the focus group element to the production method.  
In each case, a design process existed that only allowed participants to develop symbol 
designs one at a time, with no interaction or shared information between other 
participants.  Just as the focus group method produced more effective symbols in parallel 
than the serialized production method (Macbeth et al., 2000), it is anticipated that 
distributed interactive evolutionary computation, as a parallel search process, will 
produce the highest quality results.  Thus, this dissertation explores the use of distributed 
IEC, specifically a distributed interactive genetic algorithm (DIGA), to replace the 
conceptual design focus group used in the focus group method..   
 
Semantic Annotation and Clustering 
One limitation of the previous research performed by Dorris (2004) is that the 
nature of search space provided to the IEC was defined by the investigators.  While many 
have acknowledged the drawbacks associated with restricting the symbol design process 
to factors predetermined by designers (Dorris, 2004; Dowse & Ehlers, 2001; Huer, 2000; 
Smith-Jackson & Wogalter, 2007), it is understandable in this case since it is not practical 
to produce an IEC which draws on a blank canvas or searches an unbounded search 
space.  The algorithm must have design variables upon which to search and construct 
solutions.  In the case of Dorris (2004), these design variables took the form of an 
encoded vector of numerical angles and lengths which were converted to a graphical 
representation of a symbol when presented to the user.  The determination of which 
variables to make available and their upper and lower bounds provided boundaries to the 
IEC search space, and these decisions were made largely on the basis of previously 
 25 
published symbol designs (Carnahan & Dorris, 2004).  The implication to the participant 
of this encoding structure is that he or she is limited in his or her design to various 
combinations and permutations of the components already chosen by the investigators (in 
this case, to those found in the previously preferred design).   
Participatory design strategies encourage the use of design participants in all 
feasible stages of the design process.  Therefore, including participants in the 
determination of the design variables to be searched by the IEC represents an 
improvement in user participation in the design process. However, graphical symbols, 
even simple ones, represent complex pieces of data (Carneiro, Chan, Moreno, & 
Vasconcelos, 2007) for which the development of design parameters is not a simple task. 
Semantic annotation is a process which assigns qualitative attributes (i.e. descriptive 
terms) to complex pieces of information such as documents, music or photographs which 
often require a human to interpret (Carneiro & Vasconcelos, 2004; Turnbull, Liu, 
Barringon, & Lanckrie, 2007; Vasconcelos & Lippman, 2000a, 2000b).  Semantic 
annotation has primarily been used to label information in a database for later search and 
retrieval (e.g. tagging photographs).  However, the qualitative aspects of symbols (Wolff, 
1995) combined with the need for the identification of design parameters to produce them 
with an IEC make the symbol design process an interesting opportunity for semantic 
annotation.   
Hancock, Rogers, Schroeder and Fisk (2004) have already pioneered the use of 
participants to gather semantic phrases (i.e. qualitative attributes) related to symbols, 
though they used them to evaluate symbol effectiveness rather than to design symbols.  
Piper, Boelhouwer and Davis (2008) used an expert panel to attribute semantic terms to 
 26 
warning symbols in order to determine those symbols most salient characteristics.  They 
then developed a matrix of row vectors each representing one symbol in the design pool 
and containing the presence or absence of each defined attribute. By replicating this 
method in the current study, this research aims to develop semantic annotations of 
symbol drawings in order to determine the most prevalent and interesting design criteria 
offered by those symbol sketches.  From this information, the design variables for the 
proposed distributed interactive genetic algorithm (DIGA) can be determined based on 
participant design input rather than on designers? experiences or preferences. 
 Piper et al. (2008) reported the identification of at least 19, and as many as 27, 
design variables for each of the three symbol referents investigated in that study from 
only 38 symbol drawings available for each referent.  In Dorris (2004), one symbol 
referent had only 16 variables, yet it still produced a search space of size 3.1 x 1031.  
Thus, even with reduced resolution among the variables, it will quickly become necessary 
to reduce the size of the search space considerably, especially since fatigue among IEC 
users can set in quickly (Takagi, 2001).  By transforming symbol sketches to an attribute 
matrix, as previously performed by Piper et al. (2008), the most primary design variables 
can be identified, and the remainder of variables reduced, through clustering. 
It may at first seem counterintuitive or even redundant to use human subjective 
judgment to create data points and then systematically apply a formal clustering 
algorithm.  However, Aggarwal (2004) suggests that for high-dimensional data that are 
inherently sparse in their solution space, a combination of human intuition and 
computerized clustering is the most optimal method of identifying data clusters.  In the 
proposed procedure, the human panelists act as data reduction agents, greatly reducing 
 27 
the complexity of the data from millions of pixels to a simple one dimensional row of 
integers.  Then, the clustering algorithm reduces the search space further by eliminating 
columns in the matrix which do not contribute to the clustering of the data. 
Many clustering algorithms exist for grouping data into thematic families (Anil & 
Richard, 1988; H. M. Chan & Milner, 1981; Choi & Chang Hyo, 1993; Holman, 
Carnahan, & Thomas, 2006), and  Frias-Martinez, Chen, Macredie, & Liu (2007) 
reviewed numerous studies using various clustering methods to group human factors 
data.  K-means clustering is a relatively simple clustering technique that initially 
identifies a user-specified k random cluster centroids in the search space and assigns each 
solution to the nearest centroid.  After assignment, the centroids are recalculated and the 
process repeats until a residual sum of square error function converges to a minimum 
value (Manning, Raghavan, & Schutze, 2008).  Hierarchical clustering establishes a 
hierarchy or tree of clusters rather than a single layer.  While a solution may only belong 
to one cluster in the same layer, higher order clusters usually contain two or more clusters 
of the next lower order, and so forth.  Thus, a solution cannot be defined by its 
membership in a single cluster (Frias-Martinez et al., 2007). Fuzzy clustering, which 
includes the widely used Fuzzy C-means (FCM) technique, defines a fuzzy membership 
of each solution for each cluster in C.  Centroids are recalculated based upon the fuzzy 
membership set, and the cluster or clusters to which a solution most belongs when the 
algorithm converges to a minimum value depending on its user-specified fuzzifier 
parameter, m (Bezdek, 1981).  Finally, Frias-Martinez et al. (2007) introduces a novel 
method, robust clustering, which incorporates the clustering strategies of all three of the 
 28 
previous techniques, but only reports the clustering results when all three methods are in 
consensus. 
 For this research, a simple K-means algorithm was used from the Weka Data 
Mining Software suite (Hall et al., 2009) because it is simple to implement, is capable of 
handling a discrete data set and can report simple centroids of the multivariate symbol 
data which will be assumed to represent the most salient symbol attributes.  As noted, the 
Weka simple K-means algorithm does require a predetermined number of clusters as an 
input into the algorithm. This cluster number, K, can be heuristically determined, 
however, by following a process described by Manning, Raghavan, and Schutze (2008).  
In this method, several clustering runs, each with different initialization points, are 
generated at each for each value in a range of likely K?s.  The actual number of clusters, 
K, is identified by plotting the residual sum of squares as a function of K and determining 
the value of K at which the curve?s successive decreases become noticeably smaller.  
From the primary symbol attributes that can be identified using a semantic annotation and 
clustering process, the design variables and bounding criteria for the distributed 
interactive genetic algorithm (DIGA) can be identified (Roy et al., 2008). 
 
Limitations of the Existing Research 
 Three primary limitations have been identified in the review of the existing 
literature.  These limitations are reported in this section, and they are highlighted again in 
the manuscript chapters whose hypotheses address those limitations. 
 
 29 
Lack of means to determine the ease of converting a referent to a symbol 
 Many factors have been identified to qualify and quantify warning symbols 
(Isherwood et al., 2007), and some of them have been shown to affect warning 
effectiveness (McDougall et al., 2000).  While this research aims to develop and test a 
novel approach to symbol production, there is currently no direct means to identify sets 
of easy or difficult referents from which symbols can be developed.  There are factors 
that attempt to evaluate a symbol?s relationship to its referent (Hicks et al., 2003), but 
none attempt to determine which referents will be considered ?easy? or ?difficult? to turn 
into symbols.  A specific aim of this research is to determine if referents can be 
distinguished based on their ease of conversion from referent to symbol. 
 
Lack of participatory design in symbol production 
 Huer (2000) suggests that user participation in symbol production remains almost 
exclusively in evaluation of symbols rather than symbol development.  Though some 
studies have recognized the need for meaningful participatory design (Dorris, 2004), 
there is still room for greater implementation of this design strategy.  Previous research 
on the development of an interactive evolutionary computation design tool for symbol 
production using representative users made strides towards this goal, but there remains a 
gap between the current literature and complete participatory design in warning symbol 
development.  This aim of this research is to narrow this gap by involving participants in 
defining the design variables used to create the search algorithm within the distributed 
interactive genetic algorithm.  In this way, many of the restrictions placed on participants 
by the designers will be lifted in place of design criteria set by participants themselves.   
 30 
Lack of IEC to model focus group method 
 The production method of symbol development (Green, 1993) is essentially a 
serial process where participants contribute to the design in isolation, never interacting 
with other designers or seeing the final designs.  Dorris et al. (2004) used a similar 
technique with added interactive evolutionary computation to assist the individual in 
developing their design.  The focus group method (Macbeth & Moroney, 1994) enhances 
the production method by allowing parallel interaction between users as they produce 
their symbols.  A similar construct within IEC exists, known as distributed IEC (Rupela 
& Dozier, 2002), which allows for parallel searches and evaluations while working 
towards the same final solution.  However, as yet there has been no attempt to model the 
focus group method using distributed IEC. This research aims to develop and test a 
distributed interactive genetic algorithm modeled after the focus group method to allow 
participants to produce symbol designs in parallel while sharing information and working 
towards a final design solution.
 31 
 
  
 
CHAPTER 3 
DETERMINING A RANKED ORDER OF WARNING REFERENTS BY EASE 
OF CONVERSION FROM WRITTEN REFERENTS TO GRAPHICAL 
SYMBOLS 
 
Introduction 
 According to Dorris (2004), the first step in producing a graphical warning 
symbol is to determine the referent safety message the symbol should portray.  Similarly, 
when testing a new method of symbol development, it is important to carefully select the 
referents on which the design method will be evaluated.  A robust design method should 
be able to produce high quality symbols from warning referents that are both easy and 
difficult to convert into graphical symbols.  However, it is rare in the literature to find 
such a factor of association between referents and symbols.  The relationship of a symbol 
to its referent, such as its concreteness or its semantic distance, has been used in many 
studies to predict or test symbol communicative effectiveness once a symbol has been 
generated (Isherwood, McDougall, J.P, & Curry, 2007; S. J. McDougall, Curry, & de 
Bruijn, 1999; S. J. P. McDougall, de Bruijn, & Curry, 2000; Young & Wogalter, 2001).  
It is conceivable in some instances there may exist a relationship between the referent?s 
difficulty of conversion from text to graphical symbol and the developed symbol?s 
effectiveness in communicating its message.  However, evaluating an existing symbol?s 
effectiveness, while important, is certainly very different than determining how difficult 
 32 
it might be to generate a new symbol from an original referent. In fact, only one study 
was found that sought to characterize symbols before they were generated, while in the 
written referent stage.  Still with the goal of predicting symbol effectiveness rather than 
categorizing referent difficulty, Hicks, Bell and Wogalter (2003) defined the concept of 
?ease of visualization? as a scale of perception by potential users regarding the ease of 
imagining or visualizing the concept portrayed by a referent message.  The study 
compared survey responses for 50 referents? perceived ease of visualization and 
perceived concreteness, among other factors, and determined that ease of visualization 
correlated most highly with open-ended comprehension testing of the symbols produced 
from those referents. The authors recommended the use of both ease of visualization and 
concreteness perceptions as screening tools prior to symbol production to identify those 
symbols which may prove difficult to produce.   
Ease of visualization, as used by Hicks et al. (2003), is not the same concept as 
ease of conversion from referent to symbol, which is defined in the current research.  The 
previous study instructed survey respondents to rate their ease of visualizing or imagining 
the referent message itself (e.g. ?radioactive? or ?slippery surface?), whereas the current 
study focused on soliciting user perceptions of the ease of portraying a referent as a 
graphical symbol.  A few studies have considered the concept that there may be aspects 
of certain referents that make them more difficult to convert to a symbol (Hicks et al., 
2003; Mayhorn & Goldsworthy, 2007; McDougall et al., 2000).  However, those authors 
only determined that certain abstract or complex concepts (e.g. the passage of time or 
conditional states) are considered difficult to portray pictorially.  None of these studies 
 33 
attempted to assign a specific difficulty level to a particular symbol referent or to sort or 
rank a list of referents by their ease of symbol conversion, as the current study aims to do. 
 The purpose of the current research is to sort a list of written warning referents by 
their ease of conversion from referent to symbol.  By selecting referents from this list, 
warning symbol design methods can be evaluated on referents that vary substantially in 
their relative perceived difficulty.  In this way, comparisons of the quality of symbols 
produced by a one method over another will be less likely to be biased by the arbitrary 
selection of an easy or difficult referent.  In other words, when testing a new symbol 
production method, selecting test referents from the list that are dissimilarly ranked can 
help ensure that the method is robust.  
 
Methods 
Objective and Hypotheses 
 The objective of this experiment is to sort a list of written warning referents by 
their relative ease of conversion from written referent to graphical symbol based on the 
perceptions of potential symbol users of varied safety experience.  The hypotheses of the 
experiment are: 
Hypothesis 1: There is no significant difference between the mean ranks of the perceived 
ease of conversion from referent to symbol of any of the nine warning referents. 
H0:  ? referent 1 = ? referent 2 = ?= ? referent 9 
H1:  ? referent 1 ? ? referent 2 or ? referent 1 ? ? referent 2 or ?or ? referent 8 ? ? referent 9 
 34 
Hypothesis 2: There is no significant association between the ranked order of referents 
made by university students, by uncertified safety professionals, and by certified safety 
professionals. 
H0:  ? all-undergraduates = ? all-uncertified  = ? all-certified   = ? undergraduates-uncertified  
 = ? undergraduates-certified  = ? uncertified-certified  = 0 
H1:  ? all-undergraduates ? 0  or  ? all-uncertified  ? 0  or  ? all-certified  ? 0  or  
? undergraduates-uncertified ? 0  or  ? undergraduates-certified  ? 0  or  ? uncertified-certified  ? 0 
Experimental Design 
 In order to test these hypotheses, a randomized, balanced, 50% incomplete block 
experiment (Figure 3) was designed with the level of significance (?) set at 0.05.   The 
independent variables were warning referent (No Access for Persons with Metallic 
Implants, Warning: Flooring Surface Changes, Do Not Touch with Wet Hands, Confined 
Space? Entry by Permit Only, Steel-toed Shoes Required, No Reaching In, Disconnect 
Main Plug from Electrical Outlet, Hot Exhaust, Walk Down Stairs Backwards) and safety  
1 2 3 4 5 6 7 8 9
1 1
2 1
3 1
4 1
5 1
6 1
7 1
8 1
9 1
 
Figure 3.  A reciprocal table for a nine-treatment balanced incomplete block experiment. 
 35 
professional status (uncertified university students, uncertified safety professionals, BCSP 
certified safety professionals) (BCSP, 2009).  The response variable was relative rank of 
perceived ease of conversion from referent to symbol measured by pairwise comparison.  
Figure 3 illustrates the balanced, incomplete block design used in this experiment. 
Subjects 
 A volunteer sample of 174 participants was recruited to participate in the study, 
out of which 145 participants completed the protocol.  Twenty-nine participants were 
omitted from these results because they terminated participation prior to completion of 
the study.  Participants were recruited in three strata.  The uncertified university student 
stratum (55 participants) was recruited by email invitation to Auburn University?s student 
population using the Department of Industrial and Systems Engineering undergraduate 
and graduate student email lists.  The uncertified safety professional (44 participants) and 
certified safety professional strata (which includes 46 participants holding either the 
Associate or Certified Safety Professional designation) were recruited using email 
invitations to the membership of the American Society of Safety Engineers Region IV 
chapters.  Participants were invited to read an online information letter approved by the 
Auburn University Institutional Review Board (IRB) prior to participation in the study.  
Participation was anonymous, with no directly identifiable information collected from 
any of the participants.  Thus, all information used to stratify the participants was self-
reported and not subject to verification by the investigators.  
Experimental Instrument 
 An online survey was designed and revised through three pilot trials involving 46, 
56 and 119 participants, respectively.  A limited validation experiment (the equivalent of 
 36 
3 blocks) was also conducted on the pilot results to estimate whether the survey results 
describing the expected perception of ease of conversion corresponded to the perceptions 
of actual symbol designers producing symbols for those same referents.  The validation 
results, though limited, provided positive evidence that the referent rank order of ease of 
conversion after symbol production was similar to the rank order estimated by the survey 
participants beforehand.  The survey was administered electronically by 
SurveyMonkey.com as a series of 18 pairwise comparisons (see Appendix 3.3 for a 
sample comparison set) in which participants compared the first listed referent message 
to the second by selecting one of these three options: 1) The first referent is more difficult 
to draw, 2) The two messages are equally difficult to draw, 3) The second referent is 
more difficult to draw.   
 The nine warning referents ranked by the survey participants were chosen to meet 
three criteria.  First, three referents were chosen from each of the following types of 
warning: prohibited actions, mandatory actions, and hazard warnings (ISO, 2004). 
Second, referents were selected from a variety of occupational safety topical areas.  
Finally, referents were needed both which already had symbols available from ANSI or 
ISO and which did not have archived symbols available.  The nine referents selected for 
the survey are shown in Table 1.  Each referent was paired randomly with four other 
referents according to the randomized, incomplete block design (i.e. the non-shaded cells 
in Figure 3).  Additionally, referents were randomly assigned as first or second member 
of each comparison pair.  A complete copy of the survey instrument is found in Appendix 
3.1.  
 37 
Table 1 
Referents used in the ease of conversion survey 
Referent Referent Type 29CFR1910  Subpart & Topic? ISO 7010 Referent 
No Access for Persons with 
Metallic Implants Prohibited Action G - Nonionizing Radiation P014 
No Reaching In Prohibited Action O - Machine Guarding P015 
Do Not Touch with Wet 
Hands Prohibited Action H - Hazardous Materials * 
Walk Down Stairs 
Backwards Mandatory Action D - Walking-Working Surfaces N/A 
Steel-toed Shoes Required Mandatory Action I - PPE M008 
Disconnect Main Plug from 
Electrical Outlet Mandatory Action S - Electrical M006 
Hot Exhaust Hazard Warning L - Fire Prevention N/A 
Warning: Flooring Surface 
Changes Hazard Warning D - Walking-Working Surfaces N/A 
Confined Space; Entry by 
Permit Only Hazard Warning J - Confined Spaces N/A 
* ?Do not touch with wet hands? is not listed in ISO 7010, but ISO 3864-2 does provide a symbol for this 
referent as an example. 
? 29 CFR 1910 includes the Occupational Safety and Health Administration (OSHA) regulations for 
general industry.  These regulations are divided into subparts A ? T & Z by safety topic. 
 
Protocol 
 Participants were invited via email through group membership lists to navigate to 
a secure link at SurveyMonkey.com.  Upon entry into the electronic survey, participants 
reviewed the study information letter (Appendix 3.2).  Participants who wished to 
continue provided basic demographic and professional information to verify their 
membership in one of the three experimental strata, and instructions were presented.   
 Participants then reviewed written referents along with brief descriptions of the 
hazards involved, why such warning information is important in an occupational setting, 
and how a symbol portraying this information might be used.  Since it might be difficult 
to articulate an absolute measurement of the difficulty of producing a symbol without 
 38 
actually doing so, participants were simply asked to estimate which of a pair of referents 
would be the more difficult from which to draw a symbol.  Specifically, participants were 
prompted to select one of the referents as the more difficult to draw or to indicate that the 
two referents were equally difficult to convert to symbols.  After evaluating each pair of 
referents, the users moved to a new page in the survey which presented a new referent 
pair for comparison using the same survey process.  Eighteen pairs of comparisons were 
evaluated in this manner by each user.  Each written referent was repeated four times 
within the survey, but two referents were paired together more than once.  Of the 36 
possible pairs of the nine referents, one half (18 pairs) were evaluated directly by each 
participant according to the balanced incomplete block design.   
A modified analytical hierarchy process (AHP) was used to generate a ranked 
order of referent difficulty from the experimental results (Chen & Pu, 2004; Duke & 
Aull-Hyde, 2002; Fielding, Riley, & Oyejola, 1998; Lenton, 2007; Saaty, 1986; 
Teknomo, 2006; Zio, 1996), including a modification of the AHP for incomplete designs 
based on Kirkwood and Sarin?s (1985) method.  According to this procedure, for each 
participant?s pairwise responses, a value of 5 is assigned to the ?more difficult? referent 
while a reciprocal value of 1/5 is assigned to the ?less difficult? referent.  In the case of 
an equally difficult pair of referents, values of 1 are assigned to each referent, and a value 
of 1 is always assigned to the diagonal in the resulting reciprocal table.  Using this 
numerical encoding system, the reciprocal table shown in Figure 3 can be completed for 
each participant, and from that table, the participant?s rank order can be produced by 
simply summing the table rows.  The highest sum receives a rank of 1, while the lowest 
sum receives a rank of 9.  Referents with equal row sums receive an average of their rank 
 39 
positions (e.g. if two referents each have the highest row sum, then they each receive a 
rank of 1.5, which is the average of ranks 1 and 2).  In this manner, each participant 
indirectly produces a ranked order of all nine referents. 
 
Results 
Using AHP, ranked orders of the nine referents? ease-of-conversion were 
produced from the survey results for each of the 145 participants, and this ranked data is 
found in Appendix 3.3.  No assumptions regarding the distribution of this ranked data 
were made, and therefore non-parametric statistical tests were used to test the hypotheses.  
For Hypothesis 1, a Friedman?s test was used to compare the mean ranks of each of the 
nine referents first across all 145 participants, then by the three individual strata.  For 
each of these tests, the response variable was rank, the treatment variable was referent, 
the blocking variable was participant, and there were 8 degrees of freedom.  At ? = 0.05, 
the Q statistic for all participants, adjusted for ties, exceeds ?28 (338.54 > 15.51), which 
implies that a significant difference (p < 0.001) exists between at least two of the mean 
referent ranks.  To determine which ranks differed significantly from one another, the 
post-hoc multiple comparisons test described by Connover (1999) and Bortz, Lienert and 
Boehnke (2000) was conducted (see Table 2 for results).  For 32 of the 36 referent pairs, 
the difference in mean ranks exceeded the critical t-value (0.523), which indicates that 
these referents differed significantly in rank from one another.  Only referents E, F and G 
and referents G and H (highlighted in Table 2) had non-significant differences in rank 
from one another, as indicated by the horizontal lines drawn above the statistically similar 
referents. 
 40 
Table 2.   
Post-hoc analysis results for All 145 participants; critical value = 0.523. 
         -------- 
       -------- --------  
     -------- -------- --------   
    --------      
   --------       
  --------        
 --------         
Referent A B C D E F G H I 
A. No Access for Persons with Metallic Implants X         
B. Do Not Touch with Wet Hands 0.790 X        
C. Warning: Flooring Surface Changes 1.345 0.555 X       
D. Confined Space; Entry by Permit Only 2.121 1.331 0.776 X      
E. Disconnect Main Plug from Electrical Outlet 2.855 2.066 1.510 0.734 X     
F. No Reaching In 3.010 2.221 1.666 0.890 0.155 X    
G. Steel-toed Shoes Required 3.317 2.528 1.972 1.197 0.462 0.307 X   
H. Hot Exhaust 3.828 3.038 2.483 1.707 0.972 0.817 0.510 X  
I. Walk Down Stairs Backwards 4.366 3.576 3.021 2.245 1.510 1.355 1.048 0.538 X 
 
Similar Friedman?s tests were conducted to compare the mean ranks of the nine 
referents for the university, uncertified, and certified participant strata.  For the 55 
participants in the university stratum, the Q statistic, adjusted for ties, exceeds ?28 
(338.54 > 15.51), which implies that a significant difference (p < 0.001) exists between at 
least two of the mean referent ranks in this stratum.  The post-hoc analysis results (Table 
3) for the university stratum revealed significant differences between 23 of 32 
comparisons.  Those mean ranks which are statistically similar are highlighted in Table 3 
and are connected by horizontal lines drawn above them.  Similarly, for the 44 
participants in the uncertified stratum (141.17 > 15.51) and the 46 participants in the 
certified stratum (118.14 > 15.51), the Q statistic exceeds ?28 (p < 0.001) in both cases.  
Post-hoc analysis results for the uncertified and certified strata are presented in Table 4 
and Table 5, respectively, and those comparisons that did not reveal significant 
 41 
differences are highlighted. The final ranked order of referents by ease of conversion 
from written referent to graphical symbol for each stratum is shown in Table 6.   
 
Table 3. 
Post-hoc analysis results for 55 University Students; critical value = 0.893. 
       -------- -------- -------- 
      -------- -------- -------- -------- 
    -------- -------- -------- --------   
  -------- -------- -------- -------- --------    
 --------         
Referent A C B D G F E H I 
A. No Access for Persons with Metallic Implants X         
C. Warning: Flooring Surface Changes 1.389 X        
B. Do Not Touch with Wet Hands 1.518 0.129 X       
D. Confined Space; Entry by Permit Only 1.833 0.444 0.315 X      
G. Steel-toed Shoes Required 2.277 0.888 0.759 0.444 X     
F. No Reaching In 2.509 1.12 0.991 0.676 0.232 X    
E. Disconnect Main Plug from Electrical Outlet 3.287 1.898 1.769 1.454 1.01 0.778 X   
H. Hot Exhaust 4.064 2.675 2.546 2.231 1.787 1.555 0.777 X  
I. Walk Down Stairs Backwards 4.12 2.731 2.602 2.287 1.843 1.611 0.833 0.056 X 
 
Table 4. 
Post-hoc analysis results for 44 Uncertified Professionals; critical value = 0.880. 
        -------- -------- 
       -------- -------- -------- 
      -------- -------- --------  
    -------- -------- -------- --------   
   --------       
 -------- --------        
Referent A B C D E F H G I 
A. No Access for Persons with Metallic Implants X         
B. Do Not Touch with Wet Hands 0.273 X        
C. Warning: Flooring Surface Changes 1.364 1.091 X       
D. Confined Space; Entry by Permit Only 2.545 2.273 1.182 X      
E. Disconnect Main Plug from Electrical Outlet 2.773 2.500 1.409 0.227 X     
F. No Reaching In 3.170 2.898 1.807 0.625 0.398 X    
H. Hot Exhaust 3.852 3.580 2.489 1.307 1.080 0.682 X   
G. Steel-toed Shoes Required 4.273 4.000 2.909 1.727 1.500 1.102 0.420 X  
I. Walk Down Stairs Backwards 4.761 4.489 3.398 2.216 1.989 1.591 0.909 0.489 X 
 42 
Table 5. 
Post-hoc analysis results for 46 Certified Professionals; critical value = 0.917. 
       -------- -------- -------- 
      -------- -------- --------  
    -------- --------     
   -------- --------      
  -------- --------       
 -------- --------        
Referent A B C D E F H G I 
A. No Access for Persons with Metallic Implants X                 
B. Do Not Touch with Wet Hands 0.554 X         
C. Warning: Flooring Surface Changes 1.391 0.837 X        
D. Confined Space; Entry by Permit Only 2.141 1.587 0.750 X       
E. Disconnect Main Plug from Electrical Outlet 2.489 1.935 1.098 0.348 X      
F. No Reaching In 3.511 2.957 2.120 1.370 1.022 X     
H. Hot Exhaust 3.554 3.000 2.163 1.413 1.065 0.043 X    
G. Steel-toed Shoes Required 3.641 3.087 2.250 1.500 1.152 0.130 0.087 X   
I. Walk Down Stairs Backwards 4.435 3.880 3.043 2.293 1.946 0.924 0.880 0.793 X 
  
 
Table 6 
Ease of conversion rank order of nine referents by strata. 
Stratum Final Ranks of Ease-of-Conversion 
University Students A C B D G F E H I 
Uncertified Safety Professionals A B C D E F H G I 
Certified Safety Professionals A B C D E F H G I 
All A B C D E F G H I 
 
 
 To test Hypothesis 2 to determine if these rankings were in agreement between 
strata, Kendall?s Tau-b (Pett, 1997) was used to determine concordance between the final 
ranked order of each pair of strata, and between each stratum and the overall rank.  These 
results are shown in Table 7, and in each case, ? > 0.7, the confidence intervals excluded 
the null value, and p < 0.01.  Thus, it can be inferred that the rank order of ease-of 
 43 
conversion between all strata are concordant and that each stratum is concordant with the 
overall ranked order of the nine referents by ease-of-conversion from referent to symbol.  
Table 7 
Kendall?s Tau-b concordance between strata for the final ranked order of referents. 
University Uncertified Certified Stratum 
?  CI p-value ?  CI p-value ?  CI p-value 
University --- --- ---       
Uncertified 0.722 0.344-1 0.006 --- --- ---    
Certified 0.722 0.344-1 0.006 1 1-1 < 0.001 --- --- --- 
All 0.778 0.441-1 0.002 0.944 0.778-1 < 0.001 0.944 0.778-1 < 0.001 
 
Discussion 
 The objective of this research was to determine whether a ranked list of written 
safety referents can be obtained based on their perceived ease of conversion from written 
message to graphical symbol.  Additionally, since this ranking survey depended entirely 
on the perception of its responders, it was also desirable to determine the effect of 
previous safety experience on the ranking process.   
 
Legend
Prohibited Action
Hazard Warning
Mandatory Action
3. Warning: Flooring Surface Changes
4. Confined Space; Entry by Permit Only
5. Disconnect Main Plug from Electrical Outlet
    No Reaching In
    Steel-toed Shoes Required
    Hot Exhaust
6. Walk Down Stairs Backwards
Referent
1. No Access for Persons with Metallic Implants
2. Do Not Touch with Wet Hands
 
 
Figure 4.  Final ranked order of warning referents by ease of conversion for all strata. 
 44 
The results of the Friedman?s and post-hoc analyses demonstrate that significant 
differences exist between the user-perceived difficulties of developing graphical symbols 
from certain warning referents.  Furthermore, as illustrated in Figure 4, not all of the nine 
sorted referents were statistically distinguishable in ease of conversion from every other 
referent for these 145 participants.  However, five of the nine referents were statistically 
different in ease of conversion from all other referents.  Therefore, the final ranked order 
of referents shown in Figure 4 was generated considering those statistically similar 
referents as ties with essentially the same ease of conversion difficulty.  While not all of 
the tied referents were statistically similar to every other tied referent, at least one was 
statistically similar to all others.  From this list, symbol designers can select several 
combinations of referents that vary statistically in relative perceived difficulty.  It is 
possible that an association exists between the type of referent (e.g. prohibited action) 
and its perceived difficulty to convert to a symbol, but this research did not investigate 
such an association. 
 
 Table 6 shows the nine referents sorted by their mean ranks regardless of 
statistical similarity for all three strata individually and combined.  Both the uncertified 
safety professional and the certified safety professional strata produced identical ranked 
orders, each differing by one discordant pair from the rankings produced by all 
participants.  In both cases, only the referents ?Hot Exhaust? and ?Steel-toed Shoes 
Required? were differently ranked from the results of all participants, and these two 
referents had statistically interchangeable mean ranks even among the entire sample of 
145 participants.  However, the university student stratum produced a ranked order that 
contained four discordant pairs of referents from those of all strata, including two 
 45 
discordant referents that were not statistically interchangeable.  Though the concordance 
analysis suggests that there is a significant positive association between ranked order of 
the university stratum and the other strata, the Kendall?s Tau value for the university-to-
all comparison (? = 0.778) is substantially less than the Tau value for the other two strata 
(? = 0.944) when compared to the overall ranked order.  This may suggest that safety 
experience, but not necessarily safety professional certification, is an important factor in 
developing a perceived ease of conversion factor. 
 
Conclusion 
 This study surveyed three groups of participants?university students, uncertified 
safety professionals, and certified safety professionals?to investigate their ability to 
produce a ranked list of safety referents by estimating the difficulty of converting them 
into graphical symbols.  Results of the study indicate that a simple ranked ordering of the 
written referents can be achieved using pairwise estimations of symbol design difficulty 
even when participants have not attempted to design an actual symbol.  Substantial 
agreement was found between all participants, with essentially identical results found 
between uncertified and certified safety professionals.  By using such a ranked list of 
referents, symbol designers can test symbol design methodology to ensure that it is 
equally valid for warning referents that are relatively easy to convert to symbols and for 
referents that may present substantial challenges.   
This study generated only a relative ease of conversion between the nine referents 
considered.  While an absolute ease of conversion factor that does not depend on any 
other referents would be valuable, such a factor developed from a perception survey 
 46 
would seem speculative.  Limiting participants to a simple trinary comparison (more, 
less, or equally difficult) of referent pairs limits the output to a relative ranked order of 
the referents.  But, this type of comparison minimizes the measurement bias that could 
occur from respondents attempting to estimate referent difficulty on a larger absolute 
scale when they have not actually attempted to draw any symbols.  Thus, while it may be 
valuable to symbol designers as an estimator, this research has not validated perceived 
ease of conversion as a predictor of the actual difficulty in drawing or designing a symbol 
for the referent of interest.  Future research should attempt to validate this estimation by 
combining a pre-design survey with an actual symbol production exercise.  Following 
such an experiment, more absolute measures of ease of conversion from referent to 
symbol may become available.  Further investigation could ascertain whether user 
perception accurately predicts user-experienced difficulty when attempting to produce a 
symbol.  Additionally, various aids (e.g. photographs, hazard descriptions, etc.) could be 
added to the estimation survey to determine if such additions improve the ease of 
conversion estimate. 
 47 
 
 
 
 
 
 CHAPTER 4 
SYNTHESIS AND CLUSTERING OF SYMBOL ATTRIBUTE MATRICES 
FROM HAND-DRAWN SAFETY SYMBOLS 
Introduction 
 Most researchers recognize two, and in some cases three, primary techniques for 
producing the graphical symbols used in safety warnings (Dorris, 2004; Macbeth et al., 
2000, 2006; Pettendorfer & Mont'alvao, 2006).  The most traditional method of 
developing symbols, icons or pictograms is also the simplest.  In this method, designers 
communicate their needs to a graphic artist who develops a set of symbols.  Sometimes 
these symbol sets are tested for comprehension; sometimes they are put directly into 
practice without evaluating their communicative effectiveness (Ringseis & Caird, 1995; 
Roberts et al., 2009).  Whether tested or not, this method is often iterative with symbols 
passed between designers, artists and test subjects multiple times before a symbol is 
finalized (Wisniewski et al., 2007).  This paper will refer to this method as the Designer 
Method.  
Another method of developing symbols involves the participation of potential 
users of the symbols in their design.  This method, pioneered in the automobile and 
defense industries (Green, 1993; Howell & Fuchs, 1968; Karsh & Mudd, 1962; Mudd & 
Karsh, 1961), has been termed the Production Method.  In this method, a sample of 
participants is asked to draw symbols individually from scratch for a set of referents.  
Rather than the symbol designers communicating their wishes and ideas to a graphic 
 48 
artist, the artist instead receives the drawings created by the participants.  It is the 
responsibility of the artist to synthesize the themes found among the symbol drawings to 
create a final symbol or symbols from those themes.  Green (1993) presents a thorough 
review of the early users of the production method, including actual line drawings 
produced in previous studies (Green, 1979; J. R. Sayer & Green, 1988).   
While there have been several variants of this method (Green, 1993; Pettendorfer 
& Mont'alvao, 2006; Ringseis & Caird, 1995), some symbol designers have suggested 
that a distinct new method has emerged from the production method referred to as the 
Focus Group method.  In this variant, rather than drawing symbols individually and 
passing them directly to a graphic artist, participants are organized into small focus 
groups where their drawing designs are revealed and discussed (Goldsworthy & Kaplan, 
2006; Macbeth et al., 2000; Mayhorn & Goldsworthy, 2007).  Based on this discussion, a 
consensus symbol design is produced within the focus group by the participants 
themselves.  In this way, the group synthesizes the themes of the various participants into 
a consensus drawing with real-time input from the original designers of the candidate 
symbols.  Its proponents suggest that this method removes from the graphic artist the 
responsibility of interpreting the thematic desires of the participants, instead placing that 
responsibility with the participants themselves.  The graphic artist is called upon only to 
clean up and professionalize the drawings produced from the focus group (Dorris, 2004).   
Human factors engineers and designers have found participatory design to 
produce better products, more suited to the needs and preferences of their potential users 
(Dewar, 1999).  Applying this principle to the design of symbols suggests that the focus 
group method may produce the most effective symbols because this method allows its 
 49 
participants the most input and control over the design process.  Some empirical research 
has supported this expectation (Macbeth et al., 2000; Pettendorfer & Mont'alvao, 2006). 
However, this method must overcome several challenges found in any focus group which 
can impede the ability of the group to perform its task.  Three of these challenges, culture 
and language barriers, variations in prior experience and conflicting personality traits, 
seem particularly relevant because of their potential to suppress design ideas and to lead 
to symbol designs that are biased towards specific participants? preferences (Easton et al., 
2003; Garmer et al., 2004; Klein et al., 2007; Newby et al., 2003; Sweeney et al., 1997).    
The current study attempts to overcome these challenges by introducing the DIGA 
method of symbol development.   
A New Method Proposed 
The proposed symbol design method involves the use of evolutionary 
computation to interact with a focus group of design participants by both producing 
suggested designs and consolidating the symbol designs of individual participants 
simultaneously, thereby acting as both a focus group participant and de facto group 
moderator.  While the development and details of the DIGA design process are discussed 
elsewhere (Chapter 5 of this dissertation), its main objectives are to provide a 
computerized design interface to receive symbol designs from participants, share design 
concepts between participants, and to even propose new designs using a distributed 
interactive genetic algorithm (Dozier et al., 2005a).  In this way, the reduction of design 
idea sharing caused by culture or language factors and dominant or quiet personalities 
(Sweeney et al., 1997) should be limited since all designs are treated equally with the 
same opportunity to be shared among the participants of the DIGA system with minimal 
 50 
need for verbal or written communication.  Rather than creating designs from scratch, 
DIGA users instead develop symbols using a predetermined set of graphical attributes 
available for incorporation into their designs.  The genetic algorithm receives, modifies 
and proposes new symbol designs to participants using group feedback from previous 
design combinations and permutations of these attributes. Since all users have the same 
attribute selections available to them regardless of referent familiarity, bias towards those 
with more experience with the safety referent should also be reduced (the background 
and design details of the genetic algorithm are discussed in Chapter 5 of this dissertation). 
The purpose of this study is to identify the symbol attributes to be made available 
to DIGA participants.  In a similar study, Dorris (2004) developed an evolutionary 
computation design tool which interacted with participants using a procedure similar to 
the production method.  Participants could manipulate the orientation and size of the 
attributes to form a symbol; however, the symbol attributes available to those participants 
were chosen in advance, limiting the design possibilities to those conceived by the 
designers.  To further minimize this bias, the current study expands the previous study?s 
theme of participatory design by developing the graphical attributes available to the 
DIGA tool using participants themselves.  To accomplish this, aspects of the original 
production method were utilized to produce symbol drawings upon a blank digital canvas 
prior to the development of the DIGA symbol design software itself.  These drawings 
were not used to design specific symbols.  Rather, they define the design parameters from 
which the DIGA design software can produce symbols.  They can therefore be thought of 
as ancestral designs, or proto-drawings, from which all symbols produced by the DIGA 
tool in the future will be able to trace their heritage. 
 51 
Methods 
Objective 
 The objective of this experiment is to produce a set of semantic attributes that are 
capable of pictorially describing the centroid member of each cluster in a clustered set of 
safety symbols.  The list of primary symbol attributes produced by this experiment will 
be used to develop the DIGA system by establishing the boundaries of the search space in 
which the DIGA algorithm is allowed to propose symbol designs.  The three phases 
involved in the determination of these boundary attributes are explained in this section. 
Phase 1 ? Producing Symbol Proto-Drawings for Analysis 
Phase 1 of this experiment recruited 72 participants to produce hand-drawn 
symbols from each of two written warning referents using a blank digital canvas, a 
method which is well established in the literature (Green, 1979, 1993; Karsh & Mudd, 
1962; Mudd & Karsh, 1961; J. R. Sayer & Green, 1988; Wisniewski et al., 2007).  Prior 
to the experiment, participants were allowed to view the information letter (Appendix 
4.1) and ask questions about their role in the study.  Each participant received both 
written and oral instructions (Appendix 4.2) and performed the experimental protocol 
individually.   
Auburn University students were recruited for this study by email invitation to 
limited membership lists such as the Department of Psychology, the Department of 
Industrial and Systems Engineering, and the International Student Organization.  In 
addition, more than 100 paper flyers were posted in public areas around the Auburn 
University campus inviting students to participate.  Previous research (Piper et al., 2008) 
found that 30-40 symbol drawings provided enough information to synthesize a robust 
 52 
list of design attributes.  However, a demographic stratification using country-of-origin 
was employed in the current study to explore cultural variation in the symbol drawing 
process.  Therefore, participants were recruited in two strata, each with 36 members.  
Stratum #1 included participants who were current students in the U.S. but who were 
born and raised in India.  India was selected because of the prevalence of its educated 
citizens who learn two or three languages simultaneously, English, Hindi and often a 
third native tribal language, in an immersive educational setting (Hadi-Tabassum, 2005; 
Raman, 2004).  Stratum #2 included participants who were current students born in the 
U.S and educated in a primarily English language environment.  All participants in both 
strata reported fluency with the English language for at least 5 years prior to the 
experiment. Participants were compensated $20 for their efforts.   
To begin the symbol drawing process, each participant selected at random one of 
two written safety messages, which included a warning referent and a brief description of 
the hazard(s) to which the referent pertained (see Chapter 3 and Appendix 3.1 of this 
dissertation for examples of these descriptions).  Since all participants were university 
students, investigators encouraged each participant to ask questions regarding the nature 
of safety warnings, symbols and of the hazards themselves.  Participants used a 
SmartBoard 600i digital whiteboard to draw their symbols and were instructed to portray 
each warning message as a simple pictogram without using any numbers, text or symbols 
(e.g. $, %, etc.).  Each participant received a tutorial on using the Smartboard 600i prior 
to making their drawing, and neither the investigators nor other participants were 
permitted to witness the drawing process.  Investigators were available to answer 
questions or assist in case of a technical problem, and investigators verified periodically 
 53 
throughout the experiment that no questions or problems had arisen.  When a participant 
announced that the first symbol was complete, the process was repeated using the second 
referent.  After both symbols were designed in this manner, the participant was excused. 
To ensure that the DIGA design tool could be tested on referents for which there 
were significant differences in expected symbol development difficulty, the two referents 
chosen for this experiment were selected from the referent list reported in the previous 
chapter.  Figure 5 shows that the two referents selected for this study have significantly 
different relative ease of conversion on this ranked list of nine referents.  In addition, 
these two referents were selected because they differed in referent type and in the 
availability of published, standardized symbols in the literature.  
In total, 140 symbol drawings were produced in Phase 1 of this experiment, 
including 70 for each referent.  Two drawings from each referent were omitted (see the 
Results section of this paper).  While these symbol drawings will not serve directly as 
candidates for final symbol designs in the remainder of this dissertation, they did assist in 
the evolution of the DIGA design tool and, therefore, serve as ancestral designs, or proto-
designs, from which future symbol designs will descend. 
Referent Type ISO Availability
Prohibited Action Available
Prohibited Action Available
Hazard Warning Not Available
Hazard Warning Not Available
Mandatory Action Available
Prohibited Action Available
Mandatory Action Available
Hazard Warning Not Available
Mandatory Action Not Available
    Hot Exhaust
6. Walk Down Stairs Backwards
Referent
1. No Access for Persons with Metallic Implants
2. Do Not Touch with Wet Hands
3. Warning: Flooring Surface Changes
4. Confined Space; Entry by Permit Only
5. Disconnect Main Plug from Electrical Outlet
    No Reaching In
    Steel-toed Shoes Required
 
Figure 5. The two warning referents selected for use in the current study, ranked by 
perceived ease of conversion from written to graphical forms (1 is the most difficult). 
 54 
Phase 2 ? Semantic Annotation 
Phase 2 of the experiment convened a panel of trained engineers to evaluate the 
symbol drawings produced in Phase 1.  Expert analysis and ratings have been used to 
evaluate symbols, including hand-drawn images, in previous studies (Dorris & Davis, 
2003; Green, 1979; J. R. Sayer & Green, 1988; T. B. Sayer, 2002), but those evaluations 
were generally used to group symbols into tiers or to cull out the top designs.  In the 
current study, expert panelists were used to develop a qualitative matrix of semantic 
attributes capable of adequately describing the significant components of each symbol 
drawing.   Similar semantic annotation processes have been performed in other research 
domains involving the assigning of qualitative descriptors to visual or auditory content, 
such as the labeling or ?tagging? of photographs for image search retrieval and the 
assigning of semantic descriptions to songs (Carneiro et al., 2007; Turnbull et al., 2007).  
In these two cases, the semantic annotation process was used to develop a list of 
keywords that could be used for later retrieval of artistic content.   
To the best knowledge of the author, only one study has utilized semantic 
annotation of a content set as an antecedent to the design of new content (Piper et al., 
2008). That research suggested that three panelists could perform this task effectively.  
Therefore, three panelists comprised the panel for the current study, each holding either 
an Associate or Certified Safety Professional designation (BCSP, 2009), and all panelists 
were trained prior to the experiment in the semantic annotation task, the nature of 
warning symbols, and the requirements of the DIGA software tool that will make use of 
the attributes found by the panel.  Each panelist produced a matrix of qualitative symbol 
attributes (e.g. ?person?s body?, ?head only?, ?fan?, ?directional arrow?, ?puddle?, etc.) 
 55 
for each referent.  The columns of the matrix represent the symbol attributes, and the 
matrix contains enough attributes to sufficiently describe each symbol drawing produced 
in Phase 1 of the experiment.  Each drawing occupies a unique row in the matrix, and 
each cell in the matrix contains a binary response to the question, ?Is this attribute present 
in this symbol drawing??  In addition to annotating symbol attributes and creating the 
attribute matrix, the panel also vetted each symbol drawing to ensure that it was not an 
example of an egregious error or critical confusion.  In this protocol, egregious error 
simply represents a drawing resulting from a substantial misunderstanding or 
misrepresentation of the referent (e.g. drawing ?slippery when wet? when the referent 
specifies ?hot exhaust?).  Critical confusion is defined as portrayal of the opposite 
message or a message that could lead to severe injury (ANSI, 2007a). 
Each panelist followed the procedure provided in Appendix 4.3 to create a matrix 
for each referent, beginning with the first symbol in the first referent and continuing until 
all symbols had been evaluated in both referents.  The symbol drawings were presented 
to panelists in random order without regard to country-of-origin, and panelists were not 
made aware of who created any of the drawings.  A sample of the panelist?s data 
collection form can be found in Appendix 4.4.  The three individual panelist matrices 
produced for each referent were then combined by summation to create a final consensus 
matrix for each referent.  A sample row from this consensus matrix, which represents the 
complete combined semantic annotation of a single symbol drawing, is shown in Figure 
6.  Only the consensus matrices were analyzed further in Phase 3 of this experiment. 
Drawing 1 3 1 3 0 3 0 0 0 0 2 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0  
Figure 6.  Row vector representing the semantic attributes of a single symbol drawing. 
 56 
Phase 3 ? Clustering of the Attribute Matrices 
From the previous phase of the experiment, each symbol drawing was represented 
by a vector, xr , of integer values ranging from 0 to 3.  The values contained in the vector 
represent the number of affirmative votes by the panelists for the presence of an attribute 
in the drawing (e.g. unanimously absent, present by minority, present by majority, 
unanimously present).  The attribute matrices containing these representations were 
clustered using a simple K-means clustering algorithm, and only those attributes 
possessed by the median of each cluster were retained.  Many clustering algorithms exist 
for grouping data into thematic families (Anil & Richard, 1988; H. M. Chan & Milner, 
1981; Choi & Chang Hyo, 1993; Holman et al., 2006).  Frias-Martinez, Chen, Macredie, 
& Liu (2007) reviewed numerous studies using various clustering methods to group 
human factors data.  For this study, a simple K-means algorithm was chosen from the 
Weka Data Mining Software suite because it easily handled the discrete data set and 
could produce simple centroid values of multivariate data.   
In their similar study, Piper et al. (2008) found six clusters among a comparable 
number of drawings using a direct clustering algorithm which does not require a prior 
assumption of the number of clusters (Holman et al., 2006).  Since the Weka simple K-
means algorithm does require a predetermined number of clusters, the cluster number, K, 
was selected considering the number of clusters found in the previous research and a 
heuristic process described by Manning, Raghavan, and Schutze (2008).  In this heuristic, 
several clusterings, each with a different initialization point, are generated at each integer 
value of K in the range K = 2, 3, ?, 8.  The minimum value of the residual sum of 
squares, RSS, defined in Equation 1, among all the clusterings at each value of K is 
 57 
recorded as RSS(K).  ?r , defined in Equation 2, is the centroid of each cluster containing 
? symbol vectors, represented by xr vectors.  
 
( )
2
1
??
= ?
?=
K
k x
k
k
xRSS
?
??
r
rr     (1) 
( ) ?
?
=
kx
x
??
??
r
rr 1      (2) 
 
In the heuristic method proposed by Manning et al. (2008), the actual number of 
clusters, K, is identified by plotting the discrete function RSS(K) and determining the 
value of K at which the curve?s successive decreases become noticeably smaller.  Using 
the ?knees? in the curve to make this decision assumes that the primary objective of 
determining cluster quality is to minimize RSS.  However, as Manning et al. (2008) 
admit, a minimal RSS may sometimes occur with clusters of only 1 symbol.  Regardless 
of the value of RSS, for this study it is useful to define a minimum and maximum cluster 
size.  The centroids of very small clusters (e.g. size 1 or 2) may overemphasize one or 
two outlying symbol drawings, while the centroids of overly large clusters (containing 
more than 50% of the symbols) may mask some of the interesting symbol design 
attributes.  
To address this concern, in addition to the minimization of RSS, a second 
objective for determining the optimal cluster number, K, was defined. For each clustering 
run, the percentage of symbols, s, contained by the smallest cluster, ?small, was compared 
to the percentage of symbols, l, contained in the largest cluster, ?large.  Equation 3 defines 
 58 
the ratio, r, where a value of 1.0 is considered optimal in which all clusters, ?k, are the 
same size.  Like RSS, direct comparisons are only meaningful between clusterings runs 
that have the same number of clusters (e.g. K=4).  For this reason, the best (e.g. lowest) 
value of r for each set of clusterings, i, was denoted as 
i
rmin .  Likewise, the smallest value 
of RSS for each set of clusterings was denoted as 
i
RSSmin .  By comparing each RSSi and 
ri to the best values in that set of clusterings, d is defined in Equation 4 as a normalized 
larger-the-better decision variable used to determine the correct number of clusters, K.   
i
i
i s
lr =       (3) 
max di = geometric mean 
??
?
??
?
ii RSS
RSS
r
r
ii minmin ,    (4) 
 
By conducting i clustering runs on a set of symbols at each value of K, the run 
producing the highest value of d was selected as the best clustering of the data for that K.  
However, since d is a relative factor valid only for comparison within a set of i 
clusterings at the same value of K, each winning clustering run, iK, was placed in set J.  
The overall best clustering was determined by the run in J containing the lowest value of 
r.  At this point, all attributes which were absent in all cluster centroids were ignored, and 
the clusters were reproduced considering only the remaining attributes.  These final 
attributes present in at least one cluster centroid in the final clustering run comprise the 
primary attribute set for that referent. 
 
 59 
Results 
 The purpose of this study was to determine sets of qualitative symbol attributes to 
be used to create design boundaries for the production of graphical symbols for the 
warning referents ?Hot Exhaust? and ?Do Not Touch with Wet Hands.?  A total of 72 
Auburn University student participants joined the study (36 from India and 36 from the 
U.S.).  Each participant created two symbol drawings, one for each referent.  However, 
two drawings from each referent were excluded due to a malfunction of the Smartboard 
system.  For the ?Hot Exhaust? referent, 35 drawings were recorded for both the U.S. and 
Indian strata; however, for the ?Do Not Touch with Wet Hands? referent, both system 
failures occurred during drawings made by Indian participants.  Thus, for this referent, 
there are 36 drawings from the U.S. stratum and 34 drawings for the Indian stratum.  
Appendix 4.5 contains these drawings, and Appendix 4.6 contains the attribute matrices 
produced by the expert panel?s analysis, including both the individual panelist matrices 
and the combined summation matrix for each referent.  Table 8 summarizes the results of 
the panelists? evaluations, including the percent disagreement, which is the percentage of 
 
Table 8. 
Semantic annotation summary of two drawing sets by a three-member panel. 
Referent Stratum 
Total 
Symbols 
Considered 
Discarded for 
Critical 
Confusion 
Discarded for 
Egregious 
Error 
Surviving 
Symbols 
% 
Disagreement 
Indian 35 0 4 31 5.7% 
American 35 0 0 35 6.2% Hot Exhaust 
All 70 0 4 66 6.0% 
Indian 34 4 1 29 3.8% 
American 36 0 4 32 2.9% 
Do Not Touch 
With Wet 
Hands All 70 4 5 61 3.3% 
 
 60 
 
Hot Exhaust
Nearest Drawings to
Cluster Medians
Cluster 1 #8
Cluster 2 #3
Cluster 3 #23
Cluster 4 #6
Cluster 5 #13
8
13
23
6
3
Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 All
Emissions Dissension Unanimous Unanimous Unanimous Unanimous Unanimous
Pipe Unanimous Dissension Unanimous Unanimous Unanimous
Flame Unanimous Unanimous
Attributes present in Cluster MedianAttributes
# of Symbols 16 12 11 11 16 66
 
Figure 7.  K-means cluster results from ?Hot Exhaust? attribute matrix with combined 
strata, and the clustered drawings most nearly representing the centroids (medians) of 
each cluster. 
 
all attribute ratings for which one dissenting panelist voted differently than the other two 
regarding the presence of that attribute in a particular symbol.  Appendix 4.7 shows the 
symbols discarded for critical confusion and egregious error. 
Figure 7 summarizes the results of the clustering analysis performed on the ?Hot 
Exhaust? drawings.  The drawings were grouped into five clusters whose centroids could 
be constructed from only three symbol attributes: Emissions, Pipe and Flame.  Similar 
analysis was performed on the ?Do Not Touch with Wet Hands? drawings, and a 
summary of those results is shown in Figure 8.  These drawings were grouped into four 
 61 
Do Not Touch with 
Wet Hands
Nearest Drawings to
Cluster Medians
Cluster 1 #8
Cluster 2 #5
Cluster 3 #21
Cluster 4 #70
8 5
7021
Cluster 1 Cluster 2 Cluster 3 Cluster 4 All
Single Hand Unanimous Unanimous Unanimous Unanimous Unanimous
Water Drops Unanimous Unanimous Unanimous Unanimous Unanimous
Prohibition Symbol Unanimous Dissension
Faucet Unanimous
Prohibition "X" Unanimous Unanimous Dissension
61
Attributes Attributes present in Cluster Median
# of Symbols 7 23 25 6
 
Figure 8.  K-means cluster results from ?Do Not Touch with Wet Hands? attribute matrix 
with combined strata, and the clustered drawings most nearly representing the centroids 
(medians) of each cluster. 
 
clusters whose centroids could be constructed from only five symbol attributes.  The 
results of the clustering analysis are found in Appendix 4.8.  The attribute matrices were 
also stratified by country-of-origin and clustered using the same technique.  The results of 
the stratified clustering are summarized in Figures 9-10, and the detailed analysis results 
are available in Appendix 4.7. 
 
 62 
Hot Exhaust, US
Nearest
Drawings to
Cluster Medians
Cluster 1 #4
Cluster 2 #24
Cluster 3 #13
13
44 24
Cluster 1 Cluster 2 Cluster 3 All
Emissions Unanimous Unanimous Unanimous
Pipe Unanimous Unanimous Unanimous
Person Unanimous
Flame Unanimous
Attributes Attributes present in Cluster Median
# of Symbols 12 12 11 35
Do Not Touch with 
Wet Hands, U.S.
Nearest Drawings to
Cluster Medians
Cluster 1 #25
Cluster 2 #28
Cluster 3 #29
25 28
29
Cluster 1 Cluster 2 Cluster 3 All
Single Hand Unanimous Unanimous Unanimous Unanimous
Water Drops Unanimous Unanimous Unanimous Unanimous
Prohibition Symbol Unanimous Unanimous Unanimous
Prohibition "X" Unanimous
2-D Panel Unanimous
Attributes Attributes present in Cluster Median
# of Symbols 12 5 15 32
Hot Exhaust, Indian
Nearest Drawings to
Cluster Medians
Cluster 1 #8
Cluster 2 #43
Cluster 3 #4
Cluster 4 #27
Cluster 5 #14
8
Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 All
Emissions Dissension Unanimous Unanimous Unanimous Unanimous Unanimous
Pipe Unanimous Dissension Unanimous Unanimous Unanimous
Flame Unanimous Unanimous
Attributes present in Cluster MedianAttributes
# of Symbols 16 12 11 11 16 66
43 4
27 14
Do Not Touch with 
Wet Hands, Indian
Nearest Drawings to
Cluster Medians
Cluster 1 #21
Cluster 2 #34
Cluster 3 #55
Cluster 4 #27
21
27
34
55
Cluster 1 Cluster 2 Cluster 3 Cluster 4 All
Single Hand Unanimous Unanimous Unanimous Unanimous Unanimous
Water Drops Unanimous Unanimous Unanimous Unanimous Unanimous
Prohibition Symbol Unanimous
Faucet Unanimous
Prohibition "X" Unanimous Unanimous Unanimous Unanimous
Energized Equip. Unanimous
4
Attributes Attributes present in Cluster Median
# of Symbols 10 11 4 29
 
Figure 9. K-means clustering results for the U.S. stratification, ?Do not touch with wet 
hands? referent (left) and ?Hot exhaust? referent (right). Drawings most closely 
representing the centroids (medians) of each cluster are also included. 
 
 
 
 
 
 
 
 
 
Figure 10. K-means clustering results for the Indian stratification, ?Do not touch with 
wet hands? referent (left) and ?Hot exhaust? referent (right). Drawings most closely 
representing the centroids (medians) of each cluster are also included. 
 
 63 
Discussion 
 From the 70 ?Hot exhaust? symbol proto-drawings, 35 qualitative graphical 
attributes were defined by the expert panel.  From the clustering analysis of those 70 
attribute vectors, three primary attributes were identified from which all five centroids of 
the five clusters can be constructed.  Similarly, the 70 ?Do not touch with wet hands? 
proto-drawings yielded 28 graphical attributes which were reduced to five centroidal, or 
primary, attributes by the clustering process.  Table 9 lists these primary attributes for 
both referents. 
 The purpose of this study was to develop lists of primary attributes for 
incorporation into the DIGA symbol design tool.  The referents in these lists are all that 
are needed to produce each cluster centroid k?r , meaning these attribute lists are sufficient 
to produce at least k different symbol families representing the cluster centroids.  
Specifically, the three primary attributes identified for hot exhaust, when incorporated 
into the DIGA software, should allow at least five substantially different families of 
symbol designs to be produced (See Figure 7).  Likewise, the five primary attributes 
identified for ?Do not touch with wet hands? should allow at least four different families 
of symbols to be created (See Figure 8). 
 When the pool of symbol proto-drawings was stratified by country of origin and 
clustered separately, the resulting constituents of the primary attribute sets differed from 
those of the original clustering.  In addition, the number of clusters varied by stratum, 
even for the same referent.  Table 9 lists the primary attribute sets for the stratified data, 
and Figures 9 and 10 show a sample drawing for each cluster, the centroidal attributes 
describing the clusters, and the unanimity of each centroidal attribute. 
 64 
Table 9. 
Primary attribute sets that describe the centroid vectors of each symbol cluster. 
    Stratum 
Total Attributes in 
Stratum Primary Attributes 
Symbol Families 
(Clusters) 
  
 Emission Lines 
 Pipe/Stack 
 Arrow 
 Vent / Grate 
 
Indian 25 
  
5 
   
 Emission Lines 
 Pipe/Stack 
 Person 
 Flame 
 
U.S. 33 
  
3 
   
 Emission Lines 
 Pipe/Stack 
 Flame 
Hot 
Exhaust 
 
All 35 
  
5 
    
 Single Hand 
 Water Drops 
 Prohibition Symbol 
 Faucet 
 Prohibition "X" 
 Energized Equip. 
 
Indian 22 
  
4 
   
 Single Hand 
 Water Drops 
 Prohibition Symbol 
 Prohibition "X" 
 2-D Surface 
 
U.S. 27 
  
3 
   
 Single Hand 
 Water Drops 
 Prohibition Symbol 
 Faucet 
 Prohibition "X" 
Do Not 
Touch 
with Wet 
Hands 
  
All 28 
  
4 
 
 65 
Table 10. 
Attribute subsets by stratum. 
    Recessive Dominant 
    Universal Indian U.S. Indian U.S. 
Emission Lines Arrow Person   Flame Hot 
Exhaust   Pipe/Stack Vent/Gate       
  Single hand Energized Equip. 2-D Surface Faucet   
 Water Drops     
 
Prohibition 
Symbol     
Do Not 
Touch 
with Wet 
Hands 
  Prohibition "X"         
 
For each referent, there was a universal subset of attributes that appeared in both 
strata as well as in the combined data.  This implies that the attributes in the universal 
subset may be less sensitive to cultural or country of origin factors.  For the ?Hot 
Exhaust? referent, the universal subset included two attributes: ?Emission Lines? and 
?Pipe/Stack?.  For the ?Do Not Touch with Wet Hands? referent, the universal subset 
included four attributes:  ?Single Hand?, ?Water Drops?, ?Prohibition Symbol? and 
?Prohibition ?X??.  A second subset of attributes identified for each referent can be 
referred to as the dominant attribute subset.  The attributes in this subset appear in both 
the combined data analysis as well as one of the strata.  However, these attributes do not 
appear in the other stratum.  There was one member of the dominant attribute subset for 
each referent, ?Flame? for ?Hot Exhaust and ?Faucet? for ?Do Not Touch with Wet 
Hands?.  Finally, a recessive subset of attributes was also identified.  As the name 
implies, these attributes only appear in an individual stratum.  They do not appear in the 
opposite stratum or in the combined data set.  For ?Hot Exhaust?, there were three total 
recessive attributes?two found in Indian stratum (?Arrow? and ?Vent/Gate?) and one 
found in the U.S. stratum (Person).  Only two total recessive attributes were identified for 
 66 
?Do Not Touch with Wet Hands?, ?Energized Equipment? in the Indian stratum and ?2-D 
Surface? In the U.S. stratum.  Table 10 summarizes these attribute subsets. 
Certainly, the universal subsets of attributes should demand primary interest when 
designing warning symbols for a diverse population since they were found in the 
centroids of both strata of participants.  The insensitivity of some attributes to country of 
origin suggests that symbols may be able to bridge at least some of the cultural barriers to 
risk communication.  However, the presence of the recessive and dominant subsets of 
attributes seems also to reinforce the notion that symbols are not completely culturally 
neutral.   Nevertheless, the process demonstrated in this study of identifying the universal 
and non-universal attributes should be valuable to symbol designers attempting to work 
with diverse populations 
 
Conclusions 
 Developing a symbol design tool that utilizes evolutionary computation to assist 
design participants has the potential to capitalize on the benefits of participatory design.  
However, in order to receive the best design concepts from the participant designers, 
investigators must do everything possible to minimize investigator bias.  By developing 
primary attribute subsets in this experiment, a new symbol design tool can be constructed 
that will both engage the participant designer in innovative ways and reduce the 
investigator?s input in selecting the design parameters.  Future research should investigate 
the incorporation of these symbol attributes into an actual distributed interactive genetic 
algorithm interface.  In order to do so, certain decisions must be made regarding the 
 67 
manner in which the attributes found in this study should be encoded in the software.  For 
example, how should the attribute ?Single Hand? be portrayed by the DIGA design tool?  
The results of this study imply that there may be a relationship between the 
specific graphical attributes appearing in symbol drawings and country of origin.  This 
study did not investigate the nature of this relationship, should it exist.  Future research 
should explore this relationship across a variety of nationalities as well as other similar 
factors, such as cultural and language experience. Furthermore, this study included only 
novice university students with relatively little design experience as the generators of the 
symbol drawings.  While participatory design principles suggest that the inclusion of 
realistic users in the design process, in this case the general population, is likely to 
improve the design, it is possible that participants unfamiliar with the hazards but skilled 
in industrial or graphic design might produce different symbol drawings for these 
referents.  Future studies should compare the attribute matrices generated from 
participant groups of various experiences in product or system design.  Finally, though 
the sample size used in this study proved adequate in previous research, the multivariate 
nature of the computational analysis would benefit from more data.  Future research 
should consider producing additional symbol drawings for the same two referents used in 
this study so that greater clustering resolution can be achieved.
 68 
 
 
 
CHAPTER 5 
DEVELOPING AND TESTING A DISTRIBUTED INTERACTIVE GENETIC 
ALGORITHM TO DESIGN SAFETY WARNING SYMBOLS 
Introduction 
 Evolutionary computation (EC) is a form of artificial intelligence that has been 
typically used to solve complex optimization problems by utilizing principles of 
biological evolution to evolve problem solutions in a large solution space (Dreo et al., 
2003; Rees & Koehler, 2006).  EC has been applied to human factors and safety 
problems such as avoiding pilot error (Chouraqui & Doniat, 2003), estimating chemical 
exposures (Johnston et al., 2005; Nomen et al., 2003; Northage, 2005), detecting sensor 
faults (Klim?nek & ?ulc, 2004, 2005; Lo et al., 2006), and predicting crowd dynamics 
(Garrett et al., 2006; Langston et al., 2006; Muhdi et al., 2006).  A genetic algorithm 
(GA) is a particular implementation of evolutionary computation that emphasizes natural 
selection and random mutation to search a population of solutions using a survival of the 
fittest approach (Goldberg, 1989).  Genetic algorithms are among the more common 
forms of EC. 
While traditional evolutionary computation attempts to optimize a mathematical 
function, interactive evolutionary computation (IEC) instead attempts to optimize 
performance of a system that requires subjective human evaluation (Takagi, 2001).  In 
human factors, there is often an element to system performance that depends on human 
preference or subjectivity.  IEC allows machine and human to work together to optimize 
 69 
these systems and to design solutions to these kinds of problems.  Furthermore, the 
involvement of potential users of a product or system in its design, known as 
participatory design, is believed to improve the quality of the final  products (Schuler & 
Namioka, 1993).  Parmee, Abraham and Machwe (2008) suggest that IEC is particularly 
suited to exploring open-ended concepts in participatory design because the high level of 
human/machine interaction stimulates creativity and innovation.   
The design of safety warning symbols has long made use of participatory design 
to develop and evaluate symbol candidates because it is believed to produce the highest 
likelihood of meeting symbol comprehension criteria (ANSI, 2007a; Green, 1993; ISO, 
2007).  The design process generally includes the identification of a safety message to 
portray as a symbol, the production of simple sketches of possible designs, the analysis of 
these designs for thematic elements, and the evolution of final designs representing the 
identified themes (ANSI, 2007a; Dorris, 2004; Green, 1993).  Depending on the design 
method, human participants representative of potential symbol users can be involved in 
one or more of these design phases (Green, 1979; Macbeth et al., 2000).  The process of 
evaluating potential symbol designs to determine one or more best designs is essentially a 
search task that incorporates user subjective assessment of symbol quality as its fitness 
function. 
Carnahan and Dorris (2004) were the first to apply evolutionary computational 
search to the design of safety warnings when they developed an IEC design tool to allow 
both English and Spanish-speaking sawmill workers to produce their own graphic 
symbols for two warning messages, or referents.  While these users had no previous 
experience designing hazard communication, Dorris (2004) was able to demonstrate that 
 70 
their individually-created symbol designs were statistically equivalent in estimated 
comprehension to symbols currently in use in industry.  The Carnahan and Dorris (2004) 
IEC search algorithm was similar in design intent to the production method of symbol 
design which recruits many participants to produce independent symbol designs that are 
evaluated by designers to produce a final design.  By replacing the human search task of 
identifying the best symbol from an infinite set of undrawn possibilities with an IEC 
search that evolves a symbol design fit to each user, Carnahan and Dorris (2004) began 
the process of transforming the symbol design system. 
A new approach to IEC design was developed by Dozier et al. (2005b), which 
involves the evolution of design solutions using input from multiple participants 
simultaneously.  The process uses an interactive distributed evolutionary algorithm 
(IDEA) to simultaneously evolve solutions of multiple participants by incorporating the 
judgment of one participant into the genetic material available to other participants.  The 
design space shared by the participants where symbol designs are mated and mutated was 
labeled Meme Space (Dozier et al., 2005b).    Prior to this work, distributed evolutionary 
computation had primarily focused on decreasing computation time for complex 
problems by running the EC search simultaneously on many processors (Rupela & 
Dozier, 2002). The IDEA algorithm of Dozier et al. (Dozier et al., 2005b) is ?distributed? 
because, rather than allowing only a series of individual participants to interact with the 
algorithm and evolve their own solution, many participants may interact in parallel.  This 
allows the IDEA to converge to single solutions that have incorporated multiple 
participants? judgments (Dozier et al., 2005a).   
 71 
Participatory design of warning symbols has also progressed in its design 
strategies.  Macbeth et al (2000) proposed the focus group method of symbol 
development which allows a group of 6-12 participants to develop symbol designs in 
parallel, sharing and critiquing ideas verbally and on paper until a final group design is 
chosen.  Just as the production method was analogous to IEC, the focus group method of 
parallel, shared symbol design is similar in strategy to the distributed IEC pioneered by 
Dozier et al. (Dozier et al., 2005a).  Thus, this study explores the use of distributed 
interactive evolutionary computation, specifically a distributed interactive genetic 
algorithm (DIGA), to computationally model the focus group method of symbol design 
developed by Macbeth et al. (2000).   
 
The Algorithm 
The search for high quality symbol designs is almost certainly a non-polynomial 
hard, multivariate problem involving an unknown mathematical formulation of a single 
participant?s judgment.  Furthermore, the problem becomes multi-objective when it must 
attempt to optimize the various subjective judgments of a group of 6-12 participants, in 
the case of the focus group method (Macbeth & Moroney, 1994; Macbeth et al., 2000). 
Fortunately, Dozier et al (2005a) developed a distributed interactive evolutionary 
algorithm (IDEA) to computationally model a very similar process.  The IDEA was 
designed to evolve emoticons (e.g. ?smilies?), and the IDEA pseudo code is shown in 
Figure 11.  To design emoticons, IDEA participants received 9 randomized initial 
emoticons from the system.  The user responded by selecting their favorite emoticon, e, 
and a preferred mutation operator, o.  The user submitted emoticon e to the Meme space 
 72 
Procedure IDEA_Client { 
 t = 0 
 Initialize Pop(t)  // Randomly generate initial emoticons 
 Present Pop(t) to User; 
 While (Not Done) 
 { 
  Allow user to select an emoticon(e); 
  Allow user to select a mutation_op(o); 
   
  Send(e) to MEME space; 
  Receive(m) from MEME space; 
 
  Parents(t) =  {e,m} 
  Offspring(t) = { 
    Create 4 Mutants(e,o); 
    Create 3 Recombinations(e,m,o); 
    } 
 
  Pop(t+1) = Parents(t) ?Offspring(t): 
  t = t+1; 
 } 
} 
server and received a random emoticon from Meme space, m.  Emoticons e and m 
became the parents for 7 daughter emoticons, 4 generated by mutation using the specified 
operator o, and 3 by recombination, which in this case was blend crossover (Eshelman & 
Schaffer, 1993).  This process repeated until the user determined the process was 
complete. 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 11.  IDEA pseudo code (Dozier et al., 2005a). 
Selection, Crossover and Mutation in the DIGA Algorithm 
In order to computationally model the focus group method of Macbeth et al. 
(2000), a simple genetic algorithm employing two-point crossover and single-point 
mutation (Goldberg, 1989) at the server level was combined with a client graphical 
interface similar to that used by Dozier et al. (2005a).  However, this distributed 
 73 
interactive genetic algorithm (DIGA), unlike the IDEA emoticon algorithm shown in 
Figure 11, handles the majority of the computational steps on the server side.  The client 
interface is primarily used to solicit participant design evaluations.  Flow charts of the 
server and client portions of the algorithm are shown in Figures 12 and 13, respectively.   
Compile master
set of symbols
from all n clientsClient 3
Client 2
Client 1
Client n
IN
TE
RN
ET
Selection
Most fit symbols
have highest
chance of
reproducing
Meme Space
m best global 
solutions found so far
by all users
Crossover 
Selected symbols
reproduce with symbols
from Meme Space
Mutation
Mutation operator
Is applied to 
daughter symbols
Global Replacement
Most elite parent
Symbols sent to
Meme Space
Client Replacement
Daughter symbols
returned to clients
with random from m
IN
TE
RN
ET
 
Figure 12.  Server-side DIGA Flow Chart. 
 
 
IN
TE
RN
ET
Meme Space
Global best solutions
+ reproduction
& replacement 
Local Search
User adjusts
parameters for 
symbol population
Initialize
Generate initial symbol with 
random parameter values
for each simultaneous user
Local Search
Desired?
Yes
Fitness
rate current population 
(aka fitness assessment)
No Stoppingcriteria
reached?
Local Elitism
Update client?s best
symbol so far
Send Solutions
to Meme Space
No
Yes
End
Record 
Symbol Data 
Receive new
generation
from Meme
Space IN
TE
RN
ET
 
 
Figure 13.  Client-side DIGA Flow Chart
 
74 
Procedure DIGA_Server { 
 Initialize Server(g,c,)  // Max # of generations & # of clients 
 t = 0; 
For all clients(c) { 
 Initialize Pop(c,t); // Randomly generate initial 9 symbols 
} 
 
While (t < g) { 
  Send Pop(c,t);  // Deliver symbols to clients 
For all clients ? c { 
   For all i ? [1,9] { 
User selects ith most favorite symbol, s(c,i) 
   } 
  } 
  Send[s(c,i)] to Meme space;  
   
  Elitism(t) = s(c,1) + s(c,2) for all clients ? c; 
// Preserve 1st & 2nd favorites of each client 
   
Begin_Tournament(t) { 
   P[s(c,i)] = (10-i)/45; // Assign selection probabilities 
   While j < 7*c { // Make 7 offspring per client 
    Select Candidate1(P); 
    Select Candidate2(P); 
    Parent(j) = candidate with lower rank, i; 
    j = j+1 
  } 
   
While k < 7*c { 
   Offspring(t) = Crossover[Parent(k), Parent(k+1)]; 
 Mutate_Offspring(t); 
 k = k+2; 
} 
   
  Pop(t+1) = Elitism(t) ? Offspring(t): 
  t = t+1; 
 } 
} 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 14. Pseudo code for DIGA symbol design algorithm 
 
 75 
DIGA pseudo code is shown in Figure 14.  To design warning symbols, each 
DIGA participant receives 9 randomized initial symbols from the system.  The participant 
responds by selecting their favorite symbol based on how well it portrays the written 
message (i.e. the referent) provided on the screen.  The participant c repeats the ranking 
of the next best symbol until all symbols (i=1-9) have been ranked, s(c,i).  The symbols 
are then submitted to the server, which is analogous to Meme space of Dozier et al. 
(2005a). Symbols s(c,1) and s(c,2) from each participant are preserved; the 1st ranked 
symbol returns to its original participant and the 2nd ranked symbol is submitted to any 
participant at random.  The remaining symbols in the population are replaced by the two-
point crossover shown in Figure 15.  Each parent in the crossover is the winner of a 
selection tournament in which a pair of symbols is chosen randomly and compared.  The 
symbol ranked higher by its participant wins the job, and ties are broken randomly. 
 
Figure 15.  Illustration of two-point crossover. 
Crossover Step 1: Parent solutions S1 and S2 are chosen for crossover. 
S1  p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12  
S2   q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 q11 q12  
 
Crossover Step 2: Select 2 points at random [0,12] ? 3 and 9  
 
S1 p1 p2 p3  p4 p5 p6 p7 p8 p9     p10 p11 p12  
S2 q1 q2 q3  q4 q5 q6 q7 q8 q9     q10 q11 q12  
 
Crossover Step 3: Exchange values (3,9] in S1 with values (3,9] in S2 to form 
new offspring solutions 
 
O1 p1 p2 p3        q4 q5 q6 q7 q8 q9        p10 p11 p12  
O2 q1 q2 q3        p4 p5 p6 p7 p8 p9       q10 q11 q12  
 
 76 
At the conclusion of the crossover, parents have been selected and recombined to 
replace the seven lowest ranked symbols for each client, (e.g. ranks 3-9).  These symbols 
are now subject to mutation.  A mutation probability specific to the experiment is applied 
so that only a fraction of newly formed offspring experience mutation.  If an offspring 
symbol is selected for mutation, a single variable allele in the genome (Figure 16) is 
changed to a new value in its range with uniform probability for any single value.  After 
all applicable offspring are mutated, the offspring are combined with the elitist symbols 
preserved from the previous generation and resubmitted to the clients as generation t+1.  
Each client receives nine symbols to evaluate as the next generation, including his 
pervious top ranked symbol, a randomly chosen 2nd ranked symbol from any client, and 7 
randomly chosen offspring who have just undergone crossover and mutation.  To 
maintain continuity in the number of symbol designs searched from one participant 
session to the next, the algorithm repeats until a maximum number of generations 
specified at the start of each experiment is reached.   
 
Figure 16.  Illustration of single-point mutation. 
Mutation Step 1: Some solutions are chosen for mutation based on specified 
mutation probability. 
 
O1  p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12  
 
Mutation Step 2: Select 1 point at random [1,12]  to mutate ? 7  
 
O1 p1 p2 p3 p4 p5 p6        p7         p8 p9 p10 p11 p12  
 
Mutation Step 3: Replace selected variable?s value, p7, with randomly chosen 
value from that variable?s range, p7*. 
 
O1* p1 p2 p3 p4 p5 p6   p7*   p8 p9 p10 p11 p12   
 
 77 
Encoding the DIGA Genotype for Warning Symbols 
Two implementations of the DIGA algorithm were developed, each 
implementation capable of designing a different safety symbol.  Based on the procedure 
performed in Chapter 4 of this dissertation, the warning referents ?Hot Exhaust? and ?Do 
Not Touch with Wet Hands? were selected.  Evolutionary computation involves both 
solution encoding into a genotype and decoding into a phenotype.  In this case, the 
solution phenotype for each implementation is the graphical symbol presented to the 
participant on the client side of the system.  However, in order for the server side of the 
algorithm to perform its computations, the solution must be encoded into its genotype.   
To ensure that the DIGA produces symbols representative of the design 
participants wishes from the previous experiment, the primary design variables 
determined by the clustering process (Chapter 4, Table 9), such as ?flame? or ?pipe? for 
Hot Exhaust, are included in the phenotype.  However, the design parameters themselves 
must be converted to a range of realistic parameters.  Returning to the participants? 
symbol drawings and expert panel analyses presented in Appendices 4.5 and 4.6, the 
range of each primary design parameter was determined.  From a review of this 
information, the solutions for each DIGA implementation were encoded as vectors of 
integer values shown in Figure 17. The design parameters, their description, their ranges 
and their resolutions between adjacent values are presented in Tables 11 and 12.   
 
 
 
 
 
Figure 17.  Genotype encoding of both symbol phenotypes.  
 
       Hot Exhaust:     
Do Not Touch with Wet Hands:  
p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 q11 q12 q13
 78 
Table 11. 
Design Parameters of the Hot Exhaust genotype. 
Parameters Description Range Resolution 
p1 Size of the Flame (width in pixels) [20,70] 10 
p2 Horizontal position of Flame (pixels) [60,440] 20 
p3 Vertical position of Flame (pixels) [60,400] 20 
p4 Diameter of the Pipe (pixels) [20,100] 10 
p5 Length of the Pipe (pixels) [25,65] 5 
p6 Angular Orientation of Pipe (degrees) [0,360] 30 
p7 Breadth of Pipe Spray (pixels) [5,65] 5 
p8 Length of Pipe Spray (pixels) [25,65] 5 
p9 Pipe Visibility Toggle (on/off) [0,1] 1 
p10 Spray Visibility Toggle (on/off) [0,1] 1 
p11 Flame visibility Toggle (on/off) [0,1] 1 
p12 Type of Spray Lines (dashed, dotted, solid, wavy) [1,4] 1 
 
Table 12. 
Design Parameters of the Do Not Touch with Wet Hands genotype. 
Parameter Description Range Resolution 
q1 Size of the Hand (width in pixels) [50,120] 10 
q2 Angular Orientation of Hand (degrees) [0,360] 30 
q3 Size of the Water Droplets (width in pixels) [20,80] 10 
q4 Size of the Faucet (width in pixels) [20,60] 10 
q5 Horizontal Position of Faucet (pixels) [50,240] 10 
q6 Vertical Position of Faucet (pixels) [50,150] 10 
q7 Diameter of the Prohibition Symbol (pixels) [30,100] 10 
q8 Hand visibility Toggle (on/off) [0,1] 1 
q9 Droplet Visibility Toggle (on/off) [0,1] 1 
q10 Faucet Visibility Toggle (on/off) [0,1] 1 
q11 Prohibition Symbol Visibility Toggle (on/off) [0,1] 1 
q12 Type of Hand (flat palm, reaching, pointing) [1,3] 1 
q13 Type of Prohibition Symbol (circle/slash, circle/x, lone x) [1,3] 1 
 
 The genotype described above for ?Hot Exhaust? produces a search space that 
includes 12 parameters and more than 8.2 x 109 possible solutions.  Similarly, the ?Do 
 79 
Not Touch with Wet Hands? genotype includes 13 parameters and more than 4.6 x 106 
possible solutions.  By developing first the phenotype, then the genotype, based on 
participatory design methods rather than on simply the insights of the researchers, the 
search space for each of these referents has been constrained by more representative 
boundaries. 
 
Methods 
Objectives and Hypotheses 
 The objectives of these three experiments are to determine whether a DIGA can 
be used by novice participants to develop and converge warning symbol designs and to 
compare symbols designed by DIGA with those designed by more traditional methods.  
The hypotheses of the experiment are: 
Hypothesis 1: There is no significant difference between subjects in the coefficient of 
variation of symbol parameter values of favorite symbols between the first and final 
generations. 
H0:  
20
20
1
1
Gen
Gen
Gen
Gen
?
?
?
? =  for all design parameters and all referents
 
H1:  
20
20
1
1
Gen
Gen
Gen
Gen
?
?
?
? >  for all design parameters and all referents 
Hypothesis 2: There is no significant difference in the preference ranking between DIGA 
designed symbols and Focus Group designed symbols. 
H0:  FGDIGA ?? =  
H1:  FGDIGA ?? ?  
 80 
Hypothesis 3: There is no significant difference in the preference ranking between DIGA 
designed symbols and published symbols. 
H0:  publishedDIGA ?? =  
H1:  publishedDIGA ?? ?  
Hypothesis 4 There is no significant difference in the preference ranking between DIGA 
symbols when stratified by country of origin. 
H0:  ChinaIndiaSUMulti ???? === ..  
H1:  ChinaIndiaSUMulti ???? ??? ..  
Experiment #1 ? DIGA 
Subjects.  The DIGA algorithm is modeled after the focus group method of 
participatory symbol design proposed by Macbeth and Moroney and Macbeth et al. 
(1994; 2000).  They found that using 6 ? 12 participants provided enough design 
diversity without the process becoming cumbersome.  For comparison, Experiment #1 
recruited four groups of 12 Auburn University students each (N=48) to participate in a 
DIGA design session, each group representing a different treatment of the independent 
variable country of origin.  Participants were recruited by email to departmental mailing 
lists within the university as well as a by flyers posted on public boards around the 
university campus.  Each participant was paid $40 for two hours of effort. 
Group 1 consisted of a heterogeneous mix of participants that included two 
participants from China, Turkey, Sri Lanka and the U.S., and one participant each from 
India, Mauritius, Korea, and Chile.  Groups 2 ? 4 were homogeneous groups of 
participants who hailed from the same nation.  Group 2 consisted of 12 participants from 
the U.S.  Group 3 consisted of 10 participants from India (two participants withdrew 
 81 
before the study began) who were attending graduate school in the U.S., and Group 4 
consisted of 12 participants from China also attending graduate school in the U.S.  All 
participants reported at least moderate fluency with the English language, and all 
participants were at least 19 years of age. 
Experimental Apparatus.  The experiment was conducted using a series of 
networked computers running the Linux operating system.  The DIGA was coded in the 
Java programming language, and the graphical user interface operated by the participants 
is shown in Figure 18.  One computer served as the server and was operated by the 
investigators.  Each participant performed the experiment individually on his or her own 
client computer, and the participants? only interactions with the system were to select the  
 
Figure 18.  Client interface for Hot Exhaust operated by DIGA participants. 
 82 
?Rank? button to rank a symbol as the best symbol remaining, to ?Reset Rank? if a 
mistake was made, and to submit the ?Next Generation? when all symbols were ranked.  
The ?Modify? button was used by the investigators only when an error was made.
 Protocol.  Prior to participating in the experiment, each participant reviewed the 
information letter approved by the Auburn University IRB (Appendix 5.1) and read and 
completed the research instruction form (Appendix 5.2).  The investigators also read the 
instructions aloud in English and fielded questions and requests for explanations of the 
warning referents.  Participants were assigned random seats in the research lab, and each 
was instructed on the operation of the DIGA program.  Without consulting other 
participants, each individual first ranked the best symbol displayed on the monitor for the 
initial generation, and then they ranked each successive ?best remaining? symbol.  When 
all nine symbols in the initial generation were ranked in this manner, the individual 
clicked the ?Next Generation? button and waited on delivery of the next generation of 
symbols for evaluation.  Once all members of the experimental group had submitted their 
generation of symbols in this fashion, the DIGA processed them and submitted a new 
generation to each participant for evaluation.  Takagi (2001) recommended a maximum 
of 10-20 generations for interactive evolutionary computation to avoid fatigue-related 
bias in participant responses.  Therefore, after the submission of the 20th generation, the 
experiment terminated, and the process was repeated using the second referent.  
Participants then viewed anonymously the favorite symbol of each participant in the 
group, and evaluated them by three methods: comprehension estimation, a Likert-type 
scale of perceived effectiveness, and a ranking of most to least effective.  Since open-
ended comprehension was not feasible because the participants already knew the 
 83 
intended meaning, this composite evaluation analysis should provide adequate 
information to determine the most representative symbol from each DIGA group. 
Experiment #2 ? Development of Comparison Symbols by the Focus Group Method 
Macbeth et al. (2000) and Pettendorfer and Mont?alvao (2006) reported that the 
focus group method produced more effective symbols than previous non-IEC methods, 
while Dorris (2004) demonstrated that an IEC symbol design tool could produce symbols 
that performed at least as well as those produced by other methods.  To provide a means 
of comparison, Experiment #2 developed a companion set of symbols using the focus 
group method with similar demographic stratification to Experiment #1. 
Subjects.  Four groups of 12 Auburn University students each (N=48) were 
recruited using the same emails and flyers for Experiment #1, each group again 
representing a different treatment of the independent variable country of origin.  
However, 11 participants withdrew before completing the experiment.  Group 1 consisted 
of a heterogeneous mix of 10 participants (two participants withdrew) that included three 
participants from the U.S., two participants each from Turkey and Korea, and one 
participant each from India, Zimbabwe, and Japan.  Groups 2 ? 4 were homogeneous by 
country-of-origin.  Group 2 consisted of 12 participants from the U.S.  Group 3 consisted 
of six participants from India (six withdrawals) who were attending graduate school in 
the U.S., and Group 4 consisted of nine participants from China (three withdrawals) also 
attending graduate school in the U.S.  All participants reported at least moderate fluency 
with the English language, and all participants were 19 years of age or older.  Each 
participant was paid $40 for two hours of effort. 
 84 
Experimental apparatus.  For the first portion of Experiment #2, participants were 
assigned to random seats in a conference room, and each was provided a pencil, a pen 
and a blank page for drawing a symbol (See Appendix 5.3).  After completion of the 
drawing, the hand drawn symbols were scanned and converted into electronic images and 
a Smartboard 600i digital whiteboard was used to evaluate the drawings.   
Protocol.  Prior to participating in the focus group phase of the experiment, each 
participant reviewed the approved information letter (Appendix 5.1), read and completed 
the research instruction form (Appendix 5.2).  The instructions were also read aloud in 
English by the investigator, and questions and requests for explanations of the warning 
referents from the participants were fielded.  Participants were assigned random seats in 
the research lab, and each was provided several copies of the blank symbol drawing form 
for the first referent (chosen at random).  Without consulting other participants, each 
individual sketched a simple drawing of a symbol that portrayed the referent without 
using words.  After all participants had completed their drawings, the symbols were 
scanned and converted to an electronic image.  The participants then introduced 
themselves to one another and selected a moderator from among the group.  The 
moderator presented the symbol drawings anonymously and solicited feedback from the 
group, while also providing feedback on the symbol designs herself.  The best ideas were 
recorded, and one participant was nominated to sketch a final consensus symbol drawing 
based on the group?s collective preferences.  The investigators were present during this 
process to answer questions and assist with any problems that arose, but they avoided 
direct participation in the focus group.  Once the group consensus symbol was designed 
and saved, the process was repeated with the second referent. 
 85 
Experiment #3 ? Comparing DIGA and Focus Group Symbols 
Subjects.  To compare the symbol designs, 501 participants were recruited using 
the Amazon Mechanical Turk anonymous electronic task recruitment system which 
recruits respondents globally via requests for assistance.  One hundred participants 
withdrew before completing the experiment, resulting in a completion rate of 80%.  The 
countries of origin represented by the 401 participants who completed the survey 
included 249 from the U.S., 105 from India, 11 from Canada, 5 from the U.K., 4 from 
China, 3 from Nigeria, 3 from The Philippines, 2 from Mexico, and 18 participants from 
18 other countries.  All participants reported moderate fluency with the English language, 
and all participants were 19 years of age or older.  Each participant was paid $0.10 for 
their efforts. 
Experimental Instrument.  For Experiment #3, an electronic survey was designed 
that included symbols placed in a photographic context appropriate for their hazard.  
Since the DIGA and focus group symbols would also be compared along side of 
previously published symbols, a graphic artist was employed to standardize the designs 
(Dorris, 2004) based upon published design criteria (ANSI, 2007a; ISO, 2003).  A sample 
symbol design with context was presented first, followed by comprehension questions 
inquiring of the precise meaning of the symbol and the most appropriate response action 
that should be taken.  Photographic context was also provided for the eighteen actual 
symbols, and similar questions were asked.  Participants were randomly assigned to only 
two of the eighteen contextualized symbols for evaluation, one from each referent, so that 
previous experience with that referent would not bias their responses.  The final portion 
of the survey included a single presentation of nine symbols for each referent: four 
 86 
produced by the DIGA, four produced by the focus group method, and one already in use.  
One final question asked participants to select the symbol for each referent which most 
effectively communicates its intended message.  Examples of the comprehension 
questions with context and of the comparison ranking portions of the survey are located 
in Appendix 5.4. 
Protocol.  Each participant recruited by Amazon Mechanical Turk was directed to 
SurveyMonkey.com to complete only a portion of the evaluation survey.  Respondents 
were asked to provide their country-of-origin and the date, but not the month or year, in 
which they were born.  This date was used as a surrogate to approximate a uniformly 
distributed random variable to assign respondents to a portion of the survey since the 
survey software in SurveyMonkey.com does not provide other means of partial 
assignment of survey portions.  Each participant was directed first to a sample symbol 
design with photographic context and asked to give a precise meaning for the symbol as 
well as a response action that should be taken.  Participants were then shown the actual 
symbol meaning as well as an appropriate response.  Then, one random symbol for each 
referent was shown to the participant with photographic context, and the same questions 
were asked.  Finally, for each referent, participants were shown the four symbols 
designed in Experiment #1, the four symbols designed in Experiment #2, and a symbol 
design already in use, and they were asked to select the symbol that they feel is most 
effective at communicating the intended message without knowledge of the source of 
each symbol. 
 
 87 
Multinational Group   U.S. Group
Indian Group Chinese Group
Multinational Group   U.S. Group
Indian Group Chinese Group
 
 
 
 
 
 
 
 
 
 
 
Figure 19. Best ?Hot Exhaust? symbols from each DIGA group. 
 
 
 
 
 
 
 
 
 
 
 
Figure 20. Best ?Do Not Touch with Wet Hands? symbols from each DIGA group.
 88 
Results 
 The symbols produced by DIGA Groups 1-4 of Experiment #1 are found in 
Appendix 5.5.  Each symbol was evaluated using the composite analysis discussed in the 
previous section by its design group, and the top performing symbols from each group 
are presented in Figures 19 and 20.  A summary of the results of these evaluations are 
listed in Table 13.   
Table 13. 
Composite evaluation results of the best symbol by each DIGA design group 
  Design Group Quality Rating  (1-7)* Comprehension Estimate* Subjective Rank 
     
 Multinational 5.75 (1st) 81.25% (1st) 6.5 
 U.S. 5.25 (1st) 74.58% (2nd) 3 
 Indian 5.60 (1st) 76.00% (1st) 2 
Hot 
Exhaust 
 Chinese 5.25 (1st) 78.33% (1st) 4 
     
 Multinational 5.00 (1st) 60.00% (1st) 1 
 U.S. 5.50 (1st) 78.75% (1st) 1 
 Indian 5.50 (1st) 74.50% (1st) 1 
Do Not 
Touch 
with Wet 
Hands 
 Chinese 5.50 (1st) 75.83% (3rd) 1 
* Parenthetical ranks are in comparison to other symbols produced in the same group. 
 Though Takagi (2001) recommends a limit of 10-20 generations to avoid 
participant fatigue, this recommendation does not guarantee that the algorithm has begun 
to converge.  To demonstrate convergence, Figures 21 and 22 plot the number of times 
participants selected a new favorite symbol (i.e. found a better design) across all 
participants in a particular design group for each referent.  In the first few generations, it 
was not uncommon for more than half of the participants to find a new favorite symbol in 
a given generation.  In each case, however, the function of design changes per generation 
decreases, though not necessarily to zero. 
 89 
 
 
Figure 21.  Convergence of ?Hot Exhaust? algorithm based on number of times a 
participant selected a new best symbol per generation.   
  
 One-tail paired t-tests were performed to determine whether a significant 
reduction in symbol diversity occurred between the top-ranked symbols of each 
participant from generation 1 to generation 20.  The coefficient of variation (CV) was 
calculated for each design parameter, since they have very different ranges, to normalize 
the variance, and the CV of each parameter was found at generation 1 and at generation 
20.  Table 14 summarizes the paired t-tests for each of the eight trials.  Only two trials did 
not result in a significant decrease in the coefficient of variance between the highest 
ranked symbols of each participant from the first to the final generation. 
DIGA Group 1 - Multinational
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 2 - U.S.
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 3 - Indian
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 4 - Chinese
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
 90 
 
Figure 22.  Convergence of ?Do Not Touch with Wet Hands? algorithm based on number 
of times a participant selected a new best symbol per generation.   
 
Table 14. 
Convergence analysis of one-tail paired t-test statistics comparing the coefficient of 
variation of the first and final generations between subjects for each DIGA trial. 
 
 
 
 
 
 
* Indicates results that were not significant at the ? = 0.05 level. 
  DIGA Design Groups 
  
Multinational 
(n=12) 
U.S. 
(n=12) 
Indian 
(n=10) 
Chinese 
(n=12) 
Hot 
Exhaust 
2.13 
(p=0.028) 
1.20* 
(p=0.128) 
3.96 
(p=0.001) 
3.31 
(p=0.003) 
Do Not 
Touch with 
Wet Hands 
-0.36* 
(p=0.362) 
2.53 
(p=0.013) 
1.87 
(p=0.042) 
1.98 
(p=0.036) 
DIGA Group 1 - Multinational
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 2 - U.S.
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 3 - Indian
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
DIGA Group 4 - Chinese
0
2
4
6
8
10
0 5 10 15 20
Generations
Nu
mb
er
 of
 N
ew
 Fa
vo
rit
es
 91 
Multinational Group   U.S. Group
Indian Group Chinese Group
Multinational Group               U.S. Group
Indian Group Chinese Group
Multinational Group   U.S. Group
Indian Group Chinese Group
Multinational Group                 U.S. Group
Indian Group Chinese Group
 The symbols produced by the focus group participants in Experiment #2 served as 
a control for comparison to the DIGA symbols.  An additional control symbol for each 
referent found in the literature was also included.  The focus group symbols are shown in 
Figures 23 and 24, and Figure 25 contains the previously published symbols. 
 
 
 
 
 
 
 
 
 
 
Figure 23.  Draft and final focus group symbols for ?Hot Exhaust?. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 24.  Draft and final focus group symbols for ?Do Not Touch with Wet Hands?. 
 92 
Hot Exhaust   Do Not Touch with Wet Hands
Figure 25.  Previously published manufacturer?s symbol for ?Hot Exhaust? (Lewis, 
2008) and ISO symbol for ?Do Not touch with Wet Hands? (ISO, 2004). 
 
 A one-way Analysis of Variance (ANOVA) revealed that significant differences 
existed in the ranking of the symbols by the evaluation survey participants (N=401).  
Tukey HSD multiple comparisons analysis revealed that Hot Exhaust symbols DIGA-
Group4, FG-Group1 and FG-Group2 were preferred significantly more frequently than 
the other symbols.  Similarly, Do Not Touch with Wet Hands Symbol FG-Group2 was 
preferred significantly more frequently than the others.  The ANOVA and Tukey HSD 
results are shown in Tables 15 and 16.   
Table 15. 
Results of ANOVA and Tukey HSD analysis for ?Hot Exhaust? Symbol Evaluations. 
 
Subsets of Equivalent Symbols (? = 0.05) Symbol % 
Preferred 1 2 3 4 5 
HE DIGA Chinese 23.2% X X    
HE FG U.S. 22.2% X X    
HE FG Multinational 19.7% X X X   
HE FG Indian 14.2%  X X X  
HE FG Chinese 11.0%   X X  
HE DIGA U.S. 4.2%     X 
HE DIGA Indian 2.5%     X 
HE DIGA Multinational 1.8%     X 
HE Manufacturer's 1.2%         X 
 93 
Table 16.   
Results of ANOVA and Tukey HSD analysis for ?Do Not Touch with Wet Hands? 
Symbol Evaluations. 
 
Subsets of Equivalent Symbols (? = 0.05) Symbol % 
Preferred 1 2 3 4 5 6 7 8 
WH FG U.S. 32.4% X        
WH ISO 19.0%  X X      
WH FG Multinational 16.2%  X X X     
WH FG Chinese 11.2%  X X X X X   
WH DIGA U.S. 8.5%   X X X X   
WH DIGA Chinese 5.2%    X X X X  
WH FG Indian 4.7%    X X X X X 
WH DIGA Indian 2.5%     X X X X 
WH DIGA Multinational 0.2%           X X X 
 
 
Discussion 
 Interactive evolutionary computation must balance two competing constraints 
when human judgment serves as the fitness evaluation for design solutions.  Like any 
form of evolutionary computation, the convergence velocity (Back, Fogel, & 
Michalewicz, 2000) must be gradual enough to allow for adequate diversity of designs 
and exploration of the search space (Dumitrescu, Lazzerini, Jain, & Dumitrescu, 2000).  
However, Takagi (2001) insists that fatigue can set in and degrade the design process if 
convergence takes longer than 10-20 generations.  Figures 19 and 20 suggest that the 
DIGA did not converge too quickly since many new best solutions were being discovered 
well after the 10th generation in all 8 trials.  Furthermore, the algorithm successfully 
converged to near zero (2 or less) best symbol replacements in three of the eight trials.  
However, in the other trials, the DIGA did not produce consistent near zero symbol 
replacements by generation 20, though the number of replacements was decreasing in all 
 94 
but one trial.  This finding implies that it is possible, but not necessarily probable, that the 
DIGA can come to convergence between generations 10-20 for symbol referents similar 
to those tested.  The fact that two of the three converging trials occurred among the two 
U.S. groups of participants may suggest that factors related to the participants, not just 
the algorithm, affected convergence.  Further implications that participant related factors 
might affect convergence likelihood arise from the observation that the poorest 
convergence occurred in the two homogeneous Indian participant trials.  In addition to 
the country-of-origin factor, these two trials were also the only trials that differed in 
number of participants (i.e. 10 participants instead of 12).  Thus, there may also be an 
association between the size of the participant group and the rate of convergence.  
Finally, a 0.10 mutation probability was utilized in all eight trials, and it is certainly 
possible that this parameter, or the two-point crossover method used in each trial, could 
be adjusted to optimize convergence velocity.   
Despite these considerations, six out of eight symbols produced by these 
experiments were shown to have converged between participants with significantly less 
diversity between each participant?s favorite symbol at generation 20 than at generation 
1.  This implies that the symbol chosen by the group as most comprehensible following 
the design session represents well the group?s consensus design ideas since they were 
converging well at the end of the experiment.  Once again, it is possible that an 
association exists between country-of-origin and the convergence of the participant?s 
symbols over time; but if so, it is not an obvious association.  The two trials which failed 
to significantly reduce symbol diversity over the course of the 20 generations were not 
found within the same demographic stratum nor the same warning referent.  Further 
 95 
investigation is necessary to determine the distinguishing factors between the 
significantly convergent and non-significantly convergent results. 
 For the ?Hot Exhaust? referent, a DIGA-produced symbol (DIGA-Group 4) was 
found to be more preferred, along with Focus Group 1 and Focus Group 2 symbols, than 
the others, including the manufacturer?s symbol currently in use.  It should be noted that 
no design or evaluation information was published with the manufacturer?s symbol, so 
there is no way to know if it was designed and evaluated according to ANSI or ISO 
testing standards.  In addition, the remaining 3 DIGA-produced symbols, along with the 
FG-2 symbol, were statistically least preferred.  These results suggest that it is possible to 
design a symbol using a distributed interactive genetic algorithm that is understood at 
least as well as symbols designed by other means.  It is also possible to produce a symbol 
from a DIGA design experiment that is perceived to be inferior to symbols designed by 
other means.  The statistical difference between the DIGA-Group 4 symbol and the other 
DIGA symbols implies that the design parameters existed in the algorithm to produce a 
viable symbol.  However, since the algorithm converged upon symbols that were poorly 
perceived as well as on symbols understood well by the evaluation group, it seems that 
local rather than global optimums were sometimes found.  More investigation is 
necessary to determine what design factors in the DIGA experiment affect the eventual 
evaluation results of the symbol it produces.  Interestingly, the most preferred DIGA 
symbol for ?Hot Exhaust? was produced in DIGA Group 4 (i.e. group of all Chinese 
participants), which did not appear to be converging to near zero changes of favorites at 
generation 20 according to the plot of best symbol changes vs. generation number shown 
in Figure 20.   
 96 
 For the ?Do Not Touch with Wet Hands? referent, the symbol from Focus Group 
2 was preferred far more frequently than the others with nearly a third of the evaluation 
participants selecting it as the most comprehensible.  In this case, the four DIGA-
produced symbols performed poorly.  Among the five control symbols, only the Focus 
Group 3 symbol was preferred by fewer people than the best of the DIGA symbols, and 
three of the control symbols (FG-2, ISO, and FG1) were statistically more preferred than 
all of the DIGA symbols.  The poor performance of the DIGA for this referent is not well 
understood.  However, one potential cause is the inclusion of the ?Circle and X? 
prohibition symbol, in addition to the traditional ?Circle and slash? symbol.  Although 
this variant of the prohibition symbol was found to be a valuable addition in to the 
attribute matrix during the semantic annotation process (see Chapter 4 of this 
dissertation), it may produce a negative effect on symbol effectiveness since it appears in 
three of the four least preferred symbols.  A negative effect associated with the ?X? 
portion of this prohibition symbol would be consistent with findings in the literature 
which suggest that prohibitives that obscure the image the least are typically the most 
preferred (Murray, Magurno, Glover, & Wogalter, 1998; Shieh & Huang, 2003). 
 
Conclusions 
 Developing warning symbols using a distributed interactive evolutionary 
algorithm can be an effective means of engaging diverse participants in sharing designs 
without the need for verbal communication.  The experiments conducted in this study 
have demonstrated that convergence is possible in the 10-20 generations for which 
human participants can be expected to contribute judgments meaningfully.  However, the 
 97 
experiment also revealed that the DIGA can be inconsistent in its convergence and the 
symbols it produces, potentially due to both programming decisions within the algorithm 
as well as personal factors associated with the participants.   
Several limitations in this experimental protocol are noted.  First, several factors 
which may have a large impact on the convergence of the DIGA as well as the content of 
the final designs were not allowed to vary in this experiment.  The mutation probability 
can have a large impact on the convergence velocity as well as the diversity of the 
solutions, especially in the early generations.  Research should be performed to determine 
whether there an optimal value or set of values for this and other DIGA parameters.  In 
addition, only a maximum of 12 participants were allowed to interact with the DIGA in a 
given trial.  A larger cohort of participants would have been able to search a larger 
percentage of the design space, potentially developing more diverse symbol designs.  
Finally, the evaluation survey did not use open-ended comprehension testing, the best 
approach for assessing the ability of a symbol to communicate its message.  This 
limitation was primarily due to the 50 unique evaluators recommended per symbol 
variant for a given referent, and the need to replicate this evaluation in three countries.  
For nine variants, this would require 1350 participants, 450 in each of three countries.  
Although this quantity of human participation was beyond the capacity of this research, 
future investigation into the comprehension of DIGA-produced symbols should attempt 
to employ this evaluation technique.  
 98 
 
 
 
CHAPTER 6 
CONCLUSIONS 
Introduction 
Participatory design of safety warning symbols has progressed from the early 
days of the designer?s method with very little user involvement to the production method 
of generating many symbol candidates from user-drawn images to the focus group 
method that allows users to analyze designs and determine a consensus symbol.  
Unfortunately, not all of the benefit of participatory design is captured when the 
designers must intervene in the content development or design reduction portion of the 
process or when the challenge of overcoming participant diversity discourages the 
adequate sharing of ideas.  However, since comprehension of and compliance with 
warning symbols depends significantly on their design, there is a significant need to 
improve the participatory design of warning symbols to increase the likelihood of 
effective communication to those who need this important protective information. 
In essence, the design of symbols constitutes a search task of all symbol design 
possibilities (both real and hypothetical) to determine the most effective.  Given the 
incredibly broad scope of possible symbol designs for a given warning referent, various 
techniques have been proposed to bound this search space, either by developing a short 
list of candidate symbols or by evolutionary computational searches using predetermined 
graphical search boundaries.  Evidence exists that the focus group method performs 
better than other methods at producing comprehensible symbols.  It has also been shown 
 99 
that interactive evolutionary computation can produce symbols of similar quality to more 
traditional methods for users of diverse cultural and national demographics.  The aim of 
this study was to remodel the top performing traditional symbol design method (i.e. the 
focus group method) using distributed (i.e. multiuser) interactive evolutionary 
computation that was designed based on user-contributed search boundary criteria. 
 
Summary of Findings 
Three primary experiments were performed in this dissertation.  First, in order to 
ensure that the proceeding distributed interactive genetic algorithm (DIGA) symbol 
design method could be tested on symbol referents that were considered both easy and 
difficult from which to design a symbol, a relative ?ease of conversion? factor was 
defined using a perception survey.  Next, an experiment was conducted to define the 
search parameters to be made available to the DIGA by examining and clustering hand 
drawn versions of the symbols to be designed.  Finally, the DIGA experiment allowed 
participants to interact with the distributed genetic algorithm to propose ideas, share them 
with one another, and receive new symbol ideas from the system.  The final symbols 
produced in this experiment were compared to symbols produced by the focus group 
method and to symbols currently in the field. 
The findings of the research are summarized below. 
1. Survey participant perception can be used to develop a ranked list of referents with 
regard to their ease of conversion from written warning referent to graphical symbol.  
New and existing symbol design methods should be evaluated based on their ability 
to produce quality symbols from referents both easy and difficult to convert. 
 100 
2. Similar perceptions of ease of conversion were noted for novice participants as well 
as for two groups of safety professionals, though the perceptions of these groups were 
not statistically identical.  Certified Safety Professional and non-certified members of 
the American Society of Safety Engineers did have identical perceptions of warning 
referent difficulty, while a novice group of university students differed in their 
perceptions by a small, yet statistically significant amount.  It appears that the 
experience gained from working as a safety professional affects the perception of 
warning referents significantly, though not to such a degree as to invalidate the 
perceptions of others. 
3. Hand-made drawings of safety symbols can be analyzed by a three-member panel to 
transform the set of graphical images to a set of binary attribute matrices.  The 
attribute matrices completely define the gross characteristics of each symbol drawing. 
Agreement between panelist evaluations was high (~95%). 
4. When the symbol attribute matrices are clustered using a simple K-means clustering 
algorithm, the centroidal characteristics of each symbol cluster creates a reduced set 
of symbol attributes capable of completely defining the gross characteristics of each 
cluster of symbols.  In this way, general concentrations of symbol designs can be 
identified for each safety referent without having to sort the symbols into themes by 
human judgment alone, as has been required by previous design methods. 
5. Distributed interactive genetic algorithms can be used as a substitute for the focus 
group method of symbol development that does not require communication in a 
common language between participants.  In general, the algorithms demonstrated 
solution convergence for 10-12 participants after 20 generations, which falls within 
 101 
the recommendations for both interactive evolutionary computation and for symbol 
design in groups.   
6. Convergence appeared to differ between the easy to convert referent (?Hot Exhaust?) 
and the difficult to convert referent (?Do Not Touch with Wet Hands?).  The more 
difficult referent followed an interesting profile, decreasing in diversity during the 
first 10 generations and increasing in diversity again during the last 10 generations, 
regardless of participants.  However, the easier referent either increased or decreased 
with a linear trend across all 20 generations, depending on the demographics of the 
participants.   
7. Convergence of the DIGA algorithm appeared to depend on country of origin.  The 
rate of selection of new top symbols decreased more rapidly among Multinational and 
homogeneous U.S. experimental groups than it did for homogeneous Chinese and 
homogeneous Indian groups, when compared across both symbol referents. 
8. One of the symbols designed by the DIGA method for the easier safety referent, ?Hot 
Exhaust?, received the highest preference ranking when compared to eight other 
variants, and it was statistically tied for the highest ranking with two symbols 
designed via the focus group method.  All four DIGA symbols were preferred over 
the currently used manufacturer?s symbol.  This implies that the DIGA is capable of 
producing symbols for easy to convert warning referents that are at least as effective 
as other current methods.  However, the DIGA symbols designed for the more 
difficult referent, ?Do Not Touch with Wet Hands?, did not perform well in 
comparison to symbols designed by other means, including the currently published 
symbol.  For more difficult referents, modifications to the search strategy, such as 
 102 
local searches or extended DIGA sessions, may be needed to ensure a quality design.  
This also reinforces the need to evaluate warning referents for ease of conversion and 
to ensure that design methods function adequately for all difficulty levels. 
 
Limitations of the Research 
The key limitations of this research are described below. 
1. The ease of conversion factor attempted to ascertain the perceived difficulty a survey 
respondent would have in attempting to produce a symbol from a written safety 
referent.  However, the study did not attempt to validate the authenticity of this 
difficulty estimate, and the respondents were not asked to actually attempt to produce 
any symbols to verify whether their perception was accurate. 
2. The participatory design strategy of ascertaining symbol design criteria from actual 
participant symbol drawings involved only two demographic strata: U.S.-born and 
Indian-born students attending college in the Southeastern United States.  To build 
confidence that the semantic annotation and attribute matrix clustering processes 
produced symbol clusters representing adequate demographic diversity, additional 
drawings should be analyzed from other regional, international and cultural groups. 
3. Only one replication of the DIGA experiment was performed with each research 
group and referent.  This study did not attempt to measure whether similar designs 
were repeatable by the same individuals in a second trial or whether a learning curve 
effect would alter the convergence velocity in a second trial. 
4. The DIGA parameters and algorithm design decisions were held constant during the 
experimentation.  It is likely that adjustments to these parameters and to the algorithm 
 103 
design could significantly affect performance of the system with regard to 
convergence and to the diversity of the search space explored.   
5. Though the DIGA demonstrated good convergence in most trials, the final generation 
still resulted in separate designs for each participant which necessitated a vote by the 
group members to determine which final symbol would serve as the group?s design.  
A method within the algorithm of selecting a representative design out of the final 12 
candidates should be pursued. 
6. The best approach for assessing the ability of symbols to communicate their 
messages, open-ended comprehension testing, was not used in this research, primarily 
due to the sample size needed to test nine symbol variants according to ANSI and 
ISO specifications.  The combined method of preference ranking, comprehension 
estimation and quality rating, while diverse, should be supplemented with open-ended 
comprehension testing with context cues to ensure that both the DIGA and Focus 
Group symbols are adequately evaluated. 
 
Recommendations for Future Research 
 Several opportunities for future research have arisen from this study.  First, a 
validation study to determine the accuracy of the ease of conversion factor with regard to 
predicting the difficulty of producing a graphical symbol from a written warning referent 
should be undertaken.  Such a study should recruit participants similar to those surveyed 
in this research to draw sets of symbols matching the referent pairs compared in the 
current survey.   In addition, research should be conducted to validate the robustness of 
the attribute matrices defined from the U.S. and Indian research participants.  Analyzing 
 104 
symbol drawings produced by participants from additional demographic groups for the 
same symbol referents without heed to the previously defined symbol attributes should 
produce a parallel attribute matrix for each symbol.  These parallel attribute matrices 
should then be compared to determine if they are products of statistically equivalent 
populations.  Furthermore, additional replications of the DIGA design experiment are 
recommended to test the effect of replication, algorithm and parameter adjustments on 
the performance of the system.  Another research need involves the exploration of means 
with the DIGA algorithm to narrow multiple ?final designs? to a single consensus.  While 
the algorithm itself converges towards a final design, the diversity remaining in the final 
generation makes declaring a consensus symbol difficult.  Finally, symbols designed by 
distributed interactive genetic algorithms should be tested according to ANSI and ISO 
standards, including open-ended comprehension testing, to fully determine their 
effectiveness. 
 105 
 
 
 
REFERENCES 
Aggarwal, C. C. (2004). A human-computer interactive method for projected clustering. 
Knowledge and Data Engineering, IEEE Transactions on, 16(4), 448-460. 
Anil, K. J., & Richard, C. D. (1988). Algorithms for clustering data: Prentice-Hall, Inc. 
ANSI. (2007a). Criteria for Safety Symbols: American National Standards Institute 
Z535.3 - 2007. 
ANSI. (2007b). Environmental Facility and Safety Signs: American National Standards 
Institute Z535.2 - 2007. 
ANSI. (2007c). Product Safety Signs and Labels: American National Standards Institute 
Z535.4 - 2007. 
Back, T., Fogel, D. B., & Michalewicz, Z. (2000). Evolutionary Computation: Basic 
Algorithms and Operators. New York: Taylor and Francis. 
BCSP. (2009). Board of Certified Safety Professionals.   Retrieved November 12, 2009, 
from http://www.bcsp.org 
Bezdek, J. C. (1981). Pattern Recognition with Fuzzy Objective Function Algorithms: 
Kluwer Academic Publishers. 
Boersema, T., & Zwaga, H. J. G. (1989). Selecting comprehensible warning symbols for 
swimming pool slides. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 33, 994-998. 
Bortz, J., Lienert, G. A., & Boehnke, K. (2000). Verteilungsfreie Methoden in der 
Biostatistik. Berlin: Springer. 
 106 
Campbell, J. L., Hofmeister, D. H., Kiefer, R. J., Selke, D. J., Green, P., & Richman, J. B. 
(2004). Comprehension testing of active safety symbols. In SAE International. 
Carnahan, B. J., & Dorris, N. (2004). Identifying Relevant Symbol Design Criteria Using 
Interactive Evolutionary Computation. Paper presented at the Genetic and 
Evolutionary Computing Conference (GECCO), Seattle. 
Carnahan, B. J., Dorris, N. T., & Kuntz, L. A. (2005). Designing anthromorphic symbols 
using interactive evolutionary design. Information Design Journal and Document 
Design, 13(3), 179-190. 
Carneiro, G., Chan, A. B., Moreno, P. J., & Vasconcelos, N. (2007). Supervised Learning 
of Semantic Classes for Image Annotation and Retrieval. IEEE Transactions on 
Pattern Analysis and Machine Intelligence, 29(3), 394-410. 
Carneiro, G., & Vasconcelos, N. (2004). Formulating Semantic Image Annotation as a 
Supervised Learning Problem. Unpublished Technical Report. University of 
California, San Diego. 
Chan, A. H. S., & Ng, A. W. Y. (2009). Perceptions of implied hazard for visual and 
auditory alerting signals. Safety Science, 47(3), 346-352. 
Chan, H. M., & Milner, D. A. (1981). Direct clustering algorithm. Journal of 
Manufacturing Systems, 1(1), 65-75. 
Chen, L., & Pu, P. (2004). Survey of Preference Elicitation Methods (No. IC/200467). 
Choi, H.-G., & Chang Hyo, M. (1993). Performance comparison of two-dimensional 
clustering algorithms in group technology. Computers & Industrial Engineering, 
25(1-4), 199-202. 
 107 
Chouraqui, E., & Doniat, C. (2003). The s-ethos system: a methodology for systematic 
flight analysis centered on human factors. Applied Artificial Intelligence, 17(7), 
583-629. 
Clark, D. R. (1988). Model for the Analysis, Representation and Synthesis of Hazard 
Warning Communication. Unpublished Dissertation, The University of Michigan. 
Clark, D. R., Benysh, S. A. H., & Lehto, M. R. (2003). Design of Industrial Warnings. In 
W. Karwowski & W. S. Marras (Eds.), Occupational Ergonomics: Principles of 
Word Design. Boca Raton, FL: CRC Press. 
Cohen, H. H., Cohen, J., Mendat, C. C., & Wogalter, M. S. (2006). Warning Channel: 
Modality and Media. In M. S. Wogalter (Ed.), Handbook of Warnings (pp. 841). 
Mahwah, New Jersey: Lawrence Erlbaum Associates. 
Connover, W. J. (1999). Practical Nonparametric Statistics (3rd ed.): John Wiley and 
Sons, Inc. 
Davies, S., Haines, H., Norris, B., & Wilson, J. R. (1998). Safety pictograms: are they 
getting the message across? Applied Ergonomics, 1, 15-23. 
Dejoy, D. M., Cameron, K. A., & Della, L. J. (2006). Postexposure evaluation of warning 
effectiveness: a review of field studies and population-based research. In M. S. 
Wogalter (Ed.), Handbook of Warnings (pp. 35-48). Mahwah, New Jersey: 
Lawrence Erlbaum Associates. 
Deppa, S. W. (2006). U.S. and inernational standards for safety symbols. In M. S. 
Wogalter (Ed.), Handbook of Warnings (pp. 477-486). Mahwah, New Jersey: 
Lawrence Erlbaum Associates. 
 108 
Dewar, R. (1999). Design and evaluation of public information symbols. In B. Zwaga 
(Ed.), Visual Information for Everyday Use (pp. 285-303). London Taylor & 
Francis. 
Dorris, N. (2004). The application of interactive evolutionary computation (IEC) to the 
design of safety symbols. Unpublished Dissertation, Auburn University, Auburn, 
AL. 
Dorris, N., Carnahan, B., Orsini, L., & Kuntz, L. A. (2004). Interactive evolutionary 
design of anthropomorphic symbols. Paper presented at the Congress on 
Evolutionary Computation, 2004. CEC2004. 
Dorris, N., & Davis, J. (2003). Testing the Effects of Label Deterioration on the 
Legibility and Comprehensibility of Warnings. Human Factors and Ergonomics 
Society Annual Meeting Proceedings, 47, 1726-1729. 
Dowse, R., & Ehlers, M. S. (2001). The evaluation of pharmaceutical pictograms in a 
low-literate South African population. Patient Education and Counseling, 45(2), 
87-99. 
Dozier, G., Carnahan, B., Seals, C., Kuntz, L. A., & Fu, S. G. (2005a). An IDEA for 
design. Unpublished Technical Report. 
Dozier, G., Carnahan, B., Seals, C., Kuntz, L. A., & Fu, S. G. (2005b). An interactive 
distributed evolutionary algorithm (IDEA) for design. Paper presented at the 2005 
IEEE International Conference on Systems, Man and Cybernetics. 
Dreo, J., Petrowsdki, A., Siarry, P., & Taillard, E. (2003). Metaheuristics for Hard 
Optimization: Methods and Case Studies (A. Chatterjee, Trans.). Berlin: Springer. 
 109 
Duke, J. M., & Aull-Hyde, R. (2002). Identifying public preferences for land preservation 
using the analytic hierarchy process. Ecological Economics, 42(1-2), 131-145. 
Dumitrescu, D., Lazzerini, B., Jain, L. C., & Dumitrescu, A. (2000). Evolutionary 
Computation. Boca Raton, FL: CRC Press. 
Easton, G., Easton, A., & Belch, M. (2003). An experimental investigation of electronic 
focus groups. [Article]. Information & Management, 40(8), 717-727. 
Edworthy, J., & Adams, A. (1996). Warning Design: A Research Perspective: CRC 
Press. 
Egilman, D., & Bohme, S. R. (2006). A Brief History of Warnings. In M. S. Wogalter 
(Ed.), Handbook of Warnings (pp. 11-20). Mahwah, New Jersey: Lawrence 
Erlbaum Associates. 
Eshelman, L. J., & Schaffer, J. D. (1993). Real-coded genetic algorithms and interval-
schemata. In Foundations of Genetic Algorithms (Vol. 2, pp. 187-202). San 
Mateo: Morgan Kaufman Publishers. 
Fielding, W. J., Riley, J., & Oyejola, B. A. (1998). Ranks are statistics: some advice for 
their interpretation. On PLA Notes - International Institute for Environment and 
Development. London (United Kingdom): IIED. 
Foster, J. J., & Afzalnia, M. R. (2005). International assessment of judged symbol 
comprehensibility. International Journal of Psychology, 40(3), 169-175. 
Frias-Martinez, E., Chen, S. Y., Macredie, R. D., & Liu, X. (2007). The role of human 
factors in stereotyping behavior and perception of digital library users: a robust 
clustering approach. User Modeling and User-Adapted Interaction, 17(3), 305-
337. 
 110 
Friedmann, K. (1988). The Effect of Adding Symbols to Written Warning Labels on User 
Behavior and Recall. Human Factors: The Journal of the Human Factors and 
Ergonomics Society, 30, 507-515. 
Garmer, K., Ylven, J., & Karlsson, I. C. M. (2004). User participation in requirements 
elicitation comparing focus group interviews and usability tests for eliciting 
usability requirements for medical equipment: a case study. [Article]. 
International Journal of Industrial Ergonomics, 33(2), 85-98. 
Garrett, A., Carnahan, B., Muhdi, R., Davis, J., Dozier, G., SanSoucie, M. P., et al. 
(2006). Evacuation Planning via Evolutionary Computation. Paper presented at 
the Evolutionary Computation, 2006. CEC 2006. IEEE Congress on. 
Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine 
Learning. Reading, MA: Addison-Wesley Publishing Company. 
Goldsworthy, R., & Kaplan, B. (2006). Exploratory evaluation of several teratogen 
warning symbols. Birth Defects Research Part A: Clinical and Molecular 
Teratology, 76(6), 453-460. 
Green, P. (1979). Development of pictographic symbols for vehicle controls and displays. 
In SAE Technical Paper Series (No. 790383). Warrendale, PA: Society of 
Automotive Engineers. 
Green, P. (1993). Design and evaluation of symbols for automobile controls and displays. 
In B. Peacock & W. Karwowski (Eds.), Automotive Ergonomics (pp. 237-268). 
London, UK: Taylor & Francis. 
 111 
Haas, E., & Edworthy, J. (2006). An introduction to auditory warnings and alarms. In M. 
S. Wogalter (Ed.), Handbook of Warnings (pp. 189-198). Mahwah, New Jersey: 
Lawrence Erlbaum Associates. 
Hadi-Tabassum, S. (2005). The Balancing Act of Bilingual Immersion. Educational 
Leadership, 62(4), 50-54. 
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). 
The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1). 
Hammer, W. (1989). Occupational Safety Management and Engineering (4th ed.). 
Englewood Cliffs, NJ: Prentice Hall. 
Hancock, H. E., Rogers, W. A., Schroeder, D., & Fisk, A. D. (2004). Safety Symbol 
Comprehension: Effects of Symbol Type, Familiarity, and Age. Human Factors: 
The Journal of the Human Factors and Ergonomics Society, 46, 183-195. 
Hicks, K. E., Bell, J. L., & Wogalter, M. S. (2003). On the Prediction of Pictorial 
Comprehension. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 47, 1735-1739. 
Hodgkinson, R., & Hughes, J. (1982). Developing Wordless Instructions: A case history. 
IEEE Transactions on Professional Communication, 25(2), 74-79. 
Holman, G. T., Carnahan, B. J., & Thomas, R. E. (2006). Using surveys to identify 
stressors in generalized jobs: A direct clustering method. International Journal of 
Industrial Ergonomics, 36(7), 671-677. 
Howell, W. C., & Fuchs, A. F. (1968). Population stereotypy in code design. 
Organizational Behavior and Human Performance, 3, 310-339. 
 112 
Huer, M. B. (2000). Examining perceptions of graphic symbols across cultures: 
Preliminary study of the impact of culture/ethnicity. Augmentative and Alternative 
Communication, 16(3), 180-185. 
Isherwood, S. J., McDougall, S., J.P, & Curry, M. B. (2007). Icon Identification in 
Context: The Changing Role of Icon Characteristics With User Experience. 
Human Factors: The Journal of the Human Factors and Ergonomics Society, 49, 
465-476. 
ISO. (2002). Design principles for graphical symbols for use in safety signs: International 
Organization for Standardization 3864-1: 2002. 
ISO. (2003). Graphical symbols -- Safety colours and safety signs -- Safety signs used in 
workplaces and public areas: International Organization for Standardization 
7010:2003  
ISO. (2004). Graphical Symbols -- Design principles for Product Safety Labels: 
International Organization for Standardization 3864-2: 2004. 
ISO. (2006). Graphical Symbols -- Design principles for graphical symbols for use in 
safety signs: International Organization for Standardization 3864-3: 2006. 
ISO. (2007). Graphical symbols - Test methods - Part 1: Methods for testing 
comprehensibility: International Organization for Standardization 9186-1:2007. 
ISO. (2008). Graphical symbols - Test methods - Part 2: Method for testing perceptual 
quality: International Organization for Standardization 9186-2:2008. 
Jaynes, L. S., & Boles, D. B. (1990). The Effect of Symbols on Warning Compliance. 
Human Factors and Ergonomics Society Annual Meeting Proceedings, 34, 984-
987. 
 113 
Johnston, K. L., Phillips, M. L., Esmen, N. A., & Hall, T. A. (2005). Evaluation of an 
Artificial Intelligence Program for Estimating Occupational Exposures. Annals of 
Occupational Hygiene, 49, 147-153. 
Kalsher, M. J., Wogalter, M. S., & Racicot, B. M. (1996). Pharmaceutical container 
labels: enhancing preference perceptions with alternative designs and pictorials. 
International Journal of Industrial Ergonomics, 18(1), 83-90. 
Karsh, R., & Mudd, S. A. (1962). Design of a picture language to identify vehicle 
controls. Iii. A comparative evaluation of selected picture-symbol designs. 
Retrieved 12/16/2008. from http://handle.dtic.mil/100.2/AD289544. 
Kirkwood, C. W., & Sarin, R. K. (1985). Ranking with Partial Information: A Method 
and an Application. Operations Research, 33(1), 38-48. 
Klein, E. E., Tellefsen, T., & Herskovitz, P. J. (2007). The use of group support systems 
in focus groups: Information technology meets qualitative research. [Review]. 
Computers in Human Behavior, 23(5), 2113-2132. 
Klim?nek, D., & ?ulc, B. (2004). Fault Detection in Control Systems via Evolutionary 
Algorithms. Paper presented at the Transactions of the V?B-Technical University 
of Ostrava, Mechanical series, V?B Ostrava, Czech Republic. 
Klim?nek, D., & ?ulc, B. (2005). Evolutionary Detection of Sensor Discredibility in 
Control Loops. Paper presented at the 31st Annual Conference IEEE Industrial 
Electronics, New York. 
Langston, P. A., Masling, R., & Asmar, B. N. (2006). Crowd dynamics discrete element 
multi-circle model. Safety Science, 44(5), 395-417. 
 114 
Laughery, K. R. (2006). Safety communications: Warnings. Applied Ergonomics, 37(4), 
467-478. 
Laughery, K. R., & Wogalter, M. S. (2006). Designing Effective Warnings. Reviews of 
Human Factors and Ergonomics, 2, 241-271. 
Laughery, K. R., Young, S. L., Vaubel, K. P., & Brelsford, J. J. W. (1993). The 
Noticeability of Warnings on Alcoholic Beverage Containers. Journal of Public 
Policy & Marketing, 12(1), 38-56. 
Lehto, M. R. (2006). Optimal warnings: an information and decision theoretic 
perspective. In M. S. Wogalter (Ed.), Handbook of Warnings (pp. 89-108). 
Mahwah, New Jersey: Lawrence Erlbaum Associates. 
Lehto, M. R., & Papastavrou, J. D. (1993). Models of the warning process: important 
implications towards effectiveness. Safety Science, 16(5-6), 569-595. 
Lenton, R. (2007). Using the Method of Paired Comparisons in Non-Designed 
Experiments. Unpublished Masters Thesis, Griffith University. 
Lesch, M. F. (2005). A Semantic Relatedness Paradigm for Assessing Comprehension of 
Warning Symbols. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 49, 1795-1799. 
Lesch, M. F. (2006). Consumer product warnings: Research and Recommendations. In 
M. S. Wogalter (Ed.), Handbook of Warnings (pp. 137-146). Mahwah, New 
Jersey: Lawrence Erlbaum Associates. 
Lesch, M. F. (2008a). A comparison of two training methods for improving warning 
symbol comprehension. Applied Ergonomics, 39(2), 135-143. 
 115 
Lesch, M. F. (2008b). Warning symbols as reminders of hazards: Impact of training. 
Accident Analysis & Prevention, 40(3), 1005-1012. 
Lesch, M. F., Rau, P.-L. P., Zhao, Z., & Liu, C. (2009). A cross-cultural comparison of 
perceived hazard in response to warning components and configurations: US vs. 
China. Applied Ergonomics, 40(5), 953-961. 
Lewis, D. P. (2008). Abstract closeup of a CC-115 Buffalo aircraft.   Retrieved 
November 16, 2009, 2009, from 
http://www.istockphoto.com/file_closeup/transportation/6712797-hot-blast-
exhaust.php?id=6712797 
Liu, L., Hoelscher, U., & Gruchmann, T. (2005). Symbol comprehension in different 
countries: experience gained from medical device area. Mensch Und Computer, 
81-87. 
Lo, C. H., Wong, Y. K., & Rad, A. B. (2006). Intelligent system for process supervision 
and fault diagnosis in dynamic physical systems. Industrial Electronics, IEEE 
Transactions on, 53(2), 581-592. 
Macbeth, S. A., & Moroney, W. F. (1994). Development and Evaluation of Automobile 
Symbols: The Focus Group Approach. Human Factors and Ergonomics Society 
Annual Meeting Proceedings, 38, 978. 
Macbeth, S. A., Moroney, W. F., & Biers, D. W. (2000). Development and Evaluation of 
Symbols and Icons: a Comparison of the Production and Focus Group Methods. 
Human Factors and Ergonomics Society Annual Meeting Proceedings, 44, 327-
329. 
 116 
Macbeth, S. A., Moroney, W. F., & Biers, D. W. (2006). Developing Icons and Symbols: 
A Comparison of Two Methods. Ergonomics in Design, 14, 14-18. 
Manning, C. D., Raghavan, P., & Schutze, H. (2008). An Introduction to Information 
Retrieval. Cambridge, England: Cambridge University Press. 
Mayer, D. L., & Laux, L. F. (1989). Recognizability and Effectiveness of Warning 
Symbols and Pictorials. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 33, 984-988. 
Mayhorn, C. B., & Goldsworthy, R. C. (2007). Refining teratogen warning symbols for 
diverse populations. Birth Defects Research Part A: Clinical and Molecular 
Teratology, 79(6), 494-506. 
McDougall, S. J., Curry, M. B., & de Bruijn, O. (1999). Measuring symbol and icon 
characteristics: norms for concreteness, complexity, meaningfulness, familiarity, 
and semantic distance for 239 symbols. Behav Res Methods Instrum Comput, 
31(3), 487-519. 
McDougall, S. J., de Bruijn, O., & Curry, M. B. (2000). Exploring the effects of icon 
characteristics on user performance: The role of icon concreteness, complexity, 
and distinctiveness. Journal of Experimental Psychology: Applied, 6(4), 291-306. 
Miller, J. M., & Lehto, M. R. (2001). Warnings & Safety Instructions (4th ed.). Ann 
Arbor, MI: Fuller Technical Publications. 
Mudd, S. A., & Karsh, R. (1961). Design of a picture language to identify vehicle 
controls. 1. General method. 2. Investigation of population stereotypes. Retrieved 
12/16/2008. from http://handle.dtic.mil/100.2/AD272263. 
 117 
Muhdi, R., Garrett, A., Agarwal, R., Davis, J., Dozier, G., & Umphress, D. (2006). The 
Application of Evolutionary Computation in Evacuation Planning. Paper 
presented at the Intelligent Transportation Systems Conference, 2006. ITSC '06. 
IEEE. 
Murray, L. A., Magurno, A. B., Glover, B. L., & Wogalter, M. S. (1998). Prohibitive 
pictorials:: Evaluations of different circle-slash negation symbols. International 
Journal of Industrial Ergonomics, 22(6), 473-482. 
Newby, R., Soutar, G., & Watson, J. (2003). Comparing traditional focus groups with a 
Group Support Systems (GSS) approach for use in SME research. [Article]. 
International Small Business Journal, 21(4), 421-433. 
Nomen, R., Sempere, J., Pey, A., & Alvarez, A. (2003). Solving the seveso II prediction 
problem of substance generation during loss of control by Al techniques. Journal 
of Loss Prevention in the Process Industries, 16(1), 55-60. 
Northage, C. (2005). EASEing into the Future. Annals of Occupational Hygiene, 49, 99-
101. 
OSHA. (1996). Specifications for accident prevention signs and tags (Vol. 29.1910.145): 
Occupational Safety and Health Administration. 
Otsubo, S. M. (1988). A behavioral study of warning labels for consumer products: 
perceived danger and use of pictographs. Human Factors and Ergonomics Society 
Annual Meeting Proceedings, 32, 536-540. 
Parmee, I. C., Abraham, J. A. R., & Machwe, A. (2008). User-Centric Evolutionary 
Computing: Melding Human and Machine Capability to Satisfy Multiple Criteria. 
In Multiobjective Problem Solving from Nature (pp. 263-283). 
 118 
Pett, M. A. (1997). Nonparametric Statistics for Health Care Research: Statistics for 
Small Samples and Unusual Distributions. Thousand Oaks, CA: Sage 
Publications, Inc. 
Pettendorfer, M., & Mont'alvao, C. R. (2006). Safety symbols: focus group combined with 
production test as a research method. Paper presented at the 16th World Congress 
in Ergonomics / International Ergonomics Association IEA.  
Piper, A. K., Boelhouwer, E. J., & Davis, J. (2008). Using hand drawn images to 
determine warning symbol design parameters within interactive evolutionary 
computation software. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 52, 1708-1712. 
Raman, M. (2004). English Language Teaching. New Delhi: Atlantic Publishers and 
Distributors. 
Rees, J., & Koehler, G. J. (2006). Learning genetic algorithm parameters using hidden 
Markov models. European Journal of Operational Research, 175(2), 806-820. 
Ringseis, E. L., & Caird, J. K. (1995). The Comprehensibilitly and Legibility of Twenty 
Pharmaceutical Warning Pictograms. Human Factors and Ergonomics Society 
Annual Meeting Proceedings, 39, 974-978. 
Roberts, N. J., Mohamed, Z., Wong, P.-S., Johnson, M., Loh, L.-C., & Partridge, M. R. 
(2009). The development and comprehensibility of a pictorial asthma action plan. 
Patient Education and Counseling, 74(1), 12-18. 
Rogers, W. A., Lamson, N., & Rousseau, G. K. (2000). Warning Research: An 
Integrative Perspective. Human Factors: The Journal of the Human Factors and 
Ergonomics Society, 42, 102-139. 
 119 
Rousseau, G. K., & Wogalter, M. S. (2006). Research on warning signs. In M. S. 
Wogalter (Ed.), Handbook of Warnings (pp. 147-158). Mahwah, New Jersey: 
Lawrence Erlbaum Associates. 
Roy, R., Hinduja, S., & Teti, R. (2008). Recent advances in engineering design 
optimisation: Challenges and future trends. CIRP Annals - Manufacturing 
Technology, 57(2), 697-715. 
Rupela, V., & Dozier, G. (2002). Parallel and distributed evolutionary computations for 
multimodal function optimization. Paper presented at the Automation Congress, 
2002 Proceedings of the 5th Biannual World. 
Russo, P., & Boor, S. (1993). How fluent is your interface?: designing for international 
users. Paper presented at the Proceedings of the INTERACT '93 and CHI '93 
conference on Human factors in computing systems.  
Saaty, T. L. (1986). Axiomatic foundation of the analytic hierarchy process. Management 
Science, 32(7), 841-855. 
Sanders, M. S., & McCormick, E. J. (1993). Human Factors in Engineering and Design 
(Seventh ed.). New York: McGraw-Hill. 
Sayer, J. R., & Green, P. (1988). Current IS0 Automotive S vs Alternatives: A Preference 
Studv (SAE paper 880057). In SAE Special Publication SP-752. Warrendale, PA: 
Society of Automotive Engineers. 
Sayer, T. B. (2002). The Development and Evaluation of Icons for Side Obstacle 
Warning Systems. Human Factors and Ergonomics Society Annual Meeting 
Proceedings, 46, 1794-1798. 
 120 
Schuler, D., & Namioka, A. (1993). Participatory design: principles and practices. 
Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. 
Shieh, K.-k., & Huang, S.-m. (2003). Factors affecting preference ratings of prohibitive 
symbols. Applied Ergonomics, 34(6), 581-587. 
Sloan, G., & Eshelman, P. (1981). The development and evaluation of pictographic 
symbols. Human Factors and Ergonomics Society Annual Meeting Proceedings, 
25, 198-202. 
Smith-Jackson, T. L. (2006). Culture and Warnings. In M. S. Wogalter (Ed.), Handbook 
of Warnings (pp. 363-371). Mahwah, New Jersey: Lawrence Erlbaum Associates. 
Smith-Jackson, T. L., & Wogalter, M. S. (2006). Methods and procedures in warning 
research. In M. S. Wogalter (Ed.), Handbook of Warnings (pp. 23-33). Mahwah, 
New Jersey: Lawrence Erlbaum Associates. 
Smith-Jackson, T. L., & Wogalter, M. S. (2007). Application of a mental models 
approach to MSDS design. Theoretical Issues in Ergonomics Science, 8(4), 303-
319. 
Stanton, N. (1994). Human Factors in Alarm Design. London: Taylor and Francis. 
Sweeney, J. C., Soutar, G. N., Hausknecht, D. R., Dallin, R. F., & Johnson, L. W. (1997). 
Collecting information from groups: A comparison of two methods. [Article]. 
Journal of the Market Research Society, 39(2), 397-411. 
Takagi, H. (2001). Interactive evolutionary computation: fusion of the capabilities of EC 
optimization and human evaluation. Proceedings of the IEEE, 89(9), 1275-1296. 
Teknomo, K. (2006). Analytic Hierarchy Process (AHP) Tutorial.   Retrieved July 6, 
2009, from http://people.revoledu.com/kardi/tutorial/ahp/ 
 121 
Turnbull, D., Liu, R., Barringon, L., & Lanckrie, G. (2007). A game-based approach for 
collecting semantic annotations of music. Paper presented at the International 
Symposium on Music Information Retrieval. 
Vasconcelos, N., & Lippman, A. (2000a). A Probabilistic Architecture for Content-based 
Image Retrieval. Paper presented at the IEEE Conference Computer Vision and 
Pattern Recognition, Hilton Head, North Carolina. 
Vasconcelos, N., & Lippman, A. (2000b). A Unifying View of Image Similarity. Paper 
presented at the 15th International Conference on Pattern Recognition, Barcelona, 
Spain. 
Wisniewski, E. C., Isaacson, J. J., & Hall, S. M. (2007). Development and Evaluation of 
Safety Symbols for Landscaping Products. Human Factors and Ergonomics 
Society Annual Meeting Proceedings, 51, 1109-1113. 
Wogalter, M. S. (2006). Purposes and Scope of Warnings. In M. S. Wogalter (Ed.), 
Handbook of Warnings (pp. 3-9). Mahwah, New Jersey: Lawrence Erlbaum 
Associates. 
Wogalter, M. S., Dejoy, D. M., & Laughery, K. R. (1999). Organizing theoretical 
framework: a consolidated Communication - Human Information Processing (C-
HIP) model. In M. S. Wogalter, D. M. Dejoy & K. R. Laughery (Eds.), Warnings 
and Risk Communication (pp. 15-24). Philadelphia: Taylor and Francis, Inc. 
Wogalter, M. S., Racicot, B. M., Kalsher, M. J., & Noel Simpson, S. (1994). 
Personalization of warning signs: The role of perceived relevance on behavioral 
compliance. International Journal of Industrial Ergonomics, 14(3), 233-242. 
 122 
Wogalter, M. S., Silver, N. C., Leonard, S. D., & Zaikina, H. (2006). Warning symbols. 
In M. S. Wogalter (Ed.), Handbook of Warnings (pp. 159-176). Mahwah, New 
Jersey: Lawrence Erlbaum Associates. 
Wolff, J. S. (1995). A Study of the Effect of Context and Test Method in Evaluating Safety 
Symbols. Georgia Institute of Technology, Atlanta. 
Young, S. L., & Wogalter, M. S. (2001). Predictors of pictorial symbol comprehension. 
Information Design Journal, 10, 124-132. 
Zio, E. (1996). On the use of the analytic hierarchy process in the aggregation of expert 
judgments. Reliability Engineering & System Safety, 53(2), 127-138. 
Zwaga, H. J., & Mijksenaar, P. (2000). The Development and Standardization of 
Warning Symbols; The Role of Design and Human Factors. Human Factors and 
Ergonomics Society Annual Meeting Proceedings, 44, 782-785. 
 
 
 123 
 
 
 
 
 
APPENDICES
 124 
APPENDIX 3.1 
 
Survey of ease of conversion from written referent to graphical symbol for nine referents 
compared pair-wise. 
 
 125 
 
 126 
 
 127 
 
 128 
 
 129 
 
 130 
 
 131 
 
 132 
 
 133 
 
 134 
 
 135 
 
 136 
 
 137 
 
 138 
 
 139 
 
 140 
 141 
 
 142 
 
 143 
 
 144 
 
 145 
 
 146 
 
 147 
APPENDIX 3.2 
 
Information letter approved by the auburn university institutional review board for 
conducting surveys of ease of referent-to-symbol conversion. 
 148 
 
 149 
APPENDIX 3.3 
Reciprocal table with a sample set of pairwise comparison results taken from a single 
participant (participant #51). 
 
1 2 3 4 5 6 7 8 9 Sum Rank
1. Walk Down Stairs Backwards 1 1/5 1/5 1 1 3.4 7
2. No Access for Persons with Metallic Implants 5 1 1 1 5 13 2.5
3. Steel-toed Shoes Required 1 1/5 5 5 1 12.2 4
4. Disconnect Main Plug from Electrical Outlet 1 5 1 5 5 17 1
5. No Reaching In 1/5 1/5 1 1/5 1/5 1.8 9
6. Do Not Touch with Wet Hands 5 1/5 1/5 5 1 11.4 5
7. Hot Exhaust 1 5 1 1 5 13 2.5
8. Warning: Flooring Surface Changes 1 1 1 1 5 9 6
9. Confined Space; Entry by Permit Only 1 1/5 1/5 1/5 1 2.6 8
 
A value of ?5? in the table means that the referent in the row (horizontal) is considered 
more difficult than the referent in the column (vertical) by direct comparison of a single 
participant.  A value of ?1/5? means that the referent in the column was considered more 
difficult than the referent in the row by direct comparison.  Values of ?1? in the table 
mean that both referents were considered equally difficult; a referent is always equally 
difficult when compared to itself.  
 
Red cells signify that the referent listed in the intersecting row was presented to the user 
first and the referent listed in the intersecting column was presented second in the survey 
for their comparison.  The opposite is true for the non-red numerical cells in the table.  
Finally, shaded cells represent comparisons that were not directly measured in this 
balanced, incomplete block design. 
 150 
APPENDIX 3.4 
Pairwise comparison of referent difficulty converted to ranks 
Stairs Implants Steel-toes Disconnect No Reaching Wet Hands Exhaust Flooring Confined Space
1 Certified SP 3 3 7 6 8 1 3 5 9
2 Certified SP 5.5 1.5 5.5 9 4 3 7 1.5 8
3 Certified SP 7.5 1.5 7.5 4 7.5 3 5 1.5 7.5
4 Certified SP 8.5 1.5 1.5 8.5 3.5 5 6.5 3.5 6.5
5 Certified SP 8.5 1.5 6 7 3.5 1.5 8.5 5 3.5
6 Certified SP 5.5 1.5 3 9 1.5 7.5 7.5 5.5 4
7 Certified SP 6.5 1 6.5 2.5 2.5 8.5 8.5 4.5 4.5
8 Certified SP 7.5 4 7.5 4 7.5 1.5 4 1.5 7.5
9 Certified SP 8.5 1.5 6.5 3.5 6.5 3.5 8.5 1.5 5
10 Certified SP 5.5 7 8.5 3.5 3.5 1.5 5.5 1.5 8.5
11 Certified SP 5.5 3 7 3 8.5 1 3 5.5 8.5
12 Certified SP 6 3 4.5 7 8 2 4.5 1 9
13 Certified SP 6.5 3.5 1 6.5 8.5 3.5 3.5 8.5 3.5
14 Certified SP 7 2.5 9 4.5 6 4.5 8 2.5 1
15 Certified SP 6 4 1 9 4 8 4 7 2
16 Certified SP 9 2 5.5 7.5 7.5 1 5.5 3 4
17 Certified SP 9 1 6 6 4 3 8 6 2
18 Certified SP 8.5 1 4 6 4 4 8.5 7 2
19 Certified SP 9 3.5 8 3.5 6 1.5 7 5 1.5
20 Certified SP 8 3.5 8 1 6 3.5 8 3.5 3.5
21 Certified SP 5 1 8.5 3.5 6.5 3.5 6.5 2 8.5
22 Certified SP 7 2.5 9 4 8 1 6 2.5 5
23 Certified SP 9 1.5 7.5 5 6 1.5 7.5 3 4
24 Certified SP 9 1.5 5 5 5 3 7.5 1.5 7.5
25 Certified SP 2 6.5 8.5 2 4.5 6.5 2 4.5 8.5
26 Certified SP 5.5 2.5 8 2.5 7 4 5.5 1 9
27 Certified SP 4 1.5 7 7 5 1.5 7 3 9
28 Certified SP 8 1 8 5.5 8 5.5 3.5 3.5 2
29 Certified SP 8.5 2 8.5 4 6 2 6 2 6
30 Certified SP 9 4.5 7.5 4.5 6 1.5 7.5 3 1.5
31 Certified SP 9 1.5 7.5 4 6 4 7.5 4 1.5
32 Certified SP 8.5 1 4 8.5 6 2 7 5 3
33 Certified SP 5 4 7.5 1.5 7.5 3 7.5 7.5 1.5
34 Certified SP 7 4 8.5 6 5 1.5 8.5 3 1.5
35 Certified SP 6 2.5 8.5 2.5 4.5 4.5 8.5 1 7
36 Certified SP 8.5 2.5 5.5 1 8.5 5.5 7 4 2.5
37 Certified SP 8.5 1.5 8.5 4 7 3 6 5 1.5
38 Certified SP 2 8.5 4 4 6.5 6.5 1 8.5 4
39 Certified SP 9 2 5 6 7.5 2 7.5 4 2
40 Certified SP 9 1 8 4 6.5 5 6.5 2.5 2.5
41 Certified SP 9 2.5 2.5 6 7.5 2.5 7.5 5 2.5
42 Certified SP 9 5 7 3 8 1 3 6 3
43 Certified SP 8 1 3 9 5 2 6.5 6.5 4
44 Certified SP 6.5 2.5 4 8.5 6.5 1 5 8.5 2.5
45 Certified SP 7 1 3.5 6 9 2 5 3.5 8
46 Certified SP 3 3 8.5 5.5 7 1 5.5 3 8.5
47 Uncertified SP 8 1.5 8 6 4 1.5 8 4 4
48 Uncertified SP 8.5 1.5 8.5 5 6.5 1.5 6.5 3.5 3.5
49 Uncertified SP 8.5 2 8.5 4.5 4.5 2 6.5 2 6.5
50 Uncertified SP 1.5 3 7.5 5 5 1.5 7.5 9 5
51 Uncertified SP 7 2.5 4 1 9 5 2.5 6 8
52 Uncertified SP 6.5 1.5 8.5 5 3.5 3.5 8.5 1.5 6.5
53 Uncertified SP 8.5 2.5 2.5 7 8.5 1 5 5 5
54 Uncertified SP 4.5 1.5 4.5 3 7.5 1.5 7.5 7.5 7.5
55 Uncertified SP 5 3 8 4 8 1 6 2 8
56 Uncertified SP 6.5 1.5 9 3.5 6.5 3.5 6.5 1.5 6.5
57 Uncertified SP 7.5 2.5 9 6 2.5 5 7.5 2.5 2.5
58 Uncertified SP 8 2.5 1 8 6 4 8 5 2.5
59 Uncertified SP 8.5 2 4 8.5 2 5 7 6 2
60 Uncertified SP 6 1.5 3 6 8.5 4 8.5 1.5 6
61 Uncertified SP 8.5 1.5 6.5 5 8.5 1.5 6.5 3.5 3.5
62 Uncertified SP 5.5 2 9 5.5 1 3 8 4 7
63 Uncertified SP 9 2 1 8 7 3 5 5 5
Participant Referent RanksStratum
 
 151 
Stairs Implants Steel-toes Disconnect No Reaching Wet Hands Exhaust Flooring Confined Space
64 Uncertified SP 7.5 3.5 9 5 6 1 3.5 2 7.5
65 Uncertified SP 9 2.5 7 4.5 7 1 4.5 7 2.5
66 Uncertified SP 7.5 1 9 4.5 6 2.5 4.5 2.5 7.5
67 Uncertified SP 8.5 3.5 8.5 6.5 1.5 3.5 5 1.5 6.5
68 Uncertified SP 6 2 8.5 6 4 2 6 2 8.5
69 Uncertified SP 5 4 7 7 3 1.5 9 7 1.5
70 Uncertified SP 7 3 9 4.5 7 1.5 4.5 7 1.5
71 Uncertified SP 8.5 2 7 5 6 2 8.5 2 4
72 Uncertified SP 7 4 8.5 6 5 1.5 8.5 3 1.5
73 Uncertified SP 7 1.5 8.5 5 6 3.5 8.5 1.5 3.5
74 Uncertified SP 8.5 2 6.5 4.5 8.5 2 4.5 2 6.5
75 Uncertified SP 5 2 8.5 3.5 3.5 6.5 6.5 1 8.5
76 Uncertified SP 9 4 2.5 5.5 8 2.5 5.5 7 1
77 Uncertified SP 2 2 8 6.5 2 4 5 6.5 9
78 Uncertified SP 8.5 2 8.5 5 2 4 6.5 2 6.5
79 Uncertified SP 8.5 1.5 6 8.5 1.5 4 7 5 3
80 Uncertified SP 7.5 6 9 4 5 2 2 2 7.5
81 Uncertified SP 9 4.5 7 2.5 8 1 6 2.5 4.5
82 Uncertified SP 9 2.5 8 7 6 1 5 4 2.5
83 Uncertified SP 8.5 1 6 3 8.5 4 5 2 7
84 Uncertified SP 7.5 2 1 7.5 7.5 3 4 7.5 5
85 Uncertified SP 5 1.5 5 7 8.5 3 5 1.5 8.5
86 Uncertified SP 8 3 5 3 8 6 8 3 1
87 Uncertified SP 8 2.5 9 2.5 5 1 6 4 7
88 Uncertified SP 7.5 1 5 9 6 2 7.5 4 3
89 Uncertified SP 9 4 7.5 4 6 1.5 7.5 1.5 4
90 Uncertified SP 6 4.5 8.5 1.5 3 4.5 8.5 7 1.5
91 University 8.5 3 3 7 8.5 1 6 3 5
92 University 3.5 1.5 6 6 3.5 1.5 9 8 6
93 University 3.5 6.5 3.5 8.5 1.5 6.5 5 8.5 1.5
94 University 3.5 1 2 7 7 7 3.5 5 9
95 University 6 2 7 8 4.5 1 9 3 4.5
96 University 9 4.5 1 3 6.5 8 2 6.5 4.5
97 University 4.5 6 7 1 8.5 3 8.5 2 4.5
98 University 8.5 1.5 6.5 6.5 5 4 8.5 3 1.5
99 University 9 1 2 7 6 5 8 3.5 3.5
100 University 8.5 1.5 4 8.5 6 1.5 6 6 3
101 University 8.5 2 5.5 8.5 2 5.5 7 4 2
102 University 1.5 4 3 8 1.5 8 6 5 8
103 University 7 1.5 3.5 8.5 5 1.5 8.5 6 3.5
104 University 7.5 2.5 7.5 4 2.5 5 9 6 1
105 University 8.5 1.5 3 7 5.5 5.5 8.5 4 1.5
106 University 6.5 1.5 8.5 4 3 6.5 5 1.5 8.5
107 University 8 2.5 7 5.5 2.5 5.5 9 4 1
108 University 8.5 4 8.5 7 1.5 3 6 1.5 5
109 University 4 1 4 7 2 7 7 4 9
110 University 8.5 1 3.5 6.5 6.5 3.5 8.5 3.5 3.5
111 University 9 4 1 8 7 2 4 6 4
112 University 1.5 6 8.5 1.5 7 4.5 8.5 3 4.5
113 University 4.5 1.5 9 3 7 4.5 7 1.5 7
114 University 9 1.5 5 7.5 6 1.5 7.5 4 3
115 University 8.5 2.5 6 8.5 1 4.5 7 2.5 4.5
116 University 8.5 2 8.5 4.5 6.5 2 4.5 2 6.5
117 University 2 5.5 2 5.5 2 9 4 7 8
118 University 8 1.5 3.5 8 5.5 3.5 8 1.5 5.5
119 University 9 5 5 2 5 3 7.5 7.5 1
120 University 8.5 3 5 6.5 8.5 1 6.5 3 3
121 University 8.5 1 4 8.5 2.5 5 6.5 2.5 6.5
122 University 8 1.5 7 4 6 5 9 3 1.5
123 University 8.5 1 3 8.5 7 2 5.5 5.5 4
124 University 7 3.5 3.5 8.5 8.5 1.5 5.5 5.5 1.5
125 University 4 1.5 1.5 8 7 5 3 9 6
126 University 8 1.5 3 4.5 8 6 8 1.5 4.5
127 University 9 2.5 6.5 5 6.5 1 8 4 2.5
128 University 8.5 1 6.5 8.5 2.5 2.5 6.5 4 5
129 University 7 1.5 9 6 3 1.5 8 4 5
Participant Stratum Referent Ranks
 152 
Stairs Implants Steel-toes Disconnect No Reaching Wet Hands Exhaust Flooring Confined Space
130 University 1 9 7 3 2 5 5 5 8
131 University 8.5 4 6.5 2 8.5 2 5 2 6.5
132 University 1.5 5 7.5 6 1.5 7.5 4 3 9
133 University 4.5 1 2 8 6.5 6.5 9 3 4.5
134 University 9 5 6 1.5 7.5 3.5 7.5 3.5 1.5
135 University 8.5 2.5 5 6.5 8.5 1 6.5 2.5 4
136 University 8.5 1.5 8.5 3.5 7 3.5 1.5 5.5 5.5
137 University 6.5 2.5 6.5 5 1 8.5 8.5 4 2.5
138 University 4.5 1 2 8 6.5 3 4.5 6.5 9
139 University 8.5 2.5 7 2.5 5 6 8.5 1 4
140 University 3 6 7.5 3 1 7.5 9 3 5
141 University 8.5 1.5 1.5 6.5 8.5 3.5 6.5 5 3.5
142 University 8.5 1.5 1.5 6.5 8.5 5 6.5 4 3
143 University 8.5 1.5 4 6 6 6 8.5 1.5 3
144 University 7 2 1 8.5 6 3 8.5 4.5 4.5
145 University 3 6 8.5 6 6 1 8.5 2 4
Participant Stratum Referent Ranks
 
 153 
APPENDIX 4.1 
Information letter for Phase 1 ? Symbol Proto-Drawings participants. 
 
.  
 154 
 
 155 
APPENDIX 4.2 
 
Instructions and Data Collection form for participants in the Symbol Proto-drawing 
experiment. 
 
The purpose of this research is to understand the kinds of symbols you prefer to see on a 
safety sign.  Safety signs are placed near the location of a hazard to communicate the risk 
to all see it.   
 
Your role in this study will be to draw a simple picture, or a symbol, that could be added 
to a sign to communicate a safety message without using any text at all.  This picture you 
draw should communicate each safety message I will give you as clearly and completely 
as possible.  However, do not worry about making a pretty or high-quality drawing. 
Artistic skill or well-drawn pictures are not important to this research.  As long as you 
can explain what you picture means, then it is just fine. 
 
You will be drawing your picture using a SmartBoard system, which includes four 
special marker pens (black, red, green and blue) and a special eraser.  Please do not use 
any other marker pens but the ones provided.  To erase your drawing, simply pick up the 
eraser and wipe away the marks you want to remove.  Remember, though, in order to 
draw with the markers again, the eraser must be returned to its home.   
 
You will be given three different safety messages to draw, one at a time.  To help you, 
you will be given a description of the hazards and locations where symbols like your 
drawing may be needed.   You may take up to 15 minutes to draw each picture, and the 
researchers will not be able to see your drawing until you are ready.  The researchers will 
remind you periodically of the time remaining on each picture, although you may have 
more time if you need it.  Whenever you are satisfied with your drawing, inform the 
researcher that it is complete.  After you have completed three symbols in this manner, 
the exercise will be finished.   
 
Please avoid discussing the details of your drawing ideas with anyone who you think 
might participate in this study to ensure that their results remain unbiased. 
 
Thank you for your cooperation!  Please complete the information below before you 
begin the activity.   
 
 
Age _________    Gender (circle one):  M  /  F 
In what country were you born? ____________________________ 
For how many years did you live in your birth country? __________________________ 
What country do you consider to be your home country?  _________________________ 
What language do you speak in your home most often?  __________________________ 
Do you consider yourself to speak English fluently? (circle one)  Yes  / No 
At what age did you first begin reading or speaking English fluently? ________________ 
 156 
APPENDIX 4.3 
To create the semantic annotation matrix for a referent, each panelist followed the 
following procedure. 
 
1. Determine whether newly revealed symbol exhibits critical confusion or is 
completely non-relevant (Egregious Error) 
a. Determine by majority opinion 
b. If true, set symbol aside and skip to Step 4.  If false, continue to Step 2. 
2. Which attributes already contained within the matrix are present in this 
symbol? 
a. Individually and privately recorded on data collection form. 
3. Does this symbol add new attributes to the matrix for this referent? 
a. Determine by majority opinion. 
b. If none, skip to Step 4. 
c. Presence of new attributes are individually and privately recorded on data 
collection forms 
4. Move to next symbol in this referent 
5. When all symbols are complete, move to new referent. 
6. When all referents are complete, end process. 
 
15
7 
AP
PE
ND
IX
 4.
4 
Da
ta 
co
lle
cti
on
 fo
rm
 us
ed
 by
 pa
ne
lis
ts 
to 
pe
rfo
rm
 se
ma
nti
c a
nn
ota
tio
n o
f d
raw
ing
s. 
Panelist:
                                                                        Image Name
Attributes
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
Date:Referent: Page #:
 158 
APPENDIX 4.5 
Symbol proto-drawings produced by the U.S. stratum for ?Hot Exhaust? 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2 3 5 10 
11 13 17 18 
24 28 29 30 
19 21 22 23 
32 33 36 37 
40 41 42 44 
 159 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47 50 51 52 
53 56 60 61 
62 
67 
70 
 160 
Symbol proto-drawings produced by the Indian stratum for ?Hot Exhaust?  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1 4 6 7 
8 9 12 14 
15 16 20 25 
26 27 31 34 
35 38 39 43 
45 46 48 49 
 161 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54 55 57 58 
59 63 64 65 
66 
68 
69 
 162 
Symbol proto-drawings produced by the U.S. stratum for ?Do Not Touch with Wet 
Hands?  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1 2 5 7 
8 10 11 13 
18 19 20 22 
23 24 25 26 
28 29 31 36 
38 39 40 43 
 163 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48 49 56 
57 
58 61 63 65 
66 
67 68 70 
 164 
Symbol proto-drawings produced by the Indian stratum for ?Do Not Touch with Wet 
Hands?  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3 4 6 9 
12 14 15 16 
17 21 27 30 
32 33 34 35 
37 41 42 44 
45 
46 47 50 
 165 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51 52 53 54 
55 59 60 62 
64 
69 
 166 
APPENDIX 4.6 
Consensus attribute matrix produced by summing the three individual panelists? attribute 
matrices for ?Do Not Touch with Wet Hands?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attribute U.S. U.S. Indian Indian U.S. Indian U.S. U.S. Indian U.S. U.S. Indian U.S. Indian Indian
A Single Hand EE EE 3 3 3 3 3 3 3 3 3 3 3 3 3
B 1-D Surface EE EE 3 2 0 0 2 0 0 0 0 0 0 0 0
C Multiple Water Drops EE EE 3 3 3 3 3 3 3 3 0 1 3 3 3
D Prohibition Symbol EE EE 3 3 0 0 3 0 0 0 3 0 3 0 0
E 2nd Color EE EE 3 3 3 3 3 3 3 3 3 3 2 3 2
F Skull/Crossbones EE EE 0 3 0 0 0 0 0 0 0 0 0 0 0
G Faucet EE EE 0 3 0 0 0 3 0 0 0 0 0 3 0
H Prohibition X EE EE 0 0 3 3 0 3 3 0 0 3 0 3 3
I Liquid Container EE EE 0 0 0 3 0 0 0 0 0 0 0 0 0
J 2-D Panel EE EE 0 0 0 0 3 0 0 0 0 1 0 3 0
K Lightning Bolts EE EE 0 0 0 0 0 0 0 3 0 1 0 0 0
L Single Water Drop EE EE 0 0 0 0 0 0 0 0 3 3 0 0 0
M 3-D Object EE EE 0 0 0 0 0 0 0 0 0 0 3 0 0
N Multi-Panel EE EE 0 0 0 0 0 0 0 0 0 0 0 3 0
O Water Ripple EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
P Spark EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
Q Single Lightning Bolt EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
R Energized Equipment EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
S Two Hands EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
T Puddle EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
U Person EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
V Sequence Arrow EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
W Rain Cloud EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
X Surprised Face EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
Y Permissable Circle EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
Z Happy Face EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
AA Mr. Sparky EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0
BB Heat Waves EE EE 0 0 0 0 0 0 0 0 0 0 0 0 0  
 
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attribute Indian Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. U.S. Indian U.S. U.S. Indian
A Single Hand 3 3 3 3 3 3 3 1 3 3 0 3 3 3 3
B 1-D Surface 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
C Multiple Water Drops 3 3 3 3 0 3 3 3 3 3 2 3 3 3 3
D Prohibition Symbol 1 2 0 3 3 3 3 0 3 0 3 0 3 3 0
E 2nd Color 2 3 3 3 3 0 3 3 3 0 3 3 2 3 3
F Skull/Crossbones 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3
G Faucet 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0
H Prohibition X 2 0 3 0 0 0 0 3 0 3 0 3 0 0 0
I Liquid Container 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
J 2-D Panel 0 0 3 0 3 0 0 0 0 0 0 0 3 0 0
K Lightning Bolts 0 0 0 0 1 0 0 3 0 0 0 0 0 0 0
L Single Water Drop 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
M 3-D Object 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
N Multi-Panel 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
O Water Ripple 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0
P Spark 0 0 0 0 3 0 0 0 0 0 0 0 0 0 3
Q Single Lightning Bolt 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
R Energized Equipment 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0
S Two Hands 0 0 0 0 0 0 0 3 0 0 2 0 0 0 0
T Puddle 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0
U Person 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0
V Sequence Arrow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
W Rain Cloud 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
X Surprised Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Y Permissable Circle 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Z Happy Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
AA Mr. Sparky 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
BB Heat Waves 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
 167 
 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attribute U.S. Indian Indian Indian Indian U.S. Indian U.S. U.S. U.S. Indian Indian U.S. Indian Indian
A Single Hand 3 0 0 3 3 3 CC 3 3 3 3 3 1 3 CC
B 1-D Surface 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
C Multiple Water Drops 3 3 3 3 3 3 CC 3 3 3 3 3 3 3 CC
D Prohibition Symbol 3 0 0 0 0 3 CC 3 3 3 0 0 3 0 CC
E 2nd Color 3 3 3 3 0 3 CC 3 3 3 3 3 0 3 CC
F Skull/Crossbones 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
G Faucet 0 3 0 0 0 0 CC 0 0 3 0 0 0 0 CC
H Prohibition X 0 3 3 3 3 0 CC 0 0 0 3 3 0 3 CC
I Liquid Container 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
J 2-D Panel 0 0 0 0 0 0 CC 3 0 0 0 0 0 0 CC
K Lightning Bolts 0 0 0 0 0 0 CC 3 0 0 0 0 0 0 CC
L Single Water Drop 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
M 3-D Object 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
N Multi-Panel 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
O Water Ripple 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
P Spark 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
Q Single Lightning Bolt 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
R Energized Equipment 0 0 0 0 0 0 CC 0 0 0 0 0 0 3 CC
S Two Hands 0 2 2 0 0 0 CC 0 0 0 0 0 2 0 CC
T Puddle 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
U Person 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
V Sequence Arrow 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
W Rain Cloud 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
X Surprised Face 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
Y Permissable Circle 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
Z Happy Face 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
AA Mr. Sparky 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC
BB Heat Waves 0 0 0 0 0 0 CC 0 0 0 0 0 0 0 CC  
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attribute Indian Indian U.S. U.S. Indian Indian Indian Indian Indian Indian U.S. U.S. U.S. Indian Indian
A Single Hand 3 3 3 EE 3 0 3 3 3 3 1 3 3 CC EE
B 1-D Surface 0 0 0 EE 0 3 0 0 0 0 0 0 0 CC EE
C Multiple Water Drops 3 0 3 EE 3 3 3 3 3 3 3 3 3 CC EE
D Prohibition Symbol 0 2 3 EE 0 1 1 3 1 0 1 0 0 CC EE
E 2nd Color 3 3 3 EE 1 0 3 3 3 0 3 3 3 CC EE
F Skull/Crossbones 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
G Faucet 0 3 0 EE 0 0 0 0 0 0 0 0 0 CC EE
H Prohibition X 3 1 0 EE 3 2 2 0 3 3 2 3 3 CC EE
I Liquid Container 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
J 2-D Panel 0 3 0 EE 0 0 0 3 0 0 3 3 2 CC EE
K Lightning Bolts 0 0 3 EE 0 1 0 0 0 0 0 0 0 CC EE
L Single Water Drop 0 2 0 EE 0 0 0 0 0 0 0 0 0 CC EE
M 3-D Object 0 0 0 EE 0 0 0 0 3 0 0 0 0 CC EE
N Multi-Panel 0 3 0 EE 0 0 0 0 0 0 0 3 0 CC EE
O Water Ripple 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
P Spark 0 0 0 EE 0 2 0 0 0 0 0 0 3 CC EE
Q Single Lightning Bolt 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
R Energized Equipment 0 0 0 EE 3 0 0 0 0 3 0 0 1 CC EE
S Two Hands 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
T Puddle 0 0 0 EE 0 1 0 0 0 0 0 0 0 CC EE
U Person 0 0 0 EE 0 2 0 0 0 0 3 0 0 CC EE
V Sequence Arrow 0 3 0 EE 0 0 0 0 0 0 0 0 0 CC EE
W Rain Cloud 0 0 0 EE 0 0 0 3 0 0 0 0 0 CC EE
X Surprised Face 0 0 0 EE 0 0 0 0 0 0 3 0 0 CC EE
Y Permissable Circle 0 0 0 EE 0 0 0 0 0 0 0 2 0 CC EE
Z Happy Face 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
AA Mr. Sparky 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE
BB Heat Waves 0 0 0 EE 0 0 0 0 0 0 0 0 0 CC EE  
 
 168 
 
61 62 63 64 65 66 67 68 69 70
Attribute U.S. Indian U.S. Indian U.S. U.S. U.S. U.S. Indian U.S.
A Single Hand 3 2 EE 3 0 3 3 3 CC 3
B 1-D Surface 0 0 EE 0 0 0 0 0 CC 0
C Multiple Water Drops 3 3 EE 3 3 3 3 2 CC 3
D Prohibition Symbol 3 3 EE 0 0 3 1 0 CC 1
E 2nd Color 3 3 EE 3 3 3 3 0 CC 0
F Skull/Crossbones 0 1 EE 3 2 0 0 0 CC 0
G Faucet 0 2 EE 0 3 0 0 2 CC 0
H Prohibition X 0 0 EE 0 3 0 2 0 CC 2
I Liquid Container 0 0 EE 2 0 0 3 0 CC 0
J 2-D Panel 0 3 EE 2 3 0 0 0 CC 0
K Lightning Bolts 0 0 EE 2 0 0 0 1 CC 0
L Single Water Drop 0 0 EE 0 0 0 0 0 CC 0
M 3-D Object 3 0 EE 0 0 0 0 0 CC 0
N Multi-Panel 0 3 EE 3 0 3 0 3 CC 0
O Water Ripple 0 0 EE 0 0 0 3 0 CC 0
P Spark 0 0 EE 3 0 0 0 2 CC 0
Q Single Lightning Bolt 0 3 EE 0 0 0 0 0 CC 0
R Energized Equipment 0 0 EE 3 0 0 0 3 CC 0
S Two Hands 0 3 EE 0 0 0 0 0 CC 0
T Puddle 0 0 EE 0 0 0 0 0 CC 0
U Person 0 0 EE 0 1 0 0 0 CC 0
V Sequence Arrow 0 0 EE 0 0 0 0 3 CC 0
W Rain Cloud 0 0 EE 0 0 0 0 0 CC 0
X Surprised Face 0 0 EE 0 0 0 0 0 CC 0
Y Permissable Circle 0 0 EE 0 0 0 0 0 CC 0
Z Happy Face 0 0 EE 0 0 3 0 0 CC 0
AA Mr. Sparky 0 0 EE 0 0 0 0 3 CC 0
BB Heat Waves 0 0 EE 0 0 0 0 2 CC 0
 169 
Attribute matrix produced by Panelist #1 for ?Do Not Touch with Wet Hands?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attribute U.S. U.S. Indian Indian U.S. Indian U.S. U.S. Indian U.S. U.S. Indian U.S. Indian Indian
A Single Hand EE EE 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface EE EE 1 1 1
C Multiple Water Drops EE EE 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol EE EE 1 1 1 1 1
E 2nd Color EE EE 1 1 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones EE EE 1
G Faucet EE EE 1 1 1
H Prohibition X EE EE 1 1 1 1 1 1 1
I Liquid Container EE EE 1
J 2-D Panel EE EE 1 1
K Lightning Bolts EE EE 1 1
L Single Water Drop EE EE 1 1
M 3-D Object EE EE 1
N Multi-Panel EE EE 1
O Water Ripple EE EE
P Spark EE EE
Q Single Lightning Bolt EE EE
R Energized Equipment EE EE
S Two Hands EE EE
T Puddle EE EE
U Person EE EE
V Sequence Arrow EE EE
W Rain Cloud EE EE
X Surprised Face EE EE
Y Permissable Circle EE EE
Z Happy Face EE EE
AA Mr. Sparky EE EE
BB Heat Waves EE EE  
 
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attribute Indian Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. U.S. Indian U.S. U.S. Indian
A Single Hand 1 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface
C Multiple Water Drops 1 1 1 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones 1
G Faucet 1
H Prohibition X 1 1 1 1
I Liquid Container
J 2-D Panel 1 1 1
K Lightning Bolts 1
L Single Water Drop 1
M 3-D Object
N Multi-Panel
O Water Ripple 1
P Spark 1 1
Q Single Lightning Bolt 1
R Energized Equipment 1
S Two Hands 1 1
T Puddle 1
U Person 1
V Sequence Arrow
W Rain Cloud
X Surprised Face
Y Permissable Circle
Z Happy Face
AA Mr. Sparky
BB Heat Waves  
 
 170 
 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attribute U.S. Indian Indian Indian Indian U.S. Indian U.S. U.S. U.S. Indian Indian U.S. Indian Indian
A Single Hand 1 1 1 1 CC 1 1 1 1 1 1 EE
B 1-D Surface CC EE
C Multiple Water Drops 1 1 1 1 1 1 CC 1 1 1 1 1 1 1 EE
D Prohibition Symbol 1 1 CC 1 1 1 1 EE
E 2nd Color 1 1 1 1 1 CC 1 1 1 1 1 1 EE
F Skull/Crossbones CC EE
G Faucet 1 CC 1 EE
H Prohibition X 1 1 1 1 CC 1 1 1 EE
I Liquid Container CC EE
J 2-D Panel CC 1 EE
K Lightning Bolts CC 1 EE
L Single Water Drop CC EE
M 3-D Object CC EE
N Multi-Panel CC EE
O Water Ripple CC EE
P Spark CC EE
Q Single Lightning Bolt CC EE
R Energized Equipment CC 1 EE
S Two Hands CC 1 EE
T Puddle CC EE
U Person CC EE
V Sequence Arrow CC EE
W Rain Cloud CC EE
X Surprised Face CC EE
Y Permissable Circle CC EE
Z Happy Face CC EE
AA Mr. Sparky CC EE
BB Heat Waves CC EE  
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attribute Indian Indian U.S. U.S. Indian Indian Indian Indian Indian Indian U.S. U.S. U.S. Indian Indian
A Single Hand 1 1 1 EE 1 1 1 1 1 1 1 1 CC EE
B 1-D Surface EE 1 CC EE
C Multiple Water Drops 1 1 EE 1 1 1 1 1 1 1 1 1 CC EE
D Prohibition Symbol 1 EE 1 1 1 1 CC EE
E 2nd Color 1 1 1 EE 1 1 1 1 1 1 CC EE
F Skull/Crossbones EE CC EE
G Faucet 1 EE CC EE
H Prohibition X 1 1 EE 1 1 1 1 1 CC EE
I Liquid Container EE CC EE
J 2-D Panel 1 EE 1 1 1 1 CC EE
K Lightning Bolts 1 EE CC EE
L Single Water Drop 1 EE CC EE
M 3-D Object EE 1 CC EE
N Multi-Panel 1 EE 1 CC EE
O Water Ripple EE CC EE
P Spark EE 1 1 CC EE
Q Single Lightning Bolt EE CC EE
R Energized Equipment EE 1 1 CC EE
S Two Hands EE CC EE
T Puddle EE 1 CC EE
U Person EE 1 1 CC EE
V Sequence Arrow 1 EE CC EE
W Rain Cloud EE 1 CC EE
X Surprised Face EE 1 CC EE
Y Permissable Circle EE CC EE
Z Happy Face EE CC EE
AA Mr. Sparky EE CC EE
BB Heat Waves EE CC EE  
 
 
 171 
 
61 62 63 64 65 66 67 68 69 70
Attribute U.S. Indian U.S. Indian U.S. U.S. U.S. U.S. Indian U.S.
Single Hand 1 1 EE 1 1 1 1 CC 1
1-D Surface EE CC
Multiple Water Drops 1 1 EE 1 1 1 1 1 CC 1
Prohibition Symbol 1 1 EE 1 1 CC 1
2nd Color 1 1 EE 1 1 1 1 CC
Skull/Crossbones EE 1 1 CC
Faucet 1 EE 1 CC
Prohibition X EE 1 CC
Liquid Container EE 1 CC
2-D Panel 1 EE 1 1 CC
Lightning Bolts EE CC
Single Water Drop EE CC
3-D Object 1 EE CC
Multi-Panel 1 EE 1 1 1 CC
Water Ripple EE 1 CC
Spark EE 1 1 CC
Single Lightning Bolt 1 EE CC
Energized Equipment EE 1 1 CC
Two Hands 1 EE CC
Puddle EE CC
Person EE CC
Sequence Arrow EE 1 CC
Rain Cloud EE CC
Surprised Face EE CC
Permissable Circle EE CC
Happy Face EE 1 CC
Mr. Sparky EE 1 CC
Heat Waves EE CC  
 
 
 
Attribute matrix produced by Panelist #2 for ?Do Not Touch with Wet Hands?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attribute U.S. U.S. Indian Indian U.S. Indian U.S. U.S. Indian U.S. U.S. Indian U.S. Indian Indian
A Single Hand EE EE 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface EE EE 1 1
C Multiple Water Drops EE EE 1 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol EE EE 1 1 1 1 1
E 2nd Color EE EE 1 1 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones EE EE 1
G Faucet EE EE 1 1 1
H Prohibition X EE EE 1 1 1 1 1 1 1
I Liquid Container EE EE 1
J 2-D Panel EE EE 1 1
K Lightning Bolts EE EE 1
L Single Water Drop EE EE 1 1
M 3-D Object EE EE 1
N Multi-Panel EE EE 1
O Water Ripple EE EE
P Spark EE EE
Q Single Lightning Bolt EE EE
R Energized Equipment EE EE
S Two Hands EE EE
T Puddle EE EE
U Person EE EE
V Sequence Arrow EE EE
W Rain Cloud EE EE
X Surprised Face EE EE
Y Permissable Circle EE EE
Z Happy Face EE EE
AA Mr. Sparky EE EE
BB Heat Waves EE EE  
 
 172 
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attribute Indian Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. U.S. Indian U.S. U.S. Indian
A Single Hand 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface
C Multiple Water Drops 1 1 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones 1
G Faucet 1
H Prohibition X 1 1 1 1 1
I Liquid Container
J 2-D Panel 1 1 1
K Lightning Bolts 1 1
L Single Water Drop 1
M 3-D Object
N Multi-Panel
O Water Ripple 1
P Spark 1 1
Q Single Lightning Bolt 1
R Energized Equipment 1
S Two Hands 1 1
T Puddle 1
U Person 1
V Sequence Arrow
W Rain Cloud
X Surprised Face
Y Permissable Circle
Z Happy Face
AA Mr. Sparky
BB Heat Waves  
 
 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attribute U.S. Indian Indian Indian Indian U.S. Indian U.S. U.S. U.S. Indian Indian U.S. Indian Indian
A Single Hand 1 1 1 1 CC 1 1 1 1 1 1 CC
B 1-D Surface CC CC
C Multiple Water Drops 1 1 1 1 1 1 CC 1 1 1 1 1 1 1 CC
D Prohibition Symbol 1 1 CC 1 1 1 1 CC
E 2nd Color 1 1 1 1 1 CC 1 1 1 1 1 1 CC
F Skull/Crossbones CC CC
G Faucet 1 CC 1 CC
H Prohibition X 1 1 1 1 CC 1 1 1 CC
I Liquid Container CC CC
J 2-D Panel CC 1 CC
K Lightning Bolts CC 1 CC
L Single Water Drop CC CC
M 3-D Object CC CC
N Multi-Panel CC CC
O Water Ripple CC CC
P Spark CC CC
Q Single Lightning Bolt CC CC
R Energized Equipment CC 1 CC
S Two Hands 1 1 CC 1 CC
T Puddle CC CC
U Person CC CC
V Sequence Arrow CC CC
W Rain Cloud CC CC
X Surprised Face CC CC
Y Permissable Circle CC CC
Z Happy Face CC CC
AA Mr. Sparky CC CC
BB Heat Waves CC CC  
 
 173 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attribute Indian Indian U.S. U.S. Indian Indian Indian Indian Indian Indian U.S. U.S. U.S. Indian Indian
A Single Hand 1 1 1 EE 1 1 1 1 1 1 1 EE EE
B 1-D Surface EE 1 EE EE
C Multiple Water Drops 1 1 EE 1 1 1 1 1 1 1 1 1 EE EE
D Prohibition Symbol 1 1 EE 1 1 EE EE
E 2nd Color 1 1 1 EE 1 1 1 1 1 1 EE EE
F Skull/Crossbones EE EE EE
G Faucet 1 EE EE EE
H Prohibition X 1 EE 1 1 1 1 1 1 1 1 EE EE
I Liquid Container EE EE EE
J 2-D Panel 1 EE 1 1 1 1 EE EE
K Lightning Bolts 1 EE EE EE
L Single Water Drop EE EE EE
M 3-D Object EE 1 EE EE
N Multi-Panel 1 EE 1 EE EE
O Water Ripple EE EE EE
P Spark EE 1 1 EE EE
Q Single Lightning Bolt EE EE EE
R Energized Equipment EE 1 1 EE EE
S Two Hands EE EE EE
T Puddle EE EE EE
U Person EE 1 EE EE
V Sequence Arrow 1 EE EE EE
W Rain Cloud EE 1 EE EE
X Surprised Face EE 1 EE EE
Y Permissable Circle EE 1 EE EE
Z Happy Face EE EE EE
AA Mr. Sparky EE EE EE
BB Heat Waves EE CC EE  
 
 
61 62 63 64 65 66 67 68 69 70
Attribute U.S. Indian U.S. Indian U.S. U.S. U.S. U.S. Indian U.S.
A Single Hand 1 EE 1 1 1 1 CC 1
B 1-D Surface EE CC
C Multiple Water Drops 1 1 EE 1 1 1 1 CC 1
D Prohibition Symbol 1 1 EE 1 CC
E 2nd Color 1 1 EE 1 1 1 1 CC
F Skull/Crossbones EE 1 CC
G Faucet 1 EE 1 1 CC
H Prohibition X EE 1 1 CC 1
I Liquid Container EE 1 1 CC
J 2-D Panel 1 EE 1 CC
K Lightning Bolts EE 1 CC
L Single Water Drop EE CC
M 3-D Object 1 EE CC
N Multi-Panel 1 EE 1 1 1 CC
O Water Ripple EE 1 CC
P Spark EE 1 1 CC
Q Single Lightning Bolt 1 EE CC
R Energized Equipment EE 1 1 CC
S Two Hands 1 EE CC
T Puddle EE CC
U Person EE 1 CC
V Sequence Arrow EE 1 CC
W Rain Cloud EE CC
X Surprised Face EE CC
Y Permissable Circle EE CC
Z Happy Face EE 1 CC
AA Mr. Sparky EE 1 CC
BB Heat Waves EE 1 CC  
 
 174 
Attribute matrix produced by Panelist #3 for ?Do Not Touch with Wet Hands?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attribute U.S. U.S. Indian Indian U.S. Indian U.S. U.S. Indian U.S. U.S. Indian U.S. Indian Indian
A Single Hand EE EE 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface EE EE 1 1
C Multiple Water Drops EE EE 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol EE EE 1 1 1 1 1
E 2nd Color EE EE 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones EE EE 1
G Faucet EE EE 1 1 1
H Prohibition X EE EE 1 1 1 1 1 1 1
I Liquid Container EE EE 1
J 2-D Panel EE EE 1 1 1
K Lightning Bolts EE EE 1
L Single Water Drop EE EE 1 1
M 3-D Object EE EE 1
N Multi-Panel EE EE 1
O Water Ripple EE EE
P Spark EE EE
Q Single Lightning Bolt EE EE
R Energized Equipment EE EE
S Two Hands EE EE
T Puddle EE EE
U Person EE EE
V Sequence Arrow EE EE
W Rain Cloud EE EE
X Surprised Face EE EE
Y Permissable Circle EE EE
Z Happy Face EE EE
AA Mr. Sparky EE EE
BB Heat Waves EE EE  
 
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attribute Indian Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. U.S. Indian U.S. U.S. Indian
A Single Hand 1 1 1 1 1 1 1 1 1 1 1 1 1
B 1-D Surface 1
C Multiple Water Drops 1 1 1 1 1 1 1 1 1 1 1 1 1 1
D Prohibition Symbol 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1
F Skull/Crossbones 1
G Faucet 1
H Prohibition X 1 1 1 1 1
I Liquid Container
J 2-D Panel 1 1 1
K Lightning Bolts 1
L Single Water Drop 1
M 3-D Object
N Multi-Panel
O Water Ripple 1
P Spark 1 1
Q Single Lightning Bolt 1
R Energized Equipment 1
S Two Hands 1
T Puddle 1
U Person 1
V Sequence Arrow
W Rain Cloud
X Surprised Face
Y Permissable Circle
Z Happy Face
AA Mr. Sparky
BB Heat Waves  
 
 175 
 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attribute U.S. Indian Indian Indian Indian U.S. Indian U.S. U.S. U.S. Indian Indian U.S. Indian Indian
A Single Hand 1 1 1 1 CC 1 1 1 1 1 1 1 CC
B 1-D Surface CC CC
C Multiple Water Drops 1 1 1 1 1 1 CC 1 1 1 1 1 1 1 CC
D Prohibition Symbol 1 1 CC 1 1 1 1 CC
E 2nd Color 1 1 1 1 1 CC 1 1 1 1 1 1 CC
F Skull/Crossbones CC CC
G Faucet 1 CC 1 CC
H Prohibition X 1 1 1 1 CC 1 1 1 CC
I Liquid Container CC CC
J 2-D Panel CC 1 CC
K Lightning Bolts CC 1 CC
L Single Water Drop CC CC
M 3-D Object CC CC
N Multi-Panel CC CC
O Water Ripple CC CC
P Spark CC CC
Q Single Lightning Bolt CC CC
R Energized Equipment CC 1 CC
S Two Hands 1 1 CC CC
T Puddle CC CC
U Person CC CC
V Sequence Arrow CC CC
W Rain Cloud CC CC
X Surprised Face CC CC
Y Permissable Circle CC CC
Z Happy Face CC CC
AA Mr. Sparky CC CC
BB Heat Waves CC CC  
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attribute Indian Indian U.S. U.S. Indian Indian Indian Indian Indian Indian U.S. U.S. U.S. Indian Indian
A Single Hand 1 1 1 EE 1 1 1 1 1 1 1 CC EE
B 1-D Surface EE 1 CC EE
C Multiple Water Drops 1 1 EE 1 1 1 1 1 1 1 1 1 CC EE
D Prohibition Symbol 1 1 EE 1 CC EE
E 2nd Color 1 1 1 EE 1 1 1 1 1 1 1 CC EE
F Skull/Crossbones EE CC EE
G Faucet 1 EE CC EE
H Prohibition X 1 EE 1 1 1 1 1 1 1 1 CC EE
I Liquid Container EE CC EE
J 2-D Panel 1 EE 1 1 1 CC EE
K Lightning Bolts 1 EE 1 CC EE
L Single Water Drop 1 EE CC EE
M 3-D Object EE 1 CC EE
N Multi-Panel 1 EE 1 CC EE
O Water Ripple EE CC EE
P Spark EE 1 CC EE
Q Single Lightning Bolt EE CC EE
R Energized Equipment EE 1 1 1 CC EE
S Two Hands EE CC EE
T Puddle EE CC EE
U Person EE 1 1 CC EE
V Sequence Arrow 1 EE CC EE
W Rain Cloud EE 1 CC EE
X Surprised Face EE 1 CC EE
Y Permissable Circle EE 1 CC EE
Z Happy Face EE CC EE
AA Mr. Sparky EE CC EE
BB Heat Waves EE CC EE  
 
 176 
 
61 62 63 64 65 66 67 68 69 70
Attribute U.S. Indian U.S. Indian U.S. U.S. U.S. U.S. Indian U.S.
A Single Hand 1 1 EE 1 1 1 1 CC 1
B 1-D Surface EE CC
C Multiple Water Drops 1 1 EE 1 1 1 1 1 CC 1
D Prohibition Symbol 1 1 EE 1 CC
E 2nd Color 1 1 EE 1 1 1 1 CC
F Skull/Crossbones 1 EE 1 1 CC
G Faucet EE 1 1 CC
H Prohibition X EE 1 1 CC 1
I Liquid Container EE 1 1 CC
J 2-D Panel 1 EE 1 1 CC
K Lightning Bolts EE 1 1 CC
L Single Water Drop EE CC
M 3-D Object 1 EE CC
N Multi-Panel 1 EE 1 1 1 CC
O Water Ripple EE 1 CC
P Spark EE 1 CC
Q Single Lightning Bolt 1 EE CC
R Energized Equipment EE 1 1 CC
S Two Hands 1 EE CC
T Puddle EE CC
U Person EE CC
V Sequence Arrow EE 1 CC
W Rain Cloud EE CC
X Surprised Face EE CC
Y Permissable Circle EE CC
Z Happy Face EE 1 CC
AA Mr. Sparky EE 1 CC
BB Heat Waves EE 1 CC  
 
 177 
Consensus attribute matrix produced by summing the three individual panelists? attribute 
matrices for ?Hot Exhaust?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attributes Indian U.S. U.S. Indian U.S. Indian Indian Indian Indian U.S. U.S. Indian U.S. Indian Indian
A Directional Arrow 3 0 0 0 0 0 0 0 0 1 3 0 0 0 0
B Safety Alert Symbol 3 0 0 0 0 2 0 0 0 0 0 0 0 0 0
C Emmission Lines 3 3 3 3 3 3 3 2 3 3 1 2 3 3 0
D Pipe or Stack 3 1 1 1 3 0 3 3 3 0 3 3 3 3 1
E 2nd Color 3 0 0 2 0 2 0 3 3 3 3 3 2 0 2
F Negative Face 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
G Person 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
H Vat 0 2 2 2 0 0 0 0 0 0 0 0 0 0 0
I Thermometer 0 0 0 0 3 3 0 0 0 0 0 3 0 0 0
J Flame 0 0 0 0 0 0 0 3 0 3 3 3 0 0 3
K Cloud 0 0 0 0 0 0 0 0 3 0 0 3 0 0 0
L Exclamation Point 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0
M Vented Object 0 0 0 0 0 0 0 0 0 3 0 1 0 0 0
N Particulates 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0
O Prohibition Symbol 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0
P Emphasis Arrows 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0
Q Structure 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0
R Skull/Crossbones 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
S Vulnerable Object 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
T Vent Grate 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
U Positive Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
V Vector 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
W Prohibited X 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
X Hand 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Y Thermos 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Z Hood 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
AA Degree Symbol 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
BB Fan 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CC Surface 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
DD Radiant Heat Lines 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
EE Multi-Panel 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
FF Ground 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GG Surprise Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
HH Relief Valve 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
II Movement Lines 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attributes Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. Indian Indian Indian U.S. U.S. U.S.
A Directional Arrow 1 0 0 0 0 0 0 0 0 2 0 3 2 0 0
B Safety Alert Symbol 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
C Emmission Lines 3 3 0 3 3 3 2 3 0 1 3 3 1 2 0
D Pipe or Stack 3 3 0 2 3 3 0 3 0 3 3 3 3 0 3
E 2nd Color 0 0 3 3 3 2 3 3 2 3 2 0 3 3 2
F Negative Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
G Person 0 0 0 3 3 0 0 0 0 0 0 0 2 0 0
H Vat 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
I Thermometer 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
J Flame 3 3 3 0 0 0 2 3 3 0 0 0 3 1 3
K Cloud 1 0 1 0 0 0 1 0 3 0 0 0 0 0 3
L Exclamation Point 0 0 0 1 0 0 0 0 0 0 0 0 0 0 3
M Vented Object 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
N Particulates 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
O Prohibition Symbol 0 3 0 0 0 0 0 0 0 0 0 0 0 3 0
P Emphasis Arrows 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
Q Structure 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
R Skull/Crossbones 3 0 0 0 0 0 0 0 0 3 3 0 0 0 0
S Vulnerable Object 0 3 3 1 0 0 0 0 0 0 0 0 3 0 0
T Vent Grate 0 0 3 3 0 0 0 0 0 0 0 0 0 3 0
U Positive Face 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0
V Vector 0 0 0 2 2 0 0 0 0 0 0 0 2 0 0
W Prohibited X 0 0 0 0 3 0 3 0 0 0 0 0 0 0 0
X Hand 0 0 0 0 0 0 3 0 0 0 0 0 0 2 0
Y Thermos 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0
Z Hood 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0
AA Degree Symbol 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3
BB Fan 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CC Surface 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
DD Radiant Heat Lines 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
EE Multi-Panel 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
FF Ground 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
GG Surprise Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
HH Relief Valve 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
II Movement Lines 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 
 178 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attributes Indian U.S. U.S. Indian Indian U.S. U.S. Indian Indian U.S. U.S. U.S. Indian U.S. Indian
A Directional Arrow 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
B Safety Alert Symbol 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0
C Emmission Lines 3 3 2 3 3 3 3 3 3 0 2 0 3 3 3
D Pipe or Stack 0 3 0 2 1 0 3 3 0 1 3 0 0 3 0
E 2nd Color 3 3 0 0 3 3 3 2 3 3 3 2 2 2 3
F Negative Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
G Person 0 0 0 0 0 3 2 0 0 3 3 0 0 3 0
H Vat 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0
I Thermometer 0 0 0 3 0 0 0 0 3 0 0 0 0 0 0
J Flame 0 0 3 0 0 0 1 0 0 1 2 2 3 0 0
K Cloud 0 0 0 0 0 0 1 1 0 0 3 1 0 0 0
L Exclamation Point 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
M Vented Object 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0
N Particulates 0 0 0 0 0 0 0 2 0 0 2 0 0 0 2
O Prohibition Symbol 0 0 0 0 0 0 0 0 0 3 3 0 0 3 0
P Emphasis Arrows 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
Q Structure 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
R Skull/Crossbones 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
S Vulnerable Object 0 0 2 0 0 0 0 0 0 3 2 0 0 0 0
T Vent Grate 1 0 0 3 0 0 0 0 3 1 0 0 3 0 0
U Positive Face 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
V Vector 0 0 0 0 0 3 0 0 0 1 0 0 0 0 0
W Prohibited X 0 0 0 0 0 3 0 0 0 0 0 3 0 0 0
X Hand 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0
Y Thermos 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Z Hood 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
AA Degree Symbol 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
BB Fan 3 0 0 0 0 0 0 0 0 0 0 0 0 0 3
CC Surface 2 0 1 0 0 2 0 0 1 0 0 0 0 0 0
DD Radiant Heat Lines 0 3 1 0 0 0 0 1 0 0 0 0 2 0 0
EE Multi-Panel 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
FF Ground 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
GG Surprise Face 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0
HH Relief Valve 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
II Movement Lines 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attributes Indian U.S. Indian Indian U.S. U.S. U.S. U.S. Indian Indian U.S. Indian Indian Indian U.S.
A Directional Arrow 1 1 0 2 0 0 0 0 EE EE 0 EE 1 3 0
B Safety Alert Symbol 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
C Emmission Lines 2 3 3 0 2 3 3 1 EE EE 3 EE 3 2 3
D Pipe or Stack 1 1 0 3 1 3 3 2 EE EE 3 EE 3 1 0
E 2nd Color 2 3 3 3 3 3 3 0 EE EE 3 EE 2 0 2
F Negative Face 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
G Person 0 0 0 0 0 2 0 3 EE EE 0 EE 3 0 0
H Vat 0 0 0 0 2 0 0 0 EE EE 0 EE 0 0 0
I Thermometer 0 0 0 0 0 0 3 0 EE EE 0 EE 0 0 0
J Flame 0 3 0 3 0 3 0 0 EE EE 3 EE 3 0 2
K Cloud 3 0 0 3 0 0 3 3 EE EE 0 EE 0 0 2
L Exclamation Point 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
M Vented Object 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
N Particulates 2 0 3 3 0 0 2 0 EE EE 0 EE 0 0 0
O Prohibition Symbol 0 1 0 0 0 3 0 0 EE EE 0 EE 0 0 0
P Emphasis Arrows 0 1 0 1 0 0 0 0 EE EE 0 EE 1 0 0
Q Structure 0 0 0 3 0 0 0 0 EE EE 0 EE 1 0 0
R Skull/Crossbones 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
S Vulnerable Object 0 0 2 0 0 3 0 0 EE EE 3 EE 3 0 0
T Vent Grate 0 3 3 0 2 0 0 0 EE EE 0 EE 0 0 3
U Positive Face 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
V Vector 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
W Prohibited X 0 0 3 0 0 0 0 0 EE EE 0 EE 3 0 0
X Hand 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
Y Thermos 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
Z Hood 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
AA Degree Symbol 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
BB Fan 3 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
CC Surface 0 0 1 0 0 0 0 0 EE EE 0 EE 0 3 0
DD Radiant Heat Lines 1 1 0 0 1 0 1 0 EE EE 3 EE 0 1 1
EE Multi-Panel 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
FF Ground 0 0 0 0 0 0 0 0 EE EE 0 EE 0 0 0
GG Surprise Face 0 0 0 0 0 0 0 3 EE EE 0 EE 0 0 0
HH Relief Valve 0 0 0 0 0 0 0 3 EE EE 0 EE 0 0 0
II Movement Lines 0 0 0 0 0 0 0 3 EE EE 0 EE 0 0 0
 
 179 
 
61 62 63 64 65 66 67 68 69 70
Attributes U.S. U.S. Indian Indian Indian Indian U.S. Indian Indian U.S.
A Directional Arrow 3 0 3 0 EE 0 0 0 1 0
B Safety Alert Symbol 0 3 0 0 EE 0 0 0 2 0
C Emmission Lines 3 3 3 3 EE 2 3 2 2 3
D Pipe or Stack 2 0 3 0 EE 3 3 3 0 0
E 2nd Color 0 3 3 3 EE 3 3 3 0 1
F Negative Face 0 3 0 0 EE 0 0 0 0 3
G Person 0 0 0 0 EE 0 3 0 0 1
H Vat 0 0 0 0 EE 0 0 0 0 0
I Thermometer 0 3 0 0 EE 0 0 0 0 0
J Flame 3 0 2 0 EE 0 0 3 3 0
K Cloud 0 0 0 0 EE 0 1 3 0 0
L Exclamation Point 0 3 0 0 EE 0 0 0 0 0
M Vented Object 0 0 0 0 EE 1 0 0 0 0
N Particulates 0 0 0 0 EE 0 0 0 0 0
O Prohibition Symbol 0 0 0 0 EE 1 2 1 0 0
P Emphasis Arrows 0 0 0 0 EE 0 0 0 0 0
Q Structure 0 0 0 0 EE 0 0 0 0 0
R Skull/Crossbones 0 0 3 0 EE 0 0 0 0 0
S Vulnerable Object 0 0 3 0 EE 0 0 0 0 0
T Vent Grate 0 3 0 0 EE 3 0 0 0 1
U Positive Face 0 0 0 0 EE 0 0 0 0 0
V Vector 0 0 0 0 EE 0 0 0 2 0
W Prohibited X 0 0 0 0 EE 0 0 0 0 0
X Hand 0 0 0 0 EE 0 2 3 0 0
Y Thermos 0 0 0 0 EE 0 0 0 0 0
Z Hood 0 0 0 0 EE 0 0 0 0 0
AA Degree Symbol 0 0 0 0 EE 0 0 0 0 0
BB Fan 0 0 0 3 EE 0 0 0 3 3
CC Surface 2 0 0 0 EE 0 0 0 0 0
DD Radiant Heat Lines 2 0 0 0 EE 0 3 0 0 3
EE Multi-Panel 0 0 0 0 EE 0 0 0 0 0
FF Ground 0 0 0 0 EE 0 0 0 0 0
GG Surprise Face 0 0 0 0 EE 0 0 0 0 0
HH Relief Valve 0 0 0 0 EE 0 0 0 0 0
II Movement Lines 0 0 0 0 EE 0 0 0 0 0  
 
 
 180 
Attribute matrix produced by Panelist #1 for ?Hot Exhaust?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attributes Indian U.S. U.S. Indian U.S. Indian Indian Indian Indian U.S. U.S. Indian U.S. Indian Indian
A Directional Arrow 1 1
B Safety Alert Symbol 1 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1
F Negative Face 1
G Person 1
H Vat 1 1 1
I Thermometer 1 1 1
J Flame 1 1 1 1 1
K Cloud 1 1
L Exclamation Point 1 1
M Vented Object 1 1
N Particulates 1
O Prohibition Symbol 1
P Emphasis Arrows 1
Q Structure
R Skull/Crossbones
S Vulnerable Object 1
T Vent Grate
U Positive Face
V Vector
W Prohibited X
X Hand
Y Thermos
Z Hood
AA Degree Symbol
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attributes Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. Indian Indian Indian U.S. U.S. U.S.
A Directional Arrow 1 1
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1 1
H Vat
I Thermometer
J Flame 1 1 1 1 1 1 1 1
K Cloud 1 1
L Exclamation Point 1 1
M Vented Object
N Particulates
O Prohibition Symbol 1 1
P Emphasis Arrows
Q Structure
R Skull/Crossbones 1 1 1
S Vulnerable Object 1 1 1 1
T Vent Grate 1 1 1
U Positive Face 1
V Vector 1 1
W Prohibited X 1 1
X Hand 1
Y Thermos 1
Z Hood 1
AA Degree Symbol 1
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 181 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attributes Indian U.S. U.S. Indian Indian U.S. U.S. Indian Indian U.S. U.S. U.S. Indian U.S. Indian
A Directional Arrow
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1 1 1 1
H Vat 1
I Thermometer 1 1
J Flame 1 1 1 1
K Cloud 1 1
L Exclamation Point
M Vented Object
N Particulates 1 1
O Prohibition Symbol 1 1 1
P Emphasis Arrows
Q Structure
R Skull/Crossbones
S Vulnerable Object 1 1
T Vent Grate 1 1 1
U Positive Face
V Vector 1
W Prohibited X 1 1
X Hand 1
Y Thermos
Z Hood
AA Degree Symbol
BB Fan 1 1
CC Surface 1 1 1
DD Radiant Heat Lines 1
EE Multi-Panel 1
FF Ground 1
GG Surprise Face 1
HH Relief Valve
II Movement Lines
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attributes Indian U.S. Indian Indian U.S. U.S. U.S. U.S. Indian Indian U.S. Indian Indian Indian U.S.
A Directional Arrow 1 EE EE EE 1
B Safety Alert Symbol EE EE EE
C Emmission Lines 1 1 1 1 1 1 EE EE 1 EE 1 1 1
D Pipe or Stack 1 1 1 1 1 EE EE 1 EE 1
E 2nd Color 1 1 1 1 1 1 1 EE EE 1 EE 1
F Negative Face EE EE EE
G Person 1 1 EE EE EE 1
H Vat 1 EE EE EE
I Thermometer 1 EE EE EE
J Flame 1 1 1 EE EE 1 EE 1 1
K Cloud 1 1 1 1 EE EE EE 1
L Exclamation Point EE EE EE
M Vented Object EE EE EE
N Particulates 1 1 1 EE EE EE
O Prohibition Symbol 1 1 EE EE EE
P Emphasis Arrows 1 EE EE EE
Q Structure 1 EE EE EE
R Skull/Crossbones EE EE EE
S Vulnerable Object 1 1 EE EE 1 EE 1
T Vent Grate 1 1 EE EE EE 1
U Positive Face EE EE EE
V Vector EE EE EE
W Prohibited X 1 EE EE EE 1
X Hand EE EE EE
Y Thermos EE EE EE
Z Hood EE EE EE
AA Degree Symbol EE EE EE
BB Fan 1 EE EE EE
CC Surface 1 EE EE EE 1
DD Radiant Heat Lines EE EE 1 EE
EE Multi-Panel EE EE EE
FF Ground EE EE EE
GG Surprise Face 1 EE EE EE
HH Relief Valve 1 EE EE EE
II Movement Lines 1 EE EE EE
 
 182 
61 62 63 64 65 66 67 68 69 70
Attributes U.S. U.S. Indian Indian Indian Indian U.S. Indian Indian U.S.
A Directional Arrow 1 1 EE
B Safety Alert Symbol 1 EE 1
C Emmission Lines 1 1 1 1 EE 1 1 1 1
D Pipe or Stack 1 EE 1 1 1
E 2nd Color 1 1 1 EE 1 1 1 1
F Negative Face 1 EE 1
G Person EE 1 1
H Vat EE
I Thermometer 1 EE
J Flame 1 EE 1 1
K Cloud EE 1
L Exclamation Point 1 EE
M Vented Object EE
N Particulates EE
O Prohibition Symbol EE 1 1
P Emphasis Arrows EE
Q Structure EE
R Skull/Crossbones 1 EE
S Vulnerable Object 1 EE
T Vent Grate 1 EE 1
U Positive Face EE
V Vector EE
W Prohibited X EE
X Hand EE 1 1
Y Thermos EE
Z Hood EE
AA Degree Symbol EE
BB Fan 1 EE 1 1
CC Surface 1 EE
DD Radiant Heat Lines 1 EE 1 1
EE Multi-Panel EE
FF Ground EE
GG Surprise Face EE
HH Relief Valve EE
II Movement Lines EE  
 
Attribute matrix produced by Panelist #2 for ?Hot Exhaust?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attributes Indian U.S. U.S. Indian U.S. Indian Indian Indian Indian U.S. U.S. Indian U.S. Indian Indian
A Directional Arrow 1 1
B Safety Alert Symbol 1 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1
F Negative Face 1
G Person 1
H Vat 1 1 1
I Thermometer 1 1 1
J Flame 1 1 1 1 1
K Cloud 1 1
L Exclamation Point 1 1
M Vented Object 1
N Particulates 1
O Prohibition Symbol 1
P Emphasis Arrows 1
Q Structure 1
R Skull/Crossbones
S Vulnerable Object
T Vent Grate
U Positive Face
V Vector
W Prohibited X
X Hand
Y Thermos
Z Hood
AA Degree Symbol
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 183 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attributes Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. Indian Indian Indian U.S. U.S. U.S.
A Directional Arrow 1 1 1
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1
H Vat
I Thermometer
J Flame 1 1 1 1 1 1 1 1 1
K Cloud 1 1 1
L Exclamation Point 1
M Vented Object
N Particulates 1
O Prohibition Symbol 1 1
P Emphasis Arrows
Q Structure 1
R Skull/Crossbones 1 1 1
S Vulnerable Object 1 1 1
T Vent Grate 1 1 1
U Positive Face 1
V Vector 1 1 1
W Prohibited X 1 1
X Hand 1 1
Y Thermos 1
Z Hood 1
AA Degree Symbol 1
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 
 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attributes Indian U.S. U.S. Indian Indian U.S. U.S. Indian Indian U.S. U.S. U.S. Indian U.S. Indian
A Directional Arrow
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1 1 1 1
H Vat 1
I Thermometer 1 1
J Flame 1 1 1 1 1
K Cloud 1
L Exclamation Point
M Vented Object
N Particulates 1 1 1
O Prohibition Symbol 1 1 1
P Emphasis Arrows
Q Structure
R Skull/Crossbones
S Vulnerable Object 1 1
T Vent Grate 1 1 1
U Positive Face
V Vector 1 1
W Prohibited X 1 1
X Hand 1
Y Thermos
Z Hood
AA Degree Symbol
BB Fan 1 1
CC Surface 1 1 1
DD Radiant Heat Lines 1 1 1 1
EE Multi-Panel 1
FF Ground 1
GG Surprise Face 1
HH Relief Valve
II Movement Lines
 
 184 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attributes Indian U.S. Indian Indian U.S. U.S. U.S. U.S. Indian Indian U.S. Indian Indian Indian U.S.
A Directional Arrow 1 1 EE EE EE 1 1
B Safety Alert Symbol EE EE EE
C Emmission Lines 1 1 1 1 1 1 EE EE 1 EE 1 1
D Pipe or Stack 1 1 1 1 EE EE 1 EE 1 1
E 2nd Color 1 1 1 1 1 1 1 EE EE 1 EE 1 1
F Negative Face EE EE EE
G Person 1 1 EE EE EE 1
H Vat 1 EE EE EE
I Thermometer 1 EE EE EE
J Flame 1 1 1 EE EE 1 EE 1 1
K Cloud 1 1 1 1 EE EE EE
L Exclamation Point EE EE EE
M Vented Object EE EE EE
N Particulates 1 1 1 1 EE EE EE
O Prohibition Symbol 1 EE EE EE
P Emphasis Arrows EE EE EE
Q Structure 1 EE EE EE 1
R Skull/Crossbones EE EE EE
S Vulnerable Object 1 EE EE 1 EE 1
T Vent Grate 1 1 1 EE EE EE 1
U Positive Face EE EE EE
V Vector EE EE EE
W Prohibited X 1 EE EE EE 1
X Hand EE EE EE
Y Thermos EE EE EE
Z Hood EE EE EE
AA Degree Symbol EE EE EE
BB Fan 1 EE EE EE
CC Surface EE EE EE 1
DD Radiant Heat Lines 1 EE EE 1 EE 1
EE Multi-Panel EE EE EE
FF Ground EE EE EE
GG Surprise Face 1 EE EE EE
HH Relief Valve 1 EE EE EE
II Movement Lines 1 EE EE EE
 
 
61 62 63 64 65 66 67 68 69 70
Attributes U.S. U.S. Indian Indian Indian Indian U.S. Indian Indian U.S.
A Directional Arrow 1 1 EE
B Safety Alert Symbol 1 EE 1
C Emmission Lines 1 1 1 1 EE 1 1 1 1
D Pipe or Stack 1 1 EE 1 1 1
E 2nd Color 1 1 1 EE 1 1 1
F Negative Face 1 EE 1
G Person EE 1
H Vat EE
I Thermometer 1 EE
J Flame 1 1 EE 1 1
K Cloud EE 1
L Exclamation Point 1 EE
M Vented Object EE
N Particulates EE
O Prohibition Symbol EE 1 1
P Emphasis Arrows EE
Q Structure EE
R Skull/Crossbones 1 EE
S Vulnerable Object 1 EE
T Vent Grate 1 EE 1 1
U Positive Face EE
V Vector EE 1
W Prohibited X EE
X Hand EE 1 1
Y Thermos EE
Z Hood EE
AA Degree Symbol EE
BB Fan 1 EE 1 1
CC Surface 1 EE
DD Radiant Heat Lines 1 EE 1 1
EE Multi-Panel EE
FF Ground EE
GG Surprise Face EE
HH Relief Valve EE
II Movement Lines EE  
 185 
Attribute matrix produced by Panelist #3 for ?Hot Exhaust?. 
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Attributes Indian U.S. U.S. Indian U.S. Indian Indian Indian Indian U.S. U.S. Indian U.S. Indian Indian
A Directional Arrow 1 1 1
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1
F Negative Face 1
G Person 1
H Vat
I Thermometer 1 1 1
J Flame 1 1 1 1 1
K Cloud 1 1
L Exclamation Point 1 1
M Vented Object 1
N Particulates 1
O Prohibition Symbol 1
P Emphasis Arrows 1
Q Structure 1
R Skull/Crossbones
S Vulnerable Object
T Vent Grate
U Positive Face
V Vector
W Prohibited X
X Hand
Y Thermos
Z Hood
AA Degree Symbol
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Attributes Indian U.S. U.S. U.S. Indian U.S. U.S. U.S. U.S. Indian Indian Indian U.S. U.S. U.S.
A Directional Arrow 1 1 1
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1 1
H Vat 1
I Thermometer
J Flame 1 1 1 1 1 1 1
K Cloud 1 1 1 1
L Exclamation Point 1
M Vented Object 1
N Particulates
O Prohibition Symbol 1 1
P Emphasis Arrows 1
Q Structure
R Skull/Crossbones 1 1 1
S Vulnerable Object 1 1 1
T Vent Grate 1 1 1
U Positive Face 1
V Vector 1
W Prohibited X 1 1
X Hand 1 1
Y Thermos 1
Z Hood 1
AA Degree Symbol 1
BB Fan
CC Surface
DD Radiant Heat Lines
EE Multi-Panel
FF Ground
GG Surprise Face
HH Relief Valve
II Movement Lines
 186 
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Attributes Indian U.S. U.S. Indian Indian U.S. U.S. Indian Indian U.S. U.S. U.S. Indian U.S. Indian
A Directional Arrow
B Safety Alert Symbol 1
C Emmission Lines 1 1 1 1 1 1 1 1 1 1 1 1
D Pipe or Stack 1 1 1 1 1 1 1
E 2nd Color 1 1 1 1 1 1 1 1 1
F Negative Face
G Person 1 1 1 1
H Vat
I Thermometer 1 1
J Flame 1 1 1
K Cloud 1 1 1
L Exclamation Point
M Vented Object 1
N Particulates 1
O Prohibition Symbol 1 1 1
P Emphasis Arrows 1
Q Structure
R Skull/Crossbones
S Vulnerable Object 1 1 1
T Vent Grate 1 1 1 1 1
U Positive Face
V Vector 1
W Prohibited X 1 1
X Hand 1
Y Thermos
Z Hood
AA Degree Symbol
BB Fan 1 1
CC Surface
DD Radiant Heat Lines 1 1
EE Multi-Panel 1
FF Ground 1
GG Surprise Face 1
HH Relief Valve
II Movement Lines
 
 
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
Attributes Indian U.S. Indian Indian U.S. U.S. U.S. U.S. Indian Indian U.S. Indian Indian Indian U.S.
A Directional Arrow 1 EE EE EE 1
B Safety Alert Symbol EE EE EE
C Emmission Lines 1 1 1 1 1 EE EE 1 EE 1 1 1
D Pipe or Stack 1 1 1 1 1 EE EE 1 EE 1
E 2nd Color 1 1 1 1 1 1 EE EE 1 EE 1
F Negative Face EE EE EE
G Person 1 EE EE EE 1
H Vat EE EE EE
I Thermometer 1 EE EE EE
J Flame 1 1 1 EE EE 1 EE 1
K Cloud 1 1 1 1 EE EE EE 1
L Exclamation Point EE EE EE
M Vented Object EE EE EE
N Particulates 1 1 1 EE EE EE
O Prohibition Symbol 1 EE EE EE
P Emphasis Arrows 1 EE EE EE 1
Q Structure 1 EE EE EE
R Skull/Crossbones EE EE EE
S Vulnerable Object 1 1 EE EE 1 EE 1
T Vent Grate 1 1 1 EE EE EE 1
U Positive Face EE EE EE
V Vector EE EE EE
W Prohibited X 1 EE EE EE 1
X Hand EE EE EE
Y Thermos EE EE EE
Z Hood EE EE EE
AA Degree Symbol EE EE EE
BB Fan 1 EE EE EE
CC Surface EE EE EE 1
DD Radiant Heat Lines 1 1 1 EE EE 1 EE 1
EE Multi-Panel EE EE EE
FF Ground EE EE EE
GG Surprise Face 1 EE EE EE
HH Relief Valve 1 EE EE EE
II Movement Lines 1 EE EE EE
 
 187 
61 62 63 64 65 66 67 68 69 70
Attributes U.S. U.S. Indian Indian Indian Indian U.S. Indian Indian U.S.
A Directional Arrow 1 1 EE 1
B Safety Alert Symbol 1 EE
C Emmission Lines 1 1 1 1 EE 1 1 1 1
D Pipe or Stack 1 1 EE 1 1 1
E 2nd Color 1 1 1 EE 1 1 1
F Negative Face 1 EE 1
G Person EE 1
H Vat EE
I Thermometer 1 EE
J Flame 1 1 EE 1 1
K Cloud EE 1 1
L Exclamation Point 1 EE
M Vented Object EE 1
N Particulates EE
O Prohibition Symbol EE
P Emphasis Arrows EE
Q Structure EE
R Skull/Crossbones 1 EE
S Vulnerable Object 1 EE
T Vent Grate 1 EE 1
U Positive Face EE
V Vector EE 1
W Prohibited X EE
X Hand EE 1
Y Thermos EE
Z Hood EE
AA Degree Symbol EE
BB Fan 1 EE 1 1
CC Surface EE
DD Radiant Heat Lines EE 1 1
EE Multi-Panel EE
FF Ground EE
GG Surprise Face EE
HH Relief Valve EE
II Movement Lines EE  
 
 188 
APPENDIX 4.7 
?Hot Exhaust? Symbols voted as Egregious Error by expert panel 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Each of these symbols was drawn for the referent ?Hot Exhaust? and was discarded by 
majority vote of the expert panel.  The panel perceived that the artists did not understand 
the intent of the referent or did not portray the intent in their picture. 
 
These symbols were not included in the clustering process to prevent passing potentially 
erroneous symbol attributes into the DIGA symbol design tool, perhaps at the expense of 
attributes contributing to adequate symbol designs. 
 
Note: No symbols were labeled ?Critical Confusion? for the ?Hot Exhaust? referent. 
54 55 
57 65 
 189 
60 49 
63 
1 2 
?Do Not Touch with Wet Hands? Symbols voted as Egregious Error by expert panel 
 
 
 
Each of these symbols was drawn for the 
referent ?Do Not Touch with Wet Hands? 
and was discarded by majority vote of the 
expert panel.  The panel perceived that the 
artists did not understand the intent of the 
referent or did not portray the intent in their 
picture. 
 
These symbols were not included in the 
clustering process to prevent passing 
potentially erroneous symbol attributes into 
the DIGA symbol design tool, perhaps at the 
expense of attributes contributing to 
adequate symbol designs. 
 190 
37 
45 
59 
 ?Do Not Touch with Wet Hands? Symbols voted as Critical Confusion by expert panel 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Each of these symbols was drawn for the referent ?Do Not Touch with Wet Hands? and 
was discarded by majority vote of the expert panel.  The panel perceived that the artists 
portrayed an opposite meaning to the intent of the referent, or that the symbol encouraged 
unsafe behavior. 
 
These symbols were not included in the clustering process to prevent passing potentially 
critically confusing symbol attributes into the DIGA symbol design tool, perhaps at the 
expense of attributes contributing to adequate symbol designs. 
69 
 191 
APPENDIX 4.8 
K-means clustering results 
 
Attributes Referent Stratum K RSS
min rmin d wsmall wlarge 
Total Eliminated* Final 
American 3 42 1.1 0.976 31% 34% 35 31 4 
Indian 5 19 1.63 1 16% 26% 35 31 4 
Hot 
Exhaust 
All 5 37 1.41 1 17% 24% 35 32 3 
American 3 21 2.94 0.976 16% 47% 28 23 5 
Indian 4 20 2.71 0.949 14% 38% 28 22 6 
Do Not 
Touch 
with Wet 
Hands All 4 27 3.73 1 11% 41% 28 23 5 
* An attribute was eliminated if it did not appear in the centroid of any cluster in the final clustering run 
reported in this table.   
 
Key to the semantic symbol attributes for ?Hot exhaust?  
 
Pictorial Description
A Directional Arrow An arrow indicating the direction of flow or movement
B Safety Alert Symbol A standard triangle with an exclamation point indicating danger or hazard
C Emmission Lines Any straight, dotted, wavy or other lines representing pneumatic flow
D Pipe or Stack A cylindrical or conical transmission line with an open end
E 2nd Color * The deliberate use of an additional color to emphasize part of the drawing
F Negative Face A facial expression meant to specifically indicate negative feelings
G Person All or part of a human body
H Vat A tank or wide-mouthed opening that is the source of the exhaust
I Thermometer A traditional mercury thermometer intended to indicate high temperatures
J Flame A flame or fire intended to indicate high temperatures or combustion
K Cloud A fine mist or emission cloud
L Exclamation Point An exclamation point symbol found outside of a safety alert symbol
M Vented Object A 3-D object with a vent or grate on one side
N Particulates A type of emission that is intended to indicate solid particles
O Prohibition Symbol The traditional circle/slash intended to indicate "Do Not?"
P Emphasis Arrows Arrows drawn to point or call attention to a portion of the symbol
Q Structure All of or part of a building, such as a wall or column
R Skull/Crossbones The traditional "toxic" or "danger" symbol of a skull and crossbones
S Vulnerable Object Any non-specific shape placed in the vulnerable area of the exhaust stream
T Vent/Grate A slotted grate or vent which is the source of emissions
U Positive Face A facial expression meant to specifically indicate positive or good feelings
V Vector An arrow intended to communicate both distance and direction
W Prohibited X An "X" or cross used in place of the traditional circle/slash prohibition symbol
X Hand A hand or arm without the rest of the human body
Y Thermos A classic camping or lunch pail thermos
Z Hood A fume hood
AA Degree Symbol The " ? " symbol intended to indicat e temperature
BB Fan A rotating fan affecting the emissions
CC Surface A 2-D flat surface
DD Radiant Heat Lines Wavy lines intended to communicate "heat" rather than an emission
EE Multi-Panel More than one scene is depicted in the symbol to tell a more complete story
FF Ground The floor or earth is specifically included
GG Surprise Face A facial expression, neither positive nor negative, intended to express surprise
HH Relief Valve A valve, switch, or cut-off handle
II Movement Lines Lines, either straight or curved, intended to show that objects are in motion
Attribute Name
 
 192 
Results of K-means clustering of the consensus attribute matrix for the ?Hot exhaust? 
referent with combined strata. The bolded and underlined rows in the table indicate the 
nearest drawing to the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB CC DD EE FF GG HH II
8 1 Indian 0 0 2 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11 1 American 3 0 1 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12 1 Indian 0 0 2 3 0 0 0 3 3 3 0 1 0 3 3 2 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 1 Indian 0 0 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18 1 American 0 0 0 0 0 0 0 0 3 1 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 1 American 0 0 2 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
24 1 American 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
28 1 American 2 0 1 3 0 2 0 0 3 0 0 0 0 0 0 0 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
29 1 American 0 0 2 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 3 0 0 0 2 3 3 0 0 0 0 0 0 0 0 0
30 1 American 0 0 0 3 0 0 0 0 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0
33 1 American 0 0 2 0 0 0 0 0 3 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0
41 1 American 0 0 2 3 0 3 0 0 2 3 0 0 2 3 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
49 1 Indian 2 0 0 3 0 0 0 0 3 3 0 0 3 0 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
66 1 Indian 0 0 2 3 0 0 0 0 0 0 0 1 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
68 1 Indian 0 0 2 3 0 0 0 0 3 3 0 0 0 1 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0
69 1 Indian 1 2 2 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0 0 0
2 2 American 0 0 3 1 3 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 2 American 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 2 Indian 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19 2 American 0 0 3 2 0 3 0 0 0 0 1 1 0 0 0 0 0 1 3 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0
34 2 Indian 0 0 3 2 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35 2 Indian 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
40 2 American 0 0 0 1 0 3 0 0 1 0 0 0 0 3 1 0 0 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
46 2 Indian 1 0 2 1 0 0 0 0 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 1 0 0 0 0 0
47 2 American 1 0 3 1 0 0 0 0 3 0 0 0 0 1 1 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
50 2 American 0 0 2 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
53 2 American 0 0 1 2 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3
59 2 Indian 3 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 1 0 0 0 0 0
10 3 American 1 0 3 0 0 0 0 0 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16 3 Indian 1 0 3 3 0 0 0 0 3 1 0 0 1 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
37 3 American 0 0 3 3 0 2 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0
43 3 Indian 0 0 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0
51 3 American 0 0 3 3 0 2 0 0 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
56 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
58 3 Indian 1 0 3 3 0 3 0 0 3 0 0 0 0 0 1 1 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0
61 3 American 3 0 3 2 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 0 0 0 0 0
63 3 Indian 3 0 3 3 0 0 0 0 2 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6 4 Indian 0 2 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 4 Indian 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 3 2 0 0 0 0 0 0
36 4 American 0 0 3 0 0 3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 2 0 3 3 0 0 0
39 4 Indian 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
42 4 American 0 0 0 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
45 4 Indian 0 0 3 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0
48 4 Indian 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 2 3 0 0 3 0 0 0 0 0 1 0 0 0 0 0 0
60 4 American 0 0 3 0 0 0 0 0 2 2 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
62 4 American 0 3 3 0 3 0 0 3 0 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
64 4 Indian 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0
70 4 American 0 0 3 0 3 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 3 0 3 0 0 0 0 0
1 5 Indian 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5 5 American 0 0 3 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 5 Indian 0 0 3 3 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13 5 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20 5 Indian 0 0 3 3 0 3 1 0 0 0 0 0 0 0 1 1 0 0 0 0 2 3 0 0 0 0 0 0 0 0 0 0 0 0
21 5 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
25 5 Indian 2 3 1 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
26 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 5 Indian 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32 5 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
38 5 Indian 0 0 3 3 0 0 0 0 0 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
44 5 American 0 0 3 3 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
52 5 American 0 0 3 3 0 0 0 3 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
67 5 American 0 0 3 3 0 3 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0
Drawing 
Number Cluster Stratum
Attribute
 193 
Results of K-means clustering of the consensus attribute matrix for the ?Hot exhaust?, 
U.S. stratum. The bolded and underlined rows in the table indicate the nearest drawing to 
the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB CC DD EE FF GG HH II
2 1 American 0 0 3 1 3 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19 1 American 0 0 3 2 0 3 0 0 0 0 1 1 0 0 0 0 0 1 3 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0
28 1 American 2 0 1 3 0 2 0 0 3 0 0 0 0 0 0 0 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0
36 1 American 0 0 3 0 0 3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 2 0 3 3 0 0 0
37 1 American 0 0 3 3 0 2 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0
40 1 American 0 0 0 1 0 3 0 0 1 0 0 0 0 3 1 0 0 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
41 1 American 0 0 2 3 0 3 0 0 2 3 0 0 2 3 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
44 1 American 0 0 3 3 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
51 1 American 0 0 3 3 0 2 0 0 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
53 1 American 0 0 1 2 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3
67 1 American 0 0 3 3 0 3 0 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0
70 1 American 0 0 3 0 3 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 3 0 3 0 0 0 0 0
10 2 American 1 0 3 0 0 0 0 0 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11 2 American 3 0 1 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18 2 American 0 0 0 0 0 0 0 0 3 1 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 2 American 0 0 2 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
24 2 American 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
29 2 American 0 0 2 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 3 0 0 0 2 3 3 0 0 0 0 0 0 0 0 0
30 2 American 0 0 0 3 0 0 0 0 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0
33 2 American 0 0 2 0 0 0 0 0 3 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0
42 2 American 0 0 0 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
47 2 American 1 0 3 1 0 0 0 0 3 0 0 0 0 1 1 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
60 2 American 0 0 3 0 0 0 0 0 2 2 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
61 2 American 3 0 3 2 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 0 0 0 0 0
3 3 American 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5 3 American 0 0 3 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13 3 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21 3 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32 3 American 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
50 3 American 0 0 2 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
52 3 American 0 0 3 3 0 0 0 3 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
56 3 American 0 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
62 3 American 0 3 3 0 3 0 0 3 0 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Drawing 
Number Cluster Stratum
Attribute
 194 
Results of K-means clustering of the consensus attribute matrix for the ?Hot exhaust?, 
Indian stratum. The bolded and underlined rows in the table indicate the nearest drawing 
to the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB CC DD EE FF GG HH II
8 1 Indian 0 0 2 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12 1 Indian 0 0 2 3 0 0 0 3 3 3 0 1 0 3 3 2 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
25 1 Indian 2 3 1 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
49 1 Indian 2 0 0 3 0 0 0 0 3 3 0 0 3 0 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
66 1 Indian 0 0 2 3 0 0 0 0 0 0 0 1 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
68 1 Indian 0 0 2 3 0 0 0 0 3 3 0 0 0 1 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0
69 1 Indian 1 2 2 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0 0 0
6 2 Indian 0 2 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 2 Indian 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 3 2 0 0 0 0 0 0
34 2 Indian 0 0 3 2 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
39 2 Indian 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
43 2 Indian 0 0 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0
45 2 Indian 0 0 3 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0
48 2 Indian 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 2 3 0 0 3 0 0 0 0 0 1 0 0 0 0 0 0
64 2 Indian 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0
4 3 Indian 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 3 Indian 0 0 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35 3 Indian 0 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
46 3 Indian 1 0 2 1 0 0 0 0 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 1 0 0 0 0 0
59 3 Indian 3 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 1 0 0 0 0 0
1 4 Indian 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16 4 Indian 1 0 3 3 0 0 0 0 3 1 0 0 1 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 4 Indian 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
58 4 Indian 1 0 3 3 0 3 0 0 3 0 0 0 0 0 1 1 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0
63 4 Indian 3 0 3 3 0 0 0 0 2 0 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 5 Indian 0 0 3 3 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20 5 Indian 0 0 3 3 0 3 1 0 0 0 0 0 0 0 1 1 0 0 0 0 2 3 0 0 0 0 0 0 0 0 0 0 0 0
26 5 Indian 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
38 5 Indian 0 0 3 3 0 0 0 0 0 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
Drawing 
Number Cluster Stratum
Attribute
 
 
 195 
Key to the semantic symbol attributes for ?Do not touch with wet hands?  
 
Pictorial Description
A Single Hand One hand or arm without the rest of the human body
B 1-D Surface A single line indicating a surface
C Multiple Water Drops More than one droplet of water
D Prohibition Symbol The traditional circle/slash intended to indicate "Do Not?"
E 2nd Color * The deliberate use of an additional color to emphasize part of the drawing
F Skull/Crossbones The traditional "toxic" or "danger" symbol of a skull and crossbones
G Faucet A simple kitchen or bathroom faucet serving as a source of water
H Prohibition X An "X" or cross used in place of the traditional circle/slash prohibition symbol
I Liquid Container An enclosed volumen intended to suggest that liquid is held inside
J 2-D Panel A 2-D shape representing a surface
K Lightning Bolts Several common, jagged lines representing shock or danger
L Single Water Drop A single droplet of water
M 3-D Object A 3-D shape with a volume
N Multi-Panel More than one scene is depicted in the symbol to tell a more complete story
O Water Ripple Ripples or waves used to portray a liquid
P Spark Any small particulate emission intended to indicate shock or danger
Q Single Lightning Bolt A lone common, jagged line representing shock or danger
R Energized Equipment A generic box or device that is intended to appear electrically sensitive
S Two Hands Two hands or arms are present without the rest of the human body
T Puddle A small amount of water collected on a surface or the ground
U Person A substantial portion of the human body is visible
V Sequence Arrow An arrow inserted to show cause and effect
W Rain Cloud A cloud drawn to represent a weather phenomenon that is emitting water drops
X Surprised Face A facial expression, neither positive nor negative, intended to express surprise
Y Permissable Circle A circle without a slash or "X" intended to portray an action that is good
Z Happy Face A facial expression meant to specifically indicate positive or good feelings
AA Mr. Sparky A specific symbol design of an electric "lightning bolt" inside of a human body
BB Heat Waves Wavy lines intended to communicate "heat" rather than an emission
Attribute Name
 
 196 
Results of K-means clustering of the consensus attribute matrix for the ?Do not touch 
with wet hands? referent with combined strata. The bolded and underlined rows in the 
table indicate the nearest drawing to the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB
8 1 American 3 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14 1 Indian 3 0 3 0 0 3 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 1 Indian 3 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32 1 Indian 0 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
47 1 Indian 3 0 0 2 0 3 1 0 3 0 2 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0
65 1 American 0 0 3 0 2 3 3 0 3 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
68 1 American 3 0 2 0 0 2 0 0 0 1 0 0 3 0 2 0 3 0 0 0 3 0 0 0 0 3 2
5 2 American 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6 2 Indian 3 0 3 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10 2 American 3 0 3 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12 2 Indian 3 0 1 0 0 0 3 0 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18 2 American 3 0 3 0 0 0 3 0 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
23 2 American 1 0 3 0 0 0 3 0 0 3 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0
25 2 American 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
30 2 Indian 3 0 3 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0
33 2 Indian 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
34 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
41 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
42 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
44 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
46 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
50 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
54 2 Indian 3 0 3 1 0 0 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
55 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
57 2 American 3 0 3 0 0 0 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0
58 2 American 3 0 3 0 0 0 3 0 2 0 0 0 0 0 3 0 1 0 0 0 0 0 0 0 0 0 0
64 2 Indian 3 0 3 0 3 0 0 2 2 2 0 0 3 0 3 0 3 0 0 0 0 0 0 0 0 0 0
3 3 Indian 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 3 Indian 3 2 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 3 American 3 2 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11 3 American 3 0 0 3 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13 3 American 3 0 3 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17 3 Indian 3 1 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20 3 American 3 0 0 3 0 0 0 0 3 1 3 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
21 3 Indian 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
24 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0
26 3 American 0 0 2 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 3 0 0 0 0 0 0 0
28 3 American 3 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
29 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
36 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
38 3 American 3 0 3 3 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
39 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
40 3 American 3 0 3 3 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
43 3 American 1 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
48 3 American 3 0 3 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
53 3 Indian 3 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
61 3 American 3 0 3 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
62 3 Indian 2 0 3 3 1 2 0 0 3 0 0 0 3 0 0 3 0 3 0 0 0 0 0 0 0 0 0
66 3 American 3 0 3 3 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 3 0 0
16 4 Indian 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
51 4 Indian 0 3 3 1 0 0 2 0 0 1 0 0 0 0 2 0 0 0 1 2 0 0 0 0 0 0 0
52 4 Indian 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
56 4 American 1 0 3 1 0 0 2 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 3 0 0 0 0
67 4 American 3 0 3 1 0 0 2 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
70 4 American 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Drawing 
Number Cluster Stratum
Attribute
 
 197 
Results of K-means clustering of the consensus attribute matrix for the ?Do not touch 
with wet hands? referent, U.S. stratum. The bolded and underlined rows in the table 
indicate the nearest drawing to the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB
5 1 American 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 1 American 3 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10 1 American 3 0 3 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18 1 American 3 0 3 0 0 0 3 0 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
23 1 American 1 0 3 0 0 0 3 0 0 3 0 0 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0
25 1 American 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
57 1 American 3 0 3 0 0 0 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0
58 1 American 3 0 3 0 0 0 3 0 2 0 0 0 0 0 3 0 1 0 0 0 0 0 0 0 0 0 0
65 1 American 0 0 3 0 2 3 3 0 3 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
67 1 American 3 0 3 1 0 0 2 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0
68 1 American 3 0 2 0 0 2 0 0 0 1 0 0 3 0 2 0 3 0 0 0 3 0 0 0 0 3 2
70 1 American 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 2 American 3 2 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20 2 American 3 0 0 3 0 0 0 0 3 1 3 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0
28 2 American 3 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
38 2 American 3 0 3 3 0 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
56 2 American 1 0 3 1 0 0 2 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0 3 0 0 0 0
11 3 American 3 0 0 3 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13 3 American 3 0 3 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
24 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0
26 3 American 0 0 2 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 3 0 0 0 0 0 0 0
29 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
36 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
39 3 American 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
40 3 American 3 0 3 3 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
43 3 American 1 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
48 3 American 3 0 3 3 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
61 3 American 3 0 3 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
66 3 American 3 0 3 3 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 3 0 0
Drawing 
Number Cluster Stratum
Attribute
 
 198 
Results of K-means clustering of the consensus attribute matrix for the ?Do not touch 
with wet hands? referent, Indian stratum. The bolded and underlined rows in the table 
indicate the nearest drawing to the centroid of the cluster. 
 
A B C D F G H I J K L M N O P Q R S T U V W X Y Z AA BB
3 1 Indian 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
4 1 Indian 3 2 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16 1 Indian 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17 1 Indian 3 1 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21 1 Indian 3 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
30 1 Indian 3 0 3 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0
51 1 Indian 0 3 3 1 0 0 2 0 0 1 0 0 0 0 2 0 0 0 1 2 0 0 0 0 0 0 0
52 1 Indian 3 0 3 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
53 1 Indian 3 0 3 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0
62 1 Indian 2 0 3 3 1 2 0 0 3 0 0 0 3 0 0 3 0 3 0 0 0 0 0 0 0 0 0
6 2 Indian 3 0 3 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12 2 Indian 3 0 1 0 0 0 3 0 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
33 2 Indian 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
34 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
41 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
42 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
46 2 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
54 2 Indian 3 0 3 1 0 0 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
44 3 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
50 3 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
55 3 Indian 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0
64 3 Indian 3 0 3 0 3 0 0 2 2 2 0 0 3 0 3 0 3 0 0 0 0 0 0 0 0 0 0
14 4 Indian 3 0 3 0 0 3 3 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 4 Indian 3 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32 4 Indian 0 0 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0
47 4 Indian 3 0 0 2 0 3 1 0 3 0 2 0 3 0 0 0 0 0 0 0 3 0 0 0 0 0 0
AttributeDrawing 
Number Cluster Stratum
 
 199 
APPENDIX 5.1 
Information letter approved by the auburn university institutional review board for 
designing symbols using interactive evolutionary computation and focus groups. 
 
.
 200 
 
 201 
APPENDIX 5.2 
Instructions and Data Collection form for participants in the DIGA experiment. 
 
The purpose of this research is to understand the kinds of symbols you prefer to see on a 
safety sign.  Safety signs are placed near the location of a hazard to communicate risk. 
 
Your role in this study will be to evaluate a series of simple pictures, or symbols, that 
could be added to a sign to communicate a safety message without using any text at all.  
You will perform this task within a group of approximately 10-20 people.  You will 
evaluate symbols on a computer monitor to determine if they communicate a given safety 
message simply, clearly and completely.  The symbols will be produced by an artificial 
intelligence system on a computer based on the preferences of all members of the group.  
Therefore, they will not always be of the highest artistic quality. When you evaluate 
them, you may assume that the pictures will be redrawn by an artist who will correct any 
small glitches.  Your task will be to anticipate which symbols, if cleaned up and redrawn, 
would be preferred the most by others like you. 
 
Each participant will be assigned his/her own computer.  You will be given a simple 
safety message as well as a brief example of locations where this hazard might be found.  
If you have questions about this safety message, feel free to ask the researchers.  When 
the researchers begin the study, you will see nine symbols randomly arranged on your 
monitor.  Select the symbol that most simply, clearly and completely portrays the 
message.  Once you have selected the best symbol, select the next best remaining symbol, 
continuing in this manner until all nine symbols have been selected.  If you make a 
mistake or would like to change your response, select one of the symbols to unselect it 
and all symbols selected after it.  Reselect the best remaining symbols one at a time until 
all nine have been selected.   
 
Once you have evaluated all nine symbols in this manner, click the ?submit? button and 
wait until everyone finishes this selection round.  When the round is complete, you will 
receive a new set of nine symbols to evaluate.  Please repeat this process until the 
researchers announce that the trial is finished. 
 
Thank you for your participation!  Please complete the information below before you 
begin the activity.   
 
 
Age _________    Gender (circle one):  M  /  F 
In what country were you born? ____________________________ 
For how many years did you live in your birth country? __________________________ 
What country do you consider to be your home country?  _________________________ 
What language do you speak in your home most often?  __________________________ 
Do you consider yourself to speak English fluently? (circle one)  Yes  / No 
At what age do you first remember reading or speaking English fluently? _____________ 
 202 
Instructions and Data Collection form for participants in the Focus Group experiment. 
 
The purpose of this research is to understand the kinds of symbols you prefer to see on a 
safety sign.  Safety signs are placed near the location of a hazard to communicate risk.   
 
Your role in this study will be to design a simple picture, or a symbol, that could be 
added to a sign to communicate a safety message without using any text at all.  You will 
perform this task within a focus group of approximately 10-20 people.  This symbol you 
design should communicate the safety message I will give clearly and completely.  
However, do not worry about making a pretty or high-quality drawing. Artistic skill or 
well-drawn pictures are not important to this research.  As long as you or your group 
members can explain what you picture means, then it will be fine. 
 
You will be drawing your picture on paper at first with no input from others in your focus 
group.  Once each member of your group has created his/her own personal design on 
paper, each of you will reveal all designs to the group and discuss your favorites.  After 
reviewing each member?s designs, the group will determine the best design 
characteristics and combine them into a new, final group design.  This final symbol 
should be drawn on the SmartBoard system which will be saved by the researchers. 
 
You will be given three different symbols to design in this fashion, one at a time.  To help 
you, you will also be given a description of the hazards and locations where symbols like 
your drawing may be needed.   You may take up to 20 minutes to draw your own symbol, 
and then the group will have 20 minutes to discuss and create the final group design. The 
researchers will remind you periodically of the time remaining on each picture, although 
you may have more time if you need it.  After you have completed three symbols in this 
manner, the exercise will be finished.   
 
Thank you for your cooperation!  Please complete the information below before you 
begin the activity.   
 
 
Age _________    Gender (circle one):  M  /  F 
 
In what country were you born? ____________________________ 
 
For how many years did you live in your birth country? __________________________ 
 
What country do you consider to be your home country?  _________________________ 
 
What language do you speak in your home most often?  __________________________ 
 
Do you consider yourself to speak English fluently? (circle one)  Yes  / No 
 
At what age do you first remember reading or speaking English fluently? _____________ 
 203 
APPENDIX 5.3 
Blank forms for drawing symbols during the focus group experiment. 
 
?Hot Exhaust.? 
 
Description: Many processes and pieces of equipment release heated 
air or fumes into the working environment.   
 
WARNING
 
 204 
?Do Not Touch with Wet Hands.? 
 
Description: Many processes and products can be dangerous when 
they become wet.   
 
WARNING
 
 205 
APPENDIX 5.4 
Sample evaluation survey question for ?Do Not Touch with Wet hands? 
 
 206 
 
 
 207 
APPENDIX 5.5 
DIGA Group 1 ?Hot Exhaust? & ?Do Not Touch with Wet Hands? top-ranked symbols 
 208 
DIGA Group 2 ?Hot Exhaust? & ?Do Not Touch with Wet Hands?  top-ranked symbols 
 
 
 
 
 
 209 
DIGA Group 3 ?Hot Exhaust? & ?Do Not Touch with Wet Hands? top-ranked symbols. 
 
 
 
 
 210 
DIGA Group 4 ?Hot Exhaust? & ?Do Not Touch with Wet Hands? top-ranked symbols.