SME Differences and Their Importance on the Content Validity of Job Analysis Ratings by Jennifer Kay Landgren A dissertation submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama August 3, 2013 Keywords: job analysis, subject matter expert ratings, content validity, selection, exam development Approved by Dan Svyantek, Chair, Professor of Psychology Jinyan Fan, Professor of Psychology Hubert S. Field, Professor of Management Philip Lewis, Professor of Psychology John Veres, Professor of Psychology ii Abstract Subject matter expert ratings differences in job analyses were studied to see how these differences may impact the content validity of selection processes. Due to the increased use of content validation strategies in employment testing, it is important to examine any potential rating differences that may arise from a job analysis due to using different types of raters and raters with varying levels of experience. This study examined rating differences among incumbents and supervisors and incumbents with low versus high levels of experience for both the Entry Firefighter and Fire Captain ranks. The findings showed that there were statistically significant differences found in the Fire Captain sample for incumbents and supervisors in the areas of Communication/Documentation, Management/Supervision, Emergency Services Delivery, and Inspections and Investigations and for different levels of experience in the areas of Fire Station Administration and Maintenance, Communication/Documentation, and Management/Supervision. There were no significant differences found in the Entry Firefighter sample. iii Acknowledgments It is with immense gratitude that I acknowledge the support and help of my committee chair, Dr. Dan Svyantek, for your encouragement, guidance and support with this project. I would like to thank my parents for all of their love and support over the years. Last but not least, I would like to thank my son, Anthony Maddox Landgren, for inspiring me to finish my dissertation and to show him how hard work and perseverance will allow you to achieve your dreams and accomplish anything you desire in life. iv Table of Contents Abstract ......................................................................................................................................... ii Acknowledgments ....................................................................................................................... iii List of Tables ................................................................................................................................ v Introduction ................................................................................................................................. 1 Method ...................................................................................................................................... 31 Participants .................................................................................................................... 31 Measures ....................................................................................................................... 32 Statistical Analysis??????????????????????????...37 Results ....................................................................................................................................... 43 Discussion ................................................................................................................................. 64 References ................................................................................................................................. 73 Appendix 1 ................................................................................................................................ 82 Appendix 2 .............................................................................................................................. 106 Appendix 3 .............................................................................................................................. 128 Appendix 4 .............................................................................................................................. 133 v List of Tables Table 1 ....................................................................................................................................... 33 Table 2 ....................................................................................................................................... 34 Table 3 ....................................................................................................................................... 35 Table 4 ....................................................................................................................................... 36 Table 5 ....................................................................................................................................... 44 Table 6 ....................................................................................................................................... 44 Table 7 ....................................................................................................................................... 46 Table 8 ....................................................................................................................................... 46 Table 9 ....................................................................................................................................... 48 Table 10 ..................................................................................................................................... 48 Table 11 ..................................................................................................................................... 50 Table 12 ..................................................................................................................................... 50 Table 13 ..................................................................................................................................... 51 Table 14 ..................................................................................................................................... 52 Table 15 ..................................................................................................................................... 53 Table 16 ..................................................................................................................................... 53 Table 17 ..................................................................................................................................... 55 Table 18 ..................................................................................................................................... 55 Table 19 ..................................................................................................................................... 56 vi Table 20 ..................................................................................................................................... 56 Table 21 ..................................................................................................................................... 57 Table 22 ..................................................................................................................................... 58 Table 23 ..................................................................................................................................... 59 Table 24 ..................................................................................................................................... 59 Table 25 ..................................................................................................................................... 60 Table 26 ..................................................................................................................................... 60 Table 27 ..................................................................................................................................... 62 Table 28 ..................................................................................................................................... 62 Table 29 ..................................................................................................................................... 63 Table 30 ..................................................................................................................................... 65 1 SME Differences and Their Importance on the Content Validity of Job Analysis Ratings INTRODUCTION In the field of personnel selection, job analyses have always played an integral role in the development of selection tools and procedures. Job analysis data are fundamental to understanding the required knowledges, skills, and abilities that are needed to perform the job successfully. Organizations must realize the importance of gathering accurate job analysis data because of how they can adversely affect all employment decisions. If job analysis information is not valid, then the validity of the organization?s human resource management system then comes into question as well (Morgeson & Campion, 1997). Due to the increased use of content validation and validity generalization strategies in employment testing, it is important to examine any potential rating differences that may arise from a job analysis due to using different types of raters and raters with varying levels of experience. This understanding will allow us to be better informed when making employment decisions based on job analysis data and ensuring that every selection procedure is job related. The use of job incumbents and supervisors as the source of job analysis information is the foundation of almost all job analysis techniques (Cornelius & Lyness, 1980; Schmitt & Cohen, 1989). Incumbents and supervisors are the best sources of information because they are either performing the duties of the job or supervise the job. Subject matter experts (SMEs) play an important role in the content validation approach because of the need for accurate and reliable ratings. Interrater agreement among SMEs is a necessary but not sufficient requirement for the validity of job analysis ratings (Harvey, 1991). The content validation approach is dependent upon the accuracy of the information that SMEs provide about the job. Content validation relies 2 on the assumption that the job analysis ratings are independent of the characteristics of the people providing the ratings (Guion, 1978). The literature on job analysis rating differences has focused on differences among the respondents as well as social and environmental factors within the workplace. Research relevant to differences among the raters has assumed that ?job ratings are perceptual assessments of job characteristics and, as such, are not only driven by objective job features but also by raters? idiosyncratic frameworks of reference? (Sanchez & Levine, 1994). Understanding the factors that can lead to variability in KSA ratings can help practitioners by allowing them to address these factors and therefore enhance the validity of their KSA ratings (Burnkrant, 2003). The purpose of this research is to assess if differences exist on mean scores of tasks and KSAs between the different groups of interest (incumbent vs. supervisor and low experience vs. high experience). Identifying the sources of variation in job analysis ratings can help practitioners understand true differences among the different groups and how these individuals perceive the jobs in question. This understanding can guide organizations in how they interpret and apply the job analysis results. The literature review will outline job analysis terminology and how job analysis is used in content validation and its role in test development. I will then review the importance of reliable job analysis data in both test transportability and validity generalization approaches in the use of exams as selection tools. The legal and professional standards will be outlined to explain the need for accurate job analyses in the development of selection tools. To conclude, the different sources of interrater agreement and their potential effects on job analysis results will be discussed. 3 BACKGROUND Terminology The Uniform Guidelines define job analysis as a detailed statement of work behaviors and other information relevant to the job. Work behaviors, or tasks, are activities that are performed to achieve the objectives of the job and consist of ?observable (physical) components and unobservable (mental) components? (Gael, 1988). The knowledges, skills, and abilities that are needed for the performance of tasks are also assessed in a job analysis. A knowledge is defined as a body of information applied directly to the performance of a function. A skill is a present, observable competence to perform a learned psychomotor act. An ability is a present competence to perform an observable behavior that results in an observable product. The first major step in performing a job analysis involves identifying a job?s important tasks. After the tasks have been identified, the knowledge, skills, and abilities that are needed to perform these tasks must be identified and should include any direct reference to specific job tasks when possible (Harvey, 1991). The courts have established the importance of identifying tasks as a requirement for an acceptable job analysis (Thompson & Thompson, 1982). Types of Validity The Uniform Guidelines recognize three validation strategies in order to determine the job relatedness of a selection procedure: criterion-related, content, and construct validation. Criterion-related validation requires the statistical demonstration of the relationship between scores on a selection procedure and job performance of employees. Content validation requires the demonstration that a selection procedure is representative of important aspects of performance on the job. Having an accurate job analysis is critical to establishing content validity because ?content-oriented validation strategies depend heavily on an accurate and 4 complete identification of the behavior domain of the job in question, job analysis plays a pivotal role in the inference process? (Landy & Vasey, 1991). Construct validity is the least common validation strategy, and it requires the demonstration that the selection procedure measures a construct that is important to successful job performance. Although some psychologists believe that criterion-related validation is the most stringent strategy to use to predict job performance, the conditions that need to be met in order for this strategy to be used may not be feasible for most organizations. Most public organizations do not have the financial resources or the appropriate number of individuals to include in this type of study. In addition, organizations that tend to have high turnover would find it difficult to attain reliable measures of job performance of their employees. Content Validity In a content validation situation, job analysis data can be used to support the use of a test and the validity of the individual items (Gael, 1988). If it can be shown that a test measures the important components of a job that are needed for successful performance, it can be concluded that the test is a valid assessment. This validation approach relies heavily on the accuracy of job analysis data (Arthur, Doverspike, & Barrett, 1996). This factor increases the need for reliable ratings from job analysis respondents. The content validity of a job specification is dependent upon whether it identifies all of the KSAs that are predictive of success and excludes those KSAs that are not (Morgeson & Campion, 2000). Low interrater agreement among SMEs may lead to some critical KSAs being excluded from a selection assessment. A job analysis with unreliable and inaccurate ratings would not provide sufficient evidence to support content validation. SMEs play an important role in job analysis because they can provide information used to create job analysis questionnaires, provide ratings for tasks and KSAs, and link the critical tasks with the 5 KSAs needed to perform them. Having SMEs involved in every step of the job analysis process would likely be judged as more content valid (Buster, Roth, & Bobko, 2005; Levine, Maye, Ulm, & Gordon, 1997). Content validation is a popular method used by many organizations that have smaller numbers of employees, inadequate job performance data, and/or limited budgets. According to the Uniform Guidelines (1978), a job analysis used for content validation purposes should include an analysis of the important work behaviors that are required for successful performance on the job, their relative importance, and whether the KSAs measured are needed prior to selection into a position. The Principles (2003) describe how evidence for content validity is established when a strong linkage is demonstrated between the important work behaviors and worker requirements in a job and the content of the selection procedure. A content validity strategy is appropriate when one wants to infer the extent to which a candidate possesses ?a relatively simple proficiency that is a component of the job or knowledge required to perform the job (Lawshe, 1985). In order for a test to be content valid, it must adhere to three criteria (Kirkland v. New York State Department of Correctional Services, 1974): 1. KSAs tested for must be critical and not peripherally related to successful job performance 2. ?portions of the exam should be a ccurately weighted to reflect the relative importance to the job of the attributes for which they test 3. The level of difficulty of the exam material should match the level of difficulty of the job The use of job analysis ratings to determine the relative weighting and importance of component scores is common in content validated tests (Arthur, Doverspike, & Barrett, 1996; 6 Bobko, Roth, & Buster, 2007;). Differences in the importance or weights of job components arising from job analysis data should be integrated into the exam weighting process. The KSAs that are linked to work behaviors that comprise a larger contribution to overall job performance should be associated with a larger weighting in the exam (Arthur, Doverspike, & Barrett, 1996). Work behaviors that make a higher relative content contribution to overall job performance would include those which are deemed as important, frequently performed, take a long time to become proficient in, and have severe error consequences (Arthur et al., 1996). Keeping current job analysis data is necessary to ensure that changes in a job that occur over time are documented. These changes can be used to modify selection, assessment, and training tools as needed. Job analysis data are the foundation for the development of selection tools. Subject matter experts have the responsibility of determining the importance and frequency of tasks that are performed on the job and the KSAs that are needed to perform these tasks. Since content validation requires the use of job analysis data to determine whether a test measures the important components of the job in question, it is becoming more important for organizations to conduct appropriate job analysis studies in order to support their use of selection tools in the hiring of employees. Since many organizations do not have the resources to develop their own selection tools, some have relied on strategies such as test transportability and validity generalization in order to support their use of selection exams that have been developed externally. Test Transportability Test transportability is a solution for organizations and practitioners to use because tests can be applied broadly without the need for local validation (Hoffman & McPhail, 1998). It was 7 officially recognized as a technique to support test validation in Section 7B of the Uniform Guidelines (1978). This technique has received more attention in the research literature recently as organizations search for alternative validation solutions. Evidence for establishing the transportability of a selection procedure can be demonstrated when job specification ratings can be generalized from one organization to another (Van Iddekinge, Putka, Raymark, & Eidson, 2005). The Principles (2003) define transportability as a validation approach in which evidence about the similarity of different work settings is used to infer how the validity of a selection procedure in one setting generalizes to another setting. The most important factor for transporting validity is having an adequate job analysis because the ratings can be used to establish job similarity in order to classify similar jobs for transportability (Gael, 1988). An accurate and reliable job analysis is needed to justify both the content validation and transportability approach. Test transportability can be useful for organizations that do not have the time or financial resources to conduct a local validation study (Hoffman & McPhail, 1998; Pearlman, Schmidt, & Hunter, 1980). According to the Uniform Guidelines, there are a few requirements that must be met to transport evidence of validity from one location to another which include: 1) A criterion?related validation study must have been conducted at another location; 2) The incumbents of the job to which the validity evidence is to be transported must perform substantially the same major work behaviors as incumbents of the job that the criterion-related validation study was conducted; and 3) A test fairness study must be conducted, if feasible, to determine if there are differences among, gender, race, or ethnic group performances on the test. If a test 8 fairness study is not feasible for an organization, it would be advisable for an organization to ensure that the applicants in the new setting and original setting represent a similar applicant pool. Validity Generalization Another technique that is similar to test transportability is the concept of validity generalization. Validity generalization is a specific approach to transportability that refers to the application of meta-analysis to the correlations between an employment test and job performance (McDaniel, 2007; Tippins, 2003). It involves transporting the validity of a selection instrument from one setting to another if it can be shown that the two settings are very similar (Gael, 1988). Before validity generalization was used as a validation approach, the idea that employment test validities are situation specific was the common belief of testing professionals. Situational validity refers to the belief that validity is situational, it varies greatly across similar settings and organizations, and it also requires that a new validation study be conducted in the new setting (Hoffman & McPhail, 1998). Since it was believed that test validities differed depending on the study, local validation studies were required. Due to the work of Schmidt and Hunter (1977), the situational specificity hypothesis was disconfirmed and the results of their meta-analyses provided evidence that the variation in validities across studies that were previously conducted were principally due to sampling error, measurement error, and range restriction. The professional guidelines and research literature lend support to the use of validity generalization because it ?provides reasonable and scientifically defensible evidence for the validity of an employment test? (McDaniel, 2007). The Standards for Educational and Psychological Tests (1999) support the use of validity generalization if a ?close correspondence? of job requirements can be established. Close 9 correspondence is demonstrated when the job requirements and predictor can be shown to be essentially the same (Tippins, 2003). The SIOP Principles (2003) provide some direction on what constitutes job similarity for validity generalization purposes, ?when a systematic new job analysis is not completed, the researcher should compile reasonable evidence which establishes that the jobs in question are similar in terms of work behavior and/or required knowledges, skills, and abilities?. The key to utilizing validity generalization results is to establish a ?linkage study, in a manner very similar to the transportability requirements of the Guidelines. Where a rational chain of evidence has been given, more often than not, positive results have been obtained? (Mahaffey, 1993). The validity generalization approach offers a major advantage for organizations that have financial or resource constraints since less comprehensive job analysis information is needed and a new validation study does not have to be conducted (Hoffman & McPhail, 1998). Although the research literature sometimes discusses the concepts of test transportability and validity generalization synonymously, a major distinction between these two concepts is the extent to which one wants to infer the test validity information. Validity generalization requires a greater degree of generalization than transportability. In test transportability, the validity of a test can be assumed only if the test validation evidence is from a similar situation using the same test. For validity generalization, broader inferences can be made in that a test can be used in a new situation without a validity study if the test is the same type of test and has been shown to be valid for similar jobs to the job in which the validity is to generalize (Gael, 1988; Landy, 2003). For example, test transportability would apply in a situation where an organization is using the Hogan Personality Inventory for similar job classifications because it has been shown to be a 10 valid predictor of performance. Validity generalization refers to when an organization uses comparable personality assessments for similar job classifications, which are not the exact same assessment, but are measuring the same traits and attributes. Due to the increased use of content validation in both test transportability and validity generalization approaches, the need for accurate and reliable job analyses is integral in order for organizations to establish the job relatedness of their selection tools. This demonstration of job relatedness would be essential for organizations if they were challenged in court. If a selection exam can be shown to measure the essential KSAs needed to perform successfully on the job, it would meet the legal and professional standards for the use of job analysis results for the development of selection tools. Legal and Professional Standards Court cases stemming from the use of job analysis results in the selection process generally arise from plaintiffs asserting that there was a major flaw in the job analysis techniques, analyses, results, or inferences (Landy & Vasey, 1991). Griggs v. Duke Power (1971) is recognized as a significant case in employment discrimination law. It established the importance of the concept of job relatedness that requires the defendant to show that the selection procedure that had adverse impact is actually job-related (Harvey, 1991). The defendant?s selection practices for entry-level positions, which included two aptitude tests and required a high school diploma, was criticized by the court for disproportionately excluding blacks and they needed to be based on business necessity and related to job performance. The court ruled that Title VII protected individuals from overt discrimination as well as practices that were intended to be fair but were discriminatory in practice. 11 Albemarle Paper Co. v. Moody (1975) was a landmark case involving job analysis because it explained in detail how job analysis played a role in demonstrating job relatedness. Mill workers argued that the paper company?s testing program, which included a general intelligence test, had adverse impact and was not shown to be related to job performance. The Albemarle decision recognized job analysis as a necessity for organizations in order to defend against any challenged employment practices. This is especially true in the area of transportability where a methodical job analysis is needed in both the original and target jobs (Harvey, 1991). Without a sufficient job analysis, the relationship between the critical work behaviors required for the job and a selection procedure used to measure these behaviors cannot be established. The courts have rejected arguments that are based on an inadequate job analysis. In Dickerson v. U.S. Steel Corp. (1978), the court rejected the validity transportability argument made by the defendant that craft apprentices in two different plants were essentially similar. The court ruled that the job analysis was inadequate; therefore, the conclusion of ?no significant differences? was unfounded (Tippins, 2003). In Guardians Association of the New York Police Department v. Civil Service Commission of New York (1980), a test that was developed by reviewing documents such as job manuals to identify the relevant KSAs was not allowed by the courts because there was no formal job analysis conducted. A linkage study between the critical tasks and KSAs was not conducted to determine the appropriate weights upon which the test should be based. Although job manuals can be used to gather initial information when developing task and KSA statements, these manuals do not provide the necessary importance and frequency ratings that would be produced in a job analysis questionnaire. 12 In EEOC v. Atlas Paper Box (1987/1989), the original district court upheld the defendant?s decision to use the Wonderlic Personnel Test to screen office and clerical workers based on validity generalization. On appeal to the 6th Circuit Court, the court criticized the use of validity generalization because a sufficient demonstration of job similarity was not provided. Without a linkage study or demonstration of job similarity through a site visit, job similarity cannot be established. The court reversed the case because the expert testimony provided no evidence of similarity between the local work situation and prior work situations (Tippins, 2003). The courts have supported test transportability in cases where validation studies have been conducted for similar jobs. In Friend v. City of Richmond (1978), the court found that a firefighter written test based on a validation study in California could be used because the firefighter job in Richmond was very similar to the firefighter job in California. Therefore, the court ruled in favor of transportability for the test because requiring every city to conduct a local validation would be ?ludicrous? (Tippins, 2003). In Youngblood v. Dalzell (1991), a firefighter written exam for the City of Cincinnati was challenged because the defendant used a transportability study to show that the test was valid. Since comparable exams in other cities used for firefighters were found to be reliable, the court ruled that the validity could ?be transported to any city with similar jobs and deemed valid in that city.? In Contreras v. City of Los Angeles (1981), the court ruled that KSA requirements measured in a test must be shown to be predictive of work behaviors that are necessary for successful performance in the job. The court found that the exam in question was content valid because SMEs were used to link test items to the KSAs in the job analysis. The Lanning v. SEPTA (2002) case demonstrated the role of evidence of validity and business necessity to support the use of selection devices that are used with a minimum or lower cut-off. The Circuit 13 Court ruled that the district court had examined sufficient evidence to show that the 12-minute cutoff for a 1.5-mile run measured the qualifications necessary for successful job performance of a SEPTA transit police officer and the cutoff could be used. Various studies have been conducted to examine whether validation of a test in a certain location could be transported to another location without the need for a local validation study. Legal and professional guidelines support the use of validity generalization and test transportability when employers can demonstrate job similarity through job analysis. In addition, professional guidelines dictate that employment testing be job-related and based on job analysis results. It is essential to ensure that any results obtained from a job analysis are accurate because these results are the foundation for the development of selection procedures in every organization. Although some studies have examined whether differences among organizations necessitate the need for local validation, other studies have explored differences within the job itself and the characteristics of the people who perform and supervise the job. According to the Principles (2003), ?the success of the content-based validation study is closely related to the qualifications of the subject matter experts.? Job analysis data will only lead to valid, significant results if raters can accurately and reliably identify the necessary requirements of the job (Green & Stutzman, 1986; Voskuijl & van Sliedregt, 2002). Even if practitioners adhere to professional and legal guidelines, the data collected in the job analysis could be biased or unreliable due to rater differences, and therefore, any selection or training tool using this information as a foundation for their development would be inaccurate and unreliable. All of these factors provide support for why it is important to further analyze the sources of interrater disagreement that can influence ratings provided by job analysis respondents. 14 SOURCES OF INTERRATER DISAGREEMENT Based on the research that has been conducted, there has been no conclusive evidence as to whether these differences among SMEs are due to perceptual differences in how an individual views their job or whether these differences arise from differences in assignments within the same job. Harvey (1991) suggested that these variations arise from differences in task assignments given by supervisors that are based on the subordinate?s performance history, experience levels, job skills, and relationship with their supervisor to create true within-title differences. In certain jobs that allow the individuals a fair amount of discretion in placing emphasis on their job tasks, one could conclude that these individuals would have different perceptions of what their job entails (Conley & Sackett, 1987). In this situation, there may be significant differences in task frequency ratings because the incumbents are actually performing slightly different job duties. Some researchers have suggested that disagreement in job analysis ratings may signify differences in how jobs are perceived by incumbents (Borman, Dorsey, and Ackerman, 1992; Wexley & Silverman, 1978). SME Differences Differences among SMEs themselves may lead to unreliable job analysis data. Stutzman (1983) suggested that the unexplained variance in job analysis ratings is due to the existence of meaningful subgroups within the same job title. Landy and Vasey (1991) attributed these differences to individuals being assigned different functions even though they hold the same job title. The content validity of job specifications is in jeopardy if there are meaningful subgroups within a single job classification because there may be irrelevant KSAs included or relevant KSAs may be excluded (Burnkrant, 2003). Some studies have shown that within-title variance can lead to significant differences in data accuracy. Most studies on job analysis rating 15 differences have attributed these differences to respondent demographic variables such as experience, gender, education, and race and also to respondent performance levels (Borman, Dorsey, & Ackerman, 1992). Gender, Race, and Education Certain demographic variables, such as gender, race, and education, have been examined to determine the extent to which these variables affect job analysis ratings. The relationship of a respondent?s age on job analysis ratings has been studied the least of all the demographic variables. Silverman, Wexley, and Johnson (1984) found no significant differences between younger and older employees on job analysis ratings. The educational level of raters has also been studied. Cornelius and Lyness (1980) found that higher levels of education led to more consistent ratings. Green and Veres (1990) looked at the how education would affect ratings on an infrequency index which assessed the tendency of respondents to rate job irrelevant tasks as important. They found that mental health workers who had higher educational levels tended to score lower on the infrequency index. This finding shows that education did affect job analysis ratings and higher levels of education led to more accurate ratings. Mullins and Kimbrough (1988) examined educational differences for patrol officers in a university police department and concluded that there were no significant effects for education on job analysis ratings. Even though some studies examining education as a moderator in job analysis ratings have found no differences, it is logical to conclude that education is a ?surrogate? for some cognitive abilities (Cornelius & Lyness, 1980). Those with more education may have developed better strategies for processing and remembering information. Sanchez and Levine (1994) studied the effects of rater training in making inferential decisions on job analysis ratings for a group of job incumbents. They found that incumbents who had attained higher 16 levels of education benefited the most from the rater training and had higher levels of interrater agreement. Gender differences were found in one study of middle-level managerial jobs in a state governmental agency in which females reported less frequent involvement in tasks that were related to financial and public speaking job duties (Schmitt & Cohen, 1989). In a study using police officers, it was found that males reported higher frequency than females for tasks involving personal danger (Landy & Vasey, 1991). These differences in task frequency could be due to incumbents choosing to participate in certain tasks or being assigned these tasks by their supervisor. The impact of a respondent?s race on job analysis ratings has been studied as well. Both Landy and Vasey (1991) and Veres, Green, and Boyles (1991) found racial differences in ratings. The only significant difference found for race in Landy and Vasey?s (1991) study of police officers was that White officers were more likely than their Black counterparts to engage in administering first aid. Another study found that race had no effect on ratings (Schmitt & Cohen, 1989). Performance Levels It has been hypothesized that low and high performers may vary in the amount of time spent on certain activities and their ability to judge which knowledges, skills, and abilities are required for successful performance (Conley & Sackett, 1987). Borman, Dorsey, and Ackerman (1992) found a relationship between task frequency ratings in certain job activity areas and sales performance levels of stockbrokers. If a high performing employee is assigned more complex tasks or challenges themselves to perform more complex work behaviors, then the importance ratings of certain KSAs and frequency ratings of tasks may vary from their lower performing 17 counterparts. It is important to include incumbents of various performance levels in the job analysis process because if only high performers provide job analysis ratings, this could result in an overstatement of the KSAs that are required for the job (Conte, Dean, Ringenbach, Moran, & Landy, 2005). Some studies have found no differences in the quality of job analysis ratings for low or high performers (Conley & Sackett, 1987; Wexley & Silverman, 1978). Incumbents and Supervisors Various studies have looked at differences between incumbent and supervisor ratings to determine if those who perform the job and supervise the job perceive the job differently. Incumbents were found to be more accurate than supervisors in their task frequency ratings (Richman & Quinones, 1996). Since incumbents actually perform the tasks on a daily basis, they may have better recall as to the frequency at which tasks are performed. Supervisors have been found to be better than incumbents in specifying the KSAs needed to perform job tasks (Gatewood & Feild, 2001). If supervisors are responsible for training employees on tasks performed on the job, they will be more accurate at identifying the KSAs needed to perform the tasks. In summary, supervisors are more effective at providing KSA information, and incumbents are better at providing task information (Goldstein, Zedeck, & Schneider, 1993). Greater variation in ratings for supervisors could be attributed to these positions being influenced by the individual styles of the people who hold these supervisory positions (Guion, 1978). One study examined the interrater reliability of job analysis ratings for Fire Lieutenants and found that the level of reliability decreased as the raters were further removed from the target position (Arthur et al., 1996). As familiarity with the target position decreased, the level of reliability also decreased from r = .56 (Fire Lieutenants) to r = .17 (Assistant Chiefs). 18 Studies have shown that incumbents and supervisors tend to agree more on the tasks performed than for the attributes required to perform them (Gael, 1988; Manson et al., 2000; Mueller & Belcher, 2000). Whereas tasks are more concrete, KSAs are more abstract in nature that could lead to greater variation in KSA ratings. Manson et al. (2000) found no differences between incumbents and supervisors on task ratings for both the Fire Lieutenant and Police Communications Technician positions. A study conducted by Mueller and Belcher (2000) examined the differences between incumbents and supervisors for the Fire Captain rank. They found that there was substantial agreement for task ratings and only moderate agreement for KSA ratings. Although the authors found moderate to substantial agreement for task and KSA ratings, there were differences found in what the incumbents and supervisors considered to be the most critical tasks and KSAs. When looking at task ratings, incumbents and supervisors differed in what they considered to be the most critical tasks. Incumbent Captains considered tasks related to the management of emergency situations as the most critical whereas their supervisors considered the more routine and supervisory tasks as the most critical. There were also differences found for the importance of KSA ratings. Incumbents were more likely to rank technical knowledges as the most important attributes and supervisors tended to rank certain physical abilities and attributes as the most important. These results reflect the differences in perspectives between incumbents and supervisors in the Fire Captain rank. Supervisors are more focused on the planning and directing of departmental activities that are important to the organization?s mission and incumbents are more focused on the response to emergency situations. Even though rating differences have been found between incumbents and supervisors, it is important to assess various SME viewpoints. The different viewpoints from incumbents and 19 supervisors may just provide a more comprehensive picture of what a job actually entails (Sanchez, Zamora, & Viswesvaran, 1997). These rating differences could reflect how incumbents and supervisors ?attach different meanings to work dimensions and possibly organize work differently? (Gael, 1988). Supervisors are responsible for what the job incumbents need to know in order to perform tasks on the job, whereas incumbents are better at describing what is actually done on the job (Goldstein et al., 1993). In addition, supervisors may have more experience and a better understanding and perspective of how the job plays a role in the entire organization (Bobko, Roth, & Buster, 2008). There is more research needed to determine whether incumbents are better for some judgments such as the time spent on certain tasks and whether supervisors are more accurate in determining what skill levels are needed to perform these tasks (Morgeson & Campion, 1997). Rating differences between incumbents and supervisors have also been examined in the performance appraisal literature and have highlighted raters? inability to provide job ratings that are completely accurate. In addition, rating differences between incumbents and supervisors in the performance appraisal literature have generally shown that incumbents have lower reliabilities than supervisors (Conway & Huffcutt, 1997; Viswesvaran et al., 1996). Both of these studies found that peers had lower reliabilities than supervisors. These findings contradict the findings in the job analysis literature that have found supervisor ratings to have lower interrater reliability than incumbents. Both reliabilities and correlations between sources tended to be higher for lower complexity and nonmanagerial jobs (Conway & Huffcutt, 1997). Higher complexity and managerial jobs may have more variation in their jobs or have the discretion to choose which tasks to perform. In this situation, one would expect reliabilities should be higher for Entry-Level Firefighters than for Fire Captains. 20 Experience Levels/Tenure In addition to examining rating differences between incumbents and supervisors, many studies have focused on the rating differences among incumbents with varying levels of experience. Experience levels have been found to be predictive of job analysis ratings (Arvey et al., 1982; Borman, Dorsey, & Ackerman, 1992; Green & Stutzman, 1986; Landy & Vasey, 1991; Prien, Prien, & Wooten, 2003; Tross & Maurer, 2000). Most of the research in this area deals with task ratings (Tross & Maurer, 2000). One study found that individuals that actually perform the task (i.e., incumbents) and who have low levels of experience were able to provide more accurate task frequency ratings than supervisors or incumbents with high levels of experience (Richman & Quinones, 1996). Task frequency ratings have been shown to differ among incumbents based on their level of experience. Studies have shown that even when individuals have the same job title, the frequency and importance of tasks may differ from one incumbent to another because they may actually be performing slightly different tasks within their job (Landy & Vasey, 1991). Incumbents with more experience either perceive themselves as performing a different job than those with less experience or actually perform a slightly different job (Prien et al., 2003). When individuals with the same job title work in different departments or units within an organization, the employees could be performing some different tasks than others that might also require a different set of KSAs to perform them. One study looked at task frequency ratings among inexperienced and experienced stockbrokers and found that there were significant differences in the frequency of some job duties between the two groups (Borman et al., 1992). It is possible that more experienced incumbents choose to focus on different job activities than their less experienced coworkers or choose different methods to accomplish the job duties 21 which would lead them to define their job differently (Green & Stutzman, 1986; Landy & Vasey, 1991). More experienced incumbents may be allowed more discretion by their supervisors to choose their work activities because of their past performance history. Those incumbents with less experience may still need oversight and guidance from their supervisors and may be assigned more routine tasks that are performed by other less experienced incumbents. Burnkrant (2003) found interrater agreement to be lower among experienced incumbents and suggested that inexperienced incumbents have higher agreement due to performing similar work before beginning to specialize in their position. As incumbents become more experienced, they may choose or be chosen for particular projects and tasks that highlight their skills and strengths. According to studies on role making, new employees experience various tasks and behaviors before getting into their well-defined role (Kahn et al., 1964; Graen, 1976). Particularly with more complex jobs, there may be many different facets of the job that could take several years for an incumbent to become proficient in all areas. In a less complex job such as a clerical position, a new employee may learn all of the required job tasks within a much shorter period of time. Mueller and Belcher (2000) hypothesized that these differences in perspective due to length of experience could be explained by what the incumbents perceive as having the greatest impact on how they perform their own job. In their study of Fire Captains, incumbents who are younger and less experienced may view emergency response as the most important because they are more intrigued by the technical aspects of the job and excitable by the adrenaline rush of fighting fires and saving lives. However, Fire Captains who are older and more experienced place more of an emphasis on being able to respond appropriately, both operationally and physically (Mueller & Belcher, 2000). 22 One explanation for differences in task ratings lies in the fact that incumbents who have been on the job longer have had an opportunity to attain more knowledge about the job, whereas less experienced incumbents may be assigned more routine, simple tasks (Prien, Prien, & Wooten, 2003). Studies have shown that the more information a rater is given about a job, the higher the accuracy of job evaluation ratings (Friedman & Harvey, 1986; Hahn & Dipboye, 1988; Harvey & Lozada-Larsen, 1988; Voskuijl & van Sliedregt, 2002). In addition, incumbents with more experience are more likely to train work-group members that would give them a more comprehensive view of the position and better accuracy when judging the requirements of the job (Green & Stutzman, 1986). Based on these findings, one might conclude that those with more experience on the job would have more accurate ratings since they have had the time to gain more knowledge about the job. However, it seems like most of the literature has shown that incumbents with less experience seem to provide more reliable ratings. Less experienced incumbents have been found to provide more accurate ratings of task frequency than incumbents with higher levels of experience (Tross & Maurer, 2000; Tross & Maurer, 2002). One explanation for this finding is that incumbents with many years of experience on the job may be more likely to automatize job tasks, and therefore, are less accurate in recalling the actual frequency in which they perform these tasks (Veres, Green, & Boyle, 1991). More experienced incumbents may forget the frequency of tasks performed because they have become a part of their daily routine. More experienced incumbents also tend to give higher KSA ratings (Tross & Maurer, 2000; Tross & Maurer, 2002). These differences may be due to the fact that experienced incumbents have a better understanding of how the KSAs are linked to the job tasks and therefore give the KSAs higher importance ratings because of this acquired knowledge. The job experience of the incumbent has also been found to have little to no effect 23 on job analysis ratings (Burnkrant, 2003; Cornelius & Lyness, 1980; Mullins & Kimbrough, 1988; Sanchez & Fraser, 1992; Schmitt & Cohen, 1989; Silverman, Wexley, & Johnson, 1984). When judging task performance, it seems that more experienced SMEs rely on the amount of time spent on the tasks to evaluate and less experienced SMEs rely on the difficulty of learning the task when evaluating tasks (Sanchez & Levine, 2000). A more experienced incumbent will be more likely to give higher frequency and importance ratings to tasks that occupy a majority of their time at work. A less experienced incumbent will tend to give higher task ratings to tasks that they deem to be more difficult to learn and perform. Some studies have examined differences in skill importance ratings between SMEs with different levels of experience. Differences in skill importance ratings were found in one study in which incumbents with low levels of experience gave the lowest skill importance skill importance ratings and incumbents with higher levels of experience gave the highest skill importance ratings (Tross & Maurer, 2002; Veres et al., 1991). However, another study found no differences for skill importance ratings among incumbents (Tross & Maurer, 2000). The reliability of task and KSA ratings has been examined in various studies. Research has shown that task ratings tend to have higher interrater reliabilities than KSA ratings (Dierdorff & Wilson, 2003; Lindell, Clause, Brandt, & Landis, 1998). Interrater agreement has also been shown to be higher for task importance ratings than task frequency ratings, which could be attributed to actual differences in how individuals perform the same job (Lindell et al., 1998). However, these differences could also be attributed to error on the part of the respondent filling out the job analysis questionnaire. One study found that incumbents were indicating that they spent some time performing tasks that could not have been performed (Green & Stutzman, 1986). Pine (1995) had a similar finding where 45% of incumbents indicated that they had spent 24 time performing tasks that were not a part of the job. A study of incumbents and supervisors found that both groups of SMEs were not able to accurately estimate task coverage (Wilson, 1997). Even though some studies have found no significant differences in ratings between different SME groups, the number of studies that have found differences warrant the need for further investigation. Understanding the differences between the groups? different perspectives could assist in organizational planning and development. If an organization were to design a new training program, it would be helpful to have less experienced employees involved in the development of the training. They would be more likely to remember the difficulties they had in learning tasks, which would ensure that training for new employees was at the appropriate difficulty level. Research has shown that job analysis results can differ depending on the sample of SMEs that are utilized in the job analysis process. It is important to remember that ?selection procedures that are content-validated may be biased if the job content domain, as defined by a job analysis, is dependent on the characteristics of the people who hold the job or complete the job analysis ratings? (Veres et al., 1991). If there is strong evidence that different groups of SMEs provide different job analysis results, it could have major consequences for how job analyses are conducted, how the results are interpreted, and how organizations comply with federal guidelines regarding job related selection procedures (Mullins & Kimbrough, 1988). Practitioners should analyze rating differences among different groups to ensure that there are no significant differences. If there are significant differences, then they need to be addressed so that the job analysis results can be interpreted appropriately. 25 Job analysis data are used to justify the inferences that are made to develop selection systems, and if the data is flawed, its utility and legal defensibility may come into question (Harvey & Wilson, 2000). An organizational decision based upon ?erroneous JA data?, particularly in high stakes situations such as selection, is highly likely to be challenged in court (Sanchez & Levine, 2000). If there are differences among different groups of SMEs and how they rate tasks and KSAs, the KSA weightings that are used to develop selection processes can be biased depending upon the composition of SMEs that participate in the job analysis process. A study conducted by Veres (1983) found that racial differences in job analysis ratings led to some work behaviors being rated significantly differently by whites and blacks. Black SMEs rated typing behaviors significantly higher than white SMEs. Based on these results, if a selection process were based on the ratings of the white SMEs, the test developer would construct a selection device that would be different from one that was developed based on the results of the ratings of the black SMEs resulting in a racially biased selection device (Veres, 1983). The black SME-generated exam would increase the importance and weighting of the typing items. Disagreement among job analysis respondents should be an indication that the method used to collect job analysis data is not adequately measuring differences that exist within the position (Burnkrant, 2003). This is especially pertinent for public sector jobs where many individuals perform jobs with the same title (Veres et al., 1991). In this situation, department- specific job analyses should be conducted to capture the real task differences that exist among incumbents with the same job title across various departments. It is important for reliability of job analysis ratings to be assessed because ?reliability becomes a precondition for validity and is necessary for developing any valid prediction process on the basis of job analysis results? 26 (Dierdorff & Wilson, 2003). Both content validity and test transportability approaches to establishing job relatedness rely upon an accurate job analysis. As jobs change and evolve, it is important to maintain valid and up-to-date job analysis information to maximize the efficiency and effectiveness of their recruiting, selection, and training processes (Bobko, Roth, & Buster, 2008). The use of test transportability to establish validity evidence for the use of selection exams is becoming more popular as organizations deal with financial constraints that would prohibit them from conducting a local criterion-related validity study. Since the Uniform Guidelines require organizations to establish job similarity between the jobs in question in order to transport validity evidence, it is crucial that practitioners understand the differences that can arise from job analysis studies in order to address them. Potential outcomes of using biased job analysis data could include adverse impact on the selection procedures being used, increased recruiting costs, and the misidentification of training needs resulting in an insufficiently prepared workforce (Morgeson & Campion, 1997). This is particularly relevant for public safety positions that tend to be litigious environments. Police and fire departments must ensure that their job analyses and the subsequent selection procedures derived from the results of the job analyses follow all professional and federal guidelines to help protect themselves from discrimination lawsuits. OVERVIEW AND HYPOTHESES The present study was conducted using job analysis ratings collected as part of two nationwide job analysis studies for the Entry Firefighter and Fire Captain positions. For Entry Firefighter, the dataset included 141 incumbents and 93 supervisors that rated the importance and frequency of 213 tasks and the importance of 166 KSAs. For Fire Captain, the dataset included 160 incumbents and 48 supervisors that rated the importance and frequency of 85 tasks and the 27 importance of 192 KSAs. The respondents reported their rank and their level of experience. Based on past research findings, it is expected that there will be differences between incumbents and supervisors on both task and KSA ratings. In addition, it is expected that there will be rating differences between respondents who have low experience levels and those who have high experience levels. Rank Effects for Task Frequency: Entry Firefighters Since incumbents know what is actually done on the job and supervisors know what should be done, there will be differences in how Entry Firefighter incumbents and their supervisors rate the frequency at which tasks are performed on the job. H1: Among Entry Firefighters, differences exist in task frequency by rank (incumbent vs. supervisor) Rank Effects for Task Importance: Entry Firefighter Since supervisors have a broader organizational view of the job, they have a better perspective of how the job plays a role in the entire organization (Bobko et al., 2008). Because of this difference in perspective, there will be differences in task importance ratings by Entry Firefighter incumbents and their supervisors. H2: Among Entry Firefighters, differences exist in task importance by rank (incumbent vs. supervisor) Rank Effects for KSA Importance: Entry Firefighter Since supervisors have the responsibility of training employees, they may be better at providing KSA ratings because they are more aware of how important the KSAs are to performing the essential tasks of the job (Morgeson & Campion, 1997). Due to this difference in 28 responsibilities, there will be differences in KSA importance ratings between incumbent Entry Firefighters and their supervisors. H3: Among Entry Firefighters, differences exist in KSA importance by rank (incumbent vs. supervisor) Experience Effects for Task Frequency: Entry Firefighter Since individuals with high levels of experience may have automatized tasks, individuals with low experience levels have a better recollection of the frequency at which they perform tasks on the job. Therefore, differences in task frequency ratings will exist between Entry Firefighters with low levels of experience and those with high levels of experience. H4: Among Entry Firefighters, differences exist in task frequency by years of experience. Experience Effects for Task Importance: Entry Firefighters Individuals with low levels of experience may be more influenced by their specific role in the organization. These differences in perspective due to varying levels of experience can be explained by what individuals perceive as having the greatest impact on how they perform their job (Mueller & Belcher, 2000); therefore, differences in task importance ratings will exist between Entry Firefighters with low levels of experience and those with high levels of experience. H5: Among Entry Firefighters, differences exist in task importance by years of experience. Experience Effects for KSA Importance: Entry Firefighter Individuals with more experience have a better idea of what KSAs are needed to perform the tasks of the job. Since less experienced individuals do not have the same comprehensive view 29 as to which KSAS are needed to perform the tasks, there will be group differences in KSA importance ratings between Entry Firefighters with low experience levels and those with high experience levels. H6: Among Entry Firefighters, differences exist in KSA importance by years of experience. Rank Effects for Task Frequency: Fire Captains Since incumbents know what is actually done on the job and supervisors know what should be done, there will be differences in how Fire Captain incumbents and their supervisors rate the frequency at which tasks are performed on the job. H7: Among Fire Captains, differences exist in task frequency by rank (incumbent vs. supervisor) Rank Effects for Task Importance: Fire Captain Since supervisors have a broader organizational view of the job, they have a better perspective of how the job plays a role in the entire organization (Bobko et al., 2008). Because of this difference in perspective, there will be differences in task importance ratings by Fire Captain incumbents and their supervisors. H8: Among Fire Captains, differences exist in task importance by rank (incumbent vs. supervisor) Rank Effects for KSA Importance: Fire Captain Since supervisors have the responsibility of training employees, they may be better at providing KSA ratings because they are more aware of how important the KSAs are to performing the essential tasks of the job (Morgeson & Campion, 1997). Due to this difference in 30 responsibilities, there will be differences in KSA importance ratings between incumbent Fire Captains and their supervisors. H9: Among Fire Captains, differences exist in KSA importance by rank (incumbent vs. supervisor) Experience Effects for Task Frequency: Fire Captain Since individuals with high levels of experience may have automatized tasks, individuals with low experience levels have a better recollection of the frequency at which they perform tasks on the job. Therefore, differences in task frequency ratings will exist between Fire Captains with low levels of experience and those with high levels of experience. H10: Among Fire Captains, differences exist in task frequency by years of experience (less than five years vs. five years or more). Experience Effects for Task Importance: Fire Captain Individuals with low levels of experience may be more influenced by their specific role in the organization. These differences in perspective due to varying levels of experience can be explained by what individuals perceive as having the greatest impact on how they perform their job (Mueller & Belcher, 2000); therefore, differences in task importance ratings will exist between Fire Captains with low levels of experience and those with high levels of experience. H11: Among Fire Captains, differences exist in task importance by years of experience. Experience Effects for KSA Importance: Fire Captain Individuals with more experience have a better idea of what KSAs are needed to perform the tasks of the job. Since less experienced individuals do not have the same comprehensive view as to which KSAs are needed to perform the tasks, there will be group differences in KSA 31 importance ratings between Fire Captains with low experience levels and those with high experience levels. H12: Among Fire Captains, differences exist in KSA importance by years of experience. METHOD Participants Entry Firefighter Rank The data for the Entry Firefighter rank were collected from January 2008 through February 2008 and in April 2010 as part of a nationwide job analysis study. There were 452 job analysis questionnaire invitations sent by email to Entry Firefighter incumbents and supervisors. Of the 452 questionnaire invitations, 234 (51.8%) job analysis questionnaires were completed online from 12 agencies. There were 4 questionnaire datasets that were deleted and not used for analysis because the respondents had indicated that they had 6 months or less experience on the job. The final sample size for Entry Firefighter was 230. Fire Captain Rank The data for the Fire Captain rank were collected from April 2007 through May 2007 as part of a nationwide job analysis study. There were 388 job analysis questionnaire invitations sent by email to Fire Captain incumbents and supervisors. Of the 388 questionnaire invitations, 210 (54.1%) job analysis questionnaires were completed online from 17 agencies. There were 2 questionnaire datasets that were deleted and not used for analysis because the respondents had indicated that they had 6 months or less experience on the job. The final sample size for Fire Captain was 208. 32 Measures Entry Firefighter Job Analysis Questionnaire The tasks and KSAs for the Entry Firefighter position were identified by reviewing previous job analyses, position descriptions from public agencies, and the National Fire Protection Association standards and requirements for this position. A draft job analysis questionnaire was reviewed by 22 SMEs in January 2008. The SMEs reviewed the questionnaire to ensure that all relevant tasks and KSAs were included. The final job analysis questionnaire included 213 tasks statements and 166 KSA statements. Respondents rated the frequency of the tasks on a six-point scale (0=Not part of the job; 1=Part of the job, but not performed; 2=Performed every few months to yearly; 3=Performed every few weeks to monthly; 4=Performed every few days to weekly; 5=Performed every few hours to daily). Table 1 lists all of the Entry Firefighter tasks and definitions. 33 Table 1 Entry Firefighter Task Categories and Definitions 1 Maintain, Inspect, and Inventory Equipment Tasks involve the maintenance and inspection of personal safety gear and fire equipment and tools. 2 Forcible Entry and Rescue Operations Tasks involve the use of tools and equipment to make forced entries into buildings to rescue confined victims. 3 Training Activities Tasks involves attending and participating in training sessions, courses, and departmental training functions and programs. 4 General Firefighting Activities Tasks involve assisting in assessing fire situation, selecting appropriate tools and equipment for firefighting duties, questioning victims and witnesses, and using fire suppression equipment. 5 Station Duties and Maintenance Tasks involve maintenance of equipment and building and storing equipment and supplies. 6 Salvage and Overhaul Tasks involve reducing the damage from fire, smoke, water, heat, and weather during and after a fire. 7 Hose Evolutions and Extinguishment Tasks involve preparing hose lines, connecting hose to water sources, and operating hose lines with appropriate nozzles. 8 Operates Apparatus Tasks involve driving fire apparatus and operating equipment on apparatus including apparatus controls and aerial ladders. 9 Use and Operations of Ladders Tasks involve selecting appropriate ladder based on type of building and maneuvering ladders to carry people and equipment. 10 Ventilation Procedures Tasks involve the expulsion of heat and smoke from a fire building, permitting the firefighters to find trapped individuals and attack the fire more easily and safely. 11 Community Service Tasks involve giving fire prevention demonstrations and participating in community functions. 12 Respond to Medical Emergency Calls Tasks involve conducting survey of scene for safety and sizeup and implementing treatment protocol. 13 Hazardous Materials Tasks involve identifying hazardous materials at incident scene and performing containment operations. 34 Respondents also rated KSAs on needed at entry on a three-point scale (0=Not Needed; 1=Needed; 2=Essential). Table 2 lists the KSAs and definitions for the Entry Firefighter rank. Table 2 Entry Firefighter KSA Categories and Definitions 1 Reading Comprehension/Understanding Written Material Includes the abilities to read and comprehend written information, follow written directions, and extract specific details from complex written information. 2 Math and Arithmetic Reasoning Includes the abilities to perform mathematical calculations, perform computational conversions of measurement, and calculate friction loss, areas, and distance. 3 Written Communication/Report Writing Includes the knowledge of correct English grammar and the ability to write clear and concise reports and records. 4 Maps, Diagrams, and Mechanical Drawings Includes the abilities to read measurement devices, understand and apply basic mechanical principles, and interpret diagrams and maps. 5 Memory and Understanding Oral Information Includes the abilities to recall factual information from memory, follow oral directions, and remember specific details of past events. 6 Oral Communication Includes the abilities to speak clearly and audibly and to orally communicate effectively with people of different backgrounds. 7 Interpersonal Includes the abilities to maintain effective interactions with individuals, maintain working relationships, and knowledge of effective customer service principles. 8 Other Personal Characteristics Includes the abilities to work effectively under time pressure and stressful conditions, analyze problems quickly and take appropriate action, and adapt quickly to changing priorities. 9 Physical Includes the physical capacity to lift and carry equipment, climb ladders, and to drag and load sections of hose. 10 Medical Includes the knowledge of basic anatomy and physiology and the procedures for evaluating and treating various medical emergencies. 11 Technical Competence Includes the knowledge of fire department policies and procedures, basic chemistry of fire behavior, and skill in operating equipment and tools. 35 In addition, both tasks and KSAs were rated for importance on a four-point scale (0=Not Important to successful job performance; 1=Somewhat Important to successful job performance; 2=Important to successful job performance; 3=Very Important to successful job performance). Fire Captain Job Analysis Questionnaire The tasks and KSAs for the Fire Captain position were identified by reviewing previous job analyses, position descriptions from public agencies, and the National Fire Protection Association standards and requirements for this position. A draft job analysis questionnaire was reviewed by 22 SMEs in March 2007. The SMEs reviewed the questionnaire to ensure that all relevant tasks and KSAs were included. The final job analysis questionnaire included 85 tasks statements and 192 KSA statements. Respondents rated the frequency of the tasks on a six-point scale (0=Not part of the job; 1=Part of the job, but not performed; 2=Performed every few months to yearly; 3=Performed every few weeks to monthly; 4=Performed every few days to weekly; 5=Performed every few hours to daily). Table 3 lists the task categories and definitions for the Fire captain rank. Table 3 Fire Captain Task Categories and Definitions 1 Fire Station Administration and Maintenance Tasks include scheduling work assignments, ensuring inspections of fire equipment, monitoring radio communications, and producing operational plans. 2 Communication and Documentation Tasks include preparing reports, updating records and logs, and completing forms. 3 Management/ Supervision Tasks include administering discipline, advising subordinates on personal and job-related problems, and evaluating work performance. 4 Emergency Services Delivery Tasks include determining priority of rescue activities, determining how building should be ventilated, and implementing action plan. 5 Inspections and Investigations Tasks include conducting inspections of buildings, investigates fire safety complaints, and providing fire safety code consultation. 36 Respondents also rated KSAs on needed at entry on a three-point scale (0=Not Needed; 1=Needed; 2=Essential). Table 4 shows the KSA categories and definitions for the Fire Captain rank. Table 4 Fire Captain KSA Categories and Definitions In addition, both tasks and KSAs were rated for importance on a four-point scale (0=Not Important to successful job performance; 1=Somewhat Important to successful job performance; 2=Important to successful job performance; 3=Very Important to successful job performance). 1 Fire Station Administration and Maintenance Includes the ability to allocate resources and knowledge of organizational structure of department and departmental budget process. 2 Communication and Documentation Includes the ability to communicate to staff of decisions and changes, document activities in writing, make effective presentations, and ability to use appropriate grammar in formal business communications. 3 Management/ Supervision Includes the ability to evaluate performance, assign work to subordinates, motivate individuals, and recognize training deficiencies and knowledge of departmental human resource policies. 4 Emergency Services Delivery Includes the ability to effectively implement Incident Command System, serve in command staff, and knowledge of aerial operations, evacuation techniques, and standard operating procedures for emergency operations. 5 Inspections and Investigations Includes the ability to apply the appropriate Uniform Fire Code, determine basic fire cause, and the knowledge of common cause of fire and origin determination, types of evidence, and methods used by arsonists. 6 General Knowledge Includes knowledge of departmental policies and procedures, fire attack strategies, fire prevention techniques, and fire suppression methods. 7 Community and Government Relations Includes the ability to establish and maintain working relationships and work with diverse groups and the knowledge of community demographics and public relations. 8 Health and Safety Includes the ability to identify safety hazards and knowledge of departmental safety policies and procedures, common causes of accidents and injuries, and procedures for conducting an accident investigation. 37 Data Analysis Plan The two sets of data (Entry Firefighter and Fire Captain) were transferred into SPSS version 19.0 for Windows. Data will be examined for accuracy, including the assessment of missing data, inconsistencies in case ID response-sets, inconsistencies with regard to the range of response options provided, and presence of outliers or extreme cases. Participants that do not meet the inclusion criteria (report fewer than six months experience) will be excluded from the analysis. Descriptive statistics will be conducted to describe the sample characteristics and the research variables. Frequencies and percentages will be calculated for categorical or nominal data and means and standard deviations were calculated for interval/ratio data (Howell, 2010). Composite Variables Composite scores will be created for the three variables used in the statistical analysis (task frequency, task importance, and KSA importance). Data will be obtained from survey sections 2 and 3 of the Entry Firefighter and Fire Captain Job Analysis Questionnaires (see Appendices A and B). Task Frequency Data for task frequency for the Entry Firefighter rank will be obtained from Section 2 Task Statements (sections A-M, items 1-212) of the Entry Firefighter Job Analysis Questionnaire. A composite score will be created for each subscale to represent a frequency value for each of the thirteen categories. The composite score is calculated by summing all of the tasks within each subscale (e.g., the composite score for the task category of Maintain, Inspect, and Inventory Equipment is calculated by summing the frequency scores on all 16 of the tasks within this subscale to form the composite). 38 Data for task frequency for the Fire Captain rank will be obtained from Section 2 Task Statements (sections A-E, items 1-85) of the Fire Captain Job Analysis Questionnaire. A composite score will be created for each subscale to represent a frequency value for each of the five categories. (e.g., the composite score for the task category of Fire Station Administration/Maintenance is calculated by summing the frequency scores on all 11 of the tasks within this subscale to form the composite). Task frequency ratings are based on the following scale: 0 = not part of the job; 1 = part of the job, but not performed; 2 = performed very few months to yearly; 3 = performed every few weeks to monthly; 4 = performed every few days to weekly; and 5 = performed every few hours to daily. Task Importance Data for task importance for Entry Firefighter will be obtained from survey Section 2 Task Statements (sections A-M, items 1-213) of the Entry Firefighter Job Analysis Questionnaire. A composite score will be created for each subscale to represent an importance value for each of the 13 categories. (e.g., the composite score for the task category of Maintain, Inspect, and Inventory Equipment is calculated by summing the importance scores on all 16 of the tasks within this subscale to form the composite). Data for task importance for Fire Captain will be obtained from survey Section 2 Task Statements (sections A-E, items 1-85) of the Fire Captain Job Analysis Questionnaire. A composite score will be created for each subscale to represent an importance value for each of the five categories. (e.g., the composite score for the task category of Fire Station Administration/Maintenance is calculated by summing the importance scores on all 11 of the tasks within this subscale to form the composite). 39 Task importance ratings are based on the following scale: 0 = not important; 1 = somewhat important; 2 = important; 3 = very important. KSA Importance Data for knowledge, skills and ability (KSA) importance for Entry Firefighter will be obtained from survey Section 3 Knowledge, Skills and Ability Statements (sections A-K, items 1- 166) of the Entry Firefighter Job Analysis Questionnaire. A composite score will be created for each subscale to represent an importance to successful job performance value for each of the eleven categories (e.g., the composite score for the KSA category of Understanding Written Materials is calculated by summing the importance scores on all 11 of the KSAs within this subscale to form the composite). Data for KSA importance for Fire Captain will be obtained from survey Section 3 Knowledge, Skills and Ability Statements (sections A-H, items 1-192) of the Fire Captain Job Analysis Questionnaire. A composite score will be created for each subscale to represent an importance to successful job performance value for each of the eight categories. (e.g., the composite score for the KSA category of Fire Station Administration/Maintenance is calculated by summing the importance scores on all 8 of the KSAs within this subscale to form the composite). KSA importance ratings are based on the following scale: 0 = not important to successful job performance; 1 = somewhat important to successful job performance; 2 = important to successful job performance; 3 = very important to successful job performance. Since the goal of the proposed research is to assess if differences exist on mean scores between the different groups of interest (incumbent vs. supervisor and low experience vs. high experience), MANOVA and ANOVA are the appropriate statistical analyses. 40 To investigate hypotheses H1 and H7, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in task frequency by rank (incumbent vs. supervisor). The dependent variable is task frequency. This includes frequency of tasks associated with maintain/inspect/and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials for Entry Firefighter and frequency of tasks associated with fire station administration/maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections/investigations for Fire Captain. The independent variable is rank (incumbent vs. supervisor). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. To investigate hypotheses H2 and H8, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in task importance by rank (incumbent vs. supervisor). The dependent variable is task importance. This includes importance of tasks associated with maintain/inspect/and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials for Entry Firefighter and importance of tasks associated with fire station administration/maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections/investigations for Fire Captain. The independent 41 variable is rank (incumbent vs. supervisor). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. To investigate hypotheses H3 and H9, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in KSA importance by rank (incumbent vs. supervisor). The dependent variable is KSA importance. This includes importance of KSAs associated with reading comprehension/understanding written material, math and arithmetic reasoning, written communication/report writing, maps/diagrams/mechanical drawings, memory and understanding oral information, oral communication, interpersonal, other personal characteristics, physical, medical, and technical competence for Entry Firefighter and importance of KSAs associated with fire station administration/maintenance, communication/documentation, management/supervision, emergency services delivery, inspections/investigations, general knowledge, community/government relations, and health/safety for Fire Captain. The independent variable is rank (incumbent vs. supervisor). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. To investigate hypotheses H4 and H10, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in task frequency by years of experience (less than five years vs. five years or more). The dependent variable is task frequency. This includes frequency of tasks associated with maintaining/inspect/and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials for Entry Firefighter and frequency of tasks associated with fire station administration/maintenance, communication/documentation, 42 management/supervision, emergency services delivery, and inspections/investigations for Fire Captain. The independent variable is years of experience (less than five years vs. five years or more). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. To investigate hypotheses H5 and H11, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in task importance by years of experience (less than five years vs. five years or more). The dependent variable is task importance. This includes importance of tasks associated with maintain/inspect/and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials for Entry Firefighter and importance of tasks associated with fire station administration/maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections/investigations for Fire Captain. The independent variable is years of experience (less than five years vs. five years or more). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. To investigate hypotheses H6 and H12, two one-way multivariate analyses of variance (MANOVA) will be conducted to assess differences in KSA importance by years of experience (less than five years vs. five years or more). The dependent variable is KSA importance. This includes importance of KSAs associated with reading comprehension/understanding written material, math and arithmetic reasoning, written communication/report writing, maps/diagrams/mechanical drawings, memory and understanding oral information, oral 43 communication, interpersonal, other personal characteristics, physical, medical, and technical competence for Entry Firefighter and importance of KSAs associated with fire station administration/maintenance, communication/documentation, management/supervision, emergency services delivery, inspections/investigations, general knowledge, community/government relations, and health/safety for Fire Captain. The independent variable is years of experience (less than five years vs. five years or more). One MANOVA will be conducted for Entry Firefighters and one will be conducted for Fire Captains. RESULTS Descriptive Statistics There were 230 entry level firefighters that took part in the study. Frequencies and percentages for entry firefighter demographics are presented in Appendix Table 1. There were 208 fire captains that took part in the study. Frequencies and percentages for fire captain demographics are presented in Appendix Table 2. Hypothesis 1 Among entry level firefighters, what differences exist in task frequency by rank? To examine Hypothesis 1, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task frequency variables (maintain, inspect, and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with 13 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except training 44 activities, general firefighting activities, and operate apparatus, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was significant, violating the assumption. Because of the violation, the more robust Wilks? Lambda statistic will be used for the MANOVA. The results of the MANOVA were not significant, F (13, 167) = 0.86, p = .596, suggesting that there were no differences in the 13 task frequency variables by rank (incumbent vs. supervisor). Results of the MANOVA are presented in Table 5. Means and standard deviations for the 13 task frequency variables by rank are presented in Table 6. Table 5 Results for MANOVA on 13 Task Frequency Variables by Rank for Entry Level Firefighters Source F df1 df2 p Partial ?2 Rank 0.86 13 167 .596 .06 Table 6 Means and Standard Deviations on 13 Task Frequency Variables by Rank for Entry Level Firefighters Incumbent Supervisor ANOVA p Task Frequency M SD M SD Maintain, Inspect, and Inventory Equipment 59.07 8.68 59.03 9.16 .975 Forcible Entry and Rescue Operations 42.88 14.76 45.58 15.13 .230 Training Activities 39.33 10.79 41.32 10.55 .215 General Firefighting Activities 92.44 35.75 96.49 28.08 .411 Station Duties and Maintenance 68.03 11.30 68.90 8.00 .566 45 Salvage and Overhaul 32.58 15.96 31.29 11.26 .545 Hose Evolutions and Extinguishment 44.19 19.18 42.45 13.58 .498 Operate Apparatus 59.30 19.85 58.14 18.54 .691 Use and Operations of Ladders 11.06 5.65 10.73 4.79 .679 Ventilation Procedures 19.42 9.45 19.66 7.98 .858 Community Service 17.14 6.81 17.08 5.45 .944 Respond to Medical Emergency Calls 88.57 23.43 92.06 19.05 .285 Hazardous Materials 15.13 11.85 15.64 10.11 .760 Hypothesis 2 Among entry level firefighters, what differences exist in task importance by rank? To examine Hypothesis 2, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task importance variables (maintain, inspect, and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with 13 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except maintain, inspect, and inventory equipment, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, and community service. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s 46 M test. The result of the test was significant, violating the assumption. Because of the violation, the more robust Wilks? Lambda statistic will be used for the MANOVA. The results of the MANOVA were not significant, F (13, 167) = 0.95, p = .503, suggesting that there were no differences in the 13 task importance variables by rank (incumbent vs. supervisor). Results of the MANOVA are presented in Table 7. Means and standard deviations for the 13 task frequency variables by rank are presented in Table 8. Table 7 Results for MANOVA on 13 Task Importance Variables by Rank for Entry Level Firefighters Source F df1 df2 p Partial ?2 Rank 0.95 13 167 .503 .07 Table 8 Means and Standard Deviations on 13 Task Frequency Variables by Rank for Entry Level Firefighters Incumbent Supervisor ANOVA p Task Frequency M SD M SD Maintain, Inspect, and Inventory Equipment 39.10 6.05 36.40 6.96 .006 Forcible Entry and Rescue Operations 56.56 12.66 53.62 13.30 .133 Training Activities 34.58 7.58 31.60 8.73 .015 General Firefighting Activities 87.80 19.58 79.35 19.60 .005 Station Duties and Maintenance 35.46 7.63 33.30 7.90 .065 Salvage and Overhaul 32.89 9.24 30.44 8.00 .063 Hose Evolutions and Extinguishment 43.29 9.94 40.40 8.65 .043 Operate Apparatus 45.25 11.05 42.48 12.14 .112 Use and Operations of Ladders 11.88 3.08 11.16 2.81 .104 47 Ventilation Procedures 22.37 4.55 20.97 4.63 .045 Community Service 14.13 5.68 12.87 5.22 .127 Respond to Medical Emergency Calls 58.42 11.60 56.48 11.98 .273 Hazardous Materials 23.87 8.23 21.58 8.52 .071 Hypothesis 3 Among entry level firefighters, what differences exist in KSA importance by rank? To examine Hypothesis 3, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the KSA importance variables (reading comprehension/understanding written material, math and arithmetic reasoning, written communication/report writing, maps, diagrams, and mechanical drawings, memory and understanding oral information, oral communication, interpersonal, other personal characteristics, physical, medical, and technical competence) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with 11 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except math and arithmetic reasoning. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was significant, violating the assumption. Because of the violation, the more robust Wilks? Lambda statistic will be used for the MANOVA. The results of the MANOVA were not significant, F (11, 164) = 1.27, p = .249, suggesting that there were no differences in the 11 KSA importance variables by rank 48 (incumbent vs. supervisor). Results of the MANOVA are presented in Table 9. Means and standard deviations for the 11 KSA importance variables by rank are presented in Table 10. Table 9 Results for MANOVA on 11 Knowledge Importance Variables by Rank for Entry Level Firefighters Source F df1 df2 p Partial ?2 Rank 1.27 11 164 .249 .08 Table 10 Means and Standard Deviations on 11 Knowledge Importance Variables by Rank for Entry Level Firefighters Incumbent Supervisor ANOVA p Task Frequency M SD M SD Reading Comprehension/Understanding Written Material 14.42 3.11 14.23 2.81 .678 Math and Arithmetic Reasoning 19.53 6.34 19.29 5.38 .791 Written Communication/Report Writing 10.43 3.23 10.84 2.96 .385 Maps, Diagrams, and Mechanical Drawings 14.76 3.86 14.31 3.88 .441 Memory and Understanding Oral Information 9.46 2.05 9.28 1.93 .565 Oral Communication 9.09 2.39 9.52 1.99 .207 Interpersonal 21.95 4.41 22.59 3.60 .308 Other Personal Characteristics 29.33 6.22 29.83 4.50 .556 Physical 84.51 19.93 84.15 18.51 .901 Medical 68.34 17.90 67.96 15.62 .884 Technical Competence 97.41 22.07 96.15 19.35 .694 49 Hypothesis 4 Among entry level firefighters, what differences exist in task frequency by experience? To examine Hypothesis 4, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task frequency variables (maintain, inspect, and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with 13 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except maintain, inspect, and inventory equipment, training activities, general firefighting activities salvage and overhaul, operate apparatus, and community service, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was significant, violating the assumption. Because of the violation, the more robust Wilks? Lambda statistic will be used for the MANOVA. The results of the MANOVA were not significant, F (13, 86) = 1.21, p = .289, suggesting that there were no differences in the 13 task frequency variables by experience (low vs. high). Results of the MANOVA are presented in Table 11. Means and standard deviations for the 13 task frequency variables by experience are presented in Table 12. 50 Table 11 Results for MANOVA on 13 Task Frequency Variables by Experience for Entry Level Firefighters Source F df1 df2 p Partial ?2 Experience 1.21 13 86 .289 .15 Table 12 Means and Standard Deviations on 13 Task Frequency Variables by Experience for Entry Level Firefighters Low High ANOVA p Task Frequency M SD M SD Maintain, Inspect, and Inventory Equipment 61.67 7.12 57.08 9.34 .072 Forcible Entry and Rescue Operations 42.51 12.30 42.53 15.91 .001 Training Activities 40.25 11.09 38.76 10.21 .005 General Firefighting Activities 95.18 38.50 89.18 30.72 .007 Station Duties and Maintenance 69.49 9.04 66.98 13.03 .013 Salvage and Overhaul Hose Evolutions and Extinguishment 32.33 46.53 16.19 20.90 31.33 40.18 13.51 14.52 .001 .031 Operate Apparatus 60.80 18.29 56.86 20.35 .011 Use and Operations of Ladders 11.59 6.34 10.08 4.24 .019 Ventilation Procedures 19.47 10.41 18.51 7.10 .003 Community Service 17.04 6.76 16.94 6.20 .001 Respond to Medical Emergency Calls 88.55 24.06 87.98 23.25 .001 Hazardous Materials 14.45 12.81 15.20 9.60 .001 51 Hypothesis 5 Among entry level firefighters, what differences exist in task importance by experience? To examine Hypothesis 5, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task importance variables (maintain, inspect, and inventory equipment, forcible entry and rescue operations, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, hose evolutions and extinguishment, operate apparatus, use and operations of ladders, ventilation procedures, community service, respond to medical emergency calls, and hazardous materials) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with 13 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except maintain, inspect, and inventory equipment, training activities, general firefighting activities, station duties and maintenance, salvage and overhaul, and community service. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were not significant, F (13, 86) = 1.36, p = .195, suggesting that there were not differences in the 13 task importance variables by experience (low vs. high). Results of the MANOVA are presented in Table 13. Means and standard deviations for the 13 task frequency variables by experience are presented in Table 14. Table 13 Results for MANOVA on 13 Task Importance Variables by Experience for Entry Level Firefighters Source F df1 df2 p Partial ?2 Experience 1.36 13 86 .195 .17 52 Table 14 Means and Standard Deviations on 13 Task Importance Variables by Experience for Entry Level Firefighters Low High ANOVA p Task Importance M SD M SD Maintain, Inspect, and Inventory Equipment 39.00 6.20 39.47 5.80 .002 Forcible Entry and Rescue Operations 53.71 13.44 59.24 11.39 .048 Training Activities 34.00 8.37 35.02 6.98 .004 General Firefighting Activities 86.71 22.71 89.04 16.32 .004 Station Duties and Maintenance 35.37 7.46 35.51 7.75 .001 Salvage and Overhaul 32.37 9.57 34.04 8.31 .009 Hose Evolutions and Extinguishment 42.71 10.40 44.49 8.93 .009 Operate Apparatus 44.61 10.51 46.12 11.16 .005 Use and Operations of Ladders 11.80 3.05 12.12 3.03 .003 Ventilation Procedures 21.80 4.47 23.18 4.22 .025 Community Service 13.92 6.47 14.59 4.80 .003 Respond to Medical Emergency Calls 57.16 12.03 60.73 9.82 .026 Hazardous Materials 21.94 9.01 26.24 6.50 .071 Hypothesis 6 Among entry level firefighters, what differences exist in KSA importance by experience? To examine Hypothesis 6, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the KSA importance variables (reading comprehension/understanding written material, math and arithmetic reasoning, written communication/report writing, maps, diagrams, and mechanical drawing, memory and 53 understanding oral information, oral communication, interpersonal, other personal characteristics, physical, medical, and technical competence) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with 11 Kolmogorov Smirnov (KS) tests. The tests were significant for all variables except math and arithmetic reasoning, interpersonal, and physical. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were not significant, F (11, 86) = 0.48, p = .913, suggesting that there were no differences in the 11 KSA importance variables by experience (low vs. high). Results of the MANOVA are presented in Table 15. Means and standard deviations for the 11 KSA importance variables by experience are presented in Table 16. Table 15 Results for MANOVA on 11 KSA Importance Variables by Experience for Entry Level Firefighters Source F df1 df2 p Partial ?2 Experience 0.48 11 86 .913 .06 Table 16 Means and Standard Deviations on 11 KSA Importance Variables by Experience for Entry Level Firefighters Low High ANOVA p KSA Importance M SD M SD Reading Comprehension/Understanding Written Material 14.24 2.83 14.75 3.19 .007 Math and Arithmetic Reasoning 19.04 5.93 20.06 6.63 .007 54 Written Communication/Report Writing 9.98 3.31 10.92 3.04 .022 Maps, Diagrams, and Mechanical Drawings 14.54 3.91 15.04 3.67 .004 Memory and Understanding Oral Information 9.42 1.98 9.58 2.00 .002 Oral Communication 8.84 2.26 9.42 2.43 .015 Interpersonal 22.08 4.15 22.04 4.37 .001 Other Personal Characteristics 29.46 5.44 29.77 5.60 .001 Physical 82.66 17.97 87.73 18.76 .019 Medical 67.40 18.22 70.42 15.44 .008 Technical Competence 96.04 22.17 99.31 21.04 .006 Hypothesis 7 Among fire captains, what differences exist in task frequency by rank? To examine Hypothesis 7, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task frequency variables (fire station admin and maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections and investigations) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with five Kolmogorov Smirnov (KS) tests. The tests were all not significant, meeting the assumption. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were significant, F (5, 202) = 3.83, p = .002, suggesting there were simultaneous differences in the task frequency variables by rank. Individual ANOVAs were examined to assess where the differences lie. The ANOVAs showed that 55 communication/documentation (p = .021), management/supervision (p < .001), emergency services delivery (p < .001) and inspections and investigations (p = .014) all had significant differences by rank. All four of these scores were significantly higher for the supervisors than they were for the incumbents (see Table 6). Results of the MANOVA are presented in Table 17. Means and standard deviations for the five task frequency variables are presented in Table 18. Table 17 Results for MANOVA on Five Task Frequency Variables by Rank for Fire Captains Source F df1 df2 p Partial ?2 Rank 3.83 5 202 .002 .09 Table 18 Means and Standard Deviations on Five Task Frequency Variables by Rank for Fire Captains Incumbent Supervisor ANOVA p Task Frequency M SD M SD Fire station admin and maintenance 50.51 5.39 51.84 5.61 .180 Communication/documentation 36.33 6.63 39.14 6.59 .02 Management/supervision 36.20 6.84 40.81 7.16 .001 Emergency services delivery 110.90 19.36 124.19 20.30 .001 Inspections and investigations 25.11 6.25 28.08 8.03 .014 Hypothesis 8 Among fire captains, what differences exist in task importance by rank? To examine Hypothesis 8, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task importance variables (fire station admin 56 and maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections and investigations) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with five Kolmogorov Smirnov (KS) tests. The tests were only significant for emergency services delivery, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were not significant, F (5, 202) = 0.69, p = .631, suggesting there were no simultaneous differences in the task importance variables by rank. Results of the MANOVA are presented in Table 19. Means and standard deviations for the five task frequency variables are presented in Table 20. Table 19 Results for MANOVA on Five Task Importance Variables by Rank for Fire Captains Source F df1 df2 p Partial ?2 Rank 0.69 5 202 .631 .09 Table 20 Means and Standard Deviations on Five Task Importance Variables by Rank for Fire Captains Incumbent Supervisor ANOVA p Task Frequency M SD M SD Fire station admin and maintenance 29.36 5.37 29.27 5.05 .929 Communication/documentation 23.44 6.13 23.32 6.98 .920 Management/supervision 29.32 4.88 28.41 6.14 .328 Emergency services delivery 86.58 9.50 87.81 9.85 .478 Inspections and investigations 20.35 5.99 19.97 6.79 .735 57 Hypothesis 9 Among fire captains, what differences exist in KSA importance by rank? To examine Hypothesis 9, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the KSA importance variables (fire station admin and maintenance, communication and documentation, management/supervision, emergency services delivery, inspections and investigations, general knowledge, community and government relations, and health and safety) by rank (incumbent vs. supervisor). Prior to analysis, the assumption of normality was assessed with eight Kolmogorov Smirnov (KS) tests. The tests were only significant for health and safety, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were significant, F (8,199) = 3.17, p = .002, suggesting there were simultaneous differences in the KSA importance variables by rank. The individual ANOVAs were examined for significance. Only inspections and investigations was significant by rank, suggesting that the incumbents had significantly higher scores than the supervisors for inspections and investigations. Results of the MANOVA are presented in Table 21. Means and standard deviations for the five KSA importance variables are presented in Table 22. Table 21 Results for MANOVA on Eight KSA Importance Variables by Rank for Fire Captains Source F df1 df2 p Partial ?2 Rank 3.17 8 199 .002 .11 58 Table 22 Means and Standard Deviations on Eight KSA Importance Variables by Rank for Fire Captains Incumbent Supervisor ANOVA p KSA Importance M SD M SD Fire station admin and maintenance 18.08 4.56 18.22 5.18 .874 Communication and documentation 33.08 5.72 33.03 6.09 .958 Management/supervision 54.88 10.21 55.38 10.82 .791 Emergency services delivery 129.06 22.52 132.14 24.13 .459 Inspections and investigations 28.71 7.13 26.08 8.19 .049 General knowledge 104.18 21.07 108.27 21.65 .288 Community and government relations 13.69 4.61 14.54 5.58 .329 Health and safety 23.76 3.91 23.76 3.70 .996 Hypothesis 10 Among fire captains, what differences exist in task frequency by experience? To examine Hypothesis 10, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task frequency variables (fire station admin and maintenance, communication/documentation, management/supervision, emergency services delivery, and inspections and investigations) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with five Kolmogorov Smirnov (KS) tests. The tests were all not significant, meeting the assumption. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. 59 The results of the MANOVA were not significant, F (5, 164) = 1.05, p = .391, suggesting there were no simultaneous differences in the task frequency variables by experience. Results of the MANOVA are presented in Table 23. Means and standard deviations for the five task frequency variables are presented in Table 24. Table 23 Results for MANOVA on Five Task Frequency Variables by Experience for Fire Captains Source F df1 df2 p Partial ?2 Experience 1.05 5 164 .391 .03 Table 24 Means and Standard Deviations on Five Task Frequency Variables by Experience for Fire Captains Low High ANOVA p Task Frequency M SD M SD Fire station admin and maintenance 50.93 5.46 50.17 5.35 .005 Communication/documentation 37.03 6.77 35.79 6.53 .009 Management/supervision 37.13 7.49 35.48 6.31 .014 Emergency services delivery 110.46 17.40 111.12 20.82 .001 Inspections and investigations 24.77 6.75 25.32 5.92 .002 Hypothesis 11 Among fire captains, what differences exist in task importance by experience? To examine Hypothesis 11, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the task importance variables (fire station admin and maintenance, communication/documentation, management/supervision, emergency services 60 delivery, and inspections and investigations) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with five Kolmogorov Smirnov (KS) tests. The tests were only significant for emergency services delivery, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was not significant, meeting the assumption. The results of the MANOVA were significant, F (5, 164) = 2.76, p = .020, suggesting there were simultaneous differences in the task importance variables by experience. Individual ANOVAs were examined to assess where the differences lie. There were significant differences in fire station admin and maintenance (p = .022), communication/documentation (p = .010), and management/supervision (p = .001); those with low experience had significantly higher scores than those with high experience. Results of the MANOVA are presented in Table 25. Means and standard deviations for the five task importance variables are presented in Table 26. Table 25 Results for MANOVA on Five Task Importance Variables by Experience for Fire Captains Source F df1 df2 p Partial ?2 Experience 2.76 5 164 .020 .06 Table 26 Means and Standard Deviations on Five Task Importance Variables by Experience for Fire Captains Low High ANOVA p Task Frequency M SD M SD Fire station admin and maintenance 30.48 5.00 28.57 5.53 .022 Communication/documentation 24.86 5.72 22.41 6.26 .010 61 Management/supervision 30.75 4.29 28.31 5.06 .001 Emergency services delivery 86.76 9.93 86.39 9.26 .805 Inspections and investigations 21.07 6.15 19.84 5.88 .188 Hypothesis 12 Among fire captains, what differences exist in KSA importance by experience? To examine Hypothesis 12, a multivariate analysis of variance (MANOVA) was conducted to assess if there were differences in the KSA importance variables (fire station admin and maintenance, communication and documentation, management/supervision, emergency services delivery, inspections and investigations, general knowledge, community and government relations, and health and safety) by experience (low vs. high) for incumbents only. Prior to analysis, the assumption of normality was assessed with eight Kolmogorov Smirnov (KS) tests. The tests were only significant for health and safety, violating the assumption. However, Stevens (2007) suggests that the MANOVA is robust against violations of normality provided that there are more than 30 participants. The assumption of equality of covariance was assessed with a Box?s M test. The result of the test was significant, violating the assumption. Therefore, the Wilks? Lambda will be interpreted. The results of the MANOVA were not significant, F (8, 161) = 1.72, p = .098, suggesting there were not simultaneous differences in the KSA importance variables by experience. Results of the MANOVA are presented in Table 27. Means and standard deviations for the eight KSA importance variables are presented in Table 28. 62 Table 27 Results for MANOVA on Eight KSA Importance Variables by Experience for Fire Captains Source F df1 df2 p Partial ?2 Experience 1.72 8 161 .098 .08 Table 28 Means and Standard Deviations on Eight KSA importance Variables by Experience for Fire Captains Low High ANOVA p KSA Importance M SD M SD Fire station admin and maintenance 18.83 4.25 17.55 4.74 .071 Communication and documentation 34.13 5.19 32.33 6.01 .044 Management/supervision 57.31 8.30 53.21 11.14 .010 Emergency services delivery 128.80 21.68 129.11 23.28 .930 Inspections and investigations 29.45 6.50 28.15 7.56 .243 General knowledge 105.41 21.50 103.12 20.84 .487 Community and government relations 14.48 4.51 13.09 4.63 .053 Health and safety 23.93 3.49 23.63 4.22 .620 Summary Although there were no significant differences found among SMEs for the Entry Firefighter rank, there were significant differences found for the Fire Captain rank. These differences were found in task frequency by rank, KSA importance by rank, and task importance by experience for the variables that are listed in Table 29. Table 29 shows the breakdown of results for all of the hypotheses for both the Entry Firefighter and Fire Captain ranks. 63 Table 29 Summary of Results for Entry Firefighter and Fire Captain Overall Significance MANOVA Significant ANOVAS Comparison of Rating Differences Entry Firefighter Fire Captain Entry Firefighter Fire Captain Task Frequency by Rank (H1 & H7) F(13,167)=0.86, p = .596 F(5,202)=3.83, p = .002 None Communication/Documentation (p =.021) Management/Supervision (p = .001) Emergency Services Delivery (p = .001) Inspections & Investigations (p = .014) Task Importance by Rank (H2 & H8) F(13,167)=0.95, p = .503 F(5,202)=0.69, p = .631 None None KSA Importance by Rank (H3 & H9) F(11,164)=1.27, p = .249 F(8,199)=3.17, p = .002 None Inspections & Investigations (p = .049) Task Frequency by Experience (H4 & H10) F(13,86)=1.21, p = .289 F(5,164)=1.05, p = .391 None None Task Importance by Experience (H5 & H11) F(13,86)=1.36, p = .195 F(5,164)=2.76, p = .020 None Fire Station Admin & Maintenance (p =.022) Communication/Documentation (p = .010) Management/Supervision (p = .001) KSA Importance by Experience (H6 & H12) F(11,86)=0.48, p = .913 F(8,161)=1.72, p = .098 None None 64 DISCUSSION Comparison of Entry Firefighters and Fire Captains The ratings of both an Entry Firefighter and Fire Captain job analysis questionnaire were analyzed in this study to determine if significant differences could be found between incumbents and supervisors and those with low experience and high experience in the field. Entry Firefighter is the entry level position in all fire departments. Fire Captain is the first line supervisor in all fire departments. Firefighters who are promoted to the Fire Captain position are coming from the Entry Firefighter position. Although some departments have the Fire Lieutenant position in between the Entry Firefighter and Fire Captain rank, a majority of fire departments do not utilize this position. Entry Firefighters perform all of the tasks associated with fire suppression and prevention. Fire Captains also perform fire suppression and prevention tasks; however, they perform these tasks at a higher level. In addition, they supervise the Entry Firefighters in the performance of all of their tasks. Although it is important for incumbents in both positions to have the technical knowledge and physical skills to perform the job, the Fire Captains must also have the leadership and supervisory skills to manage their subordinates and the incident scene. Table 30 shows the differences in duties performed between the Entry Firefighter and Fire Captain position in the task categories that comprise the positions. 65 Table 30 Entry Firefighter and Fire Captain Differences in Duties Performed Although these two positions have some similarities in what is performed on the job, a clear distinction between these positions is the higher difficulty level of tasks performed for Fire Captains and the need for leadership and supervisory skills. Task Categories ENTRY FIREFIGHTER FIRE CAPTAIN Station Policy Receives orders and instructions Writes station policy; reads journal and written communications from prior tours to determine what activities were conducted; informs firefighters about changes in policy Station Duties Carries out station duties Plans daily schedule of station duties Incident Plan Positions vehicle for use Plans action and options en route to incident in order to take control upon arrival Search & Rescue Searches for victim under direction of officer; performs rescue Evaluates conditions to determine if they are safe for firefighters to proceed with rescue Tools & Equipment Operates and tests equipment Directs crew to which tools and equipment to use; demonstrates equipment and procedures during drills Ventilation Ventilates structure Confers with Incident Commander regarding proposed ventilation tactics Training Attends training Trains firefighters Community Service Presents general fire safety info to public; participates in community events Keeps firefighters up to date with current community events; approves station visits Interpersonal Serves as team member Offers support to subordinates during training and scene of incidents Performance Appraisal (does not perform) Assesses performance Record Keeping (does not perform) Records and maintains records 66 Summary of Results The results of the MANOVA analyses showed that there were significant differences between groups for three out of the twelve hypotheses. When looking at task frequency ratings by rank, there were no significant differences found for Entry Firefighters but significant differences were found among Fire Captain incumbents and supervisors. These differences were found in task areas such as Communication/Documentation, Management/Supervision, Emergency Services Delivery, and Inspections and Investigations. In all of these task areas, scores were significantly higher for supervisors than for incumbents. Supervisors know what should be done on the job and incumbents know what is actually done on the job. Since incumbents are the ones actually performing the job, they may be more accurate at determining the frequency at which tasks are performed on the job (Richman & Quinones, 1996; Goldstein et al., 1993). When examining differences in task importance ratings by rank, there were no significant differences found between incumbents and supervisors for either Entry Firefighters or Fire Captains. When examining differences in KSA importance ratings by rank, there were no differences found among Entry Firefighter incumbents and supervisors but differences were found among Fire Captain incumbents and supervisors for Inspections and Investigations. Incumbents rated the importance of these KSAs significantly higher than supervisors. Although Fire Captain supervisors rated the frequency of Inspection and Investigation tasks significantly higher than incumbents, it seems like the incumbents rated the various knowledge, skills, and abilities needed to perform these tasks significantly higher. Studies have shown that supervisors are better at providing KSA ratings since they are more knowledgeable of what the incumbents 67 need to know in order to perform the tasks of the job since they have a better understanding and perspective of how the job fits into the entire organization and its mission (Goldstein et al., 1993; Bobko et al., 2008). For task frequency rating differences among those who have low levels of experience and those with high levels of experience, no significant differences were found for either Entry Firefighters or Fire Captains. Although several studies have shown that incumbents with less experience are more accurate at providing task frequency ratings, this study did not show any significant differences between incumbents with low levels of experience and incumbents with high levels of experience (Richman & Quinones, 1996; Tross & Maurer, 2000 & 2002; Veres et al., 1991). When examining differences in task importance ratings by level of experience, there were no significant differences found among Entry Firefighters incumbents. For Fire Captains, significant differences were found for Fire Station Administration and Maintenance, Communication and Documentation, and Management/Supervision. Incumbents with low levels of experience had significantly higher scores than incumbents with high levels of experience. When examining the tasks for Fire Station Administration and Maintenance, Communication and Documentation, and Management/Supervision, the differences found between the less experienced and more experienced may be due to the fact those who are less experienced are performing most of these tasks for the first time in their careers. Therefore, these tasks are more salient to them and deemed to be more important. Although Fire Captains may have had exposure to these task areas as entry firefighters, these tasks would have become a more integral part of their job as first line supervisors. These rating differences due to length of experience 68 could be attributed to differences in perspective in what incumbents perceive to have the greatest impact on how they perform their job (Mueller & Belcher, 2000). When investigating KSA importance ratings by levels of experience among incumbents, no significant differences were found for either Entry Firefighters or Fire Captains. Additional analyses were conducted to determine if there were differences in ratings by gender, race, and education. No significant differences were found. One drawback of examining gender and race differences in this study was the relatively small number of females and minorities in the data sample. Impact on Test Transportability and Validity Generalization These findings affect the validity generalization and test transportability of job analysis data and the exams that are developed from this data. If there are SME rating differences by rank and experience levels, the weighting of the KSAs used to develop the exam will be affected. The exam could adversely affect certain groups depending on whether mostly incumbents or supervisors were used or whether those with low versus high experience were used as raters in the job analysis process. An exam that may be valid for one organization may not be valid if used in another organization for a similar position. Test transportability refers to the same test being used in a similar situation. In the public sector, it is common for a test developed from one department for a particular position to be used to test applicants for a similar position in another department. Some departments do not have the resources or staff to develop their own exam. However, if the person conducting the job analysis does not have the appropriate sampling of SMEs to participate, the transportability of the exam may be affected. An exam developed upon a job analysis that only uses incumbents or those with 69 low experience may not be suitable for a position in another department. These groups may have a different perception of what is required of the job. Validity generalization refers to a similar test being used in a similar situation. Validity generalization could be affected if one department used an exam from another department as a foundation to develop their own exam without doing an appropriate job analysis. The two jobs may seem similar based on the job descriptions, but they actually differ enough to develop a different customized exam. Legal Implications Results from this study have shown that significant rating differences do exist for Fire Captains based on whether the respondent is an incumbent or a supervisor or the level of experience in the position. If an organization were to develop an exam process for Fire Captains and only use supervisors to complete the job analysis due to availability of subject matter experts, the written exam developed from this job analysis could be biased. The end result would be an exam that does not truly measure what is done on the job because certain KSAs would be under or overemphasized. A biased exam could have legal implications for the organization. An individual who did not perform well on the written exam could challenge the validity of the exam in court citing that the exam is not representative of the KSAs that are needed to perform successfully on the job because a comprehensive job analysis was not conducted. The exam would then be thrown out and the organization would have to gather the resources to conduct another job analysis in order to develop a new exam. The organization may also have to pay a monetary settlement to the plaintiffs who brought upon the lawsuit. Even if the exam had not been legally challenged, the biased exam could result in employees who were selected into the position who may not be able 70 to perform the tasks on the job because the wrong KSAS were assessed. The employee may stay in the position with low performance or they may get fired and the resources used to train them would be wasted. Recommendations for Best Practices Based on the results of this study and previous research, it is recommended that both incumbents and supervisors be used in the job analysis process. Although it is sometimes more difficult to get supervisors to participate due to their availability, it should be emphasized to the organization the importance of having both groups participate. In addition, supervisors should communicate the importance of participating in the job analysis to incumbents to ensure that ratings are completed in an effective, reliable manner. Otherwise, incumbents may randomly assign ratings to tasks and KSAs in order to just finish the questionnaire. It is also recommended that respondents from different units or departments be included since they may be performing slightly different tasks. If an exam developed from a job analysis is used to test all applicants for a promotional Fire Captain exam, the job analysis should be comprehensive and tap into all of the various units that have incumbents since some units may have specialized tasks. Including SMEs of various experience levels is also recommended for practitioners. Different levels of experience could significantly impact job analysis ratings. Supervisors may be inclined to recommend more experienced SMEs to participate since they are seen as more knowledgeable about the job. In contrast, some supervisors may recommend less experienced SMEs because they may have more availability to participate in the process since they have less responsibilities on the job. 71 After a job analysis is conducted, the data could be filtered based on SME demographics to see if there are significant differences in ratings. If there are significant differences, they can be addressed with the SMEs to clarify any misunderstandings before any selection tools are developed based on the data collected. Conclusion In summary, there were significant differences found between incumbents and supervisors and low experienced and high experienced incumbents for the Fire Captain rank. These rating differences are important to consider when conducting job analyses and interpreting the results. Rating differences among SMEs could lead to different definitions of the job content domain, which could affect the development of selection measures (Veres, 1983). If job analysis data is inaccurate or incomplete, the subsequent judgments based on this data would be incorrect. These judgments could then result in an invalid or even illegal selection device (Gatewood & Feild, 2001). If a practitioner were to use job analysis data in which a particular demographic group scored significantly higher than the rest of the respondents in the study in certain task and KSA areas, the resulting test weights could adversely affect certain groups if a selection process was derived from biased job analysis data. The use of job analysis ratings to determine the weighting and importance of test components is common in content validated tests (Arthur et al., 1996; Bobko et al., 2007). Because of this, the impact of incorrect or biased job analysis data could be a prevalent issue in many organizations that is not being addressed. Most of the research in the job analysis realm has focused on the accuracy of job analysis data. However, more recent job analysis research has shifted to emphasize the importance of moving away from discussing the accuracy of job analysis data and toward discussing the 72 validity of inferences made from job analyses to ensure that the inferences are correct (Harvey & Wilson, 2000; Morgeson & Campion, 2000). It is a commonly accepted belief that validity is not a characteristic of a test or assessment process, but instead of inferences made from a test or assessment process (Guion, 1980; Landy, 1986). The inferences or judgments are made from job analysis data to determine the content of selection tools such as test items, interview questions, or the contents of any other selection process (Gatewood & Feild, 2001). These inferences could be incorrect depending on the group of respondents used in the job analysis study and lead to the development of invalid selection procedures. In conclusion, future research should continue to investigate SME differences in job analysis ratings. Practitioners conducting job analysis studies for the development of selection tools should be aware of these potential differences and carefully analyze their job analysis data to determine if these differences exist within their own study. In particular, organizations that rely heavily on content validated selection tests need to examine the practices they employ to support the inferential leaps made from job analysis data (Sanchez & Levine, 2000). If the decisions and inferences can be supported by job analysis data, the resulting selection tools are more likely to withstand scrutiny by those who challenge the validity and appropriateness of the selection procedures. 73 REFERENCES Albemarle Paper Co. v. Moody (1975). 422 US 405. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. APA, Washington DC. Arthur, W., Doverspike, D., & Barrett, G.V. (1996). Development of a job analysis-based procedure for weighting and combining content-related tests into a single test battery score. Personnel Psychology, 49, 971-985. Arvey, R., Davis, G., McGowen, S., & Dipboye, R. (1982). Potential sources of bias in job analytic processes. Academy of Management Journal, 25(3), 618-629. Bobko, P., Roth, P.L., & Buster, M.A. (2007). The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta- analysis. Organizational Research Methods, 10(4), 689-709. Bobko, P., Roth, P.L., & Buster, M.A. (2008). A systematic approach for assessing the currency (?up-to-dateness?) of job-analytic information. Public Personnel Management, 37(3), 261-277. Borman, W.C., Dorsey, D., & Ackerman,L. (1992).Time-spent responses as time allocation strategies: Relations with sales performance in a stockbroker sample. Personnel Psychology, 45, 763-777. Burnkrant, S.R. (2004). Interrater agreement of incumbent job specification importance ratings: Rater, occupation, and item effects. (Doctoral dissertation, Virginia Polytechnic Institute and State University, 2003). Proquest Digital Dissertations, AAI 3110266. 74 Buster, M.A., Roth, P.L., & Bobko, P. (2005). A process for content validation of education and experience based minimum qualifications: An approach resulting in Federal Court approval. Personnel Psychology, 58, 771-799. Conley, P.R. & Sackett, P.R. (1987). Effects of using high- versus low-performing job incumbents as sources of job analysis information. Journal of Applied Psychology, 72, 434-437 Conte, J.M., Dean, M.A., Ringenbach, K.L., Moran, S.K., & Landy, F.L. (2005). The relationship between work attitudes and job analysis ratings: Do rating scale type and task discretion matter? Human Performance, 18 (1), 1-21. Contreras v. City of Los Angeles (1981). 25 FEP 867. Conway, J.M. & Huffcutt, A.I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, and self-ratings. Human Performance, 10, 331-360. Cornelius, E.T., III, & Lyness, K.S. (1980). A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents. Journal of Applied Psychology, 65, 155-163. Dickerson v. U.S. Steel Corp. (1978). 472 F.Supp. 1304. Dierdorff, E.C. & Wilson, M.A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88, 635-646. EEOC v. Atlas Paper Box Co. (1987). 680 F.Supp.1184. EEOC v. Atlas Paper Box Co. (1989). 868 F.2d 1487. 75 Friedman, L. & Harvey, R.J. (1986). Can raters with reduced job descriptive information provide accurate Position Analysis Questionnaire (PAQ) ratings? Personnel Psychology, 39, 779-790. Friend v. City of Richmond (1978). 588 F.2d 61. Gael, S. (1988). The job analysis handbook for business, industry, and government. New York: John Wiley & Sons. Gatewood, R.D. & Feild, H.S. (2001). Applying job analysis techniques. Human Resource Selection (5th ed.). Fort Worth, TX: Harcourt Brace. Goldsein, I.L., Zedeck, S., & Schneider, B. (1993). An exploration of the job analysis- content validity process. In N. Schmitt & W.C. Borman (Eds.), Personnel selection in organizations. San Francisco, CA: Jossey-Bass. Graen, G. (1976). Role-making processes within complex organizations. In M.D. Dunnette (Ed.), Handbook of industrial and organizational psychology. Chicago: Rand McNally. Green, S.B. & Stutzman, T.M. (1986). An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology, 39, 543-564. Green, S.B. & Veres, J.G. (1990). Evaluation of an index to detect inaccurate respondents to a task analysis inventory. Journal of Business and Psychology, 5, 47-61. Griggs v. Duke Power Co. (1971). 401 US 424. Guardians Association of the New York City Police Department v. Civil Service Commission of the City of New York (1980). 23 FEP 909. Guion, R.M. (1978). Scoring of content domain samples: The problem of fairness. Journal of Applied Psychology, 63(4), 499-506. 76 Guion, R.M. (1980). On trinitarian doctrines of validity. Professional Psychology, 11, 385-398. Hahn, D.C. & Dipboye, R.L. (1988). Effects of training and information on the accuracy and reliability of job evaluations. Journal of Applied Psychology, 73, 146-153. Harvey, R.J. (1991). Job analysis. In M.Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed.). Palo Alto, CA: Consulting Psychologists Press. Harvey, R.J. & Lozada-Larsen, S.R. (1988). Influence of amount of job descriptive information on job analysis rating accuracy. Journal of Applied Psychology, 73, 457-461. Harvey, R.J. & Wilson, M.A. (2000). Yes Virginia, there is an objective reality in job analysis. Journal of Organizational Behavior, 21, 829-854. Hoffman, C.C. & McPhail, S.M. (1998). Exploring options for supporting test use in situations precluding local validation. Personnel Psychology, 51(4), 987-1003. Howell, D.C. (2010). Statistical methods for psychology (7th ed.). Belmont, CA: Wadsworth Cengage Learning. Kahn, R.L., Wolfe, D.M., Quinn, R.P., Snoek, J.D., & Rosenthal, R.A. (1964). Organizational stress: Studies in role conflict and role ambiguity. New York: Wiley. Kirkland v. New York State Department of Correctional Services (1974). 628 F.2d 796. Landy, F.J. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41, 1183-1192. Landy, F.J. (2003). Validity generalization: Then and now. In K.R. Murphy (Ed.) Validity generalization: A critical review, 155-195, Mahwah, NJ: Erlbaum. 77 Landy, F.J. & Vasey, J. (1991). Job analysis: The composition of SME samples. Personnel Psychology, 44, 27-50. Lanning v. Southeastern Pennsylvania Transportation Authority SEPTA (2002). 308 F.3d 286. Lawshe, C.H. (1985). Inferences from personnel tests and their validity. Journal of Applied Psychology, 70(1), 237-238. Levine, E.L., Maye, D.M., Ulm, R.A., & Gordon, T.R. (1997). A methodology for developing and validating minimum qualifications (MQs). Personnel Psychology, 50, 1009-1023. Lindell, M.K., Clause, C.S., Brandt, C.J., & Landis, R.S. (1998). Relationship between organizational context and job analysis task ratings. Journal of Applied Psychology, 83, 769-776. Mahaffey, C. (1993). Validity Generalization: Will it become the fourth major line of validity evidence or a pipe dream? Glendale: Psychological Services, Inc. Manson, T.M., Levine, E.L., & Brannick, M.T. (2000). The construct validity of task inventory ratings: A multitrait-multimethod analysis. Human Performance, 13, 1-22. McDaniel, M.A. (2007). Validity generalization as a test validation approach. In S.M. McPhail (Ed.) Alternative Validation Strategies. San Francisco: Jossey-Bass: 159-180. Morgeson, F.P. & Campion, M.A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82, 627-655. 78 Morgeson, F.P. & Campion, M.A. (2000). Accuracy in job analysis: Toward an inference-based model. Journal of Organizational Behavior, 21, 819-827. Morgeson, F.P., Delaney-Klinger, K., Mayfield, M.S., Ferrara, P., & Campion, M.A. (2004). Self presentation processes in job analysis: A field experiment investigating inflation in abilities, tasks, and competencies. Journal of Applied Psychology, 89(4), 674-686. Mueller, M. & Belcher, G. (2000). Observed divergence in the attitudes of incumbents and supervisors as subject matter experts in job analysis: A study of the Fire Captain rank. Public Personnel Management, 29(4), 529-555. Mullins, W.C. & Kimbrough, W.W. (1988). Group composition as a determinant of job analysis outcomes. Journal of Applied Psychology, 73, 657-664. Pearlman, K., Schmidt, F., & Hunter, J. (1980). Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology, 65(4), 373-406. Pine, D. (1995). Assessing the validity of job ratings: An empirical study of false reporting in task inventories. Public Personnel Management, 24(4), 451-460. Prien, K.O., Prien, E.P., & Wooten, W.A. (2003). Interrater reliability in job analysis: Differences in strategy and perspective. Public Personnel Management, 32(1), 125-141. Richman, W.L. & Quinones, M.A. (1996). Task frequency rating accuracy: The effect of task engagement and experience. Journal of Applied Psychology, 81, 512-524. Sanchez, J.I. & Fraser, S.L. (1992). On the choice of scales for task analysis. Journal of Applied Psychology, 77, 545-553. 79 Sanchez, J.I. & Levine, E.L. (1994). The impact of raters? cognition on judgment accuracy: An extension to the job analysis domain. Journal of Business and Psychology, 9, 47-57. Sanchez, J.I. & Levine, E.L. (2000). Accuracy or consequential validity: Which is the better standard for job analysis data? Journal of Organizational Behavior, 21, 809-818. Sanchez, J.I., Zamora, A., & Viswesvaran, C. (1997). Moderators of agreement between incumbent and non-incumbent ratings of job characteristics. Journal of Occupational and Organizational Psychology, 70, 209-218. Schmidt, F.L. & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. Schmitt, N. & Cohen, S.A. (1989). Internal analysis of task ratings by job incumbents. Journal of Applied Psychology, 74, 96-104. Silverman, S.B., Wexley, K.N., & Johnson, J.C. (1984). The effects of age and job experience on employee responses to a structured job analysis questionnaire. Public Personnel Management, 13, 355- 359. Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of personnel selection procedures. Stevens, J. (2002). Applied Multivariate Statistics for the Social Sciences 4th edition. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. Stutzman, T.M. (1983). Within classification job differences. Personnel Psychology, 36, 503-516. 80 Thompson, D.E. & Thompson, T.A. (1982). Court standards for job analysis in test validation. Personnel Psychology, 35, 865-874. Tippins, N. (2003). Transporting the Validity of Assessments. Presented at the International Assessment Congress, Atlanta, GA. Tross, S.A & Maurer, T.J. (2000). The relationship between SME job experience and job analysis ratings: Findings with and without statistical control. Journal of Business and Psychology, 15, 97-110. Tross, S.A. & Maurer, T.J. (2002). Supervisor as SMEs: The relationship between supervisor job experience and ratings of subordinate skill importance. Journal of Business and Psychology, 16, 413-430. United States Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, Department of Justice. (1978). Uniform guidelines on employee selection procedure. Federal Register, 43, (166), 38290-38315. Van Iddekinge, C.H., Putka, D.J., Raymark, P.H., & Eidson, C.E., Jr. (2005). Modeling error variance in job-specification ratings: The influence of rater, job, and organization-level effects. Journal of Applied Psychology, 90(2), 323-334. Veres, J.G. (1983). Bias in Content Validity: An Investigation into Sources of Racial Differences in Job Analysis. (Doctoral Dissertation, Auburn University, 1983). Veres, J.G., Green, S.B., & Boyles, W.R. (1991). Racial differences on job analysis questionnaires: An empirical study. Public Personnel Management, 20, 135-144. Viswesvaran, C., Ones, D., & Schmidt, F. (1996). Comparative Analysis of the Reliability of Job Performance Ratings, Journal of Applied Psychology, 81(5), 557-574. 81 Voskuijl. O.F. & van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: A meta-analysis. European Journal of Psychological Assessment, 18(1), 52-62. Wexley, K.N. & Silverman, S.B. (1978). An examination of differences between managerial effectiveness and response patterns on a structured job analysis questionnaire. Journal of Applied Psychology, 63, 646-649. Wilson, M.A. (1997). The validity of task coverage ratings by incumbents and supervisors: Bad news. Journal of Business and Psychology, 12(1), 85-95. Youngblood v. Dalzell (1991). 925 F.2d 954. 82 APPENDIX 1 ENTRY FIREFIGHTER JOB ANALYSIS QUESTIONNAIRE 241 Lathrop Way Sacramento, CA 95815 (916) 263-3600 Copyright ? 2010 CPS Human Resource Services 83 INSTRUCTIONS CPS is conducting an internal job analysis in an effort to develop an accurate description of the job requirements for the position of Entry Firefighter. The information gathered during this process will be used as the foundation for the creation of a new exam to be utilized in the selection process for future applicants of the Entry Firefighter position. This job analysis questionnaire was developed with the assistance of incumbents and supervisors who are familiar with the requirements of the Entry Firefighter position. They reviewed and edited the task and knowledge, skill, and ability (KSA) statements contained in this questionnaire for appropriateness and comprehensiveness. To obtain an accurate description of the job performed by Entry Firefighters, we need your cooperation in completing this job analysis questionnaire. Individuals familiar with the position of Entry Firefighter (i.e., those who perform the job or supervise those in the position of Entry Firefighter) are required to accomplish this goal. If you are a Entry Firefighter, please respond to the questions in reference to the job that you perform. If you are the immediate supervisor of one or more Entry Firefighters, please respond only in reference to the job of Entry Firefighter within your department/office. Please do not respond in reference to your own supervisory job. The information you provide throughout this questionnaire will be kept confidential and will be pooled with other data. This questionnaire is organized into the following four sections. Section 1: Background/Supplemental Information This section requests general information about the demographic characteristics of people who participate in this survey. Section 2: Task Statements This section includes 213 job task statements to be rated based on their Frequency and Importance. Section 3: Knowledge, Skill, and Ability (KSA) Statements ? This section includes 166 statements describing the knowledge, skills, and abilities (KSAs) necessary for performing the job tasks. These KSAs are to be rated according to (1) their Importance to successful job performance (Importance) and (2) whether an individual is expected to possess the KSA upon entry to the job (Needed at Entry). 84 Selecting Responses Before beginning each section of the questionnaire, please read any directions and rating scales carefully. If you do not understand the directions or the rating scales, please contact Clinton Kelly (ext. 3385) in the CPS Test Rental Department at (916) 263-3600. The goal of this study is to identify the tasks and KSAs that are most important to the job of Entry Firefighter. When assigning ratings, it is important that you consider the entire rating scale. It is likely that some tasks or KSAs are more important than others and your ratings should reflect these differences. SECTION 1. BACKGROUND INFORMATION Before you begin completing the questionnaire, we would like to request some information about your background. This information will allow us to describe the demographic characteristics of the respondents as a whole. For example, questions regarding age will be pooled to describe the average age of participants completing the questionnaire. REMEMBER: Your information will be kept confidential and will be pooled with other respondent?s data. 1. Please indicate your educational level? 01. Less than high school 02. GED/High school proficiency 03. High school graduate 04. Some college education without degree 05. Associates degree 06. Bachelors degree 07. Some post-graduate education without advanced degree 08. Masters degree 09. Doctorate 10. Prefer not to answer 2. What is your age? 01. 30-34 02. 35-39 03. 40-44 04. 45-49 05. 50-54 06. 55-59 07. 60-64 85 3. What is your ethnicity? 01. African American 02. Asian/Pacific Islander 03. Caucasian 04. Filipino 05. Hispanic 06. Native American 07. Other 08. Prefer not to answer 4. What is your gender? 01. Male 02. Female 03. Prefer not to answer 5. What is your relationship to this job? ? Please indicate whether you perform the job now (i.e., Entry Firefighter) or directly supervise the person in the job (i.e., Fire Captain, Battalion Chief) 01. I perform the job now. 02. I directly supervise the person(s) performing the job. 03. I am a higher level manager of the job. 04. Other 6. How long have you been employed by the Fire Department? 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 86 7. Please indicate the length of time you have supervised the job of Entry Firefighter. Only respond to this question if you currently supervise Entry Firefighters (i.e., you are a Fire Captain). 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 8. Please indicate the length of time you have been or were employed performing the job of Captain. For example, if you are currently a Captain, please indicate how long you have been employed as a Captain. If you are a Battalion Chief and are rating the Captain position, please indicate how long you were employed as Captain. 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 87 9. What is your current rank? 01. Entry Firefighter 02. Fire Lieutenant 03. Fire Captain 04. Battalion Chief 05. Other 06. Fire Medic 10. Please indicate how long you have been in your current rank? 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 88 SECTION 2. TASK STATEMENTS For each task, you will be asked to make two ratings, Frequency and Importance. Supervisors: Remember to evaluate the tasks performed by Entry Firefighters, not the tasks that you perform. Frequency Rating Please read each task statement carefully and decide, based on your observations over the last 12 months, how frequently the task is performed in the job you are rating. It is not expected that all tasks are performed on every job. Some tasks are performed more frequently than others. Please indicate how frequently each of the following tasks is performed. Some tasks are part of the job but are not currently performed. For these tasks, it would be appropriate to give a frequency rating of 1 ?Part of the job, but not performed.? Please use the following scale to indicate how frequently each task is performed: FREQUENCY 0 = Not part of the job 1 = Part of the job, but not performed 2 = Performed every few months to yearly 3 = Performed every few weeks to monthly 4 = Performed every few days to weekly 5 = Performed every few hours to daily Importance Rating For each task, after you rate the Frequency of the task, you should next rate the Importance of the task. To assess Importance, ask yourself how seriously overall job performance would be compromised if a task was not performed adequately. Some tasks are more important to overall successful job performance than others. Do not let your Frequency rating influence your Importance rating for the task. Tasks that require a lot of time are not necessarily the tasks that are most important. Please indicate how important each of the following tasks is to successful job performance. Please use the following scale to indicate how important each task is to successful job performance: IMPORTANCE 0 = Not Important to successful job performance 1 = Somewhat Important to successful job performance 2 = Important to successful job performance 3 = Very Important to successful job performance 89 Task Statements A. MAINTAIN, INSPECT, AND INVENTORY EQUIPMENT 1. Places personal safety gear on apparatus (e.g., turnouts, helmet, gloves, flashlight). 2. Locates, inspects, cleans, and properly secures firefighting tools and equipment for readiness (e.g., breathing apparatus, oxygen, nozzles, rescue equipment, hoses, and hand tools) on apparatus. 3. Changes hose loads on all apparatus if they have not been rotated within specified times. 4. Inspects and cleans firefighting protective clothing and safety gear (e.g., helmet, safety boots, gloves, turnouts). 5. Cleans, checks, waxes, and polishes fire apparatus to maintain appearance and inspects for damage (exterior and interior). 6. Checks all fluid levels (e.g., fuel, oil, transmission fluid, radiator fluid, booster tank water, battery fluid) and ensures that they are filled as needed. 7. Cleans, tests, inspects, and refuels electrical generators and rescue tools. 8. Cleans, dries, inspects, and properly secures medical equipment and replaces used medical supplies and equipment. 9. Performs periodic pressure tests of hoses and couplings. 10. Washes, dries, inspects, hangs, and racks hoses and couplings for future use. 11. Makes minor repairs to equipment and tools. 12. Starts apparatus to check engine and pump, ladder, or other systems for proper operation. 13. Cleans, dries, and inspects oxygen or compressed air tanks. 14. Records tools and equipment missing or needing repair or replacement. 15. Tests radios to ensure they are operating properly (e.g., battery level, morning radio check). 16. Determines if ladders are operating properly (e.g., aerial ladder, jack ladder, extension ladder). B. FORCIBLE ENTRY AND RESCUE OPERATIONS 17. Makes forced entries into grounds or buildings by climbing walls, fences, cuts locks, chains, hasps, and bolts to gain entry to locked areas; and breaks or cuts doors, windows, walls, or roofs using hand and power tools. 18. Uses cutting tools (e.g., pickhead and flathead axe, wire and bold cutters, pneumatic cutting tool, rescue saw) to cut through wood or metal barriers. 19. Breaches concrete, brick, or block walls using a sledge hammer. 20. Moves and carries heavy objects, equipment or materials to and from emergency scene to gain access to or free trapped firefighters or victims. 21. Cuts, lifts, or pries open vehicles, machinery or enclosed areas to free persons trapped or pinned inside using appropriate extrication tools. 90 22. Carries or assists victims down ladder or stairs using drags, cots, or improvised equipment and proper lifting techniques. 23. Responds to mass casualty incidents (e.g., train derailment). 24. Performs high angle rescues using recue rope and repel techniques. 25. Removes victims using Stokes? basket, lifelines, and belts. 26. Performs rescue in rough or inaccessible terrain and from hazardous areas (e.g., chemical or gaseous areas). 27. Rescues individuals trapped in elevators. 28. Assists Technical Rescue Operations Team (TROT) by hoisting and lowering victims using the appropriate tools and equipment (e.g., haul system). 29. Removes trapped victims using aerial ladder. 30. Rescues drowning persons using poles, ropes, life rings, buoys, boats, and swimming. 31. Removes persons stranded in or by water using appropriate water rescue tools and equipment. 32. Assists citizens in gaining access or egress from buildings or cars. 33. Enters structures to determine if a rescue situation exists by systematically searching room to room for occupants who are not accounted for. 34. Climbs or crawls through confined spaces. 35. Walks or crawls along joists where balance and careful foot placement are required in order to prevent unnecessary structural damage. 36. Enters, walks, or crawls through smoke-filled areas providing little or no visibility sometimes with charged or uncharged hose in hand. 37. Coils or throws rope and ties various knots and hitches. 38. Locates and rescues victims trapped in burning, smoke-filled buildings. 39. Locates, digs, and shores up walls to free victims in confined or unstable areas who are trapped or unconscious in tunnels, pipes, sewers, excavations, and collapsed buildings. 40. Carries victims to med units or other emergency vehicles using proper lifting techniques. C. TRAINING ACTIVITIES 41. Attends formal fire science technology courses, seminars, conferences, and other non-departmental training functions. 42. Attends daily or periodic training sessions in station; listens to/participates in discussions and demonstrations. 43. Participates in company, station, or department critique and discussion sessions following fires and other emergencies. 44. Attends Emergency Medical Technician (EMT)/Paramedic training and refresher courses. 45. Participates in hazardous materials courses as part of the hazardous materials team on a regular basis, as per policy. 46. Reads, studies, and comprehends firefighting training materials on an individual basis (e.g., essentials manuals, standard operating guidelines, protocols); keeps up to date on pertinent information related to firefighting. 47. Participates in specialty training (e.g., Dive, Hazardous Materials). 91 48. Plans and makes training presentations within the department. 49. Assists and informally trains new firefighters on the job. 50. Studies location and purpose of all equipment on apparatus. 51. Studies and becomes familiar with special equipment supplied only to specific units. 52. Studies and practices safe and appropriate use of tools. 53. Studies and comprehends maps and diagrams of direct routes, structural components of buildings, locations of streets, and hydrants in response area. 54. Observes, participates in, and repeats multi-company drills, station drills, and training programs. 55. Assists personnel in recruitment. 56. Assists personnel in test development and/or administration. D. GENERAL FIREFIGHTING ACTIVITIES 57. Receives information from dispatch (e.g., radio communication system, computer printout) for information about location of building, type of building, and other relevant information. 58. Assists Commanding Officer in determining location of emergency. 59. Assists in determining placement of apparatus considering what other companies may be responding, locations of hydrants, terrain, and traffic conditions. 60. Assesses potential effect of weather, wind, and humidity. 61. Sizes up the fire as part of first responding company, considering amount of fire, unusual odors and colors of smoke and flame. 62. Locates and operates controls to shut off or control utilities (e.g., natural gas, electricity, water). 63. Examines supporting surfaces (e.g., floor, roof, stairs) to judge their stability. 64. Notifies occupants to vacate premises and determines safest evacuation route. 65. Assist in determining location of sprinkler system, hose cabinet, and hazardous material in the building. 66. Makes recommendations to officers regarding appropriate tools, equipment, and procedures. 67. Locates fire source and extinguishes fire using appropriate agent. 68. Calms emotionally distressed or distraught victims, relatives, friends, and spectators and provides assistance to displaced or evacuated persons at emergency scenes. 69. Relays orders from officers to other firefighters. 70. Receives and follows orders and oral instructions from officer at fire scene under conditions of stress, noise, heat, and confusion. 71. Selects appropriate tool or piece of equipment for various firefighting duties. 72. Assists in determining what structures are endangered during a fire. 73. Identifies, contains, removes, or protects flammable or hazardous materials at fire scene (e.g., flammable liquids, gases, combustible solids, explosives, dust) using proper equipment and procedures or blockages from roadways. 74. Operates portable hand operated pump to extinguish grass or brush fires. 92 75. Smothers fires or potential fire restart areas using hand tools (e.g., fire rake, shovel) in order to suppress fire or cover potential fire hazards with dirt during mop up phase. 76. Recognizes conditions that may lead to back-draft or flashover. 77. Coordinates activities with oral, hand, or touch signals. 78. Changes air bottle when depleted. 79. Examines fire structures for any signs of fire extension. 80. Operates elevators using emergency procedures. 81. Questions victims and witnesses about location of fire and occupancy. 82. Stabilizes the scene of an emergency to ensure the safety of victims and rescuers (e.g., secure leaky fuel tank, disconnect battery cables). 83. Uses fire suppression equipment as a precaution to protect people from potential fire hazards (e.g., prepare hose lines for possible fire in incidents involving automobile accidents or gas leaks in buildings). 84. Wears full protective clothing and equipment, including Self Contained Breathing Apparatus (SCBA) 85. Responds to calls about gas leaks. 86. Responds to calls about downed electrical wires. 87. Makes service calls (e.g., assist individuals in lockout situations, assists persons with disabilities who have fallen from beds or wheelchairs). 88. Recognizes and preserves evidence of suspicious fires. 89. Informs officers on scene of discovery of any suspicious materials or conditions. E. STATION DUTIES AND MAINTENANCE 90. Reports inadequate quantities of station stock and medical supplies. 91. Stores equipment and supplies received. 92. Straightens up own quarters including changing linens and making own bed. 93. Locks station doors and windows and secures valuable items from theft, vandalism, and damage at station. 94. Mows, trims, weeds, and waters grass and flower beds; cleans sidewalks, removes litter. 95. Cleans assigned area of fire station and assists other firefighters by dusting, washing, mopping, and waxing. 96. Keeps officer-in-charge informed of any conditions requiring his/her attention. 97. Speaks with firefighters on other shifts to receive information regarding previous day?s activities and changes in equipment status. 98. Maintains friendly relationships with other members of shift under conditions of monotony and of extreme stress for prolonged periods. 99. Communicates with officers and other crew members to keep them apprised of changes in equipment status during non-emergencies. 100. Plans meals, buys food, prepares meals for firefighters in station, washes dishes and cooking utensils, and cleans up dining area following meals. 101. Participates in a program to maintain/improve physical strength and endurance. 102. Reports and prepares for duty on time. 103. Maintains proper attire. 93 104. Sleeps in fire station during shift at appropriate times. F. SALVAGE AND OVERHAUL 105. Protects fire department and civilian property from damage. 106. Ensures fire is extinguished by performing a final hose down. 107. Uses thermal imaging equipment to ensure fire is extinguished. 108. Carries undamaged furniture from burning buildings to reduce fire and smoke damage to building and contents. 109. Covers all furniture and hangings with tarp, moving as needed. 110. Cuts or bores holes in ceilings or walls and constructs various devices with salvage covers to catch, divert, slow, or remove water. 111. Removes water from floors, cellars, or basements using broom, squeegee, mop and bucket, water vacuum, or portable pump. 112. Wedges or clamps sprinkler heads after fire is extinguished. 113. Observes bystanders to protect salvaged property from theft, and gives any personal property found to officer or owner. 114. Locates hidden fire by feeling walls and ceiling or by using a thermal imaging device; exposes fire by opening walls, ceilings, floors, and furniture using appropriate tools. 115. Removes fire debris after fire is extinguished and investigated using various tools and equipment. 116. Performs closeup activities including securing windows and doors with nails, hasps, and locks, and covers openings with wood or reinforced plastic sheet to protect building and contents from weather, and to prevent theft and vandalism. 117. Cleans, dries, and folds tarps after salvage operations. 118. Tears down or shores up weak and dangerous structural components (e.g., floors, overhangs, cornices). 119. Locates and identifies departmental fire equipment used, using knowledge of equipment and storage locations, and replaces equipment and apparatus. 120. Inspects interior of burned buildings after fire to check for signs of rekindling. G. HOSE EVOLUTIONS AND EXTINGUISHMENT 121. Lays hose line from water source to scene of fire, or secures hose line at scene of fire for reverse lay. 122. Rolls or folds hose sections; carries to and from apparatus. 123. Carries or drags charged and uncharged hose lines to fire scene, around obstacles or up ladders. 124. Prepares hose line lays using appliances such as wyes or Siamese adapters. 125. Inspects charged hose lines and removes kinks, takes up slack, and tightens leaking couplings. 126. Pulls booster hose from booster reel and advances to fire scene. 127. Connects and disconnects hose to/from water sources (e.g., hydrants, standpipes) or other hose sections, using appropriate tools. 128. Mounts and/or operates master stream appliances from aerial ladder. 129. Carries equipment and hose to upper floors of buildings. 94 130. Decides on appropriate class of fire extinguisher and operates it. 131. Operates hose lines with straight stream or fog nozzles and with master appliances. 132. Hoists hose sections and other equipment using rope, pike poles, or straps. 133. Replaces burst hose sections with new hose sections. 134. Estimates length of hose needed. 135. Pulls hose, nozzles, and fittings from apparatus. 136. Manipulates nozzle for application of water stream to extinguish fire. 137. Operates monitor by changing nozzles; directing water flow. 138. Reloads hoses, nozzles, and other equipment onto apparatus. H. OPERATE APPARATUS 139. Selects shortest available route, using knowledge of streets and information regarding temporary obstructions. 140. Speaks or signals to driver of apparatus to assist in maneuvering apparatus in close clearance. 141. Drives apparatus according to state and local regulations governing operation of emergency vehicles. 142. Maneuvers apparatus at scene of emergency to occupy best firefighting position and to avoid interfering with other companies. 143. Pumps water to supply hand lines or master appliances, or to supply or supplement standpipe or sprinkler systems. 144. Communicates with dispatcher and other fire vehicles by operating two-way and portable radio in apparatus. 145. Operates throttle and relief valve or pressure regulator and monitors temperature, pressure, vacuum, and tachometer gauges to provide necessary volume and pressure. 146. Calculates pump pressure and friction loss taking into account location and gallons per minute of hydrant, building structure, and location of fire within structure, using established procedures, printed charts, or field hydraulics. 147. Stabilizes apparatus using wheel chocks, stabilizing pads, and stabilizing jacks or outriggers. 148. Elevates, rotates, and extends, aerial ladder bed and fly sections for supported or unsupported operation. 149. Assists truck driver in extending, raising, lowering, and rotating aerial ladder. 150. Assists in mounting and operating stream apparatus on aerial ladder. 151. Assists in operating apparatus controls. 152. Responds to emergency problems (e.g., loss of water) while operating pumper, priming engine pump as needed. 153. Drives apparatus from fire scene to station during in-service and non-emergency runs. 154. Takes assigned riding position with appropriate safety clothing on and wearing seat belt. 155. Ensures that bay doors are completely open before entering or exiting the station. 156. Operates sirens, lights, and other warning devices. 95 157. Stops at fire scene and drops off several lengths of supply hose and drives to hydrant. I. USE AND OPERATION OF LADDERS 158. Selects appropriate ladder based on the type of building, height, angle needed, and determines whether it may be used safely in accordance with weight limitations. 159. Removes, carries, maneuvers, raises, and lowers straight ladders, extension ladders, roof ladders, roof ladders, attic ladders, and A-frame ladders. 160. Climbs, works from, and descends ladders carrying people or equipment and using appropriate safety equipment or procedures. 161. Secures ladder using hose strap to tie ladder to building. 162. Ascends aerial ladder; hoists or lower tools and equipment from top; descends while carrying people or equipment for rescue or fire ground needs. J. VENTILATION PROCEDURES 163. Removes ventilation fans or smoke ejectors from apparatus and positions them in windows or doorways to safely inject fresh air and exhaust heat, smoke, and gasses. 164. Creates ventilation openings by opening or forcing doors and windows and by cutting or breaking walls and roofs using minimum necessary force. 165. Ventilates fire using fog stream application. 166. ?Sounds? to determine if roof is safe. 167. Determines what equipment is needed to ventilate and the appropriate place to cut the ventilation hole based on type of building and type of ventilation (e.g., horizontal, vertical). 168. Prepares artificial lighting to ensure adequate illumination for firefighters working at the scene. 169. Assists in determining what ladders are needed based on strategy of ventilation. 170. Assists in efforts to prevent ?backdraft? explosions by using appropriate ventilation procedures. 171. Starts and operates rescue saw or uses axe to cut hole through roof. K. COMMUNITY SERVICE 172. Collects donations for special campaigns (e.g., toys, canned goods). 173. Gives fire prevention and other demonstrations to educate the public and to promote the department. 174. Conducts station tours and ride alongs. 175. Demonstrates or explains use of fire apparatus, firefighting techniques, and fire safety survival skills. 176. Assists visitors who seek help. 177. Receives and responds to complaints from the public by directing them to the appropriate officer. 178. Participates in community functions (e.g., parades, carnivals). 179. Conducts car seat inspections, installation, and training. L. RESPOND TO MEDICAL EMERGENCY CALLS 96 180. Wears appropriate personal protective equipment according to medical protocols to protect against infectious disease. 181. Conducts a survey of the scene for safety and sizeup; determines scene safety, appropriateness of on-scene treatment versus immediate transport and treatment en route. 182. Determines and implements appropriate transport category (e.g., Code I, II, or III). 183. Recognizes and reports possible child or elder abuse to appropriate agency. 184. Determines and implements treatment protocol [e.g., stop massive bleeding, open airways, facilitate breathing, perform Cardiopulmonary Resuscitation (CPR), defibrillation]. 185. Questions patient or others (e.g., bystanders) to gather patient information (e.g., level of consciousness, current medication, or medical history) to determine nature and extent of medical emergency and treatment needed. 186. Communicates with other units or persons, via radio or cell phone, to relay status of patient, medical information, and requests for additional assistance in order to meet the demands of the emergency. 187. Positions and secures patient on backboard according to appropriate protocol for cervical and spinal immobilization. 188. Places, secures, and covers patient on cot, stair chair, or improvised stretcher for transport. 189. Carries patients on cots, backboards, or chair stretchers up or down stairs, around turns, and through openings. 190. Assists in childbirth-related procedures and ensures sanitary conditions 191. Disposes of protective equipment (e.g., gloves) and needles properly to prevent exposure to infectious diseases. 192. Sanitizes equipment (e.g., cots, backboards) and rescue vehicle after use per protocol. 193. Calms patients and bystanders by talking to them and explaining the situation. 194. Performs triage at Limited Victim Incident (LVI) or Mass Casualty Incident (MCI) scenes. 195. Initiate any additional treatment specified by med control as directed. 196. Maintains vehicles, medical equipment, and communication equipment, and replenishes first-aid equipment and supplies. 197. Secures and tags all narcotics and Intravenous (IV) equipment. 198. Administers treatment to patient while en route to hospital (e.g., administering CPR, monitoring IV, delivering oxygen, providing updates on vitals). 199. Administers the appropriate medication and dosage in order to alleviate/control the patient?s symptoms and records administered medication and dosage. 200. Observes, records, and reports patient?s condition and treatment to the ambulance crew. 201. Reports any information on victim to hospital (e.g., treatment and therapy given, medical history discovered). 202. Completes all paperwork, forms, and medical reports related to medical and rescue emergencies and enters reports into computer system. M. HAZARDOUS MATERIALS 97 203. Assumes position/responsibility during hazardous materials incidents (e.g., safety, research, supply, and support, entry team 1 or 2). 204. Performs size up of hazardous materials incidents, considering weather conditions and information provided by dispatcher and bystanders. 205. Recognizes indications of potential Weapons of Mass Destruction (WMD) incident and implements appropriate procedures. 206. Identifies hazardous material at the incident scene. 207. Refrains from taking action until the nature of the hazardous material is determined. 208. Performs containment operations within hot and warm zones. 209. Identifies and attempts to contain incident through proper means (e.g., absorbing, transferring, covering, plug/dispatching leak). 210. Operates various tools and equipment while containing incident. 211. Transfers chemicals using hoses and other equipment. 212. Performs cleanup or prepares site for cleanup by outside company by placing materials in appropriate containers. 213. Follows appropriate procedures for decontamination of personnel and unit. 98 SECTION 3. KNOWLEDGE, SKILL, AND ABILITY (KSA) STATEMENTS For each KSA you will be asked to make two ratings, Importance and Needed at Entry. Importance Rating Please read each KSA statement carefully and decide its Importance to the job you are rating. Some KSAs are more important to overall successful job performance than others. Please indicate how important each of the following KSAs is to successful job performance. Please use the following scale to indicate how important each KSA is to successful job performance: IMPORTANCE 0 = Not Important to successful job performance 1 = Somewhat Important to successful job performance 2 = Important to successful job performance 3 = Very Important to successful job performance Needed At Entry Rating For each KSA, after you rate the Importance, please rate the KSA for Needed at Entry. To assess Needed at Entry, decide whether an individual is expected to possess the KSA upon the first day of performing as a Entry Firefighter. KSAs that are not needed at entry can normally be acquired once a person is in the job. Please indicate how essential the possession of this knowledge, skill, or ability is at the time of hire for this position. Please use the following scale to indicate how essential each KSA is at the time of hire: NEEDED AT ENTRY 0 = Not Needed 1 = Needed 2 = Essential 99 KSA STATEMENTS A. UNDERSTANDING WRITTEN MATERIALS 1. Ability to extract specific details from complex written information. 2. Ability to combine details from several sources to reach a conclusion. 3. Ability to follow written directions. 4. Ability to read and comprehend written information [e.g., standard operating guidelines, material dealing with the technical aspects of firefighting, Emergency Medical Technician (EMT) and Paramedic training material, medical protocols]. 5. Ability to read and apply information contained in correspondence and reports. 6. Ability to apply logical reasoning principles. B. MATH AND ARITHMETIC REASONING 7. Ability to apply principles to solve practical problems. 8. Ability to count, add, subtract, multiply, and divide whole numbers. 9. Ability to add, subtract, multiply, and divide decimals. 10. Ability to compute fractions and ratios. 11. Ability to compute percentages. 12. Ability to perform computational conversions using metric and standard system of measurement (e.g., drug dosages in metric, convert weight of victim). 13. Ability to calculate friction loss and determine pump discharge pressure. 14. Ability to calculate areas (e.g., square footage) and volumes (e.g., cubic feet). 15. Ability to calculate distances (e.g., linear feet). C. WRITTEN COMMUNICATION/REPORT WRITING 16. Knowledge of correct English grammar, punctuation, and spelling. 17. Skill in writing legibly by hand. 18. Ability to take detailed notes during training. 19. Ability to write clear and concise reports, correspondence, and email. 20. Ability to complete forms, logs, and records. D. MAPS, DIAGRAMS, AND MECHANICAL DRAWINGS/KNOWLEDGE 21. Ability to read gauges, dials, instrumentation, and measurement devices. 22. Ability to remember visual information (e.g., building layouts, maps). 23. Ability to observe details accurately and completely. 24. Ability to recognize the purpose of basic tools. 25. Ability to understand and apply basic mechanical principles (e.g., leverage, force, acceleration, friction, fluid motion, valve systems, pulley systems). 100 26. Ability to draw diagrams and maps by hand. 27. Ability to read and interpret diagrams, maps, floor plans, and geometric figures. E. MEMORY AND UNDERSTANDING ORAL INFORMATION 28. Ability to extract important information from oral communication. 29. Ability to recall factual information from memory (e.g., codes, building locations, types of buildings in district, traffic conditions at different times of day, proper fire extinguishing techniques). 30. Ability to remember specific details of past events (e.g., fire scene, condition of building, building layout). 31. Ability to follow oral directions. F. ORAL COMMUNICATION 32. Ability to express thoughts orally in a clear, organized manner. 33. Ability to orally explain complex information in language that ensures listener understanding. 34. Ability to speak clearly and audibly. 35. Ability to orally communicate effectively with people of different backgrounds and educational levels. G. INTERPERSONAL 36. Ability to work with others as a team. 37. Ability to maintain effective interactions with individuals from diverse ethnic, socio-economic, and cultural backgrounds. 38. Ability to create and maintain working relationships with a variety of individuals and groups. 39. Ability to live harmoniously with others in a community living situation. 40. Ability to handle sensitive public contacts with tact and courtesy. 41. Ability to comfort victims of fires, accidents, or illnesses. 42. Ability to show respect for individual differences. 43. Ability to show respect for authority. 44. Knowledge of effective customer service principles. H. OTHER PERSONAL CHARACTERISTICS 45. Ability to remember information both in the short term and long term. 46. Ability to follow specific directions. 47. Ability to work effectively under time pressure. 48. Ability to analyze problems quickly and take appropriate action under stress. 49. Ability to maintain personal cleanliness. 50. Ability to tolerate adverse sights and smells. 51. Ability to work effectively under extreme weather conditions, hazardous situations, or other adverse situations. 101 52. Ability to accept job associated risk of physical injury or illness. 53. Ability to work effectively under dangerous and stressful conditions. 54. Ability to accept criticism and feedback. 55. Ability to adapt quickly to changing priorities/situations. 56. Ability to awaken frequently throughout the night to respond to emergencies with full alertness. I. PHYSICAL 57. Physical capacity to coordinate hand movements based on what is seen (e.g., eye-hand coordination). 58. Physical capacity to make repeated, rapid movements over a period of less than 15 minutes in which the rapid recovery from muscle strain is critical (e.g., chopping a hole in a roof). 59. Physical capacity to sustain a high level of muscular exertion and physical activity for 15 minutes or longer, performing a sequence of physically demanding operations quickly and safely. 60. Ability to perform a complex learned series of movements rapidly in the proper sequence. 61. Ability to see clearly with or without correction. 62. Ability to recognize and tell the difference between colors. 63. Physical capacity to lift and carry straight, folding, and roof ladders unassisted (e.g., 14-20 foot straight/roof ladders; 8-12 ft folding ladders, 24 ft. extension ladders). 64. Ability to couple and uncouple hose by hand or using appropriate equipment. 65. Physical capacity to advance charged or uncharged 1 ? ? hand lines unassisted. 66. Physical capacity to advance uncharged 2 ?? line assisted. 67. Physical capacity to advance charged, but not discharging, 2 ?? line as a member of a team. 68. Physical capacity to reposition stack in 1 ?? or 2 ?? line unassisted. 69. Physical capacity to direct stream and advance discharging 1 ?? line unassisted. 70. Physical capacity to safely and properly lift, carry, or drag equipment or victims (conscious or unconscious). 71. Physical capacity to climb a ladder carrying equipment and wearing protective clothing including Self Contained Breathing Apparatus (SCBA). 72. Physical capacity to climb an aerial ladder carrying equipment and wearing protective clothing including Self Contained Breathing Apparatus (SCBA). 73. Physical capacity to safely lift and carry one end of a two-person cot bearing the weight of the human body. 74. Physical capacity to bend, stoop, and crawl wearing full protective clothing. 75. Ability to maintain balance while carrying objects or walking on slopes, joists, ladder rungs, and narrow footings. 76. Ability to climb over obstructions (minimum of 4 feet) while wearing safety clothing and equipment. 102 77. Ability to follow orders and function in a situation of zero visibility. 78. Physical capacity to lift, carry, operate and position a smoke ejector/smoke blower. 79. Physical capacity to fully extend a 24 ft. extension ladders unassisted. 80. Physical capacity to carry a hotel pack or hose bundle up a minimum of three flights of stairs unassisted. 81. Physical capacity to perform all tasks wearing protective clothing and equipment (e.g., SCBA) including safety carrying hose and equipment up and down multiple floors. 82. Ability to hoist tools and equipment using a rope. 83. Ability to roll 50 foot sections of hose for replacement on apparatus. 84. Ability to load 50 foot sections of hose onto apparatus. 85. Ability to drag 200 feet of 1 ?? and 2 ?? hose uncharged. 86. Ability to safely lift and carry all departmental extension ladders as a member of a team. 87. Ability to shag 100 feet of 4? or 5? Large Diameter Hose (LDH) from apparatus bed. 88. Ability to judge distances with reasonable accuracy. 89. Physical capacity to hear normally with or without hearing aids. 90. Physical capacity to maintain direction orientation while making turns under conditions of severely limited visibility. 91. Ability to work effectively with or without equipment in confined spaces. 92. Ability to work effectively with equipment and tools at heights. 93. Ability to differentiate various substances and conditions using sense of smell. J. MEDICAL 94. Knowledge of basic anatomy and physiology in order to properly perform EMS assessment and care. 95. Knowledge of procedures for evaluating and treating various medical emergencies (e.g., burns, allergic reactions, seizures, shocks, cardiac ailments). 96. Knowledge of procedures for patient packaging and transport (e.g., spinal injuries, head injuries, pregnancy, hyper/hypotension, unconscious). 97. Knowledge of symptoms and treatment protocols (e.g., diabetic, cardiac, obstetrics). 98. Knowledge of assessment and management of psychiatric emergencies. 99. Knowledge of legislation affecting EMS service [e.g., Do Not Resuscitate (DNR) orders]. 100. Knowledge of how to evaluate and manage orthopedic injuries (e.g., open and closed fractures, dislocations). 101. Ability to question the victim and others to elicit medical history, problem being experienced, and other information needed to help the victim. 102. Ability to evaluate correct treatment protocol and treat various types of injuries (e.g., skull, brain, spine, soft tissue, internal). 103. Ability to administer emergency medical care [e.g., basic life support, advanced life support, Intravenous (IV) drip]. 104. Ability to perform a field assessment (primary and secondary survey). 103 105. Ability to observe scene, situation, and determine mechanism of injury. 106. Ability to assess respiratory status in order to properly treat the patient. 107. Ability to manage the airway by use of oxygen, suctioning equipment, and airway adjuncts. 108. Ability to use spinal immobilization equipment (e.g., backboard, Stokes basket, cervical collar). 109. Ability to assess and manage the normal and abnormal presentations of the OB patient (e.g., normal vertex, breech, limb presentation, miscarriage, hemorrhage). 110. Ability to assess and manage newborns [e.g., Activity, Pulse, Grimace, Appearance, Respiration (APGAR) scoring system, need for resuscitation]. 111. Ability to properly use cot and stair chair. 112. Ability to act calmly and quickly in emergency situations. 113. Ability to operate an emergency vehicle in a safe manner. 114. Knowledge of transport protocol (e.g., capabilities of hospital, medical diversion, or bypass). 115. Knowledge of medical radio procedures (e.g., hospital alert, trauma alert). 116. Ability to select and deliver appropriate medication during EMS treatment using knowledge of medications (e.g., atropine, epinephrine, lidocane) and their appropriate doses, mixes, usages, and routes. 117. Ability to perform various types of needle sticks, IVs, blood draws, and shots. 118. Ability to perform physical assessment [e.g., primary, vital signs, lung sounds, Glasgow Coma Scale (GCS)]. 119. Ability to test and use glucometer. 120. Ability to initiate, secure, and monitor IV or Intraosseous (IO) using proper aseptic technique (e.g., catheter sizes and types). 121. Ability to intubate patient using appropriate equipment (endotracheal tube/combitube, stylette, laryngoscope, Magill Forceps) and proper procedures. 122. Ability to use cardiac monitor in order to recognize arrhythmias, cardiovert, defibrillate, and pace. K. TECHNICAL COMPETENCE 123. Knowledge of tool location, selection, and safe and appropriate use according to job (e.g., power tools, hand tools). 124. Knowledge of basic chemistry of combustion and characteristics of fire behavior (e.g., signs of backdraft). 125. Knowledge of water supply hydrants, their maintenance, and how to locate them. 126. Knowledge of fire alarms and fire communications systems. 127. Knowledge of fire department policies and procedures. 128. Knowledge of fire department organization and structure. 129. Knowledge of city geography, types of buildings in district, and major street locations. 130. Knowledge of flammable materials. 131. Knowledge of pre-fire planning methods and objectives. 104 132. Knowledge of proper techniques of forcible entry. 133. Knowledge of motor vehicle construction. 134. Knowledge of basic fire prevention. 135. Knowledge of the different types of extinguishing agents and the principles of their use on different types of fires. 136. Knowledge of operational limits of vehicles. 137. Knowledge of rescue extrication procedures and techniques (e.g., ropes, pulleys, jaws of life). 138. Knowledge of water rescue techniques. 139. Knowledge of tactics and strategy of fire suppression. 140. Knowledge of search procedures. 141. Knowledge of hazardous materials and safety precautions. 142. Knowledge of basic computer operations. 143. Knowledge of salvage and overhaul techniques. 144. Knowledge of ventilation techniques, procedures, and equipment. 145. Knowledge of proper operation and maintenance of ground and aerial ladders. 146. Knowledge of proper appearance of fire apparatus (e.g., hoses, belts, tires, battery terminal pressure) and related equipment (e.g., hoses, ladders, tools, pump panel). 147. Knowledge of public utilities and other building usages. 148. Knowledge of ropes (use and maintenance). 149. Knowledge of sprinkler systems. 150. Knowledge of fire streams. 151. Knowledge of basic building construction. 152. Knowledge of maintenance and use of fire hose and appliances. 153. Skill in operation of portable generator. 154. Skill in operation and maintenance of departmental vehicles. 155. Skill in operation of atmospheric monitoring equipment. 156. Skill in operation of thermal imaging equipment. 157. Ability to recognize indications of Weapons of Mass Destruction (WMD). 158. Skill in operation of two-way radio. 159. Skill in operation and maintenance of protective breathing equipment. 160. Skill in operation of fire extinguishers and ladder pipe/deluge systems. 161. Skill in operation of extrication tools (e.g., air and hydraulic powered tools). 162. Skill in use of forcible entry tools (e.g., hand tools, axes, k-tools). 163. Ability to keep track of road conditions and construction. 164. Ability to recognize signs of arson. 165. Ability to start and operate power saws. 166. Ability to communicate in signs and signals. 105 THANK YOU! You have now completed the Entry Firefighter Job Analysis Questionnaire. Your assistance is greatly appreciated! 106 Appendix 2 FIRE CAPTAIN JOB ANALYSIS QUESTIONNAIRE 241 Lathrop Way Sacramento, CA 95815 (916) 263-3600 Copyright ? 2008 CPS Human Resource Services 107 INSTRUCTIONS CPS is conducting an internal job analysis in an effort to develop an accurate description of the job requirements for the position of Fire Captain. The information gathered during this process will be used as the foundation for the creation of a new exam to be utilized in the selection process for future applicants of the Fire Captain position. This job analysis questionnaire was developed with the assistance of incumbents and supervisors who are familiar with the requirements of the Fire Captain position. They reviewed and edited the task and knowledge, skill, and ability (KSA) statements contained in this questionnaire for appropriateness and comprehensiveness. To obtain an accurate description of the job performed by Fire Captains, we need your cooperation in completing this job analysis questionnaire. Individuals familiar with the position of Fire Captain (i.e., those who perform the job or supervise those in the position of Fire Captain) are required to accomplish this goal. If you are a Fire Captain, please respond to the questions in reference to the job that you perform. If you are the immediate supervisor of one or more Fire Captains, please respond only in reference to the job of Fire Captain within your department/office. Please do not respond in reference to your own supervisory job. The information you provide throughout this questionnaire will be kept confidential and will be pooled with other data. This questionnaire is organized into the following four sections. Section 1: Background/Supplemental Information This section requests general information about the demographic characteristics of people who participate in this survey. Section 2: Task Statements This section includes 85 job task statements to be rated based on their Frequency and Importance. Section 3: Knowledge, Skill, and Ability (KSA) Statements ? This section includes 192 statements describing the knowledge, skills, and abilities (KSAs) necessary for performing the job tasks. These KSAs are to be rated according to (1) their Importance to successful job performance (Importance) and (2) whether an individual is expected to possess the KSA upon entry to the job (Needed at Entry). 108 Selecting Responses Before beginning each section of the questionnaire, please read any directions and rating scales carefully. If you do not understand the directions or the rating scales, please contact Jennifer Wynn (ext. 3145) in the CPS Test Rental Department at (916) 263-3600. The goal of this study is to identify the tasks and KSAs that are most important to the job of Fire Captain. When assigning ratings, it is important that you consider the entire rating scale. It is likely that some tasks or KSAs are more important than others and your ratings should reflect these differences. 109 SECTION 1. BACKGROUND INFORMATION Before you begin completing the questionnaire, we would like to request some information about your background. This information will allow us to describe the demographic characteristics of the respondents as a whole. For example, questions regarding age will be pooled to describe the average age of participants completing the questionnaire. REMEMBER: Your information will be kept confidential and will be pooled with other respondent?s data. 1. Please indicate your educational level? 01. Less than high school 02. GED/High school proficiency 03. High school graduate 04. Some college education without degree 05. Associates degree 06. Bachelors degree 07. Some post-graduate education without advanced degree 08. Masters degree 09. Doctorate 10. Prefer not to answer 2. What is your age? 01. 30-34 02. 35-39 03. 40-44 04. 45-49 05. 50-54 06. 55-59 07. 60-64 3. What is your ethnicity? 01. African American 02. Asian/Pacific Islander 03. Caucasian 04. Filipino 05. Hispanic 06. Native American 07. Other 08. Prefer not to answer 110 4. What is your gender? 01. Male 02. Female 03. Prefer not to answer 5. What is your relationship to this job? ? Please indicate whether you perform the job now (i.e., Fire Captain) or directly supervise the person in the job (i.e., Battalion Chief, Chief). 01. I perform the job now. 02. I directly supervise the person(s) performing the job. 03. I am a higher level manager of the job. 04. Other 6. How long have you been employed by the Fire Department? 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 7. Please indicate the length of time you have supervised the job of Captain. Only respond to this question if you currently supervise Captains (i.e., you are a Battalion Chief). 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 111 8. Please indicate the length of time you have been or were employed performing the job of Captain. For example, if you are currently a Captain, please indicate how long you have been employed as a Captain. If you are a Battalion Chief and are rating the Captain position, please indicate how long you were employed as Captain. 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 9. What is your current rank? 01. Fire Captain 02. Battalion Chief 03. Other 10. Please indicate how long you have been in your current rank? 01. Less than 6 months 02. 6 months but less than 1 year 03. 1 year but less than 3 years 04. 3 years but less than 5 years 05. 5 years but less than 7 years 06. 7 years but less than 9 years 07. 9 years but less than 11 years 08. 11 years but less than 13 years 09. 13 years but less than 15 years 10. 15 years or more 112 SECTION 2. TASK STATEMENTS For each task, you will be asked to make two ratings, Frequency and Importance. Supervisors: Remember to evaluate the tasks performed by Fire Captains, not the tasks that you perform. Frequency Rating Please read each task statement carefully and decide, based on your observations over the last 12 months, how frequently the task is performed in the job you are rating. It is not expected that all tasks are performed on every job. Some tasks are performed more frequently than others. Please indicate how frequently each of the following tasks is performed. Some tasks are part of the job but are not currently performed. For these tasks, it would be appropriate to give a frequency rating of 1 ?Part of the job, but not performed.? Please use the following scale to indicate how frequently each task is performed: FREQUENCY 0 = Not part of the job 1 = Part of the job, but not performed 2 = Performed every few months to yearly 3 = Performed every few weeks to monthly 4 = Performed every few days to weekly 5 = Performed every few hours to daily 113 Importance Rating For each task, after you rate the Frequency of the task, you should next rate the Importance of the task. To assess Importance, ask yourself how seriously overall job performance would be compromised if a task was not performed adequately. Some tasks are more important to overall successful job performance than others. Do not let your Frequency rating influence your Importance rating for the task. Tasks that require a lot of time are not necessarily the tasks that are most important. Please indicate how important each of the following tasks is to successful job performance. Please use the following scale to indicate how important each task is to successful job performance: IMPORTANCE 0 = Not Important to successful job performance 1 = Somewhat Important to successful job performance 2 = Important to successful job performance 3 = Very Important to successful job performance 114 Task Statements A. FIRE STATION ADMINISTRATION AND MAINTENANCE 1. Briefs incoming officer of significant events and circumstances occurring during the shift (e.g., damaged hydrants, sprinklers) to ensure continuity of information available. 2. Communicates verbally with communications center in order to keep center informed of the status of the firefighting unit. 3. Coordinates and schedules station work assignments (e.g., cooking, cleaning, building and equipment maintenance) for subordinates on a daily/shift basis. 4. Ensures inspections of all firefighting equipment and medical/rescue equipment are conducted in accordance with departmental standards. 5. Gives and/or supervises the giving of presentations/demonstrations for school and civic groups; provides information and assistance to the public. 6. Investigates citizen complaints and ensures appropriate actions are taken and communication is made back to the complainant. 7. Monitors radio communications to determine status of emergency activities in the service area. 8. Notifies Battalion Chief in district if there is a personnel shortage. 9. Performs some minor mechanical modifications, installations, and repairs. 10. Prepares cost estimates for budget recommendations. 11. Produces operational plans to support fire department's goals and objectives. 12. Reads general orders, SOPs, e-mails, teletype messages or other information to learn about changes in practice, policy, or procedure. 13. Reviews station journal since last shift to familiarize self with the status of personnel, equipment, and apparatus in station. 14. Supervises the daily administration of station activities, and ensures that all station operational needs are met. B. COMMUNICATION/DOCUMENTATION 15. Conducts/assists in research for various project and program work, and provides background information and recommendations to assist and support other officers' duties. 16. Interviews witnesses and gathers on-site information needed to complete departmental vehicle accident reports. 17. Notifies supervisor, in person or by memo, when apparatus/equipment is out of service to ensure continuity of information, and to initiate replacement or repair. 18. Oversees the completion of supply requisition forms and repair slips in order to notify appropriate parties of the need to replace or repair defective equipment or supplies. 19. Participates in various committees to develop departmental policies and procedures. 20. Prepares National Fire Incident Reporting System (NFIRS) reports in accordance with departmental directives. 115 21. Prepares reports, completes forms, issues memos, and writes correspondence which are both internal and external to the Department. 22. Reviews injury, accident, and health exposure reports to identify unsafe work environments or behaviors and takes approved action to prevent reoccurrence. 23. Reviews, evaluates and approves various types of reports as to both content and form. 24. Updates records, daily logs, and files in a variety of areas such as assignment of subordinates, attendance of subordinates, training of subordinates, equipment maintenance, and supply usage in order to keep record and inform interested parties who follow. 25. Writes and submits proposals for the purchase of new equipment or modification of existing equipment. 26. Writes information in the station journal so that the other officers know about the station's personnel and equipment needs. 27. Writes reports for superiors regarding inspection results and corresponding recommendations as applicable. C. MANAGEMENT/SUPERVISION 28. Administers discipline in a fair and equitable manner to subordinates both verbally and in writing. 29. Advises and consults with subordinates on personal and job-related problems, and refers them to the appropriate resource (e.g., Employee Assistance Program or other professionals). 30. Advises subordinates as to the meaning and consequences of Departmental policies and ensures that these policies are complied with. 31. Conducts safety inspection of subordinates' PPE (e.g., Turnout Clothing, SCBA gear, accountability tags) to ensure compliance with department regulations. 32. Conducts the morning briefing to notify subordinates of various issues (such as street closings, special events, outages, etc.). 33. Directs firefighters during training evolutions to correct inefficient or improper fire fighting techniques. 34. Evaluates the work performance and behaviors of subordinates both formally and informally, including annual/quarterly/monthly ratings and day-to-day evaluations of their work to ensure they are consistent with local policy, practices, and procedures. 35. Makes recommendations to superiors regarding appropriate disciplinary action and performance improvement plans. 36. Participates in local, regional, state and national conferences and seminars on fire administration, prevention, suppression and public safety education to maintain professional and technical expertise. 37. Reads maintenance manuals, operating instructions, and current publications/texts to maintain proficiency and to instruct firefighters. 38. Responds to and mediates subordinate complaints and inquiries both verbally and in writing. 39. Supervises firefighters in routine maintenance of vehicles (e.g., to check gas, oil level, tire pressure). 116 40. Trains firefighters on departmental policies and procedures and on job-related equipment, techniques, and hazards. D. EMERGENCY SERVICES DELIVERY 41. Administers first aid using "Universal Precautions" (gloves, mask, eye protection) to reduce further injury. 42. Assesses condition of victims by questions, observation, or hands-on assessment (secondary survey). 43. Briefs superior officer of incident details during transfer of command. 44. Conducts size-up immediately upon notification of emergency (environmental, traffic expected, etc.) to prepare an effective response. 45. Coordinates activities with police and medical personnel at first aid scene. 46. Counsels and calms victims and family members at scene of emergency. 47. Determines priorities of rescue activities, apply first aid vs. performs extrication. 48. Determines search areas and search pattern by evaluating ongoing suppression activity and current conditions. 49. Determines the best placement of apparatus. 50. Determines the type and location of forcible entry, based upon structural need and minimizing damage to property. 51. Determines when additional resources are needed. 52. Determines when electricity and gas need to be turned off to maintain safety of fire scene. 53. Determines where, when, and how building should be ventilated to retard the spread of fire and to avoid backdraft based on evaluations of smoke, heat, weather conditions, and type of dwelling. 54. Develops and conducts a post-incident analysis to assist in future responses. 55. Directs firefighters in how to rescue trapped victims, potential jumpers, animals, and others in distressed situations; what equipment to use and where to use it. 56. Directs firefighters to remove furniture, appliances, material, etc., to outside to eliminate possibility of rekindle. 57. Directs firefighters to use portable lighting to illuminate structures or areas to promote safer operations. 58. Directs subordinates to search/searches for fire extension by pulling walls, ceilings, checking concealed spaces, behind molding, pipe chases and vertical openings while causing minimal damage. 59. Dons own Personal Protective Equipment (PPE) and observes firefighters, on receipt of alarm, to verify they have donned protective clothing and strapped themselves into position in vehicle to prepare for departure. 60. Dons protective equipment (e.g., latex gloves and face shields) to protect against infectious diseases. 61. Gathers information from various sources at emergency scene (e.g., communications center or witnesses) regarding location of victims and special circumstances. 117 62. Implements an action plan. 63. Instructs firefighters what equipment to use and how to proceed with overhaul, depending on type of building and extent of fire. 64. Maintains visual/radio contact with crew at incident scene to ensure accountability and personnel safety. 65. Notifies communications center and other responding units of the location, circumstances, traffic and routing hazards, need for additional equipment, and conditions upon arrival at scene. 66. Notifies appropriate personnel within the incident response organization [e.g., Incident Command System (ICS)/Incident Management System (IMS), National Incident Management System (NIMS)] of any significant information (e.g., changing conditions, condition of victims, evacuation efforts, hazards noted while conducting operations). 67. Orders evacuation of buildings when warranted by conditions. 68. Orders firefighters to assist paramedics in care of victims (e.g., obtaining equipment). 69. Reassigns and redirects subordinates at scene according to strategy, evolving operations, new information, and circumstances. 70. Reports condition of victim to communications center so that it can inform ambulance crew of nature and urgency of situation. 71. Triages patients to determine priorities for treating victims where multiple victims are involved. 72. Visually evaluates condition and type of roof to determine whether roof is safe to support the weight of firefighters and their equipment. 73. Visually inspects premises closely to determine if fire has been extinguished and if it will rekindle. E. INSPECTIONS AND INVESTIGATIONS 74. Conducts inspections of buildings to look for fire hazards, check tags on fire extinguishers and alarm systems, and determine if there is a fire code violation to ensure compliance with applicable laws, codes, and regulations. 75. Contacts Communication Center to request that a Fire Investigator respond to the scene of a suspicious fire. 76. Determines point of origin and preliminary cause of fire. 77. Directs firefighters to perform regular inspections of hydrants, cisterns, or other water supply sources. 78. Informs Battalion Chief or Fire Investigator about suspicious aspects of fire (e.g., if there was an unusual odor, more than one fire in different areas, or broken doors). 79. Investigates fire safety complaints as requested by citizens or other City departments. 80. Participates in and directs the activities of subordinates in the conduct of fire inspections and the development of pre-fire plans. 81. Preserves evidence for arson investigators by cordoning off suspicious areas. 118 82. Provides fire safety code consultation and compliance for requests by governmental agencies, private industry, and members of the community. 83. Reports obvious fire hazards and safety violations (e.g., flammable rubbish or locked doors) to owners of buildings and to Battalion Chief, Fire Inspector, or Code Enforcement Department. 84. Reviews inspectors' occupancy reports and complaint resolution to enforce City's fire code. 85. Reviews prepared pre-fire plans and access maps for properties to document buildings or transit systems (e.g., building layout, type of occupancy, and structural information) during inspections, so as to be able to respond appropriately in an emergency. 119 SECTION 3. KNOWLEDGE, SKILL, AND ABILITY (KSA) STATEMENTS For each KSA you will be asked to make two ratings, Importance and Needed at Entry. Importance Rating Please read each KSA statement carefully and decide its Importance to the job you are rating. Some KSAs are more important to overall successful job performance than others. Please indicate how important each of the following KSAs is to successful job performance. Please use the following scale to indicate how important each KSA is to successful job performance: IMPORTANCE 0 = Not Important to successful job performance 1 = Somewhat Important to successful job performance 2 = Important to successful job performance 3 = Very Important to successful job performance Needed At Entry Rating For each KSA, after you rate the Importance, please rate the KSA for Needed at Entry. To assess Needed at Entry, decide whether an individual is expected to possess the KSA upon the first day of performing as a Fire Captain. KSAs that are not needed at entry can normally be acquired once a person is in the job. Please indicate how essential the possession of this knowledge, skill, or ability is at the time of hire for this position. Please use the following scale to indicate how essential each KSA is at the time of hire: NEEDED AT ENTRY 0 = Not Needed 1 = Needed 2 = Essential 120 KSA Statements A. FIRE STATION ADMINISTRATION AND MAINTENANCE 1. Ability to allocate resources. 2. Ability to interpret data. 3. Ability to use evaluative methods. 4. Knowledge of administrative policies and procedures. 5. Knowledge of costs associated with personnel, new equipment, and apparatus maintenance. 6. Knowledge of information management and recordkeeping. 7. Knowledge of organizational structure of the department. 8. Knowledge of purchasing laws, policies, and procedures. 9. Knowledge of repairs to existing fire department facilities and equipment. 10. Knowledge of revenue sources and the departmental budget process. 11. Knowledge of the data processing system (e.g., computer applications and software). B. COMMUNICATION/DOCUMENTATION 12. Ability to communicate to staff of decisions, changes, and other relevant information in a timely manner. 13. Ability to document activities or events in writing (e.g., station log book) with sufficient legibility to be understood by all intended readers. 14. Ability to effectively organize and prepare clear, concise written material on a variety of issues. 15. Ability to effectively utilize computers, software (e.g., word processing and spreadsheet programs), and communication systems in all aspects of the job 16. Ability to follow oral directions and instructions. 17. Ability to follow written directions and instructions. 18. Ability to interview individuals to obtain accurate and complete information. 19. Ability to listen to others attentively and with comprehension. 20. Ability to make effective presentations before both small and large groups and to respond to questions. 21. Ability to read and comprehend lengthy and complicated reports and fire service material. 22. Ability to use appropriate grammar, style, format, and tone in informal and formal business communications. 23. Ability to verbally communicate in a clear, effective manner with subordinates, peers, management, allied agencies, and the public in all routine and technical aspects of the job. 24. Knowledge of the departmental communication procedures. 25. Knowledge of verbal communication techniques to facilitate learning. 121 26. Skill to operate a two-way radio to communicate with the communications center, stations, aircraft, or other vehicles. C. MANAGEMENT/SUPERVISION 27. Ability to distribute issue-guided directions to unit members during training evolutions. 28. Ability to effectively assign or delegate work to subordinates and peers (including assignments during emergency incidents, long-term projects, and routine activities). 29. Ability to establish long- and short-term goals. 30. Ability to establish procedures to monitor and regulate processes, tasks, or activities of subordinates. 31. Ability to evaluate and critique subordinates' performance in an objective and positive manner. 32. Ability to implement, evaluate, and modify tactical plans during an emergency incident. 33. Ability to issue instructions for frequently assigned unit tasks based on department policy and training. 34. Ability to maintain accurate written records and schedules of personnel and resources. 35. Ability to make a decision from multiple, alternative solutions and propose appropriate recommendations. 36. Ability to make proper assignment of personnel and appropriate use of resources. 37. Ability to motivate individuals and encourage the participation of subordinates in the accomplishment of tasks and/or arrive at solutions to problems. 38. Ability to plan and carry out effective training sessions. 39. Ability to recognize training deficiencies and/or performance problems in subordinate personnel to determine effective corrective training, developmental or disciplinary measures. 40. Ability to resolve employee issues and employee disputes. 41. Ability to set priorities, coordinate or schedule tasks or events in a logical manner so as to maximize staff and material resources and meet goals and timelines. 42. Knowledge of basic human resource management. 43. Knowledge of departmental human resource policies and procedures. 44. Knowledge of departmental supervisory policies and procedures. 45. Knowledge of group dynamics. 46. Knowledge of interpersonal dynamics. 47. Knowledge of leadership styles. 48. Knowledge of organizational behavior. 49. Knowledge of principles of supervision. 122 50. Knowledge of techniques used to make assignments under stressful and/or routine situations. 51. Knowledge of the causes, signs, and symptoms of personnel problems (e.g., stress). 52. Knowledge of types of power. D. EMERGENCY SERVICES DELIVERY 53. Ability to advance charged hose into fire building. 54. Ability to advance uncharged hose into fire building. 55. Ability to apply triage techniques during multi-casualty incidents. 56. Ability to bend, stoop, and crawl wearing full protective equipment. 57. Ability to effectively implement the ICS/IMS. 58. Ability to effectively implement the NIMS. 59. Ability to effectively operate within the ICS/IMS. 60. Ability to effectively operate within the NIMS 61. Ability to identify a potential fire or safety hazard. 62. Ability to identify potential exposures in order to prevent the incident from spreading/expanding. 63. Ability to lift and carry equipment weighing up to 150 pounds assisted by another person. 64. Ability to serve in command staff and unit supervision positions within the incident response organization (e.g., ICS/IMS, NIMS). 65. Ability to supervise and account for assigned personnel under emergency conditions. 66. Ability to understand processes required to complete a unit of work (e.g., a fireground evolution or administrative process). 67. Ability to use poles, ropes, life rings, or buoys to rescue persons who have fallen or have been swept into swiftly moving water. 68. Ability to use hand tools and equipment. 69. Ability to recognize system problems with equipment and correct any deficiency noted according to policies and procedures. 70. Knowledge of aerial operations as they pertain to firefighting, rescues, ventilation, and hose operations. 71. Knowledge of available resources for fire and other emergency incidents. 72. Knowledge of basic anatomy/physiology principles and terminology pertaining to medical aid. 73. Knowledge of customer services principles. 74. Knowledge of defibrillation techniques to assist cardiac arrest patients. 75. Knowledge of departmental dispatching procedures. 76. Knowledge of Emergency Medical Technician (EMT) level medical intervention to treat patient at the scene of an incident. 123 77. Knowledge of evacuation techniques and practices. 78. Knowledge of First Aid level medical intervention to treat patient at the scene of an incident. 79. Knowledge of First Responder level medical intervention to treat patient at the scene of an incident. 80. Knowledge of Hazardous Materials response procedures. 81. Knowledge of the ICS/IMS (e.g., local, regional, state). 82. Knowledge of the NIMS. 83. Knowledge of local personnel accountability system. 84. Knowledge of patient triage procedures and techniques. 85. Knowledge of patient vital signs and assessment to identify medical problem. 86. Knowledge of post-incident analysis procedures. 87. Knowledge of pre-incident planning. 88. Knowledge of scene safety procedures. 89. Knowledge of Standard Operating Procedures for emergency operations. 90. Knowledge of strategy, tactics, and operations. 91. Knowledge of the elements of size-up. 92. Knowledge of vehicle extrication techniques. 93. Skill in conducting physical tasks under adverse conditions (e.g., heat, limited visibility, confined space, arduous terrain, height, unstable or limited footing). 94. Skill in operating hoselines and hose appliances. 95. Skill in use of basic life support equipment (e.g., defibrillator, resuscitator, bag valve mask). 96. Skill in using salvage/overhaul tools. 97. Skill to administer CPR to patients of cardiac arrest. 98. Skill to operate a back pump to extinguish grass or brush fires. 99. Skill to position ground ladders at the incident scene to carry out fire suppression and rescue activities. 100. Skill to protect potential exposures using direct stream, fog streams, or water curtains to prevent the fire from spreading. 101. Skill to pull ceiling with pike pole or other hand tools to determine location of fire. 102. Skill to tie knots (e.g., bowline, square, becket, clove hitch, timber hitch, family of eights, chimney hitch). 103. Skill to use a fire extinguisher to suppress a fire. 104. Skill to use digging tools/equipment (e.g., shovels, picks). 105. Skill to use extrication tools and equipment (e.g., hydraulic spreading tool, pneumatic cutting tool). 124 106. Skill to use hand tools, power tools, air tools, and chain saws in a safe and effective manner. 107. Skill to use lifelines and belts to remove trapped or injured victims. 108. Skill to use tools and equipment for forcible entry purposes to gain access to secure structures. E. INSPECTION AND INVESTIGATIONS 109. Ability to apply knowledge using deductive skills. 110. Ability to apply the appropriate codes (e.g. Uniform Fire Code). 111. Ability to conduct basic interviews. 112. Ability to determine basic fire cause. 113. Ability to establish perimeters at an incident scene. 114. Knowledge of common causes of fire and origin determination. 115. Knowledge of departmental inspection procedures. 116. Knowledge of documentation of preliminary fire investigation procedures. 117. Knowledge of evidence preservation. 118. Knowledge of basic fire investigation procedures including the roles and responsibilities of a fire investigator. 119. Knowledge of marking and identification systems for hazardous materials. 120. Knowledge of methods used by arsonists. 121. Knowledge of the importance of fire scene security. 122. Knowledge of the physical science pertaining to fire behavior (e.g., growth and development). 123. Knowledge of types of evidence. F. GENERAL KNOWLEDGE 124. Knowledge of agreements in force between the organization and members (e.g., rights of management and members). 125. Knowledge of atmospheric measuring devices and their uses. 126. Knowledge of automatic fire protection systems (e.g., sprinkler systems). 127. Knowledge of basic fuel loading. 128. Knowledge of classification of fires. 129. Knowledge of combustion agents. 130. Knowledge of confined space rescue procedures. 131. Knowledge of current trends, technologies, and socioeconomic and political factors that impact the fire service. 132. Knowledge of departmental policies and procedures related to releasing information to news media and public. 133. Knowledge of departmental standard operating procedures. 125 134. Knowledge of enabling and regulatory legislation and the local, state, and federal law making process. 135. Knowledge of fire attack strategies. 136. Knowledge of fire extinguishment theory. 137. Knowledge of fire hazards. 138. Knowledge of fire hose evolutions (e.g., appliances, tools, loads). 139. Knowledge of fire prevention and building safety codes and ordinances applicable to jurisdiction. 140. Knowledge of fire prevention techniques and practices. 141. Knowledge of fire streams. 142. Knowledge of forcible entry tools and techniques. 143. Knowledge of functions of other bureaus, divisions, agencies, and organizations and their roles and responsibilities that relate to the fire service. 144. Knowledge of generally accepted ethical practices. 145. Knowledge of hoisting tools and equipment. 146. Knowledge of local geography including the location of streets, water mains, hydrants and the major fire hazards within the jurisdiction. 147. Knowledge of non-water based extinguishing systems such as carbon dioxide, halogenated agents, dry chemical, and other extinguishing agents. 148. Knowledge of procedures to operate at incidents involving suspected explosives and bomb threats. 149. Knowledge of salvage and overhaul techniques. 150. Knowledge of standpipe systems and operations. 151. Knowledge of structural and trench/excavation collapse procedures. 152. Knowledge of suppression methods for BLEVE. 153. Knowledge of suppression methods for fires and emergencies in confined enclosures. 154. Knowledge of suppression methods for fires in below ground structures. 155. Knowledge of suppression methods for fires in upper levels of structures. 156. Knowledge of suppression methods for highrise fires. 157. Knowledge of suppression methods for petroleum fires. 158. Knowledge of suppression methods for trash and dumpster fires. 159. Knowledge of suppression methods for vehicle fires. 160. Knowledge of suppression methods for wildland fires. 161. Knowledge of techniques, advantages, and disadvantages of building ventilation. 162. Knowledge of the characteristics of hazardous materials. 163. Knowledge of the different types of firefighting foams, their uses, and application techniques. 126 164. Knowledge of the extinguishing properties of water. 165. Knowledge of the organization of local government. 166. Knowledge of the principles of municipal water supply systems and rural water supply operations. 167. Knowledge of the signs and symptoms of fatigue. 168. Knowledge of the types of building construction, common building materials, structural stability, and firefighter hazards related to building construction. 169. Knowledge of the types, uses, and maintenance of portable fire extinguishers. 170. Knowledge of types of alarm systems. 171. Knowledge of types of ropes, maintenance, and knots. 172. Knowledge of uses and maintenance of various ladder types. 173. Knowledge of utility control, forced air systems, heating units and fuels, and electric service. COMMUNITY AND GOVERNMENT RELATIONS 174. Ability to establish and maintain effective working relationships with those contacted in the course of work. 175. Ability to work with diverse groups. 176. Ability to respond to public inquiries regarding the fire service in a respectful, cooperative, productive and tactful manner. 177. Ability to foster intergovernmental and interagency cooperation through respectful, professional interactions with other agencies. 178. Knowledge of community demographics. 179. Knowledge of department's public education program as it relates to the target audience. 180. Knowledge of interpersonal relationships. 181. Knowledge of public relations. 182. Knowledge of service organizations. HEALTH AND SAFETY 183. Ability to identify safety hazards. 184. Ability to read and interpret reports (e.g., accident, injury, occupational injury, death reports). 185. Knowledge of basic workplace safety. 186. Knowledge of departmental safety policies and procedures. 187. Knowledge of how to properly don and maintain personal protective clothing. 188. Knowledge of personal alert safety systems (PASS). 189. Knowledge of self contained breathing apparatus, inspection and maintenance. 190. Knowledge of the components of infectious disease control program. 191. Knowledge of the most common causes of accidents and injuries. 127 192. Knowledge of the procedures for conducting an accident investigation. THANK YOU! You have now completed the Fire Captain Job Analysis Questionnaire. Your assistance is greatly appreciated! 128 Appendix 3 Table Frequencies and Percentages for Entry Level Firefighter Demographics Demographic n % Education GED/High school proficiency 3 1.2 High school graduate 11 4.4 Some college without a degree 123 49.2 Associates degree 64 25.6 Bachelor?s degree 31 12.4 Some post education 10 4.0 Master?s degree 4 1.6 Doctorate 1 0.4 Prefer not to answer 3 1.2 Age 20-29 53 21.2 30-39 92 36.8 40-49 83 33.2 50-59 19 7.6 Prefer not to answer 3 1.2 Ethnicity African American 5 2.0 Asian/Pacific Islander 12 4.8 129 Demographic n % Caucasian 198 79.2 Filipino 1 0.4 Hispanic 4 1.6 Native American 1 0.4 Other 8 3.2 Prefer not to answer 21 8.4 Gender Male 226 90.4 Female 14 5.6 Prefer not to answer 10 4.0 Relationship to job Perform now 162 64.8 Directly supervise 87 34.8 Higher level manager 1 0.4 Rank Incumbent 152 60.8 Supervisor 98 39.2 Length of employment At least 6 months but less than 1 year 5 2 At least 1 year but less than 3 years 25 10 At least 3 years but less than 5 years 38 15.2 At least 5 years but less than 7 years 22 8.8 130 Demographic n % At least 7 years but less than 9 years 24 9.6 At least 9 years but less than 11 years 40 16 At least 11 years but less than 13 years 23 9.2 At least 13 years but less than 15 years 8 3.2 15 years or more 65 26 Length of time as supervisor Less than 6 months 5 5.2 At least 6 months but less than 1 year 3 3.1 At least 1 year but less than 3 years 7 7.2 At least 3 years but less than 5 years 17 17.5 At least 5 years but less than 7 years 28 28.9 At least 7 years but less than 9 years 6 6.2 At least 9 years but less than 11 years 9 9.3 At least 11 years but less than 13 years 3 3.1 At least 13 years but less than 15 years 8 8.2 15 years or more 11 11.3 Current rank Firefighter/EMT 61 24.4 Fire Lieutenant 78 31.2 Fire Captain 15 6.0 Other 5 2.0 Fire Medic 91 36.4 131 Demographic n % Length of time as current rank Less than 6 months 14 5.6 At least 6 months but less than 1 year At least 1 year but less than 3 years At least 3 years but less than 5 years 45 43 62 18.0 17.2 24.8 At least 5 years to but less than 7 years At least 7 years to but less than 9 years 5 39 2.0 15.6 At least 9 years to but less than 11 years 7 2.8 At least 11 years to but less than 13 years At least 13 years but less than 15 years 10 11 4.0 4.4 15 or more years 14 5.6 Size of population served 0 ? 14999 3 4.5 15000 ? 29999 6 9.1 30000 ? 49999 6 9.1 50000 ? 74999 5 7.6 75000 ? 99999 2 3.0 100000 - 149999 30 45.5 150000 ? 199999 11 16.7 200000 and up 3 4.5 Size of department 0 ? 49 7 10.6 132 Demographic n % 50 ? 99 22 33.3 100 ? 149 15 22.7 150 ? 199 10 15.2 200 or over 12 18.2 133 Appendix 4 Table Frequencies and Percentages for Fire Captain Demographics Demographic n % Education GED/High school proficiency 1 0.5 Some college without a degree 64 30.8 Associates degree 64 30.8 Bachelor?s degree 51 24.5 Some post education 16 7.7 Master?s degree 11 5.3 Prefer not to answer 1 0.5 Age 30-39 24 11.5 40-49 108 51.9 50-59 70 33.7 60+ 6 2.9 Ethnicity African American 2 1.0 Asian/Pacific Islander 3 1.4 Caucasian 155 74.5 Filipino 2 1.0 Hispanic 17 8.2 134 Demographic n % Native American 6 2.9 Other 5 2.4 Prefer not to answer 18 8.7 Gender Male 200 96.2 Female 5 2.4 Prefer not to answer 3 1.4 Relationship to job Perform now 160 76.9 Directly supervise 44 21.2 Higher level manager 1 0.4 Other 3 1.4 Rank Incumbent 171 82.2 Supervisor 37 17.8 Length of employment At least 3 years but less than 5 years 3 1.4 At least 5 years but less than 7 years 3 1.4 At least 7 years but less than 9 years 7 3.4 At least 9 years but less than 11 years 3 1.4 At least 11 years but less than 13 years 15 7.2 At least 13 years but less than 15 years 10 4.8 135 Demographic n % 15 years or more Length of time as supervisor 167 80.3 Less than 6 months 2 2.8 At least 6 months but less than 1 year 9 12.5 At least 1 year but less than 3 years 17 23.6 At least 3 years but less than 5 years 6 8.3 At least 5 years but less than 7 years 7 9.7 At least 7 years but less than 9 years 6 8.3 At least 9 years but less than 11 years 1 1.4 At least 11 years but less than 13 years 3 4.2 At least 13 years but less than 15 years 5 6.9 15 years or more 16 22.2 Length of time as fire captain 6 months but less than 1 year 21 10.1 At least 1 year but less than 3 years 26 12.5 At least 3 years but less than 5 years 26 12.5 At least 5 years but less than 7 years 33 15.9 At least 7 years but less than 9 years 15 7.2 At least 9 years but less than 11 years 12 5.8 At least 11 years but less than 13 years 7 3.4 At least 13 years but less than 15 years 9 4.3 15 years or more 59 28.4 136 Demographic n % Current rank Fire Captain 171 82.2 Battalion Chief 34 16.3 Other 3 1.4 Length of time as current rank Less than 6 months 3 1.4 At least 6 months but less than 1 years 16 7.7 At least 1 year but less than 3 years 48 23.1 At least 3 years but less than 5 years At least 5 years but less than 7 years At least 7 years but less than 9 years At least 9 years but less than 11 years At least 11 years but less than 13 years At least 13 years but less than 15 years 15 years or more 33 19 12 8 10 4 55 15.9 9.1 5.8 3.8 4.8 1.9 26.4 Size of population served 0 ? 49,999 5 17.9 50000 ? 199,999 5 17.9 200000 and up 18 64.3 Size of department 0 ? 49 3 10.7 50 ? 99 6 21.4 137 Demographic n % 200 or over 19 67.9