i A SIMULATION EXERCISE IN DECISION-MAKING IN AN ELEMENTARY CLASSROOM Except where reference is made to the work of others, the work described in this dissertation is my own or was done in collaboration with my advisory committee. This dissertation does not include proprietary or classified information. _____________________________ Robin L. Fiedler Certificate of Approval: _____________________________ _____________________________ Jill Salisbury-Glennon David M. Shannon, Chair Associate Professor Professor Educational Foundations, Leadership Educational Foundations, Leadership and Technology and Technology _____________________________ _____________________________ Margaret E. Ross Stephen L. McFarland Associate Professor Acting Dean Educational Foundations, Leadership Graduate School and Technology ii A SIMULATION EXERCISE IN DECISION-MAKING IN AN ELEMENTARY CLASSROOM Robin L. Fiedler A Dissertation Submitted to the Graduate Faculty of Auburn University in Partial Fulfillment for the Degree of Doctor of Philosophy Auburn, Alabama August 8, 2005 iii VITA Robin Lynn Fiedler, daughter of David L. Cox and Marilyn J. Anderson, was born December 18, 1957, in Albuquerque, New Mexico. She graduated from Plymouth Canton High School in 1975. She attended Marion College in Marion, Indiana, for two years. She transferred to Edinboro University in September, 1977, graduating with a Bachelor of Science degree in Elementary Education in June, 1979. Following graduation, she became employed as an elementary school teacher in Pennsylvania, working in both the public and private school sectors. She was also employed as a vice- principal and principal during this time. After teaching in Pennsylvania, she taught in Delaware and Virginia. While in Virginia, she earned the degree of Master of Education in Administration and Supervision of Instruction from Virginia Commonwealth University in 1991. After obtaining her Master?s degree, she taught in Birmingham, Alabama, worked as a teacher of talented and gifted students and as an educational specialist. She entered the doctoral program in Educational Foundations, Leadership and Technology at Auburn University in August, 2002. She married Michael Allen Fiedler on September 4, 1987. She has a daughter, Jennifer and son, Nicholas. iv DISSERTATION ABSTRACT A SIMULATION EXERCISE IN DECISION-MAKING IN AN ELEMENTARY CLASSROOM Robin L. Fiedler Doctor of Philosophy, August 8, 2005 (Master of Education, Virginia Commonwealth University, 1991) (Bachelor of Science, Edinboro University, 1979) 138 Typed Pages Directed by David M. Shannon The purpose of this study was to refine and validate the use of a Simulation Exercise in Classroom Decision Making (SID2). SID2 simulated the limited amount of time within elementary classrooms to make informed decisions concerning classroom management, presentation of content, assessment of students, preparation for instruction and maintenance of a positive learning environment. Written descriptions of typical classroom scenes were projected onto a classroom- viewing screen. Participants were presented alternative teaching responses audibly, for each classroom scene, one response at a time. Participants were given five seconds to indicate whether each course of action was appropriate or inappropriate. 1 The specifications for the revised version, SID3, were developed using a group of indicators of effective teaching specified by the Alabama Professional Education Personnel Evaluation (PEPE). A panel of experts including principals, vice-principals, teachers, administrators and college faculty approved each situation. This study was conducted to answer three major research questions which attempted to verify whether teachers learn research findings about effective teaching from their educational courses and from their classroom experiences and whether this knowledge is reflected in their performance on the SID3. It was hypothesized that: 1) scores of subjects with professional educational training would differ from those subjects without such educational training, 2) scores of subjects with classroom teaching experience (teachers and student teachers) would differ from those of subjects without classroom teaching experience and 3) scores of experienced teachers would differ from scores of student teachers. In answer to the first question, those students having received instruction about teaching were able to identify responses that reflected knowledge of pedagogy acquired in educational training as they outperformed non-education students in each competency. Secondly, participants with teaching experience outscored those without teaching experience on all six domains. Demonstrating classroom teaching experience contributed to the ability to apply the findings of effective teaching research to results by performance on the exercise. Finally, the overall performance of experienced teachers and student teachers did not show a significant difference in the current study. All three findings were consistent with Shannon?s initial study (1990). 2 It was recommended that this test continue to be refined and administered to other participants that fit into one of the four groups. Further, scores from SID3 could be correlated with GRE, student portfolios, future classroom observations, student teacher evaluations and PEPE. This test could be used to identify strengths and weaknesses of practicing teachers, allowing inservice and other professional development opportunities to provide remedial services. If this exercise can be used to predict future teaching performance, teacher education programs and local school districts could administer it as a preliminary screening device. 3 ACKNOWLEDGEMENTS The preparation and completion of this document could not have been accomplished without the help, encouragement and wonderful support of many people. Among the people I would like to thank are the members of my dissertation committee; Dr. David M. Shannon, Dr. Jill Salibury-Glennon and Dr. Margaret E. Ross. Their support, feedback, constructive assistance, and shared wisdom enabled me to finish successfully. Special thanks to my colleagues who encouraged me, provided extra ideas and helped in data collection. Melanie, Sondra, Mary Ann, Esenc and Shu-Ching your friendship was a constant positive during my time at Auburn. My children gave me encouragement all along the way. Thanks to Jennifer and Nick for editing some of my work. Jen, Scott, Nick and Leslie kept my spirits high and provided laughter along the way! Michael, you are my favorite and my heart. Without your love and support this would not have been possible. I appreciate all you have done to encourage me during this process. You never wavered in your belief in me. 4 Style manual used: American Psychological Association Manual, 5 th Edition Computer software used: SPSS 11.0 for data analysis; Microsoft Word for wordprocessing 5 TABLE OF CONTENTS LIST OF TABLES???????????????????????????xi I. INTRODUCTION????????????????????????1 Introduction Statement of Purpose Research Questions & Hypotheses Significance of Study Limitations Assumptions Definitions Organizational Overview II. LITERATURE REVIEW????.?????????????????9 Components of Effective Teaching and Teaching Requirements Past and Current Definitions Decision-making of Effective Teachers Teaching Requirements National Teaching Standards State Teaching Standards Assessment Methods Traditional Assessment Alternative Assessment Assessment of Preservice Teachers Development of a Simulation Assessment Prior Research Using SID 6 III. METHODS ?????????...?????????????????38 Purpose of Study Development of SID3 Participants Instrumentation Data Collection Data Analysis IV. ANALYSIS OF DATA AND DISCUSSION?????????????47 Sample Exclusion Criteria Final Demographic Composition Reliability Analysis Preliminary Analysis Principal Analysis Research Questions and Hypotheses Educational Training Contrast Classroom Experience Contrast Teaching Experience Contrast Follow-up Survey Summary V. SUMMARY, DISCUSSION OF FINDINGS, CONCLUSIONS, AND SUGGESTIONS???.??????????????.???????.75 Summary of Study Test Refinement Sample Reliabilities Discussion of Findings First Research Question Second Research Question Third Research Question Conclusions Summary Suggestions for Further Study REFERENCES????????????????????????????..87 APPENDICES???????????????????????????.?..96 7 LIST OF TABLES 1. Demographics of Overall Sample and Subgroups?????...??????..49 2. Undergraduate Status and Education Courses Completed????????....50 3. Classroom Teaching Experience and Education Level Attained??????..51 4. Undergraduate Status and Education Courses Completed of Final Sample???????????????????????..???54 5. Classroom Teaching Experience and Education Level Attained of Final Sample??????????????????????..??..?..55 6. Summary of Reliability Estimates Original & Final Instruments??????...57 7. Summary of Reliability Analysis???????????????????58 8. Summary of Mixed-ANOVA Results?????????????????.60 9. PEPE Domain T-Scores by Group Status???????????????..61 10.Educational Training Contrast Question One??????????????.65 11. Classroom Experience Contrast Question Two?????????????.69 12.Teaching Experience Contrast Question Three?????????????..72 8 LIST OF FIGURES 1. Educational Training Group Profiles???????????????????64 2. Classroom Experience Group Profiles??????????????????..68 3. Teaching Experience Group Profiles???????????????????71 9 INTRODUCTION What does a teacher need to know? What are the requirements for obtaining a teaching certificate? In the 1800?s an exam was given to those wanting to teach (National Center for Educational Statistics, 2003). There were three levels of certification, with the best being ?first grade certification,? meaning you were capable of teaching and received the highest commendation. Two hundred years later, in most states, teachers are still required to take an exam to see how qualified they are to teach America?s children. Preparing future educators is an important challenge at colleges and universities. The quantity, breadth and depth of education courses needed prior to certification are questions that will never be answered completely. For while an answer can be given that serves a majority of the student population, there will always be those who need ?more? in a given area. Identifying how ?much? is enough is not an easy decision to make, nor is it the only decision. As educational psychology has developed and it has become clear that learning occurs in a variety of ways, our colleges and universities have needed to restructure their curricula. Accordingly, weaving together relevant content courses and presenting teaching styles that recognize and address the individual student styles of learning is crucial. 10 With the prominent reform documents of A Nation at Risk (1983) and No Child Left Behind (2001) arguments continue concerning the best way to teach and the best way to facilitate learning. Currently topics being pursued by educators and political interest groups include teacher performance, teacher pedagogical knowledge and the effect they have, together in combination or separately, on student learning (Kuligowski, Holdsum & French, 1993). America?s dissatisfaction with its schools, their teachers and student test scores has become chronic and epidemic as evidenced by A Nation at Risk written by the National Commission on Excellence in Education, April 1983. As one question appears to earn an answer, it invokes yet another round of questions to pursue. Within the tangle of debated issues, are foundational questions concerning the preparation of teacher candidates. The need to increase standards to assess preservice teachers and the requirement of solid assessment methods or systems is stated as the second standard in the National Council of Accreditation of Teacher Education (NCATE). Teachers have long been at the center of the debates, and still are today. Paradoxically, teachers are singled out as one of the best ways in which to advance the education of society, yet are often the main focuses of criticisms in education. Testing of teachers has evolved from 1977 when only three states required testing. A steady increase of required testing followed with a total of 12 states in 1980, 28 in 1982, and 38 in 1987. In a report by Council of Chief State School Officers (CCSSO) in 2000, only four states did not have teacher standards in place or being under development; Arizona, Nevada, New York and South Carolina. The use of test indicators most likely reflects the concern and attention being paid to upgrading professional 11 standards for teachers. According to the same 2000 report, 31 states? requirements are based on or are consistent with Interstate New Teacher Assessment and Support Consortium (INTASC) standards. Created in 1987, INTASC developed standards for new teachers. The standards cover all aspects of a teacher?s professional work; classroom management, discipline, teaching strategies, motivation, lesson planning, professional growth and working with other teachers and parents. In an effort for educational equality of students across the nation, in 2001, President Bush set in motion an initiative mandating that every state should have a highly qualified teacher in every classroom by the end of the 2005 ? 2006 school year (United States Department of Education, 2003). Believing that a highly qualified teacher knows how to teach, what to teach and possesses a command of the subject matter to be taught No Child Left Behind (NCLB) gave states and school districts flexibility to improve teacher quality. To achieve its goals for improved school outcomes, the NCLB Act requires a ?highly qualified teacher? in all classrooms. States have developed criteria for identifying highly qualified teachers. As a starting point in meeting the goals, each state was required to self-report baseline data to the U. S. Department of Education (USDE) by September 1, 2003. The results of each state?s report formed the impetus for the individual state plan. In the initial report, twenty states reported that highly qualified teachers taught in at least 90% of their classrooms. Alabama, initially, was one of the four states reporting 50% or fewer classrooms being taught by highly qualified teachers. It is reasonable to believe that each state varies in terms of their definition and the qualifications of its teaching staff. 12 In a race to the finish line, the 2005 school year, most states have accepted the challenge of making sure their state has the required highly qualified teachers mandated. Yet, while trying to achieve this common goal, each state?s requirements for the assessment and documentation of its teachers varies greatly. In 1988, the Alabama State Department of Education (ASDE) adopted a resolution requiring all professional public education personnel be evaluated. The Professional Education Personnel Evaluation (PEPE) was designed with a mission to assure excellence in public schools. A task force consisting of parents, local school boards, K-12 teachers and principals, district superintendents and supervisors, as well as members of Alabama business and industry developed PEPE as an instrument for evaluation. Furthermore, the process was defined as a way to assist educators through the process of evaluation and professional growth, as a pathway for increasing student achievement (ASDE, 1999). Many first year teachers enter their first classroom with a degree and certificate in hand, but with an unrealistic view of their capabilities and the demands of teaching. Being capable of writing lesson plans, successfully using instructional materials, displaying creative bulletin boards, constructing plans for each developmental mental level within the classroom, and knowing how to stimulate higher order thinking in all students are only a small portion of expectations we have of our teachers. With the formal and informal curriculum of schools being filtered through the minds of its classroom teachers, the quality of school learning is dependent on the quality of its teachers. With national and state standards in place, the challenge of evaluating whether or not a teacher 13 meets those standards creates the need for manageable assessment tools that can identify a qualified professional endowed with the characteristics of an effective teacher. Assessment tools at the university, state and national levels usually consist of classroom observations, a pencil and paper examination and perhaps an interview. Each tool presents its own type of bias; the subjective performance rating of an observer who attends your class for a small percentage of your actual teaching time, the teacher exam containing only one right answer or the interview with an administrator while you could be having an off day. Responding to questions in an interview or correctly answering multiple choice and essay questions are indeed different from the day-to-day challenges a teacher encounters that require immediate response or feedback. In response to the assessment dilemma, Hays (as cited in Shannon, Medley, & Hays, 1993) developed the Simulation Test of Interactive Teaching Competencies-Hays (STITC-H) that focused on specifications of teacher competencies as defined by the Virginia Beginning Teacher Assistance Program (BTAP). Using the competencies identified by BTAP, Hays created realistic classroom situations with different suggestions as to how a classroom teacher might respond to the situation, requiring the respondents to agree that the response suggested was either appropriate or inappropriate. In an effort to discern differences between experienced teachers, student teachers and non-education students?, Shannon (1990) modified the STITC-H. While somewhat different in context, Shannon?s revision, SID2, was administered in much the same way of having respondents agree or disagree with responses to given classroom situations. It seems intuitive that educational training would serve as a significant foundation in the development of the general knowledge used as a schoolteacher. Integrating current 14 research into the knowledge base of future teachers is imperative. Another significant foundation begins to be addressed in educational training programs and continues as teachers glean information about classroom techniques with on-the-job training. Professional development sessions held at schools, conferences or workshops present recent findings to keep teachers abreast of current research. In belief that an assessment tool could potentially be used to measure a variety of the above-mentioned areas, SID2 was revised. Educational training programs could assess education students after completion of a degree, yet prior to certification. A revised SID2 could be used as a pre-interview assessment or screening tool for job applicants in school systems. Principals could use the assessment results to find strengths and weaknesses in the school?s teaching faculty. After completing the assessment a professional development session could be designed to address each school?s specific results. Redevelopment of SID2 was based on current PEPE competencies and INTASC standards. Statement of the Problem This study was developed to verify and extend the findings obtained during both the STITC-H and the SID2. Examining the validity of a redesigned Simulation Exercise in Classroom Decision-Making was the main focus of this study. It was theorized that this test would measure generic knowledge about research-based interactive teaching competencies. It was proposed that this knowledge might be learned through experiences as a classroom teacher, teaching training programs at the university level and the reading and application of information presented in the research literature on effective teaching, 15 It was hypothesized that results of a research study into the effects of the Simulation Exercise in Classroom Decision-Making 3 would elucidate the following: 1) the scores of participants with educational training would differ from those of participants without professional educational training, 2) the scores of participants with classroom teaching experience (teachers and student teachers) would differ from scores of those without experience (non-education students and education students) and 3) the scores of teachers would differ from student teachers. Research Questions The following research questions were addressed in this study: 1. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects who have received professional educational training to those without such educational training reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? 2. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and those without experience reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? 3. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and student teachers reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? 16 The SID2 was modified for use in this study. Items written specifically for secondary school teachers were eliminated as the new exercise was created reflecting elementary classrooms. A series of item analyses on the previously SID2 items and new items considered for the modified SID2 were conducted. Twenty-five of the previous 42 problems were kept. New problems were created bringing the total to 41 problems with 159 responses. The previous videotape format was changed to a PowerPoint presentation with audio. The revised version was entitled SID3. Limitations of the Present Study It should be realized that the competencies on which the instrument was created do not cover the entire range and complexities of effective teaching as defined in current literature. The competencies were taken from the Alabama Professional Education Personnel Evaluation (PEPE), designed for Alabama educators. While the Alabama assessment tool was based on national standards caution should be taken when making generalizations to the larger population. This study dealt with teachers? indications of how they would behave in specific classroom situations, it did not deal specifically with the teachers? actual behaviors in a classroom. It should be perceived as individual perceptions. 17 II. LITERATURE REVIEW Review of Literature The literature review for this study encompassed three major areas. The first section addresses past and current research components of effective teaching, including state and national teaching requirements. The second section investigates evaluation of teachers in regards to techniques used and areas being evaluated. The final section reviews simulation assessment tools assessing teacher pedagogy and decision-making. Components of Effective Teaching and Teaching Requirements Past and Current Definitions ?Describe the best teacher you have ever had,? was the beginning of one of the earliest pieces of research on teacher effectiveness. Posing this question to a large group of elementary students, Kratz (as cited by Medley, D. in Encyclopedia of Educational Research, 1982, p.1895) then analyzed the list of characteristics and used them to create a list that differentiated effective teachers from ineffective teachers. This design continued for the next fifty years or so, differing in whom was asked for the information parents, schools, educators or teacher educators. Studies using this design often included 18 six indicators of an effective teacher: 1) honesty, 2) good judgment, 3) considerateness, 4) adaptability, 5) enthusiasm, and 6) magnetism. While creating these lists of characteristics, no attempt was made to validate any of the characteristics by looking at the pupils taught by teachers noted to possess the qualities on the lists. Furthermore, these designs assumed that anyone who had ever been taught could judge a teacher?s effectiveness. A second assumption within these studies was that teachers are born, not made, evident in the fact that the listed indicators can be observed in people as a whole, they are not specific to teachers. No evidence exists that either the students or the researchers compiling the lists made distinctions between preexisting teacher characteristics and other variables related to teacher effectiveness. Excellence in schools has been directly related to the performance of their teachers and administrators, a fact that has consistently been shown to be true in the past two decades in a growing body of research (Kanstoroom & Finn, 1999; Darling- Hammond, 2002). Definitions of effective teaching that lead to such excellence are varied even within educational literature. Brophy and Good (as cited in Wray, Medwell, Fox, & Poulson, 2000) state that effective teachers ?demand engagement with the task, prepare well and match tasks to the ability of the students.? Entwistle and Tait (1990) listed only two factors in the definition of effective teaching: teaching ability and openness to students. Lowman and Mathie?s (1993) definition included only intellectual excitement and interpersonal rapport. Silcock (as cited in Wray, et al., 2000) defined effective instructors as those who provide pupils with maximum opportunity to learn. While each of these certainly is part of effective teaching, they lack a totality of characteristics. 19 In an effort to compile a working definition of teacher effectiveness, Giovanelli (2003) included items such as classroom management, teacher expectations, instructional behavior, content knowledge and classroom organization. Likewise, Stronge and Hindman (2003) presented six domains found in research about effective teachers: 1) prerequisites of effective teachers, 2) the teacher as a person, 3) classroom management and organization, 4) organizing for instruction, 5) implementing instruction, and 6) monitoring students progress and potential. These six domains are evident both in the current national and state standards for teachers. With the mission of assuring excellence of educators in the state of Alabama, a task force was formed to create an assessment tool; the Alabama Professional Education Personnel Evaluation (PEPE). According to Alabama State Department of Education (1999), the PEPE tool was designed to focus on competencies and knowledge/skills that effective educators possess, rather than personal traits. For purposes of this study, PEPE competencies were used to define teacher effectiveness. The competencies include: 1) preparation for instruction, 2) presentation of organized instruction, 3) assessment of student performance, 4) classroom management, 5) positive learning climate, 6) communication, 7) professional development and leadership, and 8) performance of professional responsibilities. Definitions and indicators for the PEPE tool are provided in Appendix A. Each of the mentioned definitions contain the aspects of the teacher?s knowledge of content, as well as how to organize the material and teach the material, known as pedagogical content knowledge. Pedagogical content knowledge is an important aspect of teaching. Understanding the content matter so well that explanations and demonstrations 20 are automatic is an acquired knowledge and skill base (Shulman, 1998, 2002). While the beginning of this knowledge base is formed during a student?s K-12 school experience, it continues in formation during the student?s undergraduate studies. In a longitudinal study of high school teachers Wilson, Shulman and Richert (1988) began investigating how new high school teachers learn to teach. While previous studies had focused on classroom management and other organizational issues, those were only part of the picture. Shulman developed a theory of pedagogical content knowledge as the kind of teacher understanding which allows the teacher to know how to teach their specific content area. Pedagogical content knowledge is a unique knowledge of the teacher?s content domain. Shulman (as cited in Wilson, Shulman & Richert, 1988, p.144) declared pedagogical content knowledge as a form that: ... embodies the aspects of content most germane to its teachability. Within the category of pedagogical content knowledge I include, for the most regularly taught topics in one?s subject area, the most useful forms of representation of those ideas, the most powerful analogies, illustrations, examples, explanations, and demonstrations-in a word, the ways of representing and formulating the subject that make it comprehensible to others? [It] also includes an understanding of what makes the learning of specific topics easy or difficult: the conceptions and preconceptions that students of different ages and backgrounds bring with them to learning. (Shulman, 1986, p.9) 21 Decision-making of Effective Teachers Bandura (as cited in Poulou & Norwich, 2002) argued that a teacher?s self- efficacy beliefs affect not only their activities in specific instructional strategies, but also affect their general orientation toward the education process. Bandura?s social cognitive theory aimed to construct a portrayal of emotional behavior, causal attributions and cognitive and emotional responses to students. Teachers should be aware of their own potential and feel confident about their skills in the classroom. Realizing your own capacity to bring about the desired improvement of your own classroom teaching is a powerful tool. Pajares (as cited in Poulou & Norwich, 2002) contended that education programs should help preservice teachers to develop a high sense of efficacy with the teachers continuing to investigate factors that contribute to a strong and positive teaching self-efficacy. How teachers exercise control over the events in their classroom is affected by the feelings of self-efficacy within the teacher. A teacher?s belief in his or her capabilities enables the teacher to believe in the decisions made each day within the classroom. Teachers are devoted to monitoring students, both for management and instructional purposes, however processing the multidimensional, simultaneous and immediate nature of classroom events can be overwhelming. The ability to reduce the complexity of the classroom and to cull significant events from the more incidental events is a necessary skill in teaching. Within a typical classroom day, the teacher will provide instruction, give encouragement, advise, assess, monitor, plan, and make numerous decisions. 22 Airasian and Jones (1993) state: Every time a teacher plans what to teach; disciplines or encourages a pupil; alters an instructional sequence; calls on a pupil for an oral response; makes judgments about a pupil?s interest, ability, motivation, or self-confidence; teaches a concept; forms a reading group; assigns a grade; gives homework; or revises a lesson plan, that decision is influenced by some information the teacher has acquired about the pupils and the classroom setting (p. 245). In a study by Burn, Hagger, Mutton and Everton (2000) reasoning used in decision-making by student teachers was examined. Noting beginning teachers do not have a range of past experiences and established repertoires on which to draw during a lesson, the study found the novice teachers much more dependent on detailed planning. The novices were found to plan both what to do in the lesson and how to answer student?s questions. As the study progressed, the beginning teachers became more comfortable with courses of action within the classroom that they had not planned or for which an accommodation had been written. Teaching Requirements While details vary considerably across the states, the basic process of becoming a licensed teacher is similar. Prospective teachers must usually complete required coursework from an accredited institution culminating in at least a bachelor?s degree. The type of training within these institutions also varies considerably. Some programs are 23 based on prescribed courses; others are composed of on-the-job training provided by a school district, while others have combinations of the two. To obtain a teaching license, or certification, the candidate must pass one or more competency tests. Such tests may include basic skills, pedagogical knowledge, and tests of subject-matter knowledge. A probationary time period, during which the new teacher is observed and regularly assessed, is often part of a state?s requirements. Maintenance of the teaching license requires teachers to complete a designated amount of professional development courses or activities. Many states? certification is a tiered hierarchy requiring completion of additional education, assessment and experience to obtain the next level of certification (Ludlow, 2001, Daugherty, DeAngelis & Rossi, 1997). The National Council of Accreditation of Teacher Education (NCATE) is a professional accrediting organization for schools, colleges and departments of education in the United States. With a two-fold mission of accountability and improvement in teacher education, NCATE holds accredited institutions accountable for meeting specific standards. The Council also encourages unaccredited schools to demonstrate their quality by working towards achieving accreditation (NCATE, 2004). Founded in 1954, NCATE is a performance-based system requiring institutions to provide evidence of competent teacher candidate performance. Colleges of education that have received NCATE accreditation are expected to ensure that teacher candidates not only know their subject, but know how to teach it effectively. As of 2003, 36 states have adopted or adapted NCATE standards as their state standards. 24 National Teaching Standards The Department of Normal Schools was established in 1870 within the National Education Association (Raths, 1999). The department was seeking a procedure for distinguishing excellent teachers from incompetent teachers. We still have organizations, national commissions and a variety of accrediting agencies, interested in providing standards for excellence in teaching. Unfortunately, while the reform of teacher education is nationwide, a single central authority does not set educational policy. The 50 separate states each have their own system with local school districts embedded within the state system. In many cases, the bodies that regulate teacher education are independent of the agencies that are responsible for the licensing of teachers (Oakes, 1999). The Interstate New Teacher Assessment and Consortium (INTASC, 2004) was established in 1987 to enhance collaboration among states interested in a different way of thinking about teacher licensing and assessment for education professionals. According to the Council of Chief State School Officers (1999) in 1993 the consortium proposed model standards that described what teachers just entering the work force should know and be able to do. Personnel from 17 different education agencies and representatives of the teaching profession created the standards. The consortium states an important attribute of the standards is that they are performance-based, with emphasis on the abilities teachers develop as opposed to the hours spent completing coursework. One mission of the consortium is to enable states to have greater opportunities in how they create and use their teacher education programs. Appendix B shows the ten INTASC principles for teachers. 25 INTASC is now developing subject-area standards for new teachers. Standards have been developed for English/language arts, science and mathematics. In the development stage currently are standards for elementary, art, social studies and special education. In addition, based on the model standards, INTASC has contracted with Educational Testing Services (ETS) to create a Test for Teaching Knowledge (TTK). The TTK is based on authentic situations that beginning teachers will encounter and is a constructed-response based test (Ambach, 1996). INTASC believes the TTK will be an appropriate test for issuing a provisional license for the first year or two of teaching. Furthermore, they believe that the test can be used as a screen to select candidates for internships. Efforts for equality of education for all students continue across the United States as we approach the deadline of the 2005 school year. These mandates set in motion in 2001, by President George W. Bush are the manifestations of the No Child Left Behind Act (NCLB). With a belief that a well-prepared teacher knows how and what to teach and possesses command of the subject to be taught, guidelines were put in place to ensure the quality of teachers in America (Kaplan & Owings, 2003, Ludlow, 2001, Wilson & Wood, 1996). To achieve its goals for improved school outcomes, the NCLB Act requires a ?highly qualified teacher? in all classrooms. In an attempt to provide an equitable learning environment, NCLB requirements state: ? Highly Qualified Teachers: To be deemed highly qualified, teachers must have: 1) a bachelor?s degree, 2) full certification or licensure, and 3) prove that they know each subject they teach. 26 ? State Requirements: NCLB requires states to 1) measure the extent to which all students have highly qualified teachers, particularly minority and disadvantaged students, 2) adopt goals and plans to ensure all teachers are highly qualified and, 3) publicly report plans and progress in meeting teacher quality goals. ? Demonstration of Competency: Teachers (in middle and high school) must prove that they know the subject they teach with: 1) a major in the subject they teach, 2) credits equivalent to a major in the subject, 3) passage of a state- developed test, 4) HOUSSE (for current teachers only, see below), 5) an advanced certification from the state, or 6) a graduate degree. ? High, Objective, Uniform State Standard of Evaluation (HOUSSE): NCLB allows states to develop an additional way for current teachers to demonstrate subject-matter competency and meet highly qualified teacher requirements. Proof may consist of a combination of teaching experience, professional development, and knowledge in the subject matter gathered over time in the profession. The National Board for Professional Teaching Standards (NBPTS) was created in 1987 and is directed by principals, teachers and education and political figures from across the United States (USDE, 1997). The national board offers the unique way to identify and assess teachers who are highly accomplished with a national certification process that is based on current research about effective teaching. To complete the NBPTS assessment requires approximately 150 - 200 hours of personal time. Teachers are required to videotape their own teaching and then analyze their lessons. Requirements also include time spent at the NBPTS assessment center, where they complete both 27 written and oral examinations. Teachers are then assessed in their response/performance in a real-life situation. While the certification is rigorous and demanding, for those who are successful the rewards can be considerable. Many states, including Alabama, will pay the $2000 fee required to seek certification. Other states are offering a more powerful incentive, salary supplements for nationally certified teachers (USDE, 1997b). The Educational Testing Service (ETS) has also created assessments for teachers. The Praxis Series: Professional Assessments for Beginning Teachers was developed to correspond to three steps in teacher development (Educational Testing Service, 2003). Those candidates entering a teacher training program may be required to take Praxis I, an academic skills assessment. Content-specific assessments and pedagogy tests, as part of Praxis II, coincide with licensure for entering the teaching profession. Classroom performance assessments for the first year teacher are used in the Praxis III. Almost 80% of states that require tests as part of their licensing procedure use the Praxis Series. State Teaching Standards A state?s response to national mandates provides an opportunity to improve K ? 12 educations within the state and close the gaps of educational inequality nationwide. States have designed programs aimed at different audiences; states typically began with putting standards in place for the current work force, while others also put into place a more detailed program for their universities training teachers. In response to the publication of A Nation at Risk in 1983, Alabama State Department of Education (ASDE, 2003) conducted a thorough review of the state?s 28 schools. A task force was appointed in 1988 to develop the criteria to be used in the evaluation of all professional education personnel. The criteria were adopted in 1989 with continuous field-testing and revisions taking place over the next four years. The Alabama Education Personnel Evaluation (PEPE) was implemented in 1993. Evaluators use three different method collections of teaching evaluations using PEPE as the tool. The evaluation tool includes an oral or written structured interview, a supervisor?s review form and a classroom observation record, all requiring extensive training prior to use. The multi-data source evaluation attempts to ensure an individual?s performance data is not biased. An educator?s performance over the course of a full school year is given consideration. As a three tiered hierarchy, PEPE was designed with a broad function as the top level or competency. The second level provides indicators or sub-headings with a more precise understanding of what is to be measured. The third level states the definitions and details measured. ASDE provides the following example: 1.0 Preparation for Instruction (Competency level) 1.1 Selects/states Long-Range Goals and Short-Term Measurable Objectives (Indicator level) Definition Items: ? selects long-range goals from state and/or local curriculum guides and sources ? selects/states and sequences short-term, measurable objectives in accordance with learner needs and program goals ? selects objectives from approved state and/or school system source 29 Personnel minimum scores reflect the baseline quality of educator services and differ according to position within the school setting. Novice teachers require a minimum score of 18, tenured and special area teachers 20, principals and assistant principals 36, and central office personnel 39. It is recommended that any personnel not receiving the required minimum score be placed on ?annual evaluations? with intensified professional development until professional improvement occurs. The latest published report, 1999 of novice teachers? scores are in Appendix C. The majority of novice teachers received a score of three, with four being the highest possible, in almost every area. Orienting students to the lesson, assessment and using results of the assessment and communicating high expectations had averages under three- point mark. Classroom management of both time and student behavior received scores of 3.09 and 3.06, respectively. Each of these PEPE domains could use continuing progress checks throughout the year, without consuming more time by the administrators who evaluate teachers. An instrument such as Simulation Exercise in Classroom Decision- Making 3 would allow further assessment, without taking more time away from administrators and providing feedback for self-assessment to the teachers. As a response to NCLB, Connecticut devised the Beginning Educator Support and Training (BEST) program. This two-year assessment combines accountability, extensive support and professional development (ASDE, 2003b). During the first year, new teachers must analyze a videotape of one of their own classroom teaching experiences, showing their knowledge of essential teaching skills. During the second year, the teachers collect materials forming a portfolio to document their understanding of testing strategies, teaching and good planning. Mentor teachers provide support for new teachers. 30 Beginning teachers are required to attend 3 clinics during the first year that reinforce teaching competencies. During the second year in the program, teachers attend 6 seminars specifically devoted to their subject area and grade level. Both Oklahoma and Kentucky provide new teachers with a three-member team (ASDE, 2003b). Each team consists of a representative from the beginning teacher?s college, a school administrator and a mentor teacher. Observations occur multiply times during the school year, with feedback offering support and guidance. In a pay-for-performance plan, the Douglas County school district in Colorado created collaboration among the school district, the Board of Education and business and community members (Wolf, Lichtenstein, Bartlett, & Hartman, 1996). The plan was conceptualized as a way for teachers who demonstrate proficient performance to receive annual salary increases, while providing effective teaching to its students. Each teacher would submit a portfolio with a number of components. Of interest is the survey given to parents and students. The teacher chooses 10 sets of parents and students to complete an assessment of his or her teaching. The school administration chooses 10 additional sets to survey. Within the portfolio, the teachers are asked to reflect upon the assessment results. Upon completion of the entire portfolio, teachers meet with administrators to discuss and explore aspects of the teacher?s performance. While only five state programs have been discussed here, it should be noted that the majority of the states have developed or are in the process of re-developing their programs for testing and support systems for educators. 31 Assessment Methods Traditional Assessment There is growing concern regarding the use of traditional assessment methods. Traditional methods of assessment are typically defined in the literature as multiple- choice tests, norm-referenced tests, and standardized tests (Brualdi, 1998; Hawkins, Frederiksen, Collins, Bennett, & Collins, 1993; Valencia, 1997, Worthen, 1993). Those in favor of assessment reform efforts have highlighted the disadvantages of traditional testing. A thought-provoking analogy was made by Grant Wiggins (1989) when he stated that using traditional testing to determine students? learning is like using a person?s pulse rate as a measure of a person?s total health. The obvious implication is that traditional tests measure only one aspect of a student?s learning or knowledge and that other testing formats should be considered to get a more accurate picture of ability and understanding. Wiggins (1993) also has criticized traditional tests by stating that these tests are only able to reveal whether or not students can recall or ?plug-in? what they have learned out of context. Darling-Hammond (1994) echoes this criticism by suggesting that traditional testing fails to support the learner?s capacities to perform real-world tasks. A common criticism of traditional testing is the implied expectation of the learner (Darling- Hammond, 1994; Travis, 1996). Through its very nature, traditional testing encourages students to learn specific answers versus learning and understanding abstract thinking or higher cognitive skills. 32 Other criticisms of traditional testing direct attention to the actual construction or format of the traditional test (Darling-Hammond, 1994; Wiggins, 1993). Because of the way widely-used multiple-choice tests are designed, they exclude many different kinds of knowledge and types of performance we expect from students, putting students in a passive role, rather than one that actively engages their learning capabilities. Unease with traditional testing methods continues to gain momentum in the general population and in the education sector. Conventional tests of achievement are more and more being considered as poor predictors of student abilities. It is recognized that these tests predict only about 10% of the variation among people in real-world measures of success. The question that then becomes apparent is, ?Where are the other 90%?? (Sternberg, 1998). Experts in the field of education have responded with the development of alternative assessments. Traditional tests do not give us pictures of individuals as a whole. We are not able to distinguish other virtues such as creativity, civic mindedness, or morality of an individual (Checkley, 1997). Assessment of an individual must include a variety of methods. No single method will capture the essence of an individual's learning ability. Assessing in real-world contexts necessitates moving beyond the traditional forms of assessment. Alternative Assessment Various forms of alternative assessment have been labeled as authentic, direct, performance and alternative. While there are subtle differences between each of the alternative methods, they all exhibit two central facts. First, all refer to direct examination 33 of a student?s performance on tasks that are relevant to real-life situations; second alternative assessments are viewed as alternatives to traditional multiple-choice, fill-in- the-blank and standardized achievement. Alternative assessment has been defined as an assessment that attempts to record and judge the qualities of actual performances rather than inferring an ability to perform from indirect decontextualized measures of cognitive traits and are designed to measure a student?s performance in any given subject through application of knowledge on that topic (Hawkins et al., 1993, Valencia, 1997, Worthen, 1993). They are student-centered, educational, and engaging. Their focus is on developing understanding and applying knowledge, rather than evaluating achievement only (Moorcraft, Desmarais, Hogan, & Berkowitz, 2000). Wiggins (1993, 1990) has been a leader in the authentic assessment movement and contends that with authentic assessment students have greater clarity about their responsibilities and are asked to master more engaging tasks. Additionally, his research indicates that both students and teachers believe that authentic assessment results are both meaningful and useful for improving instruction. Perhaps the most widely recognized and employed characteristic is that authentic assessments are designed to be truly representative of performance in the field (Darling- Hammond, 1994; Feiman-Nemser, S & Parker, 1990, Wiggins, 1989). Students should be given tasks that have real-world contexts in order to have deeper understanding of the subject. Baron (1998) examined research findings that can be used in the design of effective authentic assessments. Generally, students will engage more easily with problems that are embedded in challenging real-world contexts that have obvious 34 relevance to their lives (Resnick, 1987). Therefore, an important aspect of assessment design identified by Baron is understanding that students learn best when problems are interesting, meaningful, challenging, and engaging. Authentic assessment is expected to be the future of assessment of educational outcomes. The obvious advantage in the short term is that this type of assessment provides useful information about what students can do. The researcher Resnick (1987) perhaps put it best when she stated, ?What you assess is what you get.? Authentic assessment has roots in several different modern theories. One theory with obvious ties to authentic assessment is situated cognition. This theory developed by Brown, Collins, and Duguid (1988) states ?knowledge is situated, being in part a product of the activity, context, and culture in which it is developed and used.? Much of their work stems from Vygotsky?s cultural-historical theory of psychological development and from the research of Jean Lave and the apprenticeship concept. Vygotsky?s (1998) theory was to explain changes that account for emergence of higher cognitive functions and the role of culture in these changes. His process of signalization is states that key mental processes of perception and simple memory, along with others, are governed by the recognition of co-occurring stimuli in the environment. Lave and Wenger (1990) argue that learning as it normally appears is a function of activity, culture and context in which it appears. This contrasts with most of classroom learning. Social interaction is a critical component of situated learning. Learners become involved in a ?community of practice? which embodies certain beliefs and behaviors to be acquired. As the beginner or apprentice moves from the periphery of this community 35 to its center, they become more engaged and active within the culture. At that point, they assume the role of expert or old-timer. Brown, Collins and Duduid (1988) contend that education is based on the premise that there is a true division between what we know and what we do. They state that the situation itself co-produces knowledge through activity, therefore, the context must be considered in learning. It is their belief that people who use knowledge actively rather than just acquiring it build a rich understanding of the world. Assessment of Preservice Teachers The research into teacher effectiveness suggests the need for a teacher to have a varied repertoire of teaching strategies. Direct instruction has been shown to help low achieving students and those with low cognitive skills. Helping students develop higher cognitive skills, problem solving skills, creativity, self-monitoring skills and self-help skills, a plethora of teaching strategies is necessary. The most competent teacher must use his or her skills with flexibility and intelligence (Ellett & Teddle, 2003). Consequently, teacher educators must assist their preservice teachers in the development of those necessary skills. As one way of teaching doesn?t meet all needs, it is intuitive that one assessment method doesn?t meet all needs. Accountability for the educators? college or university training remains significant. Shannon and Boll (1996) investigated assessment procedures of preservice teachers. Teacher assessment for licensure and/or certification has usually taken the form of a pencil-and-paper written test. In addition to the regular written test, different institutions used performance assessment simulations, portfolios, interviews and 36 computerized tests. Twenty-two of the 23 institutions surveyed reported using some form of a written test. Fifteen of the 22 institutions used more than one form of assessment, while nine reported using three or more forms of assessment. One of the challenges facing teacher educators is defining preservice teacher experiences that are necessary to produce effective teachers. If educators all agreed on what type of teacher education program would consistently graduate effective teachers, wouldn?t they all be the same? Finding common ground on theory versus practice, a definitive knowledge base, and even what order all of these things should be sequenced continues to plague teacher training institutions. Assessment of beginning teachers was also the theme in a study with the general conviction that a teacher is at the center of any attempt to improve quality of teaching and learning in schools (Uhlenbeck, Verloop, & Beijaard, 2002). The study argues that the best way for a teacher education program to become and remain effective is to continuously evaluate the program based on the evolving nature of education. Ferguson and Womack (1993) offered a summary of existing research and added to the data by reporting the findings of a three-year study of the graduates of the teacher education program at Arkansas Tech University (ATU). A study of secondary student teachers (N= 266) over a seven semester time period added to the body of research indicating that teacher education makes a difference in teacher performance. Previous research exists which confirms the proposition that pedagogical coursework has a positive effect on teaching performance. In an effort to evaluate the effectiveness of its teacher education program, ATU assessed the extent to which education and subject matter coursework predict the teaching performance of student teachers completing the 37 program. An instrument consisting of 107 Likert-response items measured instructional competence according to 13 categories of expertise. The investigators stated that of the variables subjected to analysis in this study, coursework in education was the strongest predictor of teaching performance. Evaluation must be participatory and reflective in nature to be meaningful to teachers. ?Administrators and teachers need access to comprehensive evaluation models that capture the complexities of teaching. Congruent with an expanding knowledge base of teaching and learning, performance standards are being developed that lead to reconfigured assessment designs requiring an array of reflective, analytic skills,? stated an article by Eileen and Stephen Weiss (1998). Doebler, Roberson, and Ponder (1999) designed a study to examine the use of case study analysis in an effort to improve decision-making abilities of preservice teachers. Developed as an indicator of principles related to teaching, it sampled students at four distinct points of education, in an effort to validate the effectiveness of the teacher education program of the school?s teacher education program. Students? responses to the case studies were reviewed to determine the students? decision-making ability. The study concluded that students within its teacher education program were benefiting from such case studies and later responses were specific, logical and seemed to be derived from a combination of coursework and experience. Case based pedagogy is proving to be an effective strategy in the development of preservice teachers. Sudzina (1994) describes a team case competition in which teams of preservice teachers from five teacher preparation institutions were invited to solve classroom dilemmas, present a written analysis and prepare an oral presentation and 38 defense. Coaches encouraged the competitors to cooperate, compete, develop areas of expertise, weave theory with practice, and assess resource possibilities and problem- solve. The competition, held at the University of Virginia, reinforced the connection between theory and the real-world. Students were provided with classroom vignettes and required to provide an unbiased and multidimensional perspective in their decision- making. The Curry School of Education offered its first case study via the Internet in 1995. Prior to using the Internet, the University of Virginia offered case-based scenarios to its preservice teachers within the regular education program. With CaseNet now in place, students and professional colleagues are able to participate in online discussion groups, videoconferencing and electronic journals. One participant stated, ?I learned a lot by comparing my ideas with those of my peers in class and on the Internet. It allowed me to understand different ways of dealing with issues that may arise while I am teaching.? An interactive simulation also being used at the Curry School of Education is a software program that is a vehicle for mapping teacher decisions (Strang, & Clark, 2001, Strang, 1998). In 15 years of use Strang states that over 2000 teachers and student teachers have been using a simulation program to instruct and manage ?software students.? Lesson related feedback, pacing the flow of the lesson, managing misbehaviors and the use of wait time are all being realistically simulated allowing teachers to better assess their own teaching strategies. The simulation combines the use of the teacher?s decision-making and a clear assessment of the skills used, allowing teachers practice with typical teaching issues (Strang, Badt, & Kauffman, 1987, Strang & Loper, 1983). In review of the simulations it was shown that experienced teachers are more 39 reflective, use longer think time yet reach better decisions, show more personal control in their planning, don?t feel they must assign all activities, and spend equal time with each group or individual. ?This results in a more emphatic classroom environment, a setting in which spatial, activity, and teacher help decisions are more apt to reflect the needs of the individual pupils rather than the global scripts and/or personal apprehensions of the teacher? (Strang, 1996, p.142). Cruickshank and Metcalf (1993) advise multiple forms of assessment, at multiple times in an education student?s career. Simulation, microteaching, and reflective teaching are presented as alternatives to mass-produced, one answer only assessment tests. Microteaching is a teaching encounter that is scaled down. Education students are asked to teach a brief lesson, after which feedback is given on how well the student teacher demonstrated the desired ability according to the given criteria for the lesson. Simulations, while offered in a variety of forms, intend to create the effect or appearance of some real-life situation. Simulations allow participants the opportunity to experience the situation, make decisions about it and witness the results of their decisions. Finally, reflective teaching is similar to the microteaching form; however, students are then asked somewhat guided questions to initiate reflection on the student?s teaching practices. Western Oregon developed a standards based program in 1991 that required preservice teachers to complete two work samples as one element recommended for initial licensure (Shalock, Shalock, & Myton, 1998). The work samples are built around the concept that teachers should be reflective practioners who use research outcomes as a tool for future instruction and reporting plans. The work samples would consist of a three to five week unit of instruction and include evidence about pupil knowledge acquisition 40 during the unit. The work sample is based on the theory that learning is student centered, outcome based and context dependent. Western Oregon (Western Oregon University, 2003) also developed a Teacher Effectiveness Project designed to detect any differences that might exist in the performance of participating teachers as a consequence of their preparation programs. The program was based on the hypothesis that teachers prepared in programs that were consistent with Oregon?s design for K ? 12 schooling would be more ready to meet the state standards when they entered the work force. The Western Oregon University (WOU) website declares they have been recognized by the American Association of Colleges for Teacher Education as a national model for work on their Teacher Effectiveness research project. Current research at WOU is connecting pre-service training with effective teaching. SimSchool is a program developed between WOU and The Vermont Institutes (Vermont Institutes, 2003). SimSchool is a simulated classroom environment that allows individuals to test their skills in a ?real? classroom and receive immediate feedback. SimSchool allows the user to choose the type of school setting according to social, cultural and economic background. Users are presented with a prologue explaining a situation and challenging the user to explore the school context to understand more about the learning environment, decide what the problem or situation is and write a response using resources which are given. Different strategies are allowed, with a variety of options available, as opposed to one right answer. Further, responses written by the user are accumulated and available for review and tracking, allowing self-assessment of progress made. Their vision for the future is a ?classroom that dramatically improves 41 future teacher?s knowledge and skills in results-oriented teaching by offering them many network-based opportunities (both human and machine) to practice making instructional and assessment decisions and seeing the impact of those decisions on students.? The website states SimSchool will be nationally available in the near future. Network-based assessment is emerging in teacher assessment and instruction (Mislevy, Steinberg, & Almond, 2000). Online SAT and GRE are examples of online testing and scoring which are becoming more prevalent. Creating assessments that can combine multimedia, are responsive to learners, flexible over many situations, and unobtrusive to the natural actions of learning are allowing us to assess how students know and creating a body of evidence for further research (Mislevy, et el., 2000). Bransford, Brown and Cocking (2001) state that because technology mediates learning in new ways and includes new forms of knowledge and possibilities for documentation and analysis, we should focus our attention on expanding our assessment conceptions. Development of a Simulation Assessment Prior Research Using SID Combining case-based assessment and an alternative assessment method SID was created in an effort to provide information about teachers?, both student teachers and experienced teachers, decision-making styles. Hays (as cited in Shannon, Medley, & Hays, 1993) developed the Simulation Test of Interactive Teaching Competencies-Hays (STITC-H) that focused on specifications of teacher competencies as defined by the Virginia Beginning Teacher Assistance Program (BTAP). Using the competencies 42 identified by BTAP, Hays created realistic classroom situations with different suggestions as to how a classroom teacher might respond to the situation, requiring the respondents to either agree that the response suggested was appropriate or inappropriate. Shannon (1990) modified the STITC-H in an effort to discern differences between experienced teachers?, student teachers? and non-education students? responses. While somewhat different in context, Shannon?s revision, SID2, was administered in much the same way of having respondents agree or disagree with responses to given classroom situations. STITC-H, as the first version of SID, was developed to measure the functional knowledge of effective teaching in 13 of 14 areas defined in the development of Virginia?s Beginning Teacher Assistance Program (BTAP) (as cited in Shannon, Medley, & Hays, 1993). The competencies were empirically found to be related to effective teaching and their content could be referred to as opinions or beliefs of expert educators about the nature of effective teaching. Hays? instrument contained 65 problems with a varied number of suggested responses, with a total of 260 responses. Administration was done in two sittings of 45 minutes each, allowing a short break between the section administrations. A brief description was projected onto a screen, using a slide-projector. Each response was heard aloud from a pre-recorded cassette tape, but not shown on the screen. A response time of five seconds was allowed before the next response was read. The participant decided if the suggested response was ?appropriate? or ?inappropriate? and marked the answer on an answer sheet. 43 Each suggested response was consistent or inconsistent with a described behavior in one or more of the 13 BTAP competencies. A score on each competency was obtained by adding the number of each response consistent with the specific competency that had been marked appropriate and the responses inconsistent with the competency that had been marked inappropriate. Using all 13 competencies, a total score was obtained. Coefficients of internal consistency for individual competency scores on STITC-H ranged from 0.45 to 0.77 with a mean of 0.51 and a median of 0.61. The reliability estimate for total scores was 0.89, and can be presented as evidence for both internal consistency and construct validity. A test measuring knowledge of effective teaching should also reflect higher scores for participants with greater amount of training, experience or both. A test reflecting those differences can be shown as providing evidence that scores reflect a valid construct as a measure of knowledge of effective teaching. Construct validity was measured by obtaining scores from four different levels of participants; experienced teachers, student teachers, students in education without fieldwork experience and students not enrolled in the college of education. The group of experienced teachers did score significantly higher than did student teachers on 5 of the 13 competencies measured. Non-education students scored lower on 9 of the 13 competencies than did the education majors. The second version (SID2) was created with a follow-up study to perform item and internal consistency analyses. Problems and suggested responses that did not contribute to score variance were eliminated. 44 SID2 contained 42 problems instead of the original 64, using 125 responses rather than the initial 260. This reformatting allowed the test to be administered in a single sitting of 45 minutes. Administration was done on videotape, rather than the original slide show. Again, each problem was shown on the screen, while being read aloud. The suggested responses were also read aloud, but not shown on the videotape. In taking SID2, the teacher was given 5 seconds to record his or her decision about each suggested response. The limited response time was developed in an effort to emulate the quick decision-making, often with insufficient information, teachers must make in a real classroom situation. Answers to the suggested responses were again recorded as appropriate or inappropriate on an answer sheet. Construct validity of scores from SID2 was assessed using the same classification of participants as those in the original study. Shannon (1990) found those trained in education scored higher than those with no education training and those with classroom experience scored higher than those without experience. However, the scores between experienced teachers and student teachers showed no difference. STITC-H seemed to be more sensitive to the amount of knowledge acquired from experience as a classroom teacher. SID2 seemed more sensitive to the knowledge acquired by training. Future Research Using SID Redevelopment of SID2 based on current PEPE competencies and INTASC standards, using a computer program to administer the test would allow assessment of future and current educators. Potentially a newly created assessment tool could be used 45 after completion of a degree, yet prior to certification or as a pre-interview assessment or screening tool for job applicants in school systems. It should be realized that the competencies on which the instrument was created do not cover the entire range and complexities of effective teaching as defined in current literature. Also, while this study dealt with teachers? indications of how they would behave in specific classroom situations, it did not deal specifically with the teachers? actual behaviors in a classroom. Conclusion The ability to define effective teaching and effective decision-making in a classroom continues to grow as national efforts continue to strive for teachers of high quality in all classrooms. Teaching requirements at the national level are established yet with each state being held accountable to create a plan to accomplish the requirements, much work is to be done. Methodologies of assessment continue to be explored as the need for documentation and accountability grow. Prior research using SID2 allowed exploration of measuring students in Virginia with the standards set within their state. Future research using SID3 could provide a necessary assessment of Alabama teachers and each PEPE domain. To that end, SID 3 has begun the assessment. 46 III. METHODS Purpose of Study The Simulation Exercise in Classroom Decision-Making 3 was developed to verify and extend the findings obtained during both the STITC-H and the SID2. It was theorized that SID3 could measure generic knowledge about research-based effective teaching competencies. It was proposed that this generic knowledge might be learned through teaching training programs at the university level, the reading and use of information presented in the research literature on effective teaching, and experiences as a classroom teacher. It was hypothesized that the Simulation Exercise in Classroom Decision-Making 3 would elucidate show the following: 1) the scores of participants with educational training would differ from those of participants without professional educational training, 2) the scores of participants with classroom teaching experience (teachers and student teachers) would differ from scores of those without experience (non-education students and education students) and 3) the scores of teachers would differ from student teachers. 47 Development of SID3 Examination of Existing Items and Item Development The original STITC-H consisted of 65 problems scored on 260 alternatives in thirteen competency keys and required approximately 90 minutes to complete. A revision of STITC-H was completed by Shannon (1990) and renamed SID2. Shannon?s exercise contained 42 problems with a total of 188 items and required approximately 45 minutes to complete. Of the 188 items, Shannon had retained 125 from the original STITC-H. Examination of Shannon?s items and responses revealed ten items that were specifically written for teachers of students in grades seven through twelve. Those ten items were removed from SID3 designed for an elementary classroom. Other items were examined and redefined based on the competency indicators from the Alabama teacher evaluation system. Shannon?s SID2 was aligned with Virginia indicators. SID3 retained 25 of Shannon?s 42 items. New items were written and targeted for specific PEPE domains that needed more representation within the study. When writing new items, questions were developed to assess the following: orienting students to the lesson, assessment, communicating high expectations and classroom management. Expert Review An expert focus group consisting of two principals and three elementary teachers was formed to review the new and old items. The principals each had over ten years of experience and the elementary teachers each had at least six years of experience. Walton (as cited in Eggers and Jones, 1998) distinguished experts as people who have sufficient 48 knowledge, have mastered the advanced skills of a particular domain of knowledge or experience in the field. Thirty items and 120 responses were written by the primary investigator and reviewed by the expert focus group. For use in SID3 each item and response required four of the five experts agreement. The focus group suggested including more items in the areas of creating a positive classroom climate, classroom management, presentation of instruction and assessment. The content was shaped as a result of these suggestions. Initially the panel placed each response in only one domain, however upon the recommendation of principals in the review, items were allowed to represent more than one domain. The focus group revised the responses to reflect this decision. The 65 responses addressing the creating of a positive learning climate reflected 58% of the total instrument. Additional items were written specifically to ascertain the participants? beliefs concerning how to assess students in the classroom as well as using the assessment to individualize instruction. Assessment of student performance included 52 item responses or 46% of the instrument. Presentation of instruction included 50 items, classroom management 47, communication 32 and preparation for instruction 28 items. See Appendix E for the distribution of items according to the domain exhibited. Additional Expert Item Review School administrators were selected using systematic random sampling from the 1500 elementary and secondary schools within the Alabama state system. Using the list of schools, one random school was chosen after which additional schools were chosen at evenly spaced intervals. Each of the school systems within the state was represented. 49 Dalkey (as cited in Eggers and Jones 1998) explained that anonymous feedback when decision-making provided more accurate decisions than when such groups met face to face to reach decisions. Subjective conjectures about beliefs, ideas and decisions are best made in an atmosphere when group members are allowed to submit responses without having to consider others? reactions to responses. The Delphi technique encompasses this belief as it uses questionnaires dispersed through the postal service, electronic mail and other distribution tactics. This method allows a collection of information from experts geographically separated, while allowing anonymity. Surveys containing three sections were mailed to 280 elementary administrators and 125 secondary administrators of the 1500 schools within the state. Each packet contained a cover letter and return envelope that had been addressed with postage affixed. Section one contained ten different scenarios or vignettes that could happen in school today. Participants were asked to identify the PEPE competency, or competencies they felt the scenario best reflected. Using a scale of 1 ? 5 with 1 being not realistic and 5 being very realistic, respondents rated how realistic each scenario was in a classroom of 2005. Finally, participants were asked to provide two possible responses to each scenario, based on their experience and expertise, writing one response expected from a teacher that demonstrates excellence (i.e., scoring a ?4? on PEPE) and one response that represents an unsatisfactory teacher (i.e., scoring a ?1? on PEPE). Including open-ended questions allowed understanding of the participants? viewpoints (Patton, 1990). Half of the evaluators received newly written scenarios and half received scenarios that had been used previously in SID2. The survey addressed the professional background and demographics of the participant. The survey used is Appendix D. 50 Responding administrators were predominantly white female elementary school principals. Participants consisted of administrators (n = 67, females= 51, males= 16) from elementary and secondary schools within the state of Alabama. Responses given by the principals were used to select scenarios to be included in the new SID3 assessment tool. Items shown to correspond with PEPE competencies and having received high scores for realism were retained and/or revised. A combination of previous (SID2) items and new items was used to create a new SID assessment tool (SID3). The final simulation instrument contained 43 items with 159 responses. Participants Four groups of participants completed the simulation exercise. The four groups included 1) university students not enrolled in the college of education (NoED), 2) students within the college of education who had begun coursework but had not done student teaching (PreST), 3) students who had completed the student teaching experience (ST) and 4) experienced classroom teachers (ET). Participants (PreST and ST) were selected from within the college of education. Only students (PreST, n= 143) who had completed at least six semester hours in courses within professional education coursework were considered for this sample. The sample of student teachers (ST, n= 34) was currently completing student teaching or had completed it within the previous school year. Experienced classroom teachers (ET, n= 69) from both elementary and secondary schools were included. Teachers within this group varied in years of classroom experience from 1 to 33 years, with a mean of 8.43 years and a median of 3.5 years. 51 Participant students majoring in areas other than education (NoED, n= 69) were selected primarily from general core classes and nursing classes. Sample refinement was based upon parameters that could be compared between the groups, such as education level attained, number of education classes completed, background in elementary or secondary classrooms and classroom teaching experience. The creation of four distinct groups with little or no overlap was the goal of sampling parameters. Instrumentation A revision (SID3) of the Simulation of Interactive Decision-Making, Version 2 (SID2) was created as an assessment to measure teachers? responses to scenarios of specific classroom situations based on current research of effective teaching and decision-making. The video recording of the SID2 format was updated to a PowerPoint slide show with an accompanying answer sheet. Each classroom scenario description was shown on the screen while being read aloud. After the classroom scenario was described three or four responses were presented audibly one at a time. Respondents were asked to designate the response as an appropriate or inappropriate reaction to the scenario posed on an answer sheet. Five seconds were allowed for each response to represent the reality of the quick decision-making teachers must perform frequently throughout the school day within a regular classroom situation. The exercise began with instructions being given both visually and orally. Participants recorded their responses on an answer sheet. Administration of the exercise took approximately 45 minutes to complete. 52 Each participant was asked to complete a background questionnaire that asked age, gender, educational background, and teaching experience (Appendix E). Participants could request a profile sheet of his or her test results by including an electronic mailing address on the questionnaire. The profile would describe individual responses and responses based on group scores. Data Collection The Simulation Exercise in Classroom Decision-Making (SID3) was administered to the groups of participants during January, February and March of 2005. Student participants were administered the exercise in a classroom setting. Educators were administered the exercise during a faculty meeting or in a classroom setting as a student in a graduate level course. SID3 was administered to groups ranging from six to 34 participants using a PowerPoint slide presented on a large screen or a large television monitor placed in the front of the room. Participants? responses were recorded on a scannable answer sheet. Answers were scored as appropriate or inappropriate, depending upon the responses? given by the expert focus group. Scores from specific competencies and from the total exercise were compiled. Data Analysis Internal consistency of each of the eight competencies of PEPE as demonstrated within SID3 was examined. Group homogeneity was examined within the four major 53 groups of participants and refined as needed. Finally, three major hypotheses were examined to evaluate the construct validity of the instrument. Reliability was addressed using Cronbach?s coefficient alpha for reporting separately the internal consistency of the eight different competencies and the total test. Total test scores and scores of specific competencies were examined in answer to the research questions. Construct validity for SID3 can be evidenced if the test answers are affirmative to these questions: 1. Do scores on the revised Simulation Exercise in Classroom Decision-Making reflect differences in the ability to apply teacher effectiveness findings to teaching problems between subjects with classroom teaching experience and those without experience? 2. Do scores on the revised Simulation Exercise in Classroom Decision-Making reflect differences in the ability to apply teacher effectiveness findings to teaching problems between experienced classroom teachers and university education students would differ from non-education university students? 3. Do scores on the revised Simulation Exercise in Classroom Decision-Making reflect differences in the ability to apply teacher effectiveness findings to teaching problems between experienced classroom teachers and student teachers? Results of each question were analyzed for significance of differences between the groups. Raw scores for each PEPE domain were converted to T scores so that each domain had a mean of 50 and a standard deviation of 10. Standardizing these scores allowed for a simpler comparison of group profiles. Group profiles across the six 54 domains were statistically examined using a mixed-ANOVA design, with one between variable (GROUP) and one within variable (PEPE Domains). This design yielded three significance tests (F-tests) reflecting differences among the four sampled groups, across the six PEPE domains and the interaction between group and domain. The principal analysis incorporated three orthogonal contrasts. The first contrast compared those without professional educational training and those with at least six hours of educational training. This contrast was designed to determine the effect of professional coursework in an education program. The second contrast was made to determine the effects of classroom teaching experience, comparing non-education students and educational students without classroom experience with student teachers and teachers. The third contrast was to determine the effects of classroom teaching experience between teachers and student teachers. The main effect for each PEPE domain was eliminated due to the standardization of domain scores. Group differences were followed-up using post-hoc procedures (e.g., Tukey and Bonneferoni) while significant interaction effects were followed-up with analysis at the simple effects levels. 55 IV. RESULTS Analysis of Data and Discussion Responses from each participant?s answers to the 159 items were entered into an SPSS data file. Analyses were performed using the SPSS 11.0 version for MacIntosh. Data analysis was begun with a sample refinement within the four major groups of participants. Internal consistency of each of the eight competencies of PEPE as demonstrated within SID3 was examined. Finally, three major hypotheses were examined to evaluate the construct validity of the instrument. The results from each of these will be presented within this chapter. Sample Four groups of participants completed the simulation exercise. The four groups included 1) university students not enrolled in the college of education (NoED), 2) students within the college of education who had begun coursework yet had not done student teaching (PreST), 3) students who had completed the student teaching experience (ST) and 4) experienced classroom teachers (ET). Gender, age and ethnic demographics of the participants are provided in Table 1. Males comprised 24% (n= 77) of the original 315 participants. Within the four groups the 56 male composition ranged from 0% to 40%. Females comprised 76% (n= 238) of the original participants with group composition ranging from 59% to 100%. Educational status and education courses completed are summarized in Table 2. Participants included university students from each of the five categories in educational status the majority of educational students were juniors and seniors, with non-education students were represented primarily by freshmen and sophomores. Classroom teaching experience and highest education level attained are given in Table 3. Classroom experience was identified as no experience, field observation only, mini-teaching prior to student teaching, student teaching (internship) and classroom experience. Classroom experience is defined as those having completed undergraduate studies and working or having worked in a school classroom as the sole educator. 57 Table 1 Demographics of Overall Sample and Subgroups *NoED PreST PostST ET n= 69 n= 143 n= 34 n= 69 _____________________________________________________ n (%) n (%) n (%) n (%) Gender Female 41 (59%) 106 (74%) 34 (100%) 57 (85%) Male 28 (40%) 37 (26%) 12 (15%) Age 18 or under 10 (14%) 6 (4%) 19 ? 21 31 (44%) 98 (68%) 16 (47%) 22 ? 30 20 (28%) 37 (25%) 18 (52%) 33 (47%) 31 ? 40 6 (8%) 1 (.06%) 21 (30%) Over 40 2 (2%) 1 (.06%) 15 (21%) Ethnicity African American 8 (11%) 9 (6%) 1 (2%) 13 (18%) Asian American 1 (.06%) 1 (1%) Biracial 1 (.01%) 2 (.01%) Hispanic 2 (2%) Native American 1 (1%) 2 (1%) White, not Hispanic 59 (85%) 128 (89%) 33 (97%) 51 (73%) Misng 1 (1%) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers 58 Table 2 Undergraduate Status and Education Courses Completed *NoED PreST PostST ET n= 69 n= 143 n= 34 n= 69 _____________________________________________________ n (%) n (%) n (%) n (%) Undergraduate Status Freshman 21 (30%) Sophomore 18 (26%) 17 (11%) Junior 13 (18%) 39 (27%) Senior 6 (8%) 79 (53%) 32 (96%) 5 th Year 4 (5%) 8 (5%) 1 (3%) Missing 7 (10%) 1 (3%) Education Courses Completed None 41 (59%) 5 (3%) One 2 (2%) 6 (4%) Two 5 (7%) 23 (16%) Three 3 (4%) 21 (14%) Four 2 (2%) 27 (18%) Five or more 5 (7%) 61 (42%) 33 (97%) Missing 11 (15%) 1 (3%) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers 59 Table 3 Classroom Teaching Experience and Education Level Attained *NoED PreST PostST ET n= 69 n= 143 n= 34 n= 69 _________________________________________________ n (%) n (%) n (%) n (%) Classroom Teaching Experience None 69 (100%) 12 (8%) Field observation 27 (18%) Mini-teaching 95 (66%) (prior to student teaching) Student teaching 2 (1%) 34 (100%) (internship) Classroom teaching 7 (4%) experience 69 (100%) Education Level Pre- Bachelors 63 (91%) 143 (100%) 33 (97%) Bachelors 1 (1%) 14 (20%) Bachelors + credits 2 (2%) 1 (2%) 19 (27%) Masters 18 (26%) Masters + 15 credits 3 (4%) Masters + 30 3 (4%) Masters + 45 1 (1%) 4 (5%) Specialist 1 (1%) Missing 2 (2%) 2 (2%) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers 60 Sample Exclusion Criterion As Table 2 shows, each of the four groups was comprised of over 74% females with the exception of the non-education students, which was 59% female. Comparisons were made between females and males before any further analyses were done. Females, statistically outperformed males scoring 124.9, compared to 116.3 (t 313 = 4.65, p < .001) Secondly, there were no males in the PostST group allowing no way of comparing across groups. Given this information, it was decided to use only females for all future comparisons. In the construction of the four groups, the goal was to have little or no overlap in the education level attained, education courses completed and classroom teaching experience. Finally, SID3 was designed primarily for elementary educators therefore results from secondary level educators or students preparing to teach at the secondary level were not included within this study. Final Demographic Composition The undergraduate status and number of education classes completed are summarized in Table 4. The PreST were spread between sophomores and fifth year students with the NoED gap spread even further between freshmen and fifth year students. PostST participants were 96% seniors. The entire sample of NoEDs had completed no education courses, while all PostST had completed five or more education courses. Again, the PreST were divided between none and five or more education courses. 61 Table 5 illustrates the amount of classroom teaching differences of the four groups, as well as highest educational level attained. Classroom teaching experience ranged from none in the NoED group, to all having actual teaching experience in the ET group. The majority of PreST had experience in mini-teaching and all of the PostST had completed student teaching. Further distinction is shown in the educational level attained. The majority of NoED participants were undergraduates with only four participants having previously completed a degree. PreST and PostST contained only undergraduates. The teachers varied from having attained a bachelors degree to a specialists degree, with 32% having received a masters degree. The length of time teachers have been in the classroom ranged as follows; one to five years (n= 28, 58%), six to eleven years (n= 9, 18%), fifteen to eighteen years (n= 4, 8%), twenty to thirty-three years (n= 6, 12%) and one participant left the question unanswered. 62 Table 4 Undergraduate Status and Education Courses Completed of Final Sample *NoED PreST PostST **ET n= 41 n= 46 n= 33 n= 49 _________________________________________________ n (%) n (%) n (%) n (%) Undergraduate Status Freshman 14 (34%) Sophomore 12 (29%) 11 (23%) Junior 6 (14%) 12 (26%) Senior 3 (7%) 22 (47%) 32 (96%) 5 th Year 2 (4%) 1 (2%) 1 (3%) Missing 4 (8%) Education Courses Completed None 41 (100%) 2 (4%) One 1 (2%) Two 2 (4%) Three 2 (4%) Four 10 (21%) Five or more 29 (63%) 33 (100%) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers ** For this particular table, no information is applicable for classroom teachers. 63 Table 5 Classroom Teaching Experience and Education Level Attained of Final Sample *NoED PreST PostST ET n= 41 n= 46 n= 33 n= 49 ______________________________________________________ n (%) n (%) n (%) n (%) Classroom Teaching Experience None 41 (100%) 3 (6%) Field observation 5 (10%) Mini-teaching (prior to student teaching) 37 (80%) Student teaching 33 (100%) (internship) Classroom teaching experience 44 (100%) Education Level Pre- Bachelors 39 (95%) 46 (100%) 33 (100%) Bachelors 12 (24%) Bachelors + credits 12 (24%) Masters 16 (32%) Masters + 15 credits 1 (2%) Masters + 30 3 (6%) Masters + 45 2 (4%) Specialist 1 (2%) Missing 2 (2%) 2 (4%) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers 64 Reliability Analysis Alpha coefficients of the total test were calculated using the original 159 items. Responses resulting in a negative reliability estimate were not included in the final analysis, leaving 111 of the original 159 responses. The items with negative item-total correlations showed a poor relationship to those items targeting the same domain. The instrument as a whole had a reliability estimate of .89. The homogeneity of the four groups was examined prior to the calculation of internal consistency of each individual competency, therefore these reliability estimates are based on the final survey sample of 158 participants. Individual item analysis identified the means and standard deviations according to group status, provided in Appendix F. Items were removed from the final instrument if they showed a poor relationship with others items in that domain. PEPE competency indicators were then grouped according to the recommendations of the principals and administrators. Each item was identified as aligning with a specific domain. Upon recommendation of the principals, each response could be used for more than one domain (i.e. item 27 could be labeled for domains one and two). Specific PEPE competency indicators are listed in Appendix A. The distribution of responses as recommended by the principals is Appendix E. Six of the eight competencies were preserved having alpha coefficients ranging from .66 (preparation for instruction) to .80 (presentation of organized instruction) with a mean of .63. Tables 6 and 7 present summaries of this information. 65 Table 6 Summary of Reliability Estimates Original and Final Instruments Original Instrument Final Instrument SID3 159 items SID3 111 items _____________________________________________ Competency Items Alpha Items Alpha Preparation for instruction 36 .52 28 .66 Presentation of organized instruction 69 .67 50 .80 Assessment of student performance 65 .63 48 .74 Classroom management 72 .55 47 .76 Positive learning climate 85 .62 65 .76 Communication 47 .44 32 .72 Professional development and leadership Performance of professional responsibilities 10 .05 Averages .43 .74 66 Table 7 Summary of Reliability Analysis _____________________________________________ Competency # of Items Mean SD Alpha Preparation for Instruction 28 21.17 3.05 .6673 Presentation of Organized Instruction 50 39.13 5.51 .8080 Assessment of Student Performance 50 38.68 5.15 .7446 Classroom Management 46 37.78 4.14 .7698 Positive Learning Climate 60 50.14 5.53 .7644 Communication 32 27.11 3.31 .7242 Principal Analysis The current study investigated the construct validity of scores from the Simulation Exercise in Classroom Decision-Making. Establishment of construct validity allows the theory that this test was a measure of generic knowledge about interactive classroom teaching competencies. The attainment of this knowledge, it was proposed, could be learned through educational training programs, experience as classroom teachers and the reading of literature concerning effective teaching. The three research questions were answered both by overall group performance and patterns across the six competencies included. The null hypotheses were: 67 1. There are no differences in the ability to apply teacher effectiveness findings to classroom situations between subjects who have received professional educational training to those without such educational training reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making. 2. There are no differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and those without experience reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making. 3. There are no differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and student teachers reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making. Using SID3, two of the three hypotheses were rejected at the p < .05 level. Overall interaction between group and PEPE domains (F 15,770 = 2.262, p = .006) revealed a difference in the group profiles across the PEPE domains. The profiles were examined according to the three hypotheses stated above. Each will be discussed individually in the sections that follow. A summary of the mixed-ANOVA results is shown in Table 8. Table 9 summarizes the PEPE domain t-scores by group status. 68 Table 8 Summary of Mixed-ANOVA Results _____________________________________________ Source of Variation df MS F Sig. Between Subjects Group 3 558.864 7.512 <.001 Error 154 74.4000 Within Group PEPE 5 3.683 .221 .953 Group x PEPE 15 37.657 2.262 .004 Error 770 16.648 69 Table 9 PEPE Domain T-Scores by Group Status *NoED PreST PostST ET n= 39 n= 44 n= 33 n= 42 __________________________________________________________ Domain M (SD) M (SD) M (SD) M (SD) Preparation for instruction 45.55 (12.40) 51.29 (9.11) 51.53 (7.74) 51.53 (8.77) Presentation of organized instruction 44.01 (12.04) 51.75 (8.81) 52.98 (6.78) 52.27 (7.60) Assessment of student performance 44.50 (12.29) 51.51 (8.41) 52.37 (7.37) 50.94 (8.29) Classroom management 43.70 (13.54) 49.56 (7.93) 54.33 (5.26) 52.89 (8.09) Positive learning climate 43.86 (13.46) 51.46 (7.20) 54.08 (6.57) 51.19 (8.59) Communication 45.24 (13.64) 51.04 (7.67) 53.59 (6.85) 50.83 (8.70) TOTAL 44.46 (12.89) 51.10 (8.18) 53.14 (6.76) 51.60 (8.34) * NoED - Non-education students PreST - Education student prior to student teaching PostST - Education student after student teaching ET - Experienced classroom teachers 70 Educational Training Contrast Table 10 provides a summary of the results for research question one: 1. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects who have received professional educational training to those without such educational training reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? The group of students (n= 44) who had received educational training outperformed those having no educational training (n= 39) on the overall test and on all six of the competencies. This supports the idea that training programs are necessary for the education of teachers, as opposed to placement of individuals without training into a classroom with ?emergency certificates? or waiving licensure requirements entirely. This short-sighted policy may have a detrimental impact on children in those classrooms (Darling-Hammond, 1999). Resolving the teacher shortage in ways that strengthen rather than undermine schools requires assurance that those who enter classrooms are not only knowledgeable, but are also prepared to teach. Overall, education students obtained a mean of 51.10, p < .05; while the non- education students had a mean of 44.46 on the overall test scores. More specifically, these two groups differed significantly on all six PEPE domains (F 1,81 = 9.266, p= .0037). Differences between groups ranged from 5.74 with a small to medium effect size of .46 (preparation of instruction) to a difference of 7.74 with a stronger effect size of .64 (presentation of instruction) (Cohen, 1988, p.285). The largest difference being in presentation of instruction is representative of those not having training in education as 71 not understanding the subtleties of instruction techniques within a classroom. The education students outperformed the non-education students in each of the six domains; 1) preparation for instruction, 2) presentation of organized instruction, 3) assessment of student performance, 4) classroom management, 5) positive learning climate and 6) communication. Figure 1 illustrates the means of both groups for each competency. 72 Educational Training NoED vs. PreST PEPE CommunClimateAssessManagePresentPrep E s tim a te d M a r g in a l M e a n s 54 52 50 48 46 44 42 Group Status NoED PreST Figure 1. Educational Training Group Profiles Non-education Students vs. Educational Students Prep Preparation for Instruction Present Presentation of Organized Instruction Manage Classroom Management Assess Assessment of Students Climate Positive Learning Climate Commun Communication 73 Table 10 Educational Training Contrast Question One *NoED a PreST a n= 39 n= 44 _________________________________________________________ Domain M (SD) M (SD) Difference Effect Size b Preparation for instruction 45.55 (12.4) 51.29 (9.11) 5.74 .46 Presentation of organized instruction 44.01 (12.04) 51.75 (8.81) 7.74 .64 Assessment of student performance 44.50 (12.29) 51.51 (8.41) 7.01 .57 Classroom management 43.70 (13.54) 49.56 (7.93) 5.86 .43 Positive learning climate 43.86 (13.46) 51.46 (7.20) 7.60 .56 Communication 45.24 (13.64) 51.04 (7.67) 5.8 .42 TOTAL 44.46 (12.89) 51.10 (8.18) a NoED - Non-education students PreST - Education student prior to student teaching b Effect size is expressed in SD unit 74 Classroom Experience Contrast The second research question was examining the differences between those with classroom teaching experience and those without experience. Question two was stated: 2. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and those without experience reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? Table 11 reflects the contrast of participants without teaching experience (n= 83) and those with teaching experience (n= 75). The participants without experience obtained a mean of 47.83 while the participants with experience had a mean of 52.34 on the overall test scores. Overall, participants with experience outscored those without experience (F 1,156 = 9.129, p = .003). Differences between groups ranged from 3.41 with a small effect size of .30 (preparation of instruction) to a difference of 6.60 with a stronger effect size of .59 (classroom management) (Cohen, 1988, p.285). However, the group profiles were different as indicated by the interaction between Group and PEPE domain (F 5,780 = 4.263, p = .001). The results indicate that those with teaching experience outscored those without experience on all six domains and on the total score. While experienced participants peaked (scored higher) in the domain of classroom management, the inexperienced scored lowest in this domain. Learning required is not just academic. Teaching is intensely performance- oriented and those in teacher preparation programs need many opportunities to demonstrate their abilities, prior to being given their own classroom. Teachers must be 75 prepared to do more than disseminate information. They must possess a thorough grasp of the knowledge base foundational to teaching practice, a repertoire of instructional strategies, and skills to apply these to the education of individual students; understand and use methods of inquiry and research findings in making professional decisions. 76 Classroom Experience Inexperienced vs. Experienced PEPE CommunClimateAssessManagePresentPrep E s t i m at ed M a r g i n al M e a n s 54 52 50 48 46 Group Status Experience Inexperienced Figure 2. Classroom Experience Group Profiles Inexperienced vs. Experienced Prep Preparation for Instruction Present Presentation of Organized Instruction Manage Classroom Management Assess Assessment of Students Climate Positive Learning Climate Commun Communication 77 Table 11 Classroom Experience Contrast Question 2 *No Experience a Experience a n= 83 n= 75 ___________________________________________________________ Domain M (SD) M (SD) Difference Effect Size b Preparation for instruction 48.34 (11.29) 51.76 (8.10) 3.41 .30 Presentation of organized instruction 47.77 (11.47) 52.58 (7.21) 4.80 .41 Assessment of student performance 47.99 (11.43) 52.07 (7.79) 4.08 .35 Classroom management 46.92 (11.18) 53.52 (6.98) 6.60 .59 Positive learning climate 47.80 (11.17) 52.46 (7.85) 4.66 .41 Communication 48.19 (11.20) 52.05 (8.01) 3.86 .34 TOTAL 47. 83 (11.29) 52. 34 (7.65) a No Experience - NoEd + PreSt Experience - PostST + ET b Effect size is expressed in SD unit 78 Teaching Experience Contrast The third and final research question examined the amount of teaching experience and scores using this instrument. The final research question was: 3. Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and student teachers reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? Student teachers (n= 33) outscored the teachers (n= 42) on five of the six competency domains with a range of .71 with a small effect size of .10 (presentation of instruction) to 2.89 with a small to medium effect size of .43 (positive learning climate) (Cohen, 1988, p.285). Whereas, student teachers outperformed experienced teachers in five of the six PEPE domains follow-up analysis failed to yield statistically significant differences between student teachers and experienced teachers (F 1,73 = .902, p= .345). Differences between groups ranged from 5.74 with a small to medium effect size of .46 (preparation of instruction) to a difference of 7.74 with a stronger effect size of .64 (presentation of instruction) (Cohen, 1988, p.285). Group means of both the student teachers and teachers were exactly the same (51.53) in the preparation of instruction domain. Student teachers also outscored teachers in overall group test scores. This information is plotted on Figure 3 and summarized on Table 12. In preliminary findings, Noell (2004) identified one teacher preparation program in Louisiana whose new teachers taught children whose growth in learning in mathematics surpassed the growth of learning in mathematics of children taught by experienced teachers. He states ?once valid and reliable data are available, universities will use the 79 data to identify best practices and share information across campuses to enhance the quality of all teacher preparation programs.? Teaching Experience PostST vs. ET PEPE CommunClimateAssessManagePresentPrep E s t i m at ed M a r g i n al M e a n s 55 54 53 52 51 50 Group Status PostST ET Figure 3. Teaching Experience Group Profiles Student Teachers vs. Experienced Teachers Prep Preparation for Instruction Present Presentation of Organized Instruction Manage Classroom Management Assess Assessment of Students Climate Positive Learning Climate Commun Communication 80 Table 12 Teaching Experience Question 3 PostST a ET a n= 33 n= 42 _______________________________________________________________ Domain M (SD) M (SD) Difference Effect Size b Preparation for instruction 51.53 (7.74) 51.53 (8.77) 0 Presentation of organized instruction 52.98 (6.78) 52.27 (7.60) -0.71 .10 Assessment of student performance 52.37 (7.37) 50.94 (8.29) -1.43 .19 Classroom management 54.33 (5.26) 52.89 (8.09) -1.44 .27 Positive learning climate 54.08 (6.57) 51.19 (8.59) -2.89 .43 Communication 53.59 (6.85) 50.83 (8.70) -2.56 .37 TOTAL 53.14 (6.76) 51.60 (8.34) a Post ST - Education student after student teaching Teachers - Experienced classroom teachers b Effect size is expressed in SD units 81 Follow-up Survey Upon completion of exercise the teachers were asked to answer questions about the simulation exercise. The follow-up document is shown in Appendix G. Two open-ended questions allowed the participants to explain his or her perceptions of the strengths and weaknesses of the exercise. Three main themes emerged as strengths of this study. Thirteen of the 32 (40%) teachers who completed the follow-up identified the biggest strength as the variety of realistic situations and responses. Personal statements such as, ?that has happened in my classroom? verified the belief that the situations and responses were relevant to classrooms today. Realization of how quickly decisions are made in the classroom and reflection of how situations are handled emerged as two other strengths in this particular exercise. The length of time to complete the exercise was listed as the biggest weakness (31%). Two respondents requested complete descriptions with specific factors of each classroom, as well as background information on the students involved. Using a scale of 1 ? 5 with 1 being not very realistic and 5 being very realistic, respondents answered the question asking how the exercise resembled a normal classroom in 2005 with a 4.5 average. A second question asked if the research resembled normal classroom time of decision-making receiving an average score of 4. Seventeen of the 32 respondents believed the exercise was somewhat better than a paper-and-pencil test, seven rated it as the same and six rated it as much better than a paper-and-pencil test. In response to how the exercise could be best used with the opportunity to choose more than one answer, respondents answered with the following: 82 23 believe it could be a good mentoring tool for beginning teachers, 21 stated it could be used in professional development, 15 stated it could benefit teacher preparatory colleges to assess graduates and 11 believed it could be used as a pre-interview technique for hiring new teachers. Summary of Findings The attainment of knowledge used in elementary classrooms, it was proposed, could be learned through educational training programs, experience as classroom teachers and the reading of literature concerning effective teaching. The three research questions were answered both by overall group performance and patterns across the six competencies included. Educational students in the current study exhibited knowledge of pedagogy acquired in educational training as they outperformed non-education students in each competency. Classroom experience contributes to the ability to apply the findings of effective teaching research to results by performance on the exercise. Participants with teaching experience (n= 75) outscored those without teaching experience (n= 83) on all six domains. Interestingly, the overall performance of experienced teachers and student teachers did not show a significant difference in the current study. 83 V. SUMMARY, CONCLUSIONS, DISCUSSION OF FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Summary and Conclusions The focus of this study aimed to verify and extend the findings obtained during the SID2 test. This study was created as an effort to more realistically reflect the decision-making process within a school classroom as opposed to the traditionally administered pencil-and-paper tests. Test Refinement To fulfill the purpose of this study, the SID2 test used in the initial pilot study was refined for the current study, SID3. Items from the initial study were eliminated if they dealt specifically with secondary education. A focus group of principals and teachers met to review the items for the exercise. Surveys were mailed to principals and administrators of schools in Alabama. After results were tabulated new items were selected for the revision. A PowerPoint presentation with audio background replaced the original videotape for administration of the exercise. 84 Sample The overall sample contained 315 participants. Both secondary and elementary samples were to be analyzed, however this specific exercise was designed primarily for use in making decisions in an elementary classroom. Therefore, no secondary teachers or education students training to teach in a secondary school were used in the final sample. The student teaching group (PostST) did not contain any males. In addition females outscored males in each aspect therefore only female participants were included in the final sample of the four groups. Reliabilities Alpha coefficients of the total test were calculated using the original 159 items. Responses resulting in a negative reliability estimate were not included in the final analysis, leaving 111 of the original 159 responses. The instrument as a whole had a reliability estimate of .89. PEPE competency indicators were then grouped according to the recommendations of the principals and administrators with each item identified as aligning with a specific domain. Each response could be used for more than one domain (i.e. item 98 could be labeled for domains three and four). Six of the eight competencies were preserved having alpha coefficients ranging from .66 (preparation for instruction) to .80 (presentation of organized instruction) with a mean of .63. 85 Principal Analyses It was proposed that experiences as a classroom teacher, the reading and use of information presented in literature on effective teaching and teacher training programs at the university level would build a generic knowledge base. It was hypothesized that: 1) the scores of participants with educational training would differ from those of participants without professional educational training, 2) the scores of subjects with classroom teaching experience would differ from scores of those without experience and 3) the scores of teachers would differ from student teachers. Discussion of Findings First Research Question Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects who have received professional educational training to those without such educational training reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? Educational students in the current study exhibited knowledge of pedagogy acquired in educational training as they outperformed non-education students in each competency. The results of the present study corroborated Shannon?s 1994 study as students with training outperformed those without training. Doebler, Roberson, and Ponder (1999) designed a study to examine the use of case study analysis in an effort to improve decision-making abilities of preservice teachers. Developed as an indicator of principles related to teaching, it sampled students at four distinct points of education, in an effort to validate the effectiveness of the teacher 86 education program of the school?s teacher education program. Students? responses to the case studies were reviewed to determine the students? decision-making ability. The study concluded that students within its teacher education program were benefiting from such case studies and later responses were specific, logical and seemed to be derived from a combination of coursework and experience. Western Oregon University (2003) developed a Teacher Effectiveness Project (TEP) designed to detect any differences that might exist in the performance of participating teachers as a consequence of their preparation programs. The program was based on the hypothesis that teachers prepared in programs that were consistent with Oregon?s design for K ? 12 schooling would be more ready to meet the state standards when they entered the work force. The Western Oregon University (WOU) website declares they have been recognized by the American Association of Colleges for Teacher Education as a national model for work on their Teacher Effectiveness research project. The WOU website stated that they are currently working on statistical analyses connecting pre-service training with effective teaching. In the current study those students having received instruction about teaching were able to identify responses that prepared for instruction and presentation of the organized instruction. Sensitivities to assessment of student performance by monitoring, measuring, providing feedback and use of assessment results were shown by those same students. The abilities to manage class time by directing or redirecting students, being organized, prepared and flexible to change course during an ongoing activity were shown to be response alternatives chosen most often by the students who had participated in 87 educational training. Creating a positive learning climate, involving students in class interactions and the communications of high expectations were indicated by these same students. The ability to create an environment conducive to learning by the physical arrangement of the room was further shown by the students with training. Students having training in education outscored students without training in items identifying communication skills. While other studies used a variety of methods, conclusions were the same, the educational training students received enabled them to more often correctly identify correct answers when compared with non-education students. The educational students in the current study were able to choose the indicator competencies as defined in the PEPE competencies more often than those with no educational training. Second Research Question Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and those without experience reflected in scores received on the revised Simulation Exercise in Classroom Decision-Making? Classroom experience contributes to the ability to apply the findings of effective teaching research to results by performance on the exercise. Participants with teaching experience (n= 44) outscored those without teaching experience (n= 83) on all six domains. Differences in score ranged from 3.11(preparation of instruction) to 6.88 (classroom management). 88 Ability to manage the transition time between classroom subjects, maintain order, not allowing the class to detour from the topic and keeping the attention of the students to the task at hand are all indicators of good classroom management. Selecting choices that demonstrated positive reinforcement and giving immediate feedback were areas shown by those with experience. The scores also reflected the knowledge that an effective teacher must judge which classroom situations to ignore and which need immediate attention. The group with experience responded more consistently to classroom situations more consistent with research findings when asked to deal with situations that required them to create a positive learning environment. Participants with experience monitored a student?s progress and changed the teaching style or teaching plan as needed. Those with experience also exhibited qualities needed to present organized instruction with materials readily available and the ability to maintain a physical environment conducive to learning. Students under the instruction of persons with experience in the classroom are held accountable for classroom behavior, assignments being completed on time, remaining on task during class time. Participants with experience more often involved students in the lessons, exhibited sensitivity and avoided classroom situations that would make the student feel inferior or embarrassed. CaseNet as used by The Curry School of Education in Virginia evaluates teachers and pre-service teachers alike. The current study echoed the review of over 2000 CaseNet results (2001) finding that teachers with classroom experience are better than those without experience. Experienced teachers are able to accommodate the individual needs 89 of students, show more personal control of planning, pacing the flow of the lesson, managing misbehaviors, use of wait time, show more personal control in their planning, don?t feel they must assign all activities, and spend equal time with each group or individual. Third Research Question Are differences in the ability to apply teacher effectiveness findings to classroom situations between subjects with classroom teaching experience and student teachers reflected in scores received on the revised Simulation Exercise in Classroom Decision- Making? The overall performance of experienced teachers and student teachers did not show a significant difference in the current study. The same mean (51.53) was earned by both student teachers (n= 33) and experienced teachers (n= 42) in the PEPE domain of preparation for instruction. Preparation for instruction is evident when a teacher writes lesson plans, sets short and long term goals, identifies creative activities and plans instruction consistent with developmental levels of students. Student teachers scored higher on the five remaining domains, yet the differences in scores between student teachers and teachers were very small. The smallest difference between the two groups was in presentation of organized instruction (PEPE domain two) where student teachers had a mean of 52.98 and teachers had a mean of 52.27. Presentation of organized instruction encompasses giving clear directions, presenting content to fit objectives, using up-to-date information and providing examples of how to do a task. 90 The domain of preparing a positive learning climate would include encouraging active participation, alternative responses and expectations of quality performances. Teachers had a mean of 51.19 and student teachers a mean of 54.08 in this area. While only preliminary, as stated previously, Noell?s (2004) research in Louisiana is looking into a teacher?s classroom experience and teacher preparation programs as factors on student achievement scores. Discussion Consistent with earlier research using a simulation (SID2) Shannon?s study in 1990 those trained in education scored higher than those without education training and those with classroom teaching experience scored higher than those without experience. Also consistent with the previous study, scores between student teachers and experienced teachers were no different. Classroom teaching experience and knowledge gained from pedagogical training were shown as being useful in all six of the domains. Equipping preservice teachers with the knowledge and skills needed to be an effective teacher is the objective of teacher preparation programs. Educational courses supply teaching strategies based upon current research. Instructors that encourage students to read and apply research findings in professional journals are further aiding their students. With the realization that teachers have a long-lasting influence on their students, directly affect how students learn, what they learn, how much they learn and the ways they interact with each other and the world around them, we must strive to understand what teachers can do within their classrooms to promote positive results in the lives of 91 their students. Using that understanding, it is the job of teacher preparation programs to enlighten their students about their future duties as a classroom teacher. Blair (2000) and Darling-Hammond (2001) summarize the important outcomes related to educational coursework in teacher preparation: 1) a teacher?s formal pedagogical preparation has been shown to have a positive effect on student achievement, 2) the more methods courses in a teacher?s preparation program, the more likely the teacher is to emphasize conceptual understanding and hands-on learning techniques and 3) teachers prepared in schools of education demonstrate stronger classroom management skills and can better relate content to the needs of students. A teacher education program that is closely tied to school districts allows teacher preparation programs to form partnerships with and assist school districts as they restructure to accommodate their school improvement plan and the changing nature of teachers' work. These partnerships provide settings where candidates along with school and university faculty may work collaboratively to improve their teaching. Suggestions for Further Study Based on the results of this research study suggestions for refinement of the current study and the development of further studies will be made. While the suggestions are numbered, the position each holds does not suggest the degree of importance nor does it represent the priority in which they should be acted upon. 1. A future study should be conducted investigating the results of a more diverse population. 92 2. Redevelopment of the current study should be done to insure inclusion of all domains of the Alabama Professional Education Personnel Evaluation (PEPE). The current study addressed only six of the eight domains. 3. The current format of the exercise should be further investigated with development of simulation exercises streamlined specifically for teachers working in classrooms at different levels. For example, the early childhood exercise would be refined with specific situations found within an early childhood classroom. Other exercises could be targeted for elementary teachers, middle school or junior high teachers, high school teachers and special education teachers. 4. Similar to the redevelopment of the exercise expressed in the simulations suggested above, further simulations could be created for specific subjects. Targeting subjects taught within schools such as: Mathematics, English, History, Sciences, Physical Education, Music and others exercises could be developed allowing for comparisons of teachers within each field. 5. Redevelopment of the current study should be done to reflect the INTASC standards in conjunction with the PEPE scores. The current study relied primarily on the indicators of effective teaching as defined within the PEPE program. 6. Redevelopment of the instrument to be administered on a computer, would allow participants to complete the survey at their convenience. This would also allow the answers to be scored automatically and allow the instrument to include immediate feedback shown as scores or ratings within each domain. 93 Participants receiving low scores in a specific domain could immediately receive suggestions to improve their teacher and decision-making effectiveness. Further, the instrument could be written to allow an added section with open-ended answers to scenarios, giving the participant an opportunity to write in her or his own responses to a given set of scenarios. 7. A future study should be conducted to investigate the extent to which SID3 scores predict and correlate with future PEPE scores of teachers. Many of the participants of the current study will be employed as teachers in the state of Alabama and will be evaluated using the PEPE tool. 8. A future study should include an efficacy instrument to investigate the extent to which SID3 scores predict and correlate with a participant?s efficacy beliefs. Bandura and Pajares as cited in Poulou & Norwich, 2002) believe a teacher?s self-efficacy affects the decisions he or she makes within the classroom. Suggestions for teachers, administrators and teacher preparatory institutions follow. 1. As stated by Wiggins (1993 & 1990), authentic assessment results are meaningful and useful for improving instruction. Using SID3 as an assessment tool for teachers already within the classroom would allow administrators to develop a plan to improve each school collectively. Individual strengths and weaknesses could be identified for each teacher, making sure to include strategies for improvement. 2. Assessments need to check capacities in the real-world and be engaging, not passive (Darling-Hammond, 1994). Institutions preparing future teachers 94 could use SID3 to analyze how well they are thoroughly equipping each student (teacher) for life in a real classroom. 3. Assuming all teachers want to improve in any way possible, in the future using SID3 online, individuals could take the test at specified intervals, as a self-check to evaluate how well he or she is progressing. 4. The addition of results from SID3 to other indicators, as a pre-interview technique for teachers just beginning within the specific school or school district administrators would be able to receive added information about the qualifications of each individual. 95 REFERENCES Alabama Department of Education. (1999). Report on 1999 Professional Education Personnel Evaluation (PEPE) scores for novice teachers. Retrieved on 4/22/04 from http://www.alabamapepe.com/about.htm. Alabama Department of Education. (2003) About PEPE. Retrieved 4/22/04 from http://www.alabamapepe.com/about.htm. Alabama Department of Education. (2003). Report on the spring 2000 survey of first- and second-year teachers: Subject areas in which Alabama teachers obtain assistance, and the professional sources from which they receive it. Retrieved 4/22/04 from http://www.alsde.edu. Alabama Department of Education. (2003b) Telling the whole truth (or not) about highly qualified teachers. Retrieved 4/22/04 from http://www.alsde.edu. Airasian, P., & Jones, A. M. (1993). The teacher as applied measurer: Realities of classroom measurement and assessment. Applied Measurement in Education, 6(3), 241-254. Ambach, G. (1996). Standards for teachers [Electronic version]. Phi Delta Kappan, 78(3). Baron, J. (1998). Using learner-centered assessment on a large scale. In McCombs (Ed.), How students learn: Reforming schools through learner-centered education (pp. 211 - 240). Washington, DC: American Psychological Association. 96 Blair, J. (2000). ETS study links effective teaching methods to test-score gains. Education Week, 20 (8), 24. Brown, J., Collins, A., & Duguid, P. (1988). Situated cognition and the culture of learning. (No. BBN-R-6886). Palo Alto, CA: Palo Alto Research Center. Brualdi, A. (1998). Implementing performance assessment in the classroom. Washington, DC:ERIC Clearinghouse on Assessment and Evaluation. (ERIC Documentation Reproduction Service No. ED423312). Burn, K., Hagger, H., Mutton, T., & Everton, T. (2000). Beyond concerns with self: The sophisticated thinking of beginning student teachers. Journal of Education for Teaching, 26(3), 259 - 279. Carey, L. M. (2001). Measuring and evaluating school learning. Allyn and Bacon: Boston. Checkley, K. The first seven?and the eighth: Interview with educator Howard Gardner. Educational Leadership, 55 (1), 8 ? 14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2 nd ed). Hillsdale, NJ: Erlbaum Associates. Council of Chief State School Officers. (1999). Information on INTASC. Retrieved 4/22/04, from http://www.ccss.org. Council of Chief State School Officers. (2000). Model standards for beginning teacher licensing and development. Retrieved 4/22/04, from http://www.ccss.org. Cruickshank, D. R., & Metcalf, K. K. (1993). Improving preservice teacher assessment through on-campus laboratory experiences. Theory Into Practice, 32(2), 86 - 92. 97 Darling-Hammond, L. (1994). Setting standards for students: The case for authentic assessment. The Educational Forum, 59, 14 - 21. Darling-Hammond, L. (1999). Solving the dilemmas of teacher supply, demand and standards: how we can ensure a competent, caring, and qualified teacher for every child. New York: National Commission on Teaching and America?s Future. Darling-Hammond, L. (2001). The challenge of staffing our schools. Educational Leadership, 58 (8), 12 ? 17. Darling-Hammond, L. (2002). Research & Rhetoric on Teacher Certification: A response to ?Teacher Certification Reconsidered.? Educational Policy Archives, 10 (36). Daugherty, S., DeAngelis, K., & Rossi, R. (1997). Credentials and Tests in Teacher Hiring: What do districts require? Retrieved 4/22/04, from http://www.ed.gov/NCES . Doebler, L., Roberson, T., & Ponder, C. (1998). Preservice teacher case study responses: A prelimary attempt to describe program impact. Education. 119 (2) 349 ? 359. Eggers, R. M. & Jones, C.M. (1998) Practical considerations for conducting Delphi studies: the oracle enters a new age. Educational Research Quarterly, 23 (1). Educational Testing Services. (2003). About PRAXIS. Retrieved on 5/3/04 from http://www.ets.org. Entwistle, N. J. & Tait, H. (1990). Approaches to learning, evaluations of teaching and preferences for contrasting academic environments. Higher Education, 19, 169 ? 194. 98 Ellett, C., & Teddle, C. (2003). Teacher evaluation, teacher effectiveness and school effectiveness: Perspectives from the USA. Journal of Personnel Evaluation in Education, 17(1), 101 - 128. Feiman-Nemser, S., & Parker, M. (1990). Making subject matter part of the conversation in learning to teach. Journal of Teacher Education, 41(3), 32 - 43. Ferguson, P., & Womack, S. (1993). The impact of subject matter and education coursework on teaching performance. Journal of Teacher Education, 44(1), 53 - 61. Giovannelli, M. (2003). Relationship between reflective disposition toward teaching and effective teaching. Journal of Educational Research, 96(5), 293 - 310. Hawkins, J., Frederiksen, J., Collins, A., Bennett, D., & Collins, E. (1993). Assessment and technology: New ways to assess student learning. Communications of the ACM, 36(5), 74 - 77. Interstate New Teacher Assessment and Support Consortium (2004). Standards for new teachers. Retrieved 6/16/04 from http://www.ccsso.org/intasc.html. Kanstoroom, M. & Finn, C. (1999). Better Teachers, Better Schools. Education Leaders Council, Washington, DC: Thomas B. Fordham Foundation, Washington, DC. Kaplan, L. S., & Owings, W. A. (2003). No child left behind: The politics of teacher quality. Phi Delta Kappan, 687 - 692. Kuligowski, B., Holdzkom, D., & French, R. (1993). Teachers performance evaluation in the southeastern states: Forms and functions. Journal of Personnel Evaluation in Education, 6, 335 - 358. 99 Lave. J. & Wenger, E. (1990). Situated learning: Legitimate periperal participation. Cambridge, UK: Cambridge University Press. Louisiana Board of Regents (2004). Teacher education initiatives; value-added teacher preparation program assessment model. Retrieved 4/19/05 from http://asa.regents.state.la.usa/TE/value_added_model. Lowman, J. & Mathie, V. A. (1993). What should graduate teaching assistants know about teaching? Teaching of Psychology, 20 (2) 84 ? 89. Ludlow, L. (2001). Teacher test accountability: From Alabama to Massachusetts. Education Policy Analysis Archives, 9(6). Medley, D. M. (1982). Teacher effectiveness. In H. E. Mitzel (Ed.), Encyclopedia of Educational Research (pp. 1894 - 1903): MacMillan. Moorcroft, T., Desmarais, K., Hogan, K., & Berkowitz, A. (2000). The national environmental policy act (book review). The Journal of Environmental Education (31), 3, 55 ? 58. National Association for Early Childhood Teacher Educators (2005). Alternative preparation for licensure, a policy statement prepared by the American Association of Colleges for Teacher Education. Retrieved 4/19/05 from http://www.naecte.org/aacte.htm. National Council for Accreditation of Teacher Education. (2004). Summary data on teacher effectiveness, teacher quality, and teacher qualifications. Retrieved 3/27/04 from http://www.ncate.org/resources/factsheettq.htm . 100 Oakes, T. (1999). A guide to organizations involved with licensing and certification of teachers and accreditation of teacher education programs. ERIC Digest. (ERIC Documentation Reproduction Service No. ED19991201). Retrieved 6/16/04. Poulou, M., & Norwich, B. (2002). Cognitive, emotional and behavioural responses to students with emotional and behavioural difficulties: A model of decision- making. British Educational Research Journal, 28(1), 111 - 138. Raths, J. (1999). A consumer's guide to teacher standards [Electronic version]. Phi Delta Kappan, 81(2). Resnick, L. (1987). Learning in school and out. Educational Researcher, 16, 13 - 20. Schalock, D., Schalock, M., & Myton, D. (1998). Effectiveness--along with quality-- should be the focus [Electronic version]. Phi Delta Kappan, 79(6). Shannon, D. M. (1990). The effects of teacher training and experience in a classroom decision-making simulation. Unpublished doctoral dissertation, University of Virginia, Charlottesville. Shannon, D. M. (1994). The development of preservice teacher knowledge. Professional Educator, 17(1), 31 - 39. Shannon, D. M. & Boll, M. (1996). Assessment of preservice teachers using alternative assessment methods. Journal of Personnel Evaluation in Education, 10, 117 - 135. Shannon, D. M., Swetman, D. L. & Barry, N. H. (1996). Effective teaching behaviors for beginning teachers: a multiple perspective. Research in the Schools, 3 (1), 1 ? 11. 101 Shannon, D. M., Medley, D. M. & Hays, L. (1993) Assessing Teachers? Functional Professional Knowledge. Journal of Personnel Evaluation in Education, 7, 7 ? 20. Shulman, L. (1986). A union of insufficiencies: Strategies for teacher assessment in a period of educational reform. Educational Leadership, 36 - 41. Shulman, L. (1998). Theory, practice, and the education of professionals. The Elementary School Journal, 98(5), 511 - 526. Shulman, L. (2002). Making differences a table of learning. Change, 37- 44. Sternberg, R. (1998). Teaching and assessing for successful intelligence. School Administrator, 55(1), 26 ? 28. Strang, H. (1996). The teaching decisions simulation: An interactive vehicle for mapping teaching decisions. Journal of Technology and Teacher Education, 4 (2), 133 ? 143. Strang, H. (1998). Feedback possibilities in a lesson-planning simulation. Society for Information Technology & Teacher Education, Association for the Advancement of Computing in Education, Charlottesville, VA 958 ? 960. Strang, H., Badt, K. & Kauffman, J. (1987). Microcomputer-based simulations for training fundamental teaching skills. Journal of Teacher Education, 38, 20 ? 26. Strang, H. & Clark, R. (2001). The LPII simulation: A lesson planning tool for preservice teachers. Society for Information Technology & Teacher Educations, 2002, Association for the Advancement of Computing in Education, Norfolk, VA. 2563 ? 2566. 102 Strang, H., & Loper, A. (1983). Microcomputer-based simulation in training elementary teachers. Educational Technology, 23, 30 -31. Stronge, J. H., & Hindman, J. L. (2003). Hiring the best teachers. Educational Leadership, 48-52. Sudzina, M. R. (February 12 -16, 1994). Mentoring and collaborating with cases: Developing the skills and resources to compete in a national case competition. Paper presented at the Annual Meeting of the Association of Teacher Educators, Atlanta, GA. Travis, J. (1996). Meaningful assessment. The Clearing House, 69(5), 308 - 313. Uhlenbeck, A., Verloop, N., & Beijard, D. (2002). Requirements of an assessment procedure for beginning teachers: Implications from recent theories on teaching and assessment. Teachers College Record, 104(2), 242 - 272. United States Department of Education. (1997).Credentials and tests in teacher hiring:What do districts require? National Center for Education Statistics. (Issue Brief No. IB-3-97). Retrieved 4/4/04 from http://www.de.nces.gov . United States Department of Education. (1997). What criteria are used in considering teacher applicants? National Center for Education Statistics. (No. IB-6-96). Retrieved 4/4/04 from http://www.de.nces.gov . United States Department of Education. (2003). No child left behind; highly qualified teachers. Retrieved 4/4/04 from http://www.de.gov. Valencia, S. (1997). Authentic classroom assessment of early reading: Alternatives to standardized tests. Preventing School Failure, 41(2), 63. 103 Vygotsky, L. S. (1998). Development of thinking and formation of concepts in the adolescent. In R. W. Rieber (Ed.). The collected works of L. S. Vygotsky (pp.29 ? 81). New York: Plenum. (Orginal work published 1928 ? 1931). Weiss, E. M., & Weiss, S. G. (1998). New Directions in Teacher Evaluation (No. ERIC Number ED429052). Washington, DC: Office of Educational Research and Improvement. Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70(9), 703 - 713. Wiggins, G. (1990). The case for authentic assessment. ERIC Document Reproduction Service No. ED328611. Wiggins, G. (1993). Creating tests worth taking. Educational Leadership, 49 (8), 26 ? 34. Wilson, B., & Wood, J. (1996). Teacher evaluation: A national dilemma. Journal of Personnel Evaluation in Education, 10, 75-82. Wilson, S., Shulman, L., & Richert, A. (1988). 150 Different ways of knowing: Representations of knowledge in teaching. 104-124. Wolf, K., Lichtenstein, G., Bartlett, E., & Hartman, D. (1996). Professional development and teaching portfolios: The Douglas county outstanding teacher program. Journal of Personnel Evaluation in Education, 10, 279 - 286. Worthen, B. (1993). Critical issues that will determine the future of alternative assessment. Phi Delta Kappan, 74 (6), 444 ? 454. Wray, D., Medwell, J., Fox, R., & Poulson, L. (2000). The teaching practices of effective teachers of literacy. Educational Review, 52 (1), 75 ? 85. 104 Appendix A Alabama Professional Education Personnel Evaluation Program Teacher Competencies 105 Appendix A ALABAMA PROFESSIONAL EDUCATION PERSONNEL EVALUATION PROGRAM TEACHER COMPETENCIES 1.0 PREPARATION FOR INSTRUCTION Effective teachers plan carefully for instruction. That planning includes selection of both goals and objectives to be accomplished, identification of creative, innovative activities to accomplish the objectives and accommodate differences in learners, and preparation of instructional resources to support learning. 1.1 Selects/States Long-Range Goals and Short-Term Measurable Objectives - selects long-range goals from state and/or local curriculum guides and sources - selects/states and sequences short-term, measurable objectives in accordance with learner needs and program goals - selects objectives from approved state and/or school system sources 1.2 Identifies Various Instructional Strategies - integrates knowledge and skills across curriculum areas - plans creative and innovative activities appropriate to objectives, including the use of technology - identifies teaching-learning activities to accommodate individual differences/exceptionalities among learners (e.g., achievement and ability levels, interests, learning styles) - plans instruction consistent with developmental level of students (physical, social, emotional, and cognitive 1.3 Prepares Instructional Resources for Use - selects and uses resources that are directly related to the purpose(s) and objectives of the lesson and the skills/concepts to be mastered - selects and uses resources that further clarify the lesson (remediation, reinforcement, or enrichment) - selects and uses resources appropriate to student differences (ability, achievement, interests, learning styles) - selects and uses technology/media, bulletin boards, models, realia, and/or displays - sequences materials in appropriate order and locates them for distribution when needed - makes sure that equipment is in working order and ready for use when needed 106 2.0 PRESENTATION OF ORGANIZED INSTRUCTION Effective teachers organize instruction. They introduce students to lessons, give clear directions, develop lesson content systematically, and provide appropriate summarizations of knowledge and student practice of skills. 2.1 Orients Student to the Lesson - secures student attention - states purposes of lesson and its objectives - identifies contents/skills to be mastered - relates current lesson content to previous and future lesson content 2.2 Gives Clear Directions - gives concise, but sufficient, directions - presents directions in logical sequence - presents directions (written and oral) in easy to follow form - provides examples of how to do task - identifies steps in the task - receives minimum number of procedural questions 2.3 Develops the Lesson - explains concepts, terms, vocabulary, principles - presents content to fit objectives - presents content in logical pattern and sequence - provides examples or illustrations from life experiences and current events - questions effectively - provides smooth transitions from one activity to another - relates content to other subject areas - uses technology when appropriate 2.4 Provides Practice and Summarization - provides guided practice when appropriate - assigns independent practice (in-school, at-home activities) when appropriate - provides review at appropriate points 2.5 Demonstrates Knowledge of Subject Matter and Pedagogy - uses accurate, up-to-date information - establishes relationships among facts, concepts, principles, skills - emphasizes main ideas, central themes - identifies/questions misconceptions, and faulty logic - responds accurately to student questions - uses multiple representations and explanations 107 3.0 ASSESSMENT OF STUDENT PERFORMANCE Effective teachers skillfully evaluate student performance and use assessment results to improve their instruction. They are expert in measuring student progress, providing feedback about performance to students and reporting student progress to others who need to know. 3.1 Monitors Student Performance - checks student understanding, processes, products - solicits questions - requests student demonstration of task/skill - asks questions requiring comprehension, application, evaluation of concept/skill 3.2 Measures Student Progress Systematically - assesses level of performance and progress regularly - uses variety of appropriate assessment methods and instruments - uses assessment strategies to involve students in self-assessment activities 3.3 Provides Feedback About Student Performance - acknowledges participation and response - affirms correct responses - praises specific behaviors and accomplishments - provides specific, corrective statements to inappropriate responses - makes specific recommendations for improvement 3.4 Uses Assessment Results - uses assessment data to determine achievement of objectives - uses assessment data to modify objectives, content, instructional strategies - clarifies/elaborates direction and explanations - reteaches when necessary using alternative strategies, activities and/or materials - adjusts pacing of instruction and activities for individuals/groups when necessary - uses assessment data in reporting progress and accomplishment to students, parents/guardians, professional staff by multiple means (report cards, progress reports, notes, conferences, etc.) 4.0 CLASSROOM MANAGEMENT Effective teachers manage class time and student behavior. They maximize instructional time and minimize disruptions to instruction. 4.1 Manages Class Time - begins instruction promptly - completes non-instructional duties with minimal loss of instruction time - disseminates materials and supplies and uses equipment with minimal loss of instructional time - discourages or redirects student digressions 108 - follows planned sequence of activities with minimum teacher digressions - minimizes time students spend waiting with nothing to do - makes effective use of time - returns students to task quickly after unavoidable interruptions 4.2 Manages Student Behavior - establishes classroom rules and procedures - requires and monitors student adherence to rules and procedures - anticipates conditions which can lead to inappropriate student behavior and uses intervention strategies - uses verbal and nonverbal (proximity, eye contact, etc.) skills to control student conduct - stops inappropriate behavior using reasonable sanctions - rewards (verbally and nonverbally) appropriate student conduct 5.0 POSITIVE LEARNING CLIMATE Effective teachers establish positive learning climates by involving students in classroom interactions, constantly communicating high expectations for student performance, expressing positive affect, and minimizing negative messages. 5.1 Involves Students in Interaction - encourages active participation - ensures equitable participation - establishes and maintains effective positive rapport with/between/among students - elicits responses - encourages students to help each other and share ideas - accepts and uses student ideas, questions, and responses - seeks alternative responses - refers student ideas and questions to other students - engages students in generating knowledge and testing hypotheses - varies roles in instructional process (facilitator, coach, audience) in relation to content and purposes of instructional needs of students 5.2 Communicates High Expectations - establishes and maintains timelines for task completions - establishes and maintains standards for consistency, correctness, neatness, and form - holds students accountable for assigned activities - encourages students to deliver quality performance and products - indicates confidence in students' ability to learn 5.3 Expresses Positive Affect/Minimizes Negative Affect - expresses enthusiasm verbally and nonverbally - uses positive verbal language - uses positive nonverbal cues 109 - demonstrates respect and consideration for all students - accepts student responses without ridicule - avoids use of sarcasm/derogatory statements (verbal and nonverbal) - avoids personal criticism of students - avoids emotional outbursts 5.4 Maintains Physical Environment Conducive to Learning within Limitations of Facilities Provided - arranges furniture and equipment to facilitate movement and learning - uses assigned facilities to accommodate different types of activities - creates an attractive physical environment 6.0 COMMUNICATION Effective teachers are effective communicators. They speak and write clearly, coherently, and correctly. 6.1 Speaks Clearly, Correctly, and Coherently - uses standard speech - pronounces words correctly - adjusts rate of speaking when needed /requested - adjusts pitch for emphasis - organizes presentations - uses vocabulary and style appropriate to level of students - speaks fluently 6.2 Writes Clearly, Correctly, and Coherently - spells words correctly - uses correct grammar and mechanics - writes legibly - uses vocabulary and style appropriate to level of audience - organizes written information 7.0 PROFESSIONAL DEVELOPMENT AND LEADERSHIP Effective teachers seek to grow in professional knowledge and skills. Further, they seek ways to become leaders in improving their school, school system and profession. 7.1 Improves Professional Knowledge and Skills - participates in professional organizations - participates in school system and state professional development programs and/or attends state, regional, and national conferences - participates in a professional development program to improve job performance - takes formal course work or obtains advanced degree(s)/certification - uses ideas from books, professional journals, and professional organizations to improve teaching 110 7.2 Takes A Leadership Role in Improving Education - provides leadership in identifying and resolving issues and problems facing education (local, state, regional, national) - provides leadership in establishing and/or achieving school/school system goals - initiates activities and projects in the school/school system - conducts workshops/training sessions - shares ideas, materials, and resources with peers and others - participates in shared decision-making in the school 8.0 PERFORMANCE OF PROFESSIONAL RESPONSIBILITIES Effective teachers perform their responsibilities in an efficient and timely manner. They complete tasks on time; adhere to laws, policies, and regulations; and they consistently exhibit professionalism and cooperative behavior. 8.1 Completes Job Requirements According to Established Timelines - completes assigned task on schedule - is punctual for school, classes, meetings, conferences, and other scheduled activities - adheres to local personnel policies and procedures (e.g. attendance, leave) 8.2 Adheres to Written Local and State Board Policies and Federal Laws and Regulations - maintains accurate, up-to-date records, including student progress records - establishes procedures consistent with established policies, laws, and regulations - recommends actions in accordance with applicable laws, policies, and regulations - supports established laws, policies, and procedures when dealing with school personnel, students, and parents/guardians 8.3 Exhibits Professionalism with Peers, Administrators, Parents/Guardians - treats confidential information about students, staff, and school affairs in a professional and ethical manner - demonstrates respect, interest, and consideration for those with whom he/she interacts - assists in school planning when requested - participates in collegial efforts without giving up individual rights to dissent or to work to effect change - handles contacts with parents/guardians in a professional, ethical manner 111 8.4 Promotes Cooperation with Parents/Guardians and Between School and Community - adjusts activities and schedules when necessary to accommodate other programs or activities - holds conferences at times mutually convenient to all participants - uses community resources to supplement program - encourages parents/guardians to participate in the school - participates in school-related, parent-/guardian-directed meetings when appropriate - acts as an advocate for students - seeks outside help for students, as needed 112 Appendix B Interstate New Teacher Assessment and Support Consortium Principles 113 Appendix B Interstate New Teacher Assessment and Support Consortium Principles Principle 1: The teacher understands the central concepts, tools of inquiry, and structures of the discipline(s) he or she teachers and can create learning experiences that make these aspects of subject matter meaningful for students. Principle 2: The teacher understands how children learn and develop, and can provide learning opportunities that support their intellectual, social and personal development. Principle 3: The teacher understands how students differ in their approaches to learning and creates instructional opportunities that are adapted to diverse learners. Principle 4: The teacher understands and uses a variety of instructional strategies to encourage students? development of critical thinking, problem solving, and performance skills. Principle 5: The teacher uses an understanding of individual and group motivation and behavior to create a learning environment that encourages positive social interaction, active engagement in learning, and self-motivation. 114 Principle 6: The teacher uses knowledge of effective verbal, nonverbal, and media communication techniques to foster active inquiry, collaboration, and supportive interaction in the classroom. Principle 7: The teacher plans instruction based upon knowledge of subject matter, students, the community, and curriculum goals. Principle 8: The teacher understands and uses formal and informal assessment strategies to evaluate and ensure the continuous intellectual, social and physical development of the learner. Principle 9: The teacher is a reflective practioner who continually evaluates the effects of his/her choices and actions on others (students, parents, and other professionals in the learning community) and who actively seeks out opportunities to grow professionally. Principle 10: The teacher fosters relationships with school colleagues, parents, and agencies in the larger community to support students? learning and well-being. 115 Appendix C Alabama State Department of Education 1999 Novice Teacher Report 116 Appendix C Alabama State Department of Education 1999 Novice Teacher Report % of Novice Scores at Level Novice Teachers Average Scores 4 3 2 1 1.0 Preparation for Instruction 1,920 3.06 12% 81% 6% 0% 1.1 Selects/states long range goals and short- term objectives 1,930 3.05 14% 78% 8% 0% 1.2 Identifies various instructional strategies 1,932 3.05 15% 76% 8% 1% 1.3a Prepares instructional resources - observations 1,909 3.08 14% 80% 8% 0% 1.3b Prepares instructional resources - interview 1,936 3.07 14% 79% 8% 0% 2.0 Presentation of Organized Instruction 1,917 3.03 9% 85% 8% 0% 2.1 Orients student to lesson 1,938 2.97 9% 78% 8% 0% 2.2 Gives clear directions 1,937 3.05 13% 79% 8% 0% 2.3 Develops the lesson 1,937 3.03 13% 78% 8% 0% 2.4 Provides practice and summarization 1,935 3.09 15% 79% 8% 0% 2.5 Demonstrates Knowledge of Subject Matter and Pedagogy 1,939 3.12 16% 80% 8% 0% 3.0 Assessment of Student Performance 1,915 2.99 7% 85% 8% 0% 3.1 Monitors student performance 1,935 3.02 10% 81% 8% 0% 3.2 Measures student progress systematically 1,936 3.04 12% 80% 8% 0% 117 % of Novice Scores at Level Novice Teachers Average Scores 4 3 2 1 3.3 Provides feedback about student performance 1,939 3.03 11% 82% 8% 0% 3.4a Uses assessment results- observations 1,893 2.96 7% 83% 8% 0% 3.4b Uses assessment results- interview 1,888 3.01 11% 80% 8% 0% 4.0 Classroom Management 1,913 3.04 17% 71% 8% 1% 4.1 Manages Class time 1,936 3.09 20% 71% 8% 0% 4.2 Manages student behavior 1,935 3.06 19% 68% 8% 1% 5.0 Positive Learning Climate 1,912 3.09 14% 81% 8% 0% 5.1 Involves students in interaction 1,937 3.15 21% 73% 6% 0% 5.2 Communicates high expectations 1,938 2.99 11% 78% 10% 0% 5.3 Expresses positive affect/minimizes negative affect 1,936 3.15 20% 76% 4% 0% 5.4 Maintains physical environment conducive to learning 1,934 3.15 20% 75% 4% 0% 6.0 Communication 1,923 3.23 25% 74% 1% 0% 6.1 Speaks clearly, correctly, and coherently 1,938 3.24 26% 72% 1% 0% 118 % of Novice Scores at Level Novice Teachers Average Scores 4 3 2 1 6.2a Writes clearly, correctly, and coherently-observations 1,936 3.25 26% 72% 1% 0% 6.2b Writes clearly, correctly, and coherently- supervisor?s review 1,922 3.25 27% 72% 2% 0% 7.0 Professional Development and Leadership 8.0 Performance of Professional Responsibilities 1,917 3.21 24% 73% 2% 0% 8.1 Completes job requirements according to established timelines 1,937 3.25 29% 67% 3% 0% 8.2 Adheres to written local and state board policies and federal laws and regulations 1,936 3.20 24% 73% 3% 0% 8.3 Exhibits professionalism with peers, administrators, parents/guardians 1,937 3.26 28% 70% 2% 0% 8.4 Promotes cooperation with parents/guardians and between school and community 1,936 3.18 22% 74% 3% 0% 119 Appendix D Survey for Validation of Scenarios and Responses 120 Appendix D SECTION A ? Validation of Classroom Scenarios Please examine each of the following classroom scenarios and indicate which PEPE domain(s) it addresses, the extent to which it represents a realistic situation, and how an excellent and unsatisfactory teacher might respond to each scenario. Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 1. You have just begun a class discussion when the principal unexpectedly appears at your door, tells you to continue with whatever you were doing, and seats himself in the back of the room. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 2. Elementary students in your school are expected to line up and are not supposed to talk as they walk through the halls. You have a first- grade class. At the beginning of the school year, You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory 121 Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 3. Catherine often gets out of line when the class goes to the cafeteria. Today she stayed in line. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 4. You have been teaching about two-letter consonant blends and now you are ready to talk about blends consisting of three letters. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: 122 Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 5. The second grade reading manual says to present eight examples of finding context clues. After 30 minutes and the fifth example, you realize that most of the students are no longer participating in the lesson. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 6. In order to divide your junior-high class into two teams for a relay-type activity, You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: 123 Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 7. The P.E. class is playing a circle game on the playground. The student who is ?IT? is supposed to tag another and chase him or her the opposite way around the circle. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 8. Cynthia wore her new watch to school today. After lunch she discovered it was missing. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: 124 Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 9. At recess Ed stomps on Carlos? new shoes and gets them muddy and dirty. Carlos comes to you crying. You talk to Ed and remind him that he must be nice to the other children. As soon as you walk away, Ed steps on Carlos? shoes again. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: Response for an unsatisfactory: Classroom Scenario Which PEPE competency does this measure? (Check all that apply.) How realistic is this scenario? 10. You are teaching in a private school in a predominantly Christian community. It is December and your class?s turn to put up the hall bulletin board. Most of the students want to do a Christmas mural. You have one Jewish girl, Becky in class. You might: O 1.0 (Prep) O 2.0 (Present) O 3.0 (Assess) O 4.0 (Manage) O 5.0 (Climate) O 6.0 (Comm) O 7.0 (Pro Dev) O 8.0 (Pro Resp) 1 2 3 4 5 O O O O O Response for a teacher that demonstrates excellence: 125 SECTION B ? Your Background 1. What is your current position? O Principal O Assistant Principal O Central Office Administrator O Other, please describe: _________________________________________ 2. How many years: # of years? a. have you served in an administrative position at this school? _________ b. have you served in an administrative position at all schools? _________ c. were you a classroom teacher? _________ 3. What is the highest level of education you have completed? O Bachelor?s degree O Master?s degree O Specialist degree O Doctorate degree 4. Are you? O Female O Male 5. Are you? O African-American (Black) O Asian-American O Biracial/Multiethnic O Hispanic O Native American O White, Non-Hispanic O Other (please specify): Response for an unsatisfactory: 126 Appendix E Principals? Recommendations for Distribution of Responses 127 Appendix E Principals? Recommendations for Distribution of Responses PEPE Domain Principals? Recommendations of Items to be Combined Items Deleted Total Number of Items Reliability 1. Preparation for Instruction 27, 28, 29, 30, 34, 35, 36, 37, 38, 52, 53, 54, 55, 64, 66, 67, 68, 69, 70, 71, 80, 81, 82, 83,92, 93, 94, 95, 146, 147,152, 153, 155, 157, 158, 159 36, 37, 38, 70, 146, 147,153, 154 28 .6673 2. Presentation of Organized Instruction 5, 6, 7, 9,10, 11, 12, 13, 14, 15, 20, 21, 23, 27, 28, 29, 30, 34, 37, 38, 52, 53, 54, 55, 56, 57, 58, 59 52, 64, 65, 66, 68, 69,70, 71, 80, 81, 82, 83, 84, 85, 86, 87, 94, 95, 97,100, 101, 102, 103,112,113,114, 115,128, 129, 130, 131, 133, 144, 145, 146, 147, 152, 156, 157, 158, 159 5, 9, 10, 15, 20, 23, 37, 59, 68, 70, 84, 101, 102, 114, 129, 130, 145, 146, 147 50 .8080 3. Assessment of Student Performance 9, 10, 11, 12, 16, 17, 18,19, 20, 21, 22, 38, 39, 40, 41, 42, 52, 53, 54, 55, 56, 57, 58, 59, 64, 65, 66, 67, 68, 69, 70, 84, 85, 86, 87, 92, 93, 94, 95, 96, 97, 98, 99,108, 109, 110, 111, 112, 114, 115, 132, 133, 134, 135, 144, 145, 146, 147, 154, 156, 157, 158, 159 9, 10, 20, 39, 59, 68, 70, 84, 114, 135, 146, 147, 154 52 .7446 4. Classroom Management 1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 16, 19, 24, 25,26, 29, 30, 31, 32, 33, 46, 45, 47, 48, 49, 50, 51, 60, 61, 62, 63, 68, 69, 70, 72, 73, 76, 77, 78, 79, 88, 89, 90, 91, 98, 102, 116, 117, 118, 120, 121, 122, 123, 125, 126, 127, 128, 129, 131, 136, 137, 138,139, 148, 149, 150, 151 1, 4, 5, 6, 8, 10, 26, 32, 46, 50, 51, 61, 68, 70, 73, 77, 79, 90, 102,117, 119, 123, 129, 135, 137, 153 47 .7698 128 5. Positive learning climate 9, 17, 18, 22, 27, 28, 34, 35, 36, 37, 39, 40, 41,42,43, 44, 45,46,47,52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 68, 69, 70, 71, 72, 74, 75, 87, 88, 89, 90, 91, 96, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 113,116, 117, 118, 119, 124, 125, 126, 127, 130, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 152, 153, 154, 155, 156, 157, 158, 159 9, 17, 18, 36, 37, 44, 46, 59, 61, 68, 70, 75, 90, 101, 102, 106, 107, 117, 119, 130, 135, 137, 143, 153, 154 65 .7644 6. Communication 45,46,47, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 72, 73, 74, 75, 88, 89, 90, 91, 100, 101, 102, 103, 104, 105, 106, 107, 128, 129, 130, 131, 136, 137, 138, 139, 152, 153, 154, 155 46, 59, 61, 73, 75, 90, 101, 102, 106, 107, 129, 130, 137, 153, 154 32 .7242 7. Professional Development and Leadership 156 0 8. Performance of Professional Responsibilities 15, 42, 42, 43, 44, 48, 49, 50, 51, 104 15, 44, 50, 51 0 129 Appendix F Final Instrument Individual Items 130 Appendix F Final Instrument Individual Items *NoED PreST PostST ET n= 41 n= 46 n= 33 n= 49 ________________________________________________________________ Item M SD M SD M SD M SD ______________________________________________________________________________ A2 59 4.99 87 3.41 79 4.15 71 4.56 A3 76 4.34 76 4.31 91 2.92 76 4.35 B6 61 4.94 89 3.15 97 1.74 96 2.00 B7 71 4.61 91 2.85 100 0.00 96 2.00 C11 90 3.00 93 2.50 97 1.74 88 3.31 C12 56 5.02 74 4.44 79 4.15 78 4.42 D13 76 4.35 91 2.85 88 3.31 92 2.77 D14 20 4.01 15 3.63 15 3.64 16 3.73 E16 85 3.58 83 3.83 94 2.42 82 3.91 E18 71 4.61 83 3.83 85 3.64 86 3.54 E19 93 2.64 93 2.50 97 1.74 96 2.00 F21 56 5.02 59 4.98 61 4.96 56 5.02 F22 34 4.80 76 4.35 67 4.79 55 5.03 G24 93 2.64 98 1.47 97 1.74 93 2.64 G25 88 3.31 100 0.00 100 0.00 98 1.43 H27 93 2.64 100 0.00 97 1.74 100 0.00 H28 71 4.61 83 3.83 76 4.35 76 4.34 H29 51 5.06 61 4.93 76 4.35 65 4.81 H30 34 4.80 43 5.01 48 5.08 51 5.05 I31 76 4.35 85 3.83 82 3.92 92 2.77 I33 88 3.31 91 2.85 100 0.00 98 1.43 J34 90 3.00 93 2.50 97 1.74 81 3.94 J35 90 3.00 98 1.47 100 0.00 96 2.00 K38 63 4.88 74 4.44 79 4.15 63 4.88 131 Item NoED PreST PostST ET M SD M SD M SD M SD K40 90 3.00 100 0.00 97 1.74 94 2.42 L41 56 5.02 52 5.05 45 5.06 69 4.66 L42 85 3.58 100 0.00 97 1.74 94 2.42 L43 46 5.05 43 5.01 61 4.96 55 5.03 M45 88 3.31 91 2.85 88 3.31 90 3.06 M47 63 4.88 72 4.55 88 3.31 78 4.22 N48 66 4.80 57 5.01 82 3.92 76 4.34 N49 85 3.58 67 4.74 100 0.00 100 0.00 O52 98 1.56 98 1.47 100 0.00 96 2.00 O53 78 4.19 96 2.06 94 2.42 96 4.19 O54 95 2.18 85 3.63 94 2.42 90 3.06 055 85 3.58 91 2.85 94 2.42 98 1.43 P56 85 3.58 91 2.85 91 2.92 94 2.42 P57 88 3.31 87 3.41 97 1.74 96 2.00 P58 54 5.05 80 4.01 82 3.92 76 4.34 Q60 93 2.64 100 0.00 100 0.00 94 2.42 Q62 80 4.01 78 4.17 97 1.74 98 1.43 Q63 93 2.64 93 2.50 97 1.74 96 2.00 R64 80 4.01 72 4.55 39 4.96 69 4.66 R65 78 4.19 91 2.85 100 0.00 92 2.77 R66 83 3.81 93 2.50 97 1.74 86 3.54 R67 57 5.01 59 4.98 67 4.79 61 4.92 S69 34 4.80 41 4.98 52 5.08 55 5.03 S71 51 5.06 85 3.63 85 3.64 88 3.31 T72 95 2.21 98 1.47 100 0.00 96 2.00 T74 93 2.64 93 2.50 94 2.42 90 3.06 U78 90 3.00 91 2.85 94 2.42 94 2.42 V80 76 4.35 91 2.85 88 3.31 98 1.43 V81 90 3.00 98 1.47 97 1.74 98 1.43 132 Item NoED PreST PostST ET M SD M SD M SD M SD V82 61 4.94 76 4.31 61 4.96 88 3.31 V83 78 4.19 76 4.31 82 3.92 90 3.06 W85 61 4.94 83 3.83 67 4.79 92 2.77 W86 88 3.31 87 3.41 85 3.64 82 3.91 W87 49 5.06 78 4.17 79 4.15 92 2.77 X88 85 3.58 67 4.74 76 4.35 51 5.05 X89 98 1.56 96 2.06 100 0.00 98 1.43 X91 56 5.02 76 4.31 76 4.35 80 4.07 Y92 90 3.00 91 2.85 97 1.74 98 1.43 Y93 54 5.05 78 4.17 91 2.92 98 1.43 Y94 68 4.71 80 4.01 88 3.31 92 2.77 Y95 68 4.71 54 5.04 45 5.06 33 4.74 Z96 68 4.71 74 4.44 58 5.02 59 4.97 Z97 68 4.71 96 2.06 97 1.74 90 3.06 Z98 76 4.35 96 2.06 91 2.92 96 2.00 Z99 76 4.35 93 2.50 91 2.92 96 2.00 AA100 76 4.35 89 3.15 97 1.74 84 3.73 AA103 78 4.19 80 4.01 88 3.31 80 4.07 BB104 78 4.19 89 3.15 82 3.92 84 3.73 BB105 56 5.02 74 4.44 61 4.96 73 4.46 CC108 80 4.01 85 3.63 82 3.92 90 3.06 CC109 73 4.49 76 4.31 94 2.42 90 3.06 CC110 98 1.56 98 1.47 100 0.00 98 1.43 CC111 71 4.61 57 5.01 70 4.67 43 5.00 DD112 66 4.80 76 4.31 85 3.64 86 3.54 DD113 54 5.05 80 4.01 91 2.92 94 2.42 DD115 68 4.71 63 4.88 76 4.35 57 5.00 EE116 73 4.49 83 3.83 85 3.64 86 3.54 EE118 66 4.80 96 2.06 100 0.00 90 2.97 133 Item NoED PreST PostST ET M SD M SD M SD M SD FF120 76 4.35 72 4.55 91 2.92 80 4.07 FF121 83 3.81 89 3.15 82 3.92 84 3.73 FF122 83 3.81 93 2.50 33 4.79 88 3.31 GG124 76 4.35 78 4.17 82 3.92 80 4.07 GG125 93 2.64 96 2.06 97 1.74 84 3.73 GG126 63 4.88 85 3.63 88 3.31 96 2.00 GG127 80 4.01 83 3.83 91 2.92 96 2.00 HH128 68 4.71 96 2.06 97 1.74 98 1.43 HH131 61 4.94 80 4.01 91 2.92 86 3.54 II132 88 3.31 98 1.47 97 1.74 96 2.00 II133 78 4.19 87 3.41 88 3.31 90 3.06 II134 90 3.00 96 2.06 100 0.00 100 0.00 JJ136 59 4.99 72 4.55 82 3.92 90 3.06 JJ138 78 4.19 80 4.01 82 3.92 71 4.56 JJ139 61 4.94 72 4.55 88 3.31 92 2.77 KK140 90 3.00 80 4.01 91 2.92 86 3.54 KK141 66 4.80 58 4.99 76 4.35 53 5.04 KK142 93 2.64 98 1.49 97 1.74 100 0.00 LL144 54 5.05 67 4.74 64 4.89 53 5.04 MM148 85 3.58 93 2.50 94 2.42 90 3.06 MM149 85 3.58 78 4.17 94 2.42 90 3.06 MM150 88 3.31 91 2.85 100 0.00 100 0.00 MM151 93 2.64 96 2.06 100 0.00 98 1.43 NN152 85 3.58 96 2.06 100 0.00 96 2.00 NN155 80 4.01 96 2.06 100 0.00 96 2.00 OO156 71 4.61 91 2.85 94 2.42 100 0.00 OO157 49 5.06 57 5.01 55 5.06 41 4.97 OO158 56 5.02 76 4.31 67 4.79 71 4.56 OO159 83 3.81 93 2.50 82 3.92 78 4.22 134 Appendix G Follow-up Survey 135 Appendix G Follow-up Survey Thank you for your participation in this research. Your additional comments about a particular item or items are welcomed. Please complete the following questions only if you have had classroom teaching experience. Please indicate the number of years you have been teaching. ______________________ Please indicate your highest degree earned. _____________________________ Not Very Realistic Very Realistic 1. How well did the situations in this research resemble normal classroom experiences? O1 O2 O3 O4 O5 2. How well did the limited response time resemble the normal classroom decision-making time? O1 O2 O3 O4 O5 3. How do you believe this survey could be best used? (Please mark all that apply) O It could benefit teacher preparation colleges in assessment of their graduates O It could be used in as a professional development situation for schoolteachers. O It could be used as a ?pre-interview? technique at a school. O It could be used as a mentoring tool for new teachers, to begin discussions. Other ideas for use? ________________________________________________________________________ ________________________________________________________________________ 4. Do you believe this style of presentation reflects the actual classroom decision-making process compared to the typical pencil-and-paper test? O No, it is much worse than a pencil-and-paper test O No, it is worse than a pencil-and-paper test O It is about the same as a pencil-and-paper test O Yes, it is somewhat better than a pencil-and-paper test O Yes, it is much better than a pencil-and-paper test 5. What did you perceive as the strengths of this survey? 6. What did you perceive as the weaknesses of this survey? 136