Few teachers are evaluated on their ability to use technology in the classroom despite conclusive evidence that its effective use has a positive impact on student achievement and that large amounts of resources are dedicated to placing technology in schools. Teacher evaluations in 220 school districts were reviewed for this study. Only one-fifth of the evaluations include technology skills as a specific criterion, with the majority of these occurring in larger districts. Reasons for this low rate of technological assessment are offered as well as sample evaluation language that districts might consider. These findings may assist policy makers in developing better teacher assessment tools and improve the implementation of technology in schools.
**********
Teachers have always adopted new technologies, whether slate boards, pencil sharpeners, movie projectors, or laptop computers, and incorporated them into their repertoire. The ability of teachers to integrate today's technology--defined in this study as computer-based tools, including the Internet, software and hardware, computers, and related multi-media--into the teaching and learning process directly impacts student achievement (e.g., Waxman, Connell, & Gray, 2002).
Teachers seeking to hone their technology skills have many options. University courses, professional development, seminars and conferences, professional organizations, current literature and research, and good old-fashioned classroom experimentation form a mosaic that offers avenues for improvement. Conversely, technology has little impact if teachers are not trained properly in its use (Sandholtz, 2001).
If school officials deem teachers' technology abilities as an area of interest, evidence can be garnered several ways, including observation, surveys of teachers, use of school technology resources and plans, training, and student achievement levels. If a school district's goal is to increase adept use of technology by teachers, one powerful way is through formal evaluation.
Little, though, is known about how or even if teachers are rewarded or penalized for their technological acuity. Scant research exists that addresses one critical part of the equation: teacher evaluations that include technology competence. This report examines the degree to which public school districts in one state include teacher technology skills in their evaluations.
REVIEW OF LITERATURE
Numerous studies have attempted to measure the effect of technology on student achievement. There is also a considerable body of research on the role and purpose of teacher evaluations. A brief review of these two issues framed within the context of this article follows.
Educational Technology and Its Impact on Achievement
It is more certain than ever that the appropriate use of technology has a positive impact on student achievement. Waxman et al. (2002) conducted a meta-analysis of 20 such studies with 138 effect sizes (1) involving 4,400 students and found the "study-weighted effect sizes averaging across all outcomes (cognitive, affective, and behavioral) was +.30 (p<.05), with a 95% confidence interval of .004 to .598" (p. 2), which indicated that "teaching and learning with technology has a small, positive, significant effect on student outcomes when compared to traditional instruction" (p. 2). A report by Sivin-Kachala and Bialo (2000) commissioned by the Software and Information Industry Association studied 311 projects chosen from an original set of 3,500. They concluded that the proper use of technology has a "positive effect on student attitudes toward learning, self-confidence, and self-concept" (p. 11).
A research synthesis of 31 studies, out of 195 experimental and quasi-experimental projects considered, that met rigorous criteria found a positive impact of discrete educational software (2) technology on reading and mathematics achievement (Murphy et al., 2002). The overall weighted effect size was +0.38 (p < .05), and "the magnitude of the effects was consistent and slightly larger than the estimated effects reported in earlier meta-analyses of the effectiveness of computer-based instruction" (p. 38). WestEd (2002), a nonprofit research and development agency, found that there is a "substantial body of research that suggests technology can have a positive effect on student achievement under certain circumstances and when used for certain purposes" (p. 24). The State of Illinois commissioned McCabe and Skinner (2003) of Westat to review the connection between technology and achievement and found that "technology use had a small, but significant, impact on student achievement as measured by the Illinois testing program" (p. 51).
The Apple Classrooms of Tomorrow (ACOT) was a 10-year longitudinal inquiry that investigated how the routine, embedded use of technology impacted students. In these classrooms:
Students and teachers had immediate access to a wide range of technologies, including computers, videodisc players, video cameras, scanners, CD-ROM drives, modems, and on-line communications services. In addition, students could use an assortment of software programs and tools, including word processors, databases, spread-sheets, and graphics packages. (ACOT, 1998, p. 2)
The results showed that students exposed to this style of teaching and learning "increased initiative by going beyond requirements of assignments and by independently experimenting with and exploring new applications" (Sandholtz, Ringstaff, & Dwyer, 1997, p. 90) and that "[technology] can significantly increase the potential for learning, especially when it is used to support collaboration, information access, and the expression and representation of students' thoughts and ideas" (ACOT, 1998, p. 3).
Determining the impact on student achievement resulting from the use of technology does present difficulties. One problem involves quality of research. Cuban (2001) found methodological limitations and questionable validity in his review of such studies. The broad continuum of how technology is pedagogically deployed complicates judgment and analysis, too. For example, should one focus on learning with technology, that is, its integration into the learning and teaching process as another arrow in the teacher's quiver? Or, should the focus be on achievement resulting when students learn from technology, such as performing rote drills on a computer, a "flashcard" approach? Is the role of the teacher merely to be a facilitator or is there a more direct cause and effect relationship in the teacher, technology, and achievement milieu? Regardless of the framework, however, research appears to confirm that there is a direct relationship between effective use of technology and student achievement.
Clearly, schools are ready to take advantage of technology. More than 99% of all U.S. public schools are now wired to the Internet (National Center for Education Statistics, 2002). The percentage of instructional rooms--defined as classrooms, computer and other labs, library/media centers, and any other room used for instructional purposes--that are hooked up to the Internet increased from 3% in 1994 to 87% in 2001 (National Center for Education Statistics, 2002).
The financial, as well as educational, stakes are high. By 2000, expenditures in K-12 settings for technology had tripled over the previous decade to more than $6 billion annually (Sivin-Kachala & Bialo, 2000), not including expenditures under the federally sponsored program School and Libraries Universal Service Support Mechanism program, better known as the e-rate. Completing four steps--(a) preparing a technology plan, (b) seeking discounts on eligible services, (c) confirming receipt of services, and (d) preparing invoices--has helped many districts obtain affordable Internet services and telecommunications (Universal Service Administrative Company, 2003). More than $10 billion in federal money has thus far been spent on the e-rate (McCabe & Skinner, 2003).
The federal government encourages technology in the schools in other ways, too. Part D of Title II of the "No Child Left Behind" act authorized by Congress is titled "Enhancing Education Through Technology Act of 2001" and is another instance of federal financing of school technology. A goal of Part D is to "encourage the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by state educational agencies and local educational agencies" (U.S. Department of Education, 2002, Sec. 2402, Purposes and Goals, B2B).
In summary, schools are equipped to use technology and the government supports such use. When deployed in a sound manner technology helps to bring student learners to greater heights of academic achievement.
Teacher Evaluations
Administrators use teacher evaluations in critical ways, including improving instruction and making decisions about continued employment. These objectives were established as far back as 1939 when Cooke stated that the purposes of teacher evaluation are "(a) selection of teachers, (b) determination of salaries and promotions, (c) guidance and inservice education, and (d) dismissal, retention, and retirement decisions" (pp. 185-187).
Drake and Roe (2003) noted that evaluation is a "challenging and often frustrating topic for study because it concerns not only measures of knowledge and demonstration of proficiencies but also the anxieties and consequences of interpreting and using this information at all levels" (p. 317). They offer the following teacher evaluation framework: (a) the evaluation process must be consistent with the school's philosophy, (b) the evaluation process must encourage growth, (c) the purposes of the evaluation should be developed cooperatively, (d) criteria should be clarified for assessing performance prior to evaluation, (e) evaluation should be continual and from multiple sources, and (f) the results of each stage of the evaluation should be recorded and reported.
A search of the AskERIC database in June 2003 using keyword combinations "teacher evaluation and technology skills," "teacher evaluation and technology competency," "teacher assessment and technology skills," and "teacher evaluation and computers" generated only one article relevant to this report. Anderson (2000) discussed the need for evaluating teachers on their technology abilities because such "assessment provides a measure of the return on the investment made in hardware, software, and training, as well as a way to plan for program improvement, and to distribute information to administrators, school boards, and the community."
Despite the paucity of research about how teachers are evaluated on their technology skills, teachers today enter the classroom better prepared to use technology than their longer-serving colleagues. They are raised with it and some university teacher preparation programs include a technology course in their programs. In Michigan, for example, Standard 7 of the "Entry-Level Standards for Michigan Teachers and Related Proficiencies" requires "an ability to use information-age learning and technology operations and concepts to enhance learning and personal/professional productivity" (Michigan Department of Education, 2002). The state also recommends in a section of its technology plan that school districts "must be encouraged to include technological competency as an aspect of teacher hiring and evaluation" (Michigan's State Technology Plan, 2000). Nationally, 18 states require teachers to take a course in the use of technology or a test that measures their skills in this area prior to receiving initial licensure (Galley, 2003).
Administrators have resources that assist with evaluating teacher technology skills. Specific technology standards for schools administrators have been developed by the Collaborative for Technology Standards for School Administrators (2001), a group of leading professional school organizations in the United States. One of the 16 standards for principals focuses on teacher evaluation. Standard 12 states that effective principals "implement evaluation procedures for teachers that assess individual growth toward established technology standards and guide professional development planning" (p. 12). A study by Whale (2003) found that of all 16 standards, Standard 12 had the lowest level of agreement, an indicator that principals do not normally evaluate teachers on how they use technology nor do they use such procedures to plan professional development. Thus, while teacher technology evaluation is recommended for administrators, the actual use and application of this teacher assessment tool by administrators remains inconsistent.
METHODOLOGY
A letter requesting a blank teacher evaluation form was chosen as the most efficient method of collecting data from school districts across the state. A follow-up request was sent to selected nonresponding districts. No existing databases of current teacher evaluations for this population could be located. Letters were deemed as a quicker and more likely way to obtain the needed information than, for example, phone calls or interviews with principals.
Independent observers using a content analysis model reviewed the data. Content analysis was chosen as the most appropriate and expedient method for discovering the frequency of technology skills in teacher evaluations. The results were tabulated by district size and by context within the teacher evaluation document in order to draw conclusions about how teacher technology skills are evaluated in the field.
Instrumentation
Building-level administrators in all 554 PK-12 public school districts in Michigan were mailed requests asking for a blank copy of a district teacher evaluation form. A description of the research project was included. Recipients were informed that (a) no school district would be identified by name in resulting publications or reports, (b) the responses would remain in the sole custody of the investigator, and, (c) responses would be shredded upon completion of the project.
Data Analysis
A conceptual content analysis, as outlined by Colorado State University (2003), was used to examine the returned evaluation forms. Internet, technology, software, hardware, multi-media, and computers formed the key words for the level of analysis. These terms were selected after reviewing curriculum benchmarks in the state studied. The predefined code set was not expanded during the course of the review. Coding occurred for existence of the technology terms in any section(s) of the evaluation instrument and was done by hand by two raters to increase accuracy. Implicit terms and translation rules were not problematic, as the words in the level of analysis were straightforward. The data were categorized in a nominal fashion because evaluation instruments either did or did not reference technology skills. Cohen's kappa was calculated to determine interrater reliability.
The returned evaluations from some school districts included separate appraisals for tenured and probationary teachers. A review of both evaluation forms was included in the analysis. Several districts submitted information that was not incorporated into the results, however. Individual professional development plans, pre- and postobservation forms, and evaluations for nonteaching positions such as secretaries and paraprofessionals were discarded, as were supplementary materials explaining the evaluation process, how it was determined, who was involved with it, and particular philosophies upon which it was based. The focus of this research was on the evaluation of regular classroom teachers and the appraisal forms used during the evaluation.
A list of district student enrollment was compiled for the submitted evaluations that included a district name, which totaled 192 out of a total 220 returned. The original request letters and enclosed envelopes were not coded, thus it was not possible to identify the school district unless its name was on the evaluation form. Some teacher evaluation forms did not have a school or district name on it. The list of identified districts was ordered from largest to smallest in order to determine relationships between district size and use of technology as a teacher evaluation criterion.
Delimitations
This study was limited to traditional Michigan public schools. Religious and other private schools were not included because their missions and evaluation procedures reflect a wide range of philosophies and their administrators have a greater degree of freedom in how teachers are evaluated, especially as a basis for continued employment. Fidelity was best served by concentrating on the large majority of students served by traditional public schools. Charter schools are a growing public school presence in Michigan but they were not included for the same reasons as religious and other private schools.
Limitations
It is unknown if, but possible that, one or more of the districts that submitted an evaluation form in the follow-up request also responded in the original survey because some evaluation forms received from the first survey did not have the name of the district on it.
It is also feasible that some administrators interpreted the phrase "audio visual" during the evaluation process to include technology as defined in this report. As explained earlier, the investigator discounted this interpretation.
FINDINGS
Out of 554 public school districts initially queried, 192 evaluation forms were returned, representing 34.7% of the state's school districts. A follow-up request to 50 districts was sent to increase sample size and 28 responses (56%) were received. The follow-up request went to randomly selected school districts listed in a state education directory. In total, 220 (39.7%) of the state's public school districts are represented in this study.
A conceptual analysis was performed on the 220 evaluations by two reviewers who independently examined the returned forms for incidences of teacher technology use as a criterion in evaluations. The Cohen's kappa interobserver rating was calculated at 0.86 for the original survey (192 responses) with a standard error of .049. The follow-up survey (28 responses) found a perfect interobserver rating of 1.00 with a standard error of 0.00. Overall, the interobserver rating was strong.
Context ultimately led to the exclusion of the term "audio-visual" and its variants in the conceptual analysis. Numerous evaluation forms rated teachers on how well they protected, managed, or stored audio-visual equipment, usually in the "management" section of the document. The phrase "audio-visual" was somewhat problematic, though, as it could conceivably include multi-media or other computer-based applications. The researcher decided that given the context that it should not be counted as a technology criterion because it did not appear to meet the definition of technology in this article.
A total of 42 out of the 220 districts (19.1%) include teacher technology skills as defined in this study as an evaluation criterion. A 95% confidence level indicates a margin of error of [+ or -] 4.03%. Districts that include technology competence in their evaluations tend to be larger districts (Table 1). A total of 192 school districts submitted evaluations that included the name of the district. The total number of responding districts (192) was arbitrarily divided into quarters to give a comparison of size as measured by student enrollment and teacher technology use evaluation; each quartile contains 48 schools.
Samples of evaluation language that referenced technology use include:
1. uses a variety of resources in instruction such as Internet, community members, text, and reference materials;
2. appropriate instructional materials, including electronic technology, are selected to support the benchmarks;
3. effectively integrates technology in the teaching and learning process;
4. uses available technology appropriately to maximize student learning;
5. uses technology; and
6. demonstrates the integration of technology into his or her teaching.
Teacher evaluations generally had four sections: (a) instruction, (b) planning and preparation, (c) classroom management and environment, and, (d) professional responsibilities and communication. References to technology were found in the instruction section in 72.5% of the time, 12.5% were found in planning and preparation, 7.5% of the occurrences were in the classroom management and environment section, and 7.5% were in the professional responsibilities/communication section. In three school districts technology was mentioned in more than one section.
In no instances did evaluation documents reference established technology standards for teachers, such as the National Educational Technology Standards--Teachers document developed by a consortium of professional organizations. The only instance technology was tied to any standard was in the area of curriculum and state benchmarks that are measured by statewide tests (see Example 2 in the sample language).
DISCUSSION AND CONCLUSIONS
The technology juggernaut shows no signs of slacking. Synergy resulting from dollars funneled into technology and apt delivery by teachers has a positive impact on student academic achievement. This effectiveness results from a "match between the goals of instruction, characteristics of the learners, the design of the software and technology implementation decisions made by teachers" (Sivin-Kachala & Bialo, 2000, p. 10). Substantial public resources are devoted to this effort. The government dollars pumped into technology programs may not be a fiat but it is certainly a strong push. Teacher evaluation parameters as currently written provide enough flexibility that a cooperative team of teachers, their representatives, administrators, and school board members should be able to wordsmith language that would encompass teacher technology skills in the evaluation. However, despite the considerable body of research on teacher evaluations, little work has been done when technology use is factored into the evaluation equation.
In earlier research about the degree to which principals use and agree with the Technology Standards for School Administrators (Whale, 2003), several administrators said that they would like to use teacher technology skills in teacher evaluation but were prohibited from doing so by union contract. In other words, the absence of appraising technology abilities in evaluation schema effectively stops administrators from considering it.
Another aspect of principals' behavior regarding evaluating teachers and technology to consider is that only about one-third of principals have ever taken any kind of technology course (Whale, 2003), including graduate and undergraduate education, seminars, workshops, and inservices. It is likely that these school leaders, in general, do not feel competent in evaluating teachers in this arena. In addition, only 20% of teachers report that they believe themselves to be well prepared to use technology in their teaching (National Center for Education Statistics, 1999).
Sample Contract Language
Current language found in evaluations is generally vague. Frequently, it was no more than "uses technology." This open-ended phrasing probably results from uncertainty on the part of evaluators on how to actually evaluate teachers in this area. Districts might consider, instead, more defined wording in evaluations such as:
Teacher will show evidence of appropriate use of technology in the teaching and learning process. This includes learning "with" technology and learning "from" technology. Technology means using the Internet, computers, software and hardware, and computer-based multi-media.
SUMMARY
1. A large majority of school districts do not use teacher technology skills as part of the evaluation process. In the findings reported here, 81% of responding districts do not include technology skills of teachers in their evaluations.
2. When evaluation language does appear to support technology use, it is often vague enough that deriving meaning is difficult.
3. The closest some districts come to mentioning technology in their evaluations is the category "audio-visual," and more often than not, it is in the "management" rather than "instructional" section of the evaluation and likely meant videotape records and other noncomputer media.
4. Districts that use technology in their evaluations usually place it in the "instruction" section.
5. Larger districts are more likely to include teacher technology skills in evaluations than are smaller districts.
Recommendations
1. Only 18 states currently require teacher candidates to be tested or to take a technology course as part of their university training. This number should be expanded.
2. Some school districts have made positive steps in their effort to reward or penalize teachers for their technology skills. All districts should strive to have language in their teacher evaluations in this area.
3. Although the most important use of technology by teachers is how it impacts student achievement, the other uses of technology such as in planning, recordkeeping, and the like, by teachers indicates that evaluation documents should reference technology in more than just the "instruction" section.
4. Evaluation of teacher technology use should be tied to established standards, especially the National Educational Technology Standards--Teachers.
5. Evaluation language should recognize both learning "with" technology and learning "from" technology and teachers' role in each.
6. Administrators need training to properly gauge effective use of technology by teachers.
Implications for Future Research
1. More work needs to be done to determine what constitutes fair and proper evaluation of teachers and how they use technology.
2. Research needs to link teacher skills with student learning from and about technology.
3. Studies should be done to assess principals' own technology skills and their comfort level in including such skills in teacher evaluations.
4. Researchers in states that do not have the preservice requirements for technology training should determine the extent to which technology skills appear in teacher evaluations.
In conclusion, school districts that evaluate teachers in their technology use will be a step ahead of most of their peers and will have a powerful tool to help achieve quality education. The clear impact of technology on the teaching and learning process will ultimately lead to such inclusion in evaluations as a basic part of teacher appraisal. Given the stakes, children deserve no less.
References
Anderson, M.A. (2000). Assessing teacher technology skills. MultiMedia Schools, 7(6), 25-27.
Apple Classrooms of Tomorrow (1998). ACOT history. Retrieved August 27, 2005, from http://www.apple.com/education/k12/leadership/acot/history.html
Collaborative for Technology Standards for School Administrators (2001). Technology standards for school administrators. Retrieved August 27, 2005, from http://cnets.iste.org/tssa/pdf/tssa.pdf
Colorado State University (2003). Writing@CSU: Writing guide. Retrieved August 27, 2005, from http://writing.colostate.edu/guides/
Cooke, D. H. (1939). Administering the teaching personnel. Chicago: Benjamin H. Sanborn.
Cuban, L. (2001). Oversold and underused: Computers in the classrooms, 1980-2000. Cambridge, MA: Harvard University Press.
Drake, T.L., & Roe, W.H. (2003). The principalship (6th ed.). Upper Saddle River, NJ: Merrill Prentice Hall.
Galley, M. (2003). The teacher's new test. Education Week, 22(35), 31-33.
McCabe, M., & Skinner, R.A. (2003). Analyzing the tech effect. Education Week, 22(35), 50-52.
Michigan Department of Education. (2002). Entry-level standards for Michigan teachers and related proficiencies. Lansing, MI: State of Michigan.
Michigan's State Technology Plan (1998)--Update 2000 (2000). Lansing, MI: Michigan Department of Education.
Murphy, R.F., Penuel, W.R., Means, M., Korbak, C., Whaley, A., & Allen, J.E. (2002). E-desk: A review of recent evidence on the effectiveness of discrete educational software. Menlo Park, CA: SRI International.
National Center for Education Statistics. (1999). Teacher quality: A report on the preparation and qualifications of public school teachers. Washington, DC: U.S. Department of Education.
National Center for Education Statistics. (2002). Internet access in U.S. public schools and classrooms: 1994-2001. Retrieved August 27, 2005, from http://nces.ed.gov/pubs2002/2002018.pdf
Sandholtz, J.H. (2001). Learning to teach with technology: A comparison of teacher development programs. Journal of Technology and Teacher Education, 9(3), 349-374.
Sandholtz, J. H., Ringstaff, C., & Dwyer, D. C. (1997). Teaching with technology: Creating student-centered classrooms. New York: Teachers College Press.
Sivin-Kachala, J., & Bialo, E. (2000). 2000 research report on the effectiveness of technology in schools (7th ed.). Washington, DC: Software and Information Industry Association.
United States Department of Education. (2002). No child left behind. Retrieved August 27, 2005, from http://www.nclb.gov
Universal Service Administrative Company (2003). E-rate discounts for schools and libraries. Retrieved August 27, 2005, from http://www.sl.universalservice.org
Waxman, H. C., Connell, M. L., & Gray, J. (2002). Meta-analysis: Effects of educational technology on student outcomes. Retrieved August 27, 2005, from http://www.ncrel.org/tech/effects/
WestEd RTEC (2002). The learning return on our educational technology investment: A review of findings from research. San Francisco: Author.
Whale, D.E. (2003). The new technology standards for school administrators: Findings from the first large-scale survey of high school principals. Connections: Journal of Principal Preparation and Development, 5.
Notes
1. Effect size is a statistical metric that allows comparison of results across studies and is commonly used in meta-analyses. The magnitude of a relationship between an intervention and an outcome is determined regardless of sample sizes.
2. Discrete educational software is widely available and used for instructional areas such as reading and mathematics. It includes computer-assisted instruction, computer-based instruction and can be found on CD-ROM's as well as the Internet.
DAVID WHALE
Central Michigan University
Mount Pleasant, MI USA
whale 1 de@cmich.edu
Table 1 School Districts That Evaluate Teachers on Technology Use Based on Size of District Include Technology as Quartile District Enrollment n Evaluation Criteria Percentage 1 Over 4,196 48 12 25.0 2 1,986-4,195 48 12 25.0 3 1,331-19,85 48 7 14.5 4 Under 1330 48 5 9.6 Total 192 36 18.8
