In the first large-scale analysis of new US systems that evaluate teachers based partly on student test scores, two researchers found little or no correlation between the quality of teaching and the appraisals teachers received.
The study, published in Educational Evaluation and Policy Analysis, is the latest in a growing body of research that has cast doubt on whether it is possible for states to use empirical data in identifying good and bad teachers.
"The concern is that these state tests and these measures of evaluating teachers don't really seem to be associated with the things we think of as defining good teaching," said Morgan Polikoff, of the Rossier School of Education at the University of Southern California.
The number of US states using teacher-evaluation systems based in part on student test scores has surged in the past five years. Many states and school districts are using the evaluation systems to make decisions about hirings, firings and compensation.
The rapid adoption has been propelled by the Obama administration, which made the teacher-evaluation systems a requirement for any state that wanted to compete for Race to the Top grant money or receive a waiver from the most onerous demands of No Child Left Behind, the 2002 federal education law.
Thirty-five states and the District of Columbia require student achievement to be a "significant" or the "most significant" factor in teacher evaluations.
Most states are using "value-added models", statistical algorithms designed to figure out how much teachers contribute to their students' learning, holding constant factors such as demographics.
The researchers found that some teachers who were well-regarded based on student surveys, classroom observation by principals and other indicators of quality had students who scored poorly on tests. The opposite was also true.