Compromised testing

The Daily Howler has been running a series of posts questioning the testing gains in the NYC schools. The mayor and the school administrators have been boasting of their success as measured by a sharp rise in 4th grade test results. But questions abound in Do urban kids matter?

Could "an unprecedented increase in test preparation" lead to misleading gains—to score gains which reflect test savvy, not a real gain in reading skill? We suppose this is possible—but, at least in Winerip's telling, Tobias ignores a much larger problem. Over the course of the past forty years, teachers, principals and school administrators have routinely cheated on high-stakes tests as they struggle to drive up their test scores. Under the pressure of high-stakes testing, teachers have routinely pre-acquainted kids with the specific items on which they'd be tested. Teachers have routinely given kids too much time to take their tests. Teachers have routinely arranged for slower kids to be absent from the tests. And yes, teachers have given kids correct answers while the tests were being given—and teachers have routinely erased wrong answers, making them right, after the testing is completed and the students have all gone home! (Yes, this has been widely documented.) There is no way to cheat on these tests that hasn't been routinely observed—but testing experts like Tobias almost never mention this fact, preferring instead to mouth polite words, euphemistically saying that schools sometimes try "to beat or game a test." Sorry—as we've noted again and again (links below), teachers, principals and entire school systems routinely cheat on tests like these. But forty years into this well-documented story, experts like Tobias dare not say its name—and writers like Winerip don't bring it up either. Readers of the New York Times get to see polite, puzzling euphemisms—and the lives of urban kids keep playing second fiddle to the interests of hoaxers.

My problem with standardized testing is exactly this. Test results are fungible. Teachers can cheat. Administrations, testing companies, testing professionals... all have both means and motive to influence the outcome. When the stakes are high, there is added impetus to cheat. Without the high stakes the cheating tends to be self-cancelling, random noise, and the test results tend to be more trustworthy. As a parent and as an engineer, I like the idea of testing and measuring. But the value of the testing is compromised when so many people's careers have a vested interest in the testing outcomes.