Common questions we receive in the context of state assessments are, “How important is an erasure analysis? Is a three standard deviation difference meaningful? Do you really see major limitations with this approach?”
In this blog, I state my position on erasure analyses. I don’t kid myself that this posting will silence future questions, but I will henceforward have something to cite regarding my views.
When properly and competently performed, an erasure analysis (also known as an answer change analysis) has a definite place in the analysis of results on both computer- and paper-based tests.
That said, I think answer change analyses can be over-relied on. This is because most test security violations are NOT the result of some teacher tampering with the answer sheets; most test security problems stem from the disclosure of the items and answers. In this respect, similarity, aberrance of student responses, and analysis of gains/losses are better indicators than erasure analysis of potential misbehavior.
Similarity analyses look at how often two or more test takers provide the same answers to test questions. How often do test takers answer the same questions correctly? More importantly, do examinees choose exactly the same wrong answer when they miss a question? You can learn more about identical answers and similarity analyses in this article.
Aberrance is a measure of the reasonableness of the pattern of right and wrong answers. Why would a group of students answer hard questions correctly, but at the same time miss easy questions? Could it be that they were given answers to the hard questions?
Because it's almost always possible to use more than one method, testing programs should do just that. An answer change analysis is important, but it is only one of several possible methods.
Answer change analyses have a substantial flaw. Because the “base rate” for changing answers on state assessments typically averages less than one wrong-to-right answer change per answer sheet, legitimate outliers could lead to a class being flagged by a rigidly applied trigger such as a three standard deviation difference. These statistical outliers occur when one or two students “lose their place" on an answer sheet, discover the error, and then make the necessary corrections.
Many of those changed responses will be wrong-to-right answer changes. Some advocates of erasure analyses may assert that only rarely would such an occurrence result in a class being flagged. The problem is that false accusations do NOT “average out” for the accused individual. The teacher (who was falsely accused, only to have the school or state discover later that there was no problem) may suffer substantial harm. The publication of a later correction does not restore a tarnished reputation. As Shakespeare observed:
"Who steals my purse steals trash...
But he that filches from me my good name
Robs me of that which not enriches him
And makes me poor indeed."
Anyone involved in cheating detection wants to avoid “false positives,” classifications that prove to be in error. The way to do that is to use multiple methods, to choose them wisely, and then to verify the results. Use several appropriate data forensics methods and look for convergence of the results. Then your decisions will withstand thorough scrutiny and will provide teachers and others the protection and respect they deserve.