CAVEON SECURITY INSIGHTS BLOG

The World's Only Test Security Blog

Pull up a chair among Caveon's experts in psychometrics, psychology, data science, test security, law, education, and oh-so-many other fields and join in the conversation about all things test security.

What is the Proper Place for Erasure Analyses?

Posted by John Fremer

updated over a week ago

Frequently, I am asked in the context of state assessments, “How important is erasure analysis? Is a three standard deviation difference meaningful? Do I really see major limitations with this approach?” In this blog, I state my position on erasure analyses. I don’t kid myself that this posting will silence future questions, but I will henceforward have something to cite regarding my views.

Erasure analysis, properly and competently performed, has a definite place in the analysis of results of paper-and-pencil tests. That said, I think it is over-relied on and has a substantial, if not fatal, flaw. I say it is over-relied upon because most test security violations are NOT the result of some teacher tampering with the answer sheets—most test security problems stem from the disclosure of the items and answers. In this respect, similarity, aberrance of student responses, and analysis of gains/losses are better indicators than erasure analysis of potential misbehavior.

Similarity analyses look at how often two or more test takers provide the same answers to test questions. How often do they answer the same questions correctly? More importantly, do they choose exactly the same wrong answer when they miss a question?

Aberrance is a measure of the reasonableness of the pattern of right and wrong answers. Why would a group of students answer hard questions correctly but at the same time miss easy ones? Could it be that they were given answers to the hard questions?

Because it is almost always possible to use more than one method, testing programs should do just that. Erasure analysis is important if tests are administered by paper and pencil, but it is only one of several possible methods.

Let me turn to that major flaw. Because the “base rate” for erasing on state assessments typically averages less than one wrong-to-right erasure per answer sheet, legitimate outliers could lead to a class being flagged by a rigidly applied trigger such as a three SD difference. These statistical outliers occur when one or two students “lose their place,” on an answer sheet, discover the error, and then make the necessary corrections. Many of the changed responses will be wrong-to-right erasures. Some advocates of erasure analysis may assert that only rarely would such an occurrence result in a class being flagged. The problem is that false accusations do NOT “average out” for the accused individual. The teacher, who was falsely accused only to have the school or state discover later that there was no problem, may suffer substantial harm. I liken it to the horror of having a newspaper story printed that falsely identifies you as a spouse/child abuser or one who misuses funds. The publication of a later correction does not restore a tarnished reputation. As Shakespeare observed:

          "Who steals my purse steals trash…

But he that filches from me my good name

Robs me of that which not enriches him

And makes me poor indeed"

Anyone involved in cheating detection wants to avoid “false positives,” classifications that prove to be in error. The way to do that is to use multiple methods, to choose them wisely, and then to verify the results. Don’t be overly influenced by the fact that erasure analysis is the “flavor of the year.” Develop a more refined palate and use several appropriate methods and look for convergence of the results. Then your decisions will withstand thorough scrutiny and will provide teachers and others the protection and respect that they deserve.

John Fremer

View all articles

About Caveon

For more than 18 years, Caveon Test Security has driven the discussion and practice of exam security in the testing industry. Today, as the recognized leader in the field, we have expanded our offerings to encompass innovative solutions and technologies that provide comprehensive protection: Solutions designed to detect, deter, and even prevent test fraud.

Topics from this blog: Data Forensics K-12 Education Detection Measures