I’ve been involved in the field of testing since 1982. I started my career as a testing program manager and psychometrician, where I experienced first-hand the use of test security procedures. Eventually (after several unexpected twists and turns) I started Caveon Test Security, the first company of its kind in this industry. I have now spent nearly two decades working in the exciting (to me and other converts) and often overlooked world of exam security.
Even though I have been promoting the value of test security for nearly twenty years, it wasn’t until just a few years ago that I truly began to understand the meaning of the term “security” and why our industry needs to care about it. Oddly enough, my new understanding began by looking at how security is approached in areas other than testing. My quest took me through a wide variety of industries including military base security, home security, casino security, financial security, and (an area that currently dominates news headlines) the security of information systems that protect our personal information stored on computers and the cloud.
In learning about how these other industries approach security, I soon came to realize that even though these security efforts are vastly different in many ways, they all have core principles in common. In fact, the basic principle of security for each industry is the concept of protection— they have something valuable, an asset of some sort, that needs to be protected from those who would try and undermine it. I learned that this concept is the starting point of all effective security, whether in the field of testing or elsewhere. Shame on me for taking so long to figure it out.
Protection is a goal, it's not a method or tool. All industries that have valuable assets—and we can include our own industry on that list—should have the same goal of protecting them. If those of us who work in the field of testing accept that we offer something valuable to this world, something that needs to be protected from people who want to tarnish it, it should become our #1 priority to protect it. To do so, we must ask ourselves a few central questions:
If we can answer these questions, well, the battle is pretty much won.
You might ask, “Dave, isn’t protecting our exams what we are doing today, and what we have been doing for the last century?” I have asked myself that question many times, and it is after many hours of consideration that I have come up with an answer. Here it is:
“Not really.”
What we do today is simply accept security practices that have been in place for decades, even centuries, without considering if those practices still work and still protect our exams. In all honesty, the persistently large (and even growing) number of security incidents seem to suggest that perhaps the security measures we have relied on up until now (like proctoring and the multiple-choice item) don’t really work anymore.
We in the testing industry work hard to create exams that produce valid test scores, and we need to at least consider that we are putting our own tests at risk by not upgrading our security policies and procedures. We work too hard to just throw it all out the window on the assumption that we are doing everything in our power to keep our exams safe.
The first step is to answer the three questions listed above. As any industry involved in security knows, the continual evaluation of security practices is the first and most fundamental step towards protection. Let’s get to it.
Logically, if you don’t know what you are protecting, you will not be able to protect it. In the field of testing, what is our most important asset? Is it our exams and items? Is it our reputation as specific programs? Is it the testing industry as a whole that we are trying to protect? While these are significant assets, they are not the most important. Based on a review of the Standards of Educational and Psychological Testing, our most important assets are useful test scores.
An exam is only valuable if the test scores it produces can be confidently used for their intended purpose. Unfortunately, right now, there is a persistent doubt that surrounds the validity and usefulness of test results. Those who use test scores, as well as the general public, question whether test results are of sufficient quality, whether they have been compromised by test fraud, or whether they are fair for all test takers. With the number of cheating scandals reported every month in the media, it is hard to blame people for having this doubt. This is what we need to protect—the validity of our test scores. Without it, our exams become useless.
Another way to think about this is to ask ourselves, “What are the threats to useful test scores, and who/what is behind those threats?” Thankfully, the list of threats is relatively small, and others have already done the heavy lifting and identified them for you. (Foster and Miller, 2012; ATP, 2013; ITC, 2014—see the below reference list or view all the threats laid out in this document that identifies all possible threats). The good news is that now that we have identified the threats, we can take steps to prevent them, and once we prevent them, we have successfully protected the test scores, and they can be confidently used!
Even when threats are eliminated, the question of who is behind the threat might never be clearly answered. Many of these nefarious individuals or organizations are never uncovered (after all, they are actively working hard to not get caught.) While disappointing, the goal is to stop the threats and stop the individuals behind them. Catching those individuals and bringing them to justice is a bonus. (A guilty pleasure of mine is knowing how much frustration and anger my security procedures are causing people who want to cheat on tests or steal exam content.)
On a practical note, it is important to remember that while there are only a few categories of threats, our security resources are always limited. Budgets are a very real factor for anyone working in test security. The good news is that if funds are allocated smartly, limited budgets need not stop us from protecting our scores. For this reason, the first step of any security efforts should be to narrow the list of potential threats to those that are the most likely and would be the most damaging.
This will be different for every single testing program. For example, for one program the most dangerous threat may be having the questions stolen and shared worldwide on the Internet; for another it might be the threat of proxy test takers. There is no one-size-fits-all answer to what the worst threats are, and each program must carefully analyze and determine this for themselves.
Fortunately, determining which threats are most important to your program is easy! A quick and relatively painless risk assessment analysis (which consists of rating and ranking the threats) will do the trick. I would suggest reading more about this process in this article, or by accessing a do-it-yourself risk assessment for free at Caveon’s website.
The final question remains: what are the best ways to protect our fair, reliable, and useful test scores? How do we stop threats? How do we quickly find a breach that has just begun? How do we inhibit, discourage, deter, and even scare the individuals behind the most likely threats? The methods or protection (what I call “security solutions”) fall into three categories: Prevention, Deterrence, and Detection/Reaction. Each category is equally important. However, they work best when used together, with each one complementing the other's protective strengths. Let me talk a bit about each category to illustrate its usefulness:
If you completely prevent all threats, then, problem solved! Woohoo! Unfortunately, while I dream of a world in which all threats can be completely prevented, it isn’t quite possible. We can, however, prevent some threats completely (think of recent innovations such as the SmartItem™), and some threats partially (like the Discrete Option Multiple-Choice™ item). Best practices, combined with common sense and creativity, will help you to build and implement preventive solutions. Learn more about preventative measures with curated articles here.
Prevention, as the word implies, makes it impossible for a threat to work in a particular situation. Let me show you an example from the real world. The image below shows a picture of a bollard, a physical structure designed to control vehicle access to public areas, preventing the possibility of a car being used to attack civilians.
As for an example from testing, a computerized adaptive test (CAT) does not display all of the test questions in the question pool. A test taker attempting to harvest the questions will only be able to steal the questions they see, a small percentage of the total item pool. Simply by using this test design, you have prevented a thief from stealing all your questions. Another example from testing: cheating by “copying” off someone else’s test has been effectively prevented with today’s use of computerized exams where start times are naturally offset, and questions are administered in a random order.
Deterrence is a psychological prevention strategy that targets the motivation of individuals who are considering cheating on a test or stealing questions. To deter someone from unwanted behavior, you put in place procedures that make a person not want to carry out the threat. Your procedures could convince them that they will get caught, or that what they are planning simply won’t work, or that it simply isn’t worth the risk. For the average test takers, simply informing them of the security rules and outlining the consequences of cheating is deterrence enough. For those more determined fraudsters, you will need a stronger set of deterrents.
A security guard sign is a form of deterrence against trespassers. The sign does not prevent a person from trespassing, it merely tries to convince them that trespassing would be unwise. While the testing industry also makes use of signage, such as “no cell phone signs” in a testing room, we can also deter fraud in many other ways. One example is when the Graduate Management Admissions council publicized a successful takedown of a website/company that had stolen and sold GMAT questions. Descriptions of fines, penalties, seizures, legal sanctions, and financial loss were published and distributed as a deterrence to all other organizations and individuals that might try and do the same thing. One thing to remember is that deterrence requires purposeful communication with the individuals behind a threat. Keeping deterrent strategies quiet will not work; you must convince would-be wrongdoers that procedures are in place that will catch them, and that they will surely be punished.
The third protection solution is detection/reaction. This is a two-step process. First, a set of detection measures are put in place, usually with the goal of detecting the initial attack from a threat. Second, after an attack has been detected, a series of pre-planned and associated reactions are triggered to deal with the attack and limit its consequences. The image below shows how the collision detection system for automobiles works: sensors detect an object in front of the car and warn the drivers. If the drivers don’t react in time, the car can stop itself by applying the brakes independent of the operator. The sensors detect a possible collision and then react with warnings and positive action.
One example of a detection/reaction solution in the testing industry is a vigilant proctor. The proctor should be well-trained in how to detect someone cheating on a test, but also in how to react and deal with the cheater. These two steps always work in tandem, because it doesn’t make any sense to detect a breach without having a pre-planned response to deal with it. I also like the term “reaction” because it suggests an almost automatic response to a detected breach. Another example of a detection solution is a data forensic analysis of test results. Unusual patterns of responding, caused by various types of cheating or by theft, are often apparent only by statistical analysis. If the analysis occurs early enough, it can allow a testing program to respond quickly and avoid extensive damage.
Threats change, with new ones on the horizon each year. Imagine a national defense strategy that fails to acknowledge ever-changing threats such as cyber hacking and transnational terrorism. It would be alarming and ineffective. The same is true for our test security process. The testing community needs to acknowledge that in order to protect our most valuable assets—our exam scores—we must continually evaluate current threats, as well as our process for protecting against them.
Complacency and protection do not mix. Protection is a dynamic process, one that requires innovative and exciting solutions. To succeed, we must continually ask ourselves these three questions: (1) What are we protecting? (2) What are we protecting it from? and (3) What are the best ways to protect it? As the world changes, so will the answers. It is only through this constant evaluation that we can be confident that our solutions are working, and that we are truly reaching the ultimate goal—protecting the usefulness of our test scores.
Topics from this blog: Test Security Basics Detection Measures Deterrence Measures Prevention Measures Test Security Plan
