Addressing the Unfairness That Exists in Testing: Part 2

Part 2: How to Stop Cheating and Fraud to Improve the Fairness of your Exams

There are two very real paths we can take to keep the bad actors of the world from gaining an unfair advantage on exams. 

Here's a simple truth that's hard to hear: Cheating on exams happens. It happens a lot. It happens more than most of us who work in the testing industry are willing to admit. (You can learn more about that in Part 1 of this article.)

Yet here is the flip-side of that truth: Cheating does not need to happen. Or, at least, it doesn't need to happen nearly as often as it currently does. 

If you are thinking that cheating is as inevitable as the tide, you may be unconsciously accepting it as a necessary evil when it’s actually something highly treatable. This mentality that we can't change anything about our circumstances is one of the very things that perpetuates these sources of unfairness. 

If cheating on tests is as inevitable as the tide… then let's move the moon.

Test Security Brush-Up 

Test security can be thought of in three major categories:

1. Prevention 
2. Deterrence
3. Detection + Reaction

Those three categories—Prevention, Deterrence, and Detection + Reaction—end up functioning as a kind of holistic cycle, commonly referred to as the Protection Process. So, in a perfect world, we have prevention measures that stop people from cheating; deterrence measures that convince people not to cheat; and detection measures that will alert us to those instances of cheating that can (and will) slip through the cracks. Finally, we have reaction measures that ensure we appropriately respond to bad actors in such a way that facilitates a culture of honesty, integrity, and validity in our program.

When you consider the many different types of cheating and fraud as discussed in Part 1 of this essay, these categories make sense, don’t they? Some threats (like stealing test content with hidden cameras) are difficult to prevent, while other threats (like pre-knowledge of exam content) are difficult to detect. 

Dr. David Foster, test security expert and CEO of Caveon, has this optimistic perspective: “I believe that all fraudsters can be deterred and prevented. They are successful because we have not been organized to deter and prevent them. I believe they will run and hide if we make a greater effort.”

When combined (and proudly publicized) these measures illustrate to test takers, stakeholders, and society in general that cheating and theft will not be allowed to affect the validity of your program’s test scores. This is how you illustrate that your testing program is committed to eliminating unfairness from cheating. 

One thing to keep in mind:

One of the most common misconceptions in testing today is that proctoring (all kinds of it) is some kind of bastion of security and test integrity. Unfortunately, that simply isn’t true. While proctoring can be an effective cog in much larger test security machinery, it sure ain’t the whole machine.

Please don't rely on proctors as your sole test security method. Even the best proctors in the world cannot perform the entire security cycle. Think about it. Can proctors tell if an examinee memorized answers from a braindump site and is using pre-knowledge to cheat? Nope. Can proctors tell if someone is recording your test content with a hidden camera in their button or glasses? They can't. While we don't have time to dive into all the details right here, I invite you to read these two white papers if you want to learn more about the function and limitations of proctoring: 

• The Past, Present, and Future of Proctoring
• Stealing Test Content with Hidden Cameras

So if proctoring is not well-loved by test takers and ultimately fairly ineffective at securing tests (read more about that in this white paper), what else is there? What other options do we have to stop cheating and improve the fairness of your exams?

Two Pathways to Test Security.

Looking to stop unfairness in its tracks? You have two options: Option A, which is a three-step solution, and Option B, which is focused on using a myriad of security measures to shore up your test security.

Although the scenario I am about to describe to you in Option A may sound like a lofty idea for the future, I assure you that all of what I describe for you here is easily achievable—right now—using existing technology and new test designs. Specifically, this three-part solution will eliminate most (if not all) threats to security and the resulting unfairness that occurs. Option A is the ideal solution that all programs should be striving for—but Option B is an alternative that can still help address unfairness in testing.

Ok, here we go.

Option A: Change Your Test Design

1. Create a Random Universe of Items for Your Exams

This is the most important step. I repeat: This is the most important step. By doing just this one step, you can effectively eliminate almost all sources of cheating and unfairness on your exams.

Build tests with very large (like, extremely large) item pools that pull tests randomly for each test taker. You can manually do this. The expense of a project like that would depend on the availability of your item writers and SMEs, but it is certainly possible. It can be made easier with tools like Automated Item Generation (AIG).

However, this is a better way. While AIG generates a large pool of items for your team to review and approve, SmartItem™ technology functions differently during both development and delivery so that the item renders differently each time it is given on a test. This means that no two examinees will see the same set of items (renderings) during testing. Consequently, stealing answers to gain an advantage is pointless. 

SmartItem technology removes all six types of item theft. Just like that. Gone. Because each item renders differently each time it is given, no two examinees will see the same thing during testing. That makes stealing an item (or a bunch of items, or a whole form, or an entire test) useless. In addition, using SmartItems also stops many other types of cheating, including the scourge of pre-knowledge, copying from a neighbor, using a cheat sheet, etc. So while, yes, you can still technically steal items or copy answers—there's no value, no more ill-gotten advantages. No more unfairness.

The caveat— While SmartItem technology essentially stops the bulk of cheating that occurs on exams, it can't stop the use of proxy test taking or getting expert assistance (like administrator coaching). So now, all you have to do is shore up those vulnerabilities. 

2. Shore up the Remaining Vulnerabilities:

Start By Taking Aim at Proxies

Proxy test taking rings can be both ad-hoc or large and sophisticated. Luckily, innovations in the field of authentication provide solutions. Utilize layers of authentication, including private, safe, and top-of-the-line biometrics that align with your organization's privacy policies. 

After administration, as an extra measure of authenticity, run a data forensics analysis to determine whether you see any patterns of proxy test taking present in your program. 

Then Dig Deeper Into Organized Collusion

In this world, money talks. Indeed, money sometimes tests, too. In the Varsity Blues scandal that I mentioned in Part 1 of this article, many of those ill-gotten test scores were achieved through bribery and administrative assistance. While there still isn't a great way to solve this problem, there are a few places to start.

The best you can do is this:

• • • Control your hardware. Be sure people can't remotely hack into it.
    • Observe your test takers during test administration. Then observe the observers. Extra layers of accountability go a long way to ensure your proctors and administrators are behaving ethically and honestly. You can learn more in this case study.
    • Don't rely on lockdown browsers to secure online testing. They tend to introduce privacy concerns without doing much to solve security problems.
    • Monitor the web to see if anyone is offering proxy or expert assistance services.

3. Always, Always, Always Employ Deterrence

Finally, don't forget about deterrence. Deterrence solutions are both easy and free to implement. And, as we know, deterrence makes up an entire third of the protection process, so skipping it would be a mistake on a number of levels (but you do you).

Using whatever tools are already at your disposal, and brandishing whatever security plans and policies you've put in place, deterrence is about convincing people that trying to cheat on your exams isn't worth the risk. Why?

Let's imagine that a test taker of ours is an honest person, but has a high incentive to cheat on our exam. This rational person, with their back against the wall, will probably be running a quick risk-versus-reward audit in their head to decide whether it is worth it to cheat. It's your job to tip the scales too far toward the risk to make the potential reward appealing. For example, reminding candidates that being caught with evidence of cheating results in a lifetime ban from your program may dissuade someone from thinking that cheating "just a little" will help them achieve their dreams. Or, tell examinees all about your SmartItem exam and convince them that any attempts to cheat are going to be fruitless. Convince them that cheating isn't worth their time and effort, and in the end, you have dissuaded that individual from trying to cheat. You can learn more about deterrence communications in this article.

Option B: Double-Down on Traditional Security Methods

"But Allie, I Don't Have the Energy for Space Travel."

Ok, that's fine. If you can't join me in this new cosmos—where test designs like SmartItems make sure the certainty of cheating and fraud, just like the certainty of the tides, is no longer a universal law—then there are still ways you can fly higher into the atmosphere. 

Here are some other things you can do to protect your program from the unfairness of cheating and fraud. You need to get serious about putting in place a series of test security measures to stop individuals from cheating on your exams. Let me be quite clear though: Even all of these solutions combined won't be as effective as Option A. They'll also require ongoing maintenance to upkeep. But don't get discouraged. It is still possible to have great security. Work with experts to create a comprehensive security plan that uses a variety of security measures and best fits your unique needs and budget concerns.

Again, we find ourselves at the "Big Three"—prevent, deter, and detect + react.

More Prevention Methods

What are some other ways we can prevent the possibility of exposure, theft, and other kinds of cheating from affecting your program? Prevention methods include:

• • • Expansive item pools to create tests, such as:
      • Computerized Adaptive Testing (CAT) design
      • Linear on the fly Testing (LOFT) design
      • Multiple Forms with unique items
    • IT security
    • Secure item designs (like DOMC™, the SmartItem, and other non-multiple-choice item types)

More Deterrence Methods

What are some examples of deterrence when it comes to cheating and theft? Deterrence can look like:

• • • An airtight candidate agreement
    • The presence of trained proctors in the testing space
    • Test development protocols (including confidentiality agreements, service-level agreements, and other access permissions)
    • An agreement for candidates to sign that reminds them about the use of authorized and unauthorized test preparation materials
    • Publicizing past instances when individuals were caught cheating and the consequences that they faced
    • Communicating to test takers that social media platforms are monitored throughout the testing process
    • Reminding examinees that data analysts review testing data for evidence of cheating and fraud
    • Communicating to subject matter experts and test administrators that checks and balances exist to ensure that test materials are handled appropriately
    • Reminding all stakeholders that contracts exist so that actions will be taken against individuals who attempt to cheat, steal, or share test content
    • Etc., Etc., Etc. Get creative. The sky's the limit when it comes to ways you can communicate to your test takers how seriously you take the security and fairness of your exams. 

More Detection + Reaction Methods

Many methods exist to detect the presence of cheating and theft in your testing program. What is important to remember here is the fact that, while valuable, detection occurs only after cheating or theft has already occurred. Please note the importance of prevention and deterrence to this process. When dealing with detection and reaction, I must stress to you the importance of timely and quality reactions that are pre-determined (to preview unintentional case-by-case bias from introducing more unfairness).

Here are some detection methods:

• • • Using Data Forensics to cancel fraudulent scores
    • Basic item watermarking
    • Sophisticated item watermarking
    • Patrolling the web for exam content
    • Investigations into security incidents
    • Keystroke logs
    • Recorded Test Session Audits and Reviews
    • A Security Incident Response Plan (SIRP)

So How Do We Feel?

It's a lot to unpack, this notion that cheating is a significant source of unfairness in testing (as addressed in Part 1 of this essay). Depending on your constitution, it may even be distressing. But what I've realized in my time at Caveon is that there are some very real things we can do (starting now if we wanted) to keep the bad actors of the world from gaining an unfair advantage. This article outlined a few such ways. 

Cheaters never sleep (it must be a guilty conscience thing), so methods of cheating and theft change all the time. But let's ask ourselves—have we? To leverage our positions in the testing industry and "make it matter," we're going to need to be constantly re-evaluating what we know, and embracing the very best methods at our disposal to address unfairness today.

Stopping cheating and theft is essential to achieving fairness. It is our responsibility and my honor to step up and contribute, in some small way, to a fairer world. 

We might not be able to fix a lot of the other unfairness in society. But this little corner of the universe? It's ours. We have the ability to protect it. Here, we have the means to reward the worthy. Here, we can make it impossible to gain traction on the low road. 

Stay tuned for Parts 3 and 4 of this essay. They will address accessibility limitations in testing.