In the high-stakes testing industry, we’re fixated on ensuring the integrity of test scores—including test scores for oral examinations. It is our mission at Caveon to protect that validity, and one I am personally passionate about.
As you probably know, when people ask why testing is so important, the go-to analogy is always some version of “you don’t want an unqualified surgeon operating on you.” I can vouch from my own experience that this is so much more than a saying. I try not to get choked up when I see and talk about this experience, but one of the biggest reasons I’m so passionate about test security is because I have had to trust my newborn baby to the care of surgeons. The picture below is the last picture my wife and I took of our four-month-old son after handing him to the anesthesiologist to have open-heart surgery. On that day, the phrase that “credentials matter” became my reality. I had to trust that the qualifications my son’s medical team had received were valid. The great news is that everything went great, and my son is now a thriving (almost) 3-year-old. (You can read the whole story here.)
The beginning of the 2020 decade has been marked by upheaval, stress, and lockdowns—all the byproduct of a global pandemic. So, the question we get a lot is: “After the year+ that EVERYTHING seems to have changed, what has changed about test security?” People assume that due to the mass migration to administering tests remotely (rather than in person or in testing centers), test security must be vastly different. This is a particular concern for programs that administer oral exams and have now adapted to a remote setup. But with all that in mind, the answer might surprise you:
NOT MUCH HAS CHANGED!
As crazy as it sounds, not all that much has actually changed from the perspective of test security after the pandemic. While the method of giving tests for many programs has transitioned to the online sphere, the main principles and priorities remain largely the same—with a few small caveats.
The overall risk level of online testing is not much higher than in-person testing. Just as many of our clients have learned throughout their transition to remote online exams, the risk of cheating simply shifts to different points in the test administration process. The primary difference between testing in person and testing remotely is your control over the testing environment and examinee authentication.
Here’s a good example. In a remote testing environment, the ability to utilize notes is increased compared to an in-person scenario. At the same time, the ability for a candidate to bribe a test administrator for gain is effectively eliminated. It’s not an increase in risk, but rather a shift in risk to different parts of the process.
So, if you are giving your oral examinations online, what risks should you focus on?
There are three primary threats to online oral exams:
The first threat to online oral exams is cheating using pre-knowledge. Just like when testing in person, one of the main security threats to online oral exams is when candidates use pre-knowledge of the exam content (either wittingly or unwittingly) to improve their own exam score. Pre-knowledge is when a candidate memorizes live test content before taking the exam (by finding it online, in forums, etc.) to unfairly get a better score. (You can learn more in this article about pre-knowledge.) Pre-knowledge is the single most significant threat to the validity of any exam. Unfortunately, it is extremely difficult to detect someone using pre-knowledge, and it is absolutely impossible to detect pre-knowledge solely by using test administrators or proctors. (Learn more in this article that discusses proctoring capabilities.)
Luckily, there are a couple of other options for stopping pre-knowledge. First, track down any candidates who used pre-knowledge (you can accomplish this by utilizing a mix of web monitoring and data forensics measures). Second, invalidate their scores.
For step one, you’ll start by tracking down exposed content on the internet and 1) have that content removed from the web by issuing a DMCA notice, and 2) remove any exposed content from your future oral exams and replace them with fresh questions. You can then use data forensics analyses to compare the discovered exposed content against actual response data (through “flawed key” analysis) or make a stepwise comparison of each set of responses against every other set of responses (similarity analysis).
You’ll notice that using web patrol and data forensics only catches cheaters after the fact. It does not stop anyone from being able to steal your test content in the first place (read this fascinating white paper on how easy it is to sneak hidden cameras and recording devices into an oral testing environment). It also does not stop test takers from being able to use exposed content for personal gain.
For traditional multiple-choice exams, in order to proactively prevent individuals from cheating using pre-knowledge, you need to build security into the design of your test whenever possible. But for oral exams, it's a bit more challenging. Oral exams do have the advantage of (typically) including expert interviewers or SMEs to help guide the conversation.
In addition to catching individuals who use pre-knowledge, it is vital to stop your content from being stolen and exposed in the first place. Exam exposure is a serious threat to the validity of any exam program and has the potential to impact an entire group or test-taking cohort. Examinees can steal and expose your test content through numerous methods, including: stealing questions through hidden cameras, memorizing questions to be recalled later, and transcribing questions verbally using hidden microphones. (It is shockingly easy to steal testing content with hidden devices. Read here to learn more.)
Similar to stopping the use of pre-knowledge, you can prevent your content from being stolen by adjusting your test and item design and administration methods. You can also consider watermarking your items so that you can track down the individuals who actually stole your content.
The second threat to remote oral examinations is proxy testing. We come across solicitations every day from proxy test takers willing to take an exam in someone else’s place, guaranteeing them a passing score. Unfortunately, when testing remotely, these proxy test takers are often extremely effective and very difficult to catch. They take advantage of weak authentication processes (it is unbelievably easy to get a realistic fake ID) and are impossible for a proctor to catch.
Fortunately, if you are in medical credentialing, there is a limited pool of available proxies. This is because a proxy in an oral exam or board setting would need to be a doctor (or at least someone with commensurate education). Additionally, the likelihood that a prospective proxy test taker would be willing to risk their effort and livelihood for someone else’s credential further slims the possibilities. Many other industries do not have this advantage.
In order to stop proxy test takers, your program must utilize the highest quality authentication measures that are available before the exam. Your authentication methods should include biometric measures and keystroke analysis, to name a few. Your program should also utilize data forensics analyses that compare scores across test takers. These analyses can ensure the test results are valid and not statistically similar to another candidate (who may have also used a proxy). You can learn more about proxy testing and how to stop it from happening in your program in this in-depth article on proxy testing.
The third threat to online oral examinations is when an administrator or proctor assists the examinee during the test. This type of cheating is not easily detected and reported, especially when there is a 1:1 candidate to examiner ratio.
Test collusion is an extremely difficult threat to eliminate. However, it can be significantly reduced using basic risk assessment and redundancy principles often applied in IT security.
When I was helping Caveon through our first SOC2 audit (an IT compliance and assurance audit), one of the largest things that our auditor stressed was eliminating single points of failure. This would ensure that any ”bad actions” taken would require collusion between two or more employees. Eliminating single points of failure would effectively require multiple employees to conspire together in order to take any negative actions or interrupt business operations. This same principle should be applied in a remote oral examination setting—simply recording the session and having multiple blind-selected examiners participate when possible is one method. (Bonus—record and review proctoring acts as a deterrent measure as well!)
In addition to recording your test sessions, consider having a neutral third party conduct a quality assurance check of both your live and recorded test sessions. Auditors will take a critical and objective look at your systems, personnel, practices, and policies. Employing auditors is a great way to ensure that A) everything is working as you envisioned, and B) everyone is following the rules.
In addition to the threat-specific security recommendations above, there are five additional (and more actionable) recommendations that remote oral exams should employ:
Regardless of the technology you are using, it is vital to ensure that your online testing solutions (including your item banking, test development, test delivery, authentication, reporting, and proctoring systems) are configured appropriately. Doing this will ensure the security of your exam content and even your candidates’ personal information and privacy. Confirm that your solution comes equipped with the features necessary to protect your tests and candidates.
Establishing your test security policies and procedures ahead of time will ensure long-term success. Creating a test security plan will also ensure the continuity and consistency of your exam. Take time to ensure that your security policies are useful, fair, widely understood by relevant staff, included in new-staff trainings or annual trainings, and are accessible in an emergency.
I’m confident that everybody reading this has had the experience of making decisions under pressure, and then upon reflection, wished they could have done something differently. I don’t think that I can overstate the importance of having a security plan. Having a plan that defines things like roles, crisis communication, pre-approved actions to be taken, and processes to invoke (while adhering to your candidate agreement and the rules you’ve established) is indispensable. This way, when the fire is raging, you can remove emotion from the situation and simply follow the established processes that you created ahead of time. Having these pre-planned responses will eliminate the risk of making any rushed decisions that could impact the legal defensibility of your actions.
Additionally, from a legal perspective, the agreement between a candidate and testing program is one of contractual nature. Courts give great deference to testing programs who follow their own rules. Let me repeat for emphasis: Courts give great deference to testing programs who say what they are going to do, and then do it!
Deterrence is an underrated, but powerful tool. Publicizing your test security measures to your candidates goes hand in hand with your candidate agreement. (Your candidate agreement is often a dense legal tome that requires significant time and effort to understand—one that [let’s be honest] few candidates invest time in reading thoroughly.) Do you read the terms of use before you install a new update on your phone or configure a new device? In testing, it’s critical to publicize your security measures in a way that your candidates understand, not in dense legalese that will likely be missed.
When publicizing your test security measures to examinees, clearly broadcast the steps you’re taking to secure your oral exam. Bring them to understand there will be consequences for cheating. Consider publicizing prior issues and sanctions that have been taken, even if anonymously. Then, give candidates a way to confidentially report any suspicious behavior they become aware of.
Once you’re up-and-running with your online oral examination, ensure that it’s working as intended and that your systems and processes are actually keeping your test secure. Use your quality assurance processes to ensure that your proctors and test administrators are following established protocols. Your quality assurance process will also help to isolate issues for review, re-target candidate communications, and identify opportunities for additional training.
Conducting quality assurance checks is also a great step to ensure the user experience is a good one. Your goal should involve creating a great user experience that doesn’t add to anxiety or frustration on exam day. By investing time and energy into quality assurance checks, both initially and on a periodic basis, you will ensure that you’ve created a testing process that promotes fairness and is as stress-free as possible.
It really isn’t a matter of “if” your content will be shared online, it’s a matter of “to what degree” it will impact your program. Web monitoring lets you know what content has been exposed, where the content was posted, who stole it, who used it, and to what degree your program has been impacted. As a result, monitoring the web is an unbelievably powerful test security tool.
The nature of monitoring the web for exposed test content has changed drastically since the early days of the internet. We’ve seen a large number of programs adapt their web monitoring strategy in extremely effective ways. For some, it’s an insurance policy; “there’s nothing out there now, but that doesn’t mean we’re far from our first exposure.” For others, where exposure is regular, it’s a finger on the pulse of the lifecycle of their exam; “web monitoring allows us to establish publication and exam-refresh-schedules that keep us a step ahead.”
Keep in mind that there are people who steal or distribute test content, and there are people who seek to gain an unfair advantage from already stolen test content. Both groups cannot exist in “secret”—they each rely on content and communication channels. Their ability to sustain their actions is dependent on not being found, which gives testing programs the upper hand in the new “cat and mouse” game being played.
If you are giving your oral examinations online, there are three primary threats you should be aware of:
To counter these threats, remote oral exams should:
