The World's Only Test Security Blog
Pull up a chair among Caveon's experts in psychometrics, psychology, data science, test security, law, education, and oh-so-many other fields and join in the conversation about all things test security.
Posted by Rory McCorkle, Ph.D. & David Foster, Ph.D.
updated over a week ago
Proctoring as we know it today is as old as the hills. Since we've all taken tests, we've seen proctoring in action. proctors check identification and sign you in. They provide and read exam instructions. Then when you are settled in, they watch you and everyone else taking tests. They hope, as you do, that they are not called on to "handle an incident." Proctors, in general, are not paid that well—sometimes they are even volunteers. Training is sometimes brief, but in other instances can take many weeks and result in a certification exam. They are asked to do a difficult chore, to stop test fraud when they see it. It's difficult because some test fraud, occurring right under their noses, is virtually impossible for them to see.
Before going further, we just want to state clearly that the terms “proctoring” and “proctor” used in this paper refer to both online proctoring (sometimes referred to as “remote proctoring”) and on-site proctoring (whether computer- or paper-based). While each type of proctoring has strengths and weaknesses, overall, they perform a very similar and qualified service.
Effective test security requires an understanding of the specific dangers or threats that happen during test administration, and the knowledge of what can be done about them. There are three threats that are relatively immune to proctor vigilance. Two are types of piracy. The other is a form of cheating:
Of course, there are plenty of prohibited behaviors that a proctor can easily see and deal with. It’s their strength to see attempts at copying, examinees speaking to each other, leaving the testing center, or trying to use cheat sheets or phones. Proctors, when they can dedicate themselves to the task, are very capable of preventing most kinds of cheating and test theft, detecting and dealing with those they can see, and deterring many others simply by being and appearing to be vigilant.
We who use proctors need to understand their strengths and weaknesses. With such understanding, we are in a better position to evaluate them, to improve what they are able to do, to place them in a comprehensive security system, and by so doing, to be more effective against the many varied threats.
Several techniques and technologies are reaching an inflection point for their use in proctoring. Many of them have been used on a limited basis thus far, but as their use grows, it creates exciting possibilities for proctoring.
The first is the use of Artificial Intelligence and Machine Learning in the monitoring of candidates. When combined with facial recognition technology and a standard high-definition webcam, this technology can monitor the movements of candidates to recognize irregularities. Facial recognition can read the positioning of the candidate’s head to identify potential aberrant behavior. For example, a candidate consistently looking down may be reading off a cheat sheet. Similarly, a candidate may manipulate an object on their person, such as a button camera, to take pictures of the screen. All of these behaviors can be trained into the system, and can then alert a proctor to their occurrence. While their use has been discussed more in the arena of online proctoring, its use for on-site proctoring may be simply a matter of deploying webcams to test center workstations. The use of such technology might also enable lower proctor-to-candidate ratios, and allow proctors to focus more on providing great service to candidates while reducing the number of test room walkthroughs required.
One of the toughest jobs a proctor has is spotting hidden cameras and other devices. Successful proctoring programs spend extensive time training proctors on how to spot these devices. However, there are a variety of technologies that have been developed to detect them. Some organizations have opted to use metal-detecting wands, which detect metal-based devices on the person. However, this can create privacy concerns and is not possible in some countries/regions of the world; further, devices can pass below the sensitive range of such wands. Radio Frequency (RF) and Camera Lens Detectors are two devices that have not yet been deployed extensively. RF detectors pick up the transmission of radio waves that occur from most devices, while camera lens detectors attempt to pick up the reflection of light from a camera lens. Many devices now have both techniques built in, with amateur versions starting at around $100 and professional versions running into the thousands of dollars. However, a reasonably priced device would provide another critical tool in a proctor’s toolbox. It may also detect currently undetectable devices, including new contact lens-based cameras.
With more sophisticated testing systems, big data approaches, and more complex statistics, real-time data forensics is used to flag candidates for irregular behavior while the test is in progress. Candidate response patterns, timing, and other information can be processed by cloud-based software, enabling the data to be analyzed in real-time. Such analysis could include response similarity, response latencies, and other measures. For example, latency-based statistics could alert a proctor that the candidate is proceeding unusually quickly through their examination, noting that this could be due to cheating or attempting to steal items. Similarities indices could also flag potential collusion among the candidates, or the use of the same pre-knowledge. Such flags, particularly if used in combination with the technologies discussed already, make for a potent combination.
AIG is another technique that makes one element of the proctor’s job (addressing pre-knowledge) irrelevant. AIG continues to get more sophisticated, allowing organizations to generate questions via technology, rather than solely relying on subject matter experts. While most AIG methods currently still have the items reviewed by subject matter experts prior to their use on an examination, it is likely that—in the near future—it will be common for items to be generated on the fly.
Test designs, such as computerized adaptive tests and linear-on-the-fly testing, reduce the overall exposure rate of items, making them difficult to harvest. Tests with multiple equivalent or equated forms serve a similar security purpose. Similarly, new types of security-sensitive items are being used that make pre-knowledge less of a concern. The most notable of such new item types is the SmartItem, an item type pioneered by Caveon. The SmartItem renders uniquely, on-the-fly, for every test taker making it virtually impossible to profit by either stealing test content or using stolen content as test pre-knowledge.
Finally, as we identify and implement such techniques to narrow the scope of a proctor’s role, we must ensure that proctor training is high-quality and consistent. The best proctoring programs use continuous training to update proctors on new threats, and to remind them of their customer service responsibilities. Similarly, those programs that use an initial certification of their proctors, with regular retesting, have significant advantages. At PSI, we have found that using quarterly training videos, combined with short quizzes about their content, ensures that proctors stay up to date on key security and customer service issues.
Proctors are asked to fill a complex role, being a security lead and customer service representative, carefully attending to accommodations when needed, and more—all for limited compensation. It is critical that we use current and upcoming technologies to support their role and maximize their skills. They are the accomplished and professional “faces” for their testing organization. Such focus on proctor improvement will provide a more secure testing process for testing programs, and a smoother experience for candidates.
Rory E. McCorkle, Ph.D.
Rory is the Senior Vice President of Certification and Education Services for PSI. He helps PSI clients meet their testing, educational, and strategic goals through the suite of services offered by PSI, while leading a team of business development, account management, and consulting professionals. During his career, Rory has worked with over 750 testing organizations, including well-known universities and colleges, licensing bodies, and renowned certification programs. He serves extensively throughout the testing community, with current roles Chairing the 2019 Association of Test Publishers (ATP) Innovations in Testing Conference and the External Stakeholders Committee for the Institute for Credentialing Excellence (ICE). He received a Ph.D. in Organizational Leadership from The Chicago School of Professional Psychology and holds an MBA from Drexel University. Additionally, he is certified as a Certified Association Executive (CAE), Senior Professional in Human Resources (SPHR), Project Management Professional (PMP), and New Product Development Professional (NPDP).
David Foster, Ph.D.
A psychologist and psychometrician, David has spent 36 years in the measurement industry. During the past decade, amid rising concerns about fairness in testing, David has focused on changing the design of items and tests to eliminate the debilitating consequences of cheating and testwiseness. He graduated from Brigham Young University in 1977 with a Ph.D. in Experimental Psychology, and completed a Biopsychology post-doctoral fellowship at Florida State University. In 2003, David co-founded the industry’s first test security company, Caveon. Under David’s guidance, Caveon has created new security tools, analyses, and services to protect its clients’ exams. He has served on numerous boards and committees, including ATP, ANSI, and ITC. David also founded the Performance Testing Council in order to raise awareness of the principles required for quality skill measurement. He has authored numerous articles for industry publications and journals, and has presented extensively at industry conferences.
Check out these other resources about proctoring:
For more than 18 years, Caveon Test Security has driven the discussion and practice of exam security in the testing industry. Today, as the recognized leader in the field, we have expanded our offerings to encompass innovative solutions and technologies that provide comprehensive protection: Solutions designed to detect, deter, and even prevent test fraud.
Topics from this blog: Proctoring Detection Measures Deterrence Measures Prevention Measures
Get expert knowledge delivered straight to your inbox, including exclusive access to industry publications and Caveon's subscriber-only resource, The Lockbox.
Learn more about proctoring in our white paper:
