The World's Only Test Security Blog

Pull up a chair among Caveon's experts in psychometrics, psychology, data science, test security, law, education, and oh-so-many other fields and join in the conversation about all things test security.

How Okta Won with Threat-Based Security: A Case Study

Posted by Caveon

updated over a week ago

The Need:

Following five years of rapid growth, Okta, an industry leader in identity and access management, needed to develop a certification program to ensure that customers, partners, and internal employees have the appropriate knowledge and skills to support its products. Okta’s two major goals were straightforward:

  1. Create a best-in-class portfolio of exams.
  2. Ensure that the integrity of those exams was maintained so that the scores would be meaningful and remain so over time.

To achieve these goals, Okta connected with Caveon to find out how to build high-quality, secure exams. With security and innovation being the hallmarks of both companies, it was a natural fit. Together, the two first determined the most dangerous security threats the young certification program would be facing.

Five major threats were named:

Harvesting Threats

    1. Stealing actual test files from test administration servers
    2. Memorizing questions to be recalled later
    3. Obtaining test content from a program insider

Cheating Threats

    1. Using pre-knowledge of test content
    2. Colluding with an expert while taking the test

It was clear that if Okta could mitigate the risk associated with these threats, the security battle was won, and Okta would be able to avoid the difficulties that have plagued IT certification programs for the past two decades.

The Security Solution

Part of the Secure Customer Solution that Caveon proposed for Okta involved standard procedures such as having strong candidate agreements, developing exams in a secure environment, ensuring that only trusted individuals work on the exams, and using Web Patrol® to monitor the internet for leaked items. But the solution also included two innovative ways to defeat the anticipated threats; adopting them meant that Okta would be an industry leader in test security practices, despite their newcomer status in the certification world. Okta has always been committed to providing secure and reliable connections between people and technology, and the company exercised its innovative thought leadership and became an early Caveon adopter.

Two innovative solutions were named:

  1. Online proctoring
  2. Discrete Option Multiple Choice™ (DOMC)

Online Proctoring

Online proctoring was selected to reduce the risk of harvesting from servers (see Harvesting Threat 1 from the list of five major threats above), as the exams would be delivered online and would never reside on local servers. Online proctoring would also neutralize the threat of collusion (Cheating Threat 2) since online proctoring makes it difficult for a test taker to receive help from an expert during a testing session.

Discrete Option Multiple Choice (DOMC)

The second recommendation, DOMC, acts as a security force because the DOMC item is stingy in revealing its content during testing. With less content exposed to test takers, the threat of memorizing questions to be used later (Harvesting Threat 2) is severely reduced, as is the threat of using pre-knowledge (Cheating Threat 1). To handle the remaining threat (Harvesting Threat 3), using stronger non-disclosure agreements for employees and contractors deterred those inside of Okta from sharing test content.

With the decision made to implement these solutions, Caveon helped Okta develop the exams and provided technology so that the tests—with all of their security protections in place—could be securely administered through a popular online proctoring vendor.

The Happy Conclusion

IT certification programs are up against great odds; exam content is being leaked in some cases just weeks after publication. Those tests are posted on braindump sites for cheaters to buy and use. This has not been the case with Okta.

Because of the sensible security measures infused into Okta’s program, the company’s exams are protected; the exam scores, meaningful.

Okta’s exams have been published since August of 2016 and none have been found on the thousands of braindump sites around the world. The exams remain as protected today as they were when they were first published. The Okta certification program has even won an Innovation Award for its creative and forward-thinking solution. Not bad for the new kid on the block!


About Caveon

For more than 18 years, Caveon Test Security has driven the discussion and practice of exam security in the testing industry. Today, as the recognized leader in the field, we have expanded our offerings to encompass innovative solutions and technologies that provide comprehensive protection: Solutions designed to detect, deter, and even prevent test fraud.

Topics from this blog: Exam Development Proctoring DOMC™ Certification