CAVEON SECURITY INSIGHTS BLOG

The World's Only Test Security Blog

Pull up a chair among Caveon's experts in psychometrics, psychology, data science, test security, law, education, and oh-so-many other fields and join in the conversation about all things test security.

The SmartItem™ on SailPoint Certification Exams

Posted by Arthur Altman & David Foster

updated over a week ago

Ever wonder if the SmartItem actually delivers on its promises? SailPoint testifies that when used on two of their certification exams, the SmartItem provides benefits such as stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation.

The SailPoint Certification Program

SailPoint is a leader in identity governance which plays a critical role in modern enterprise security. One of SailPoint’s core values is “Innovation” or developing creative solutions to real challenges, and SailPoint teamed up with Caveon to do just that. In this brief article, we describe the joint, successful implementation of the new testing technology of the SmartItem for the first two SailPoint certification exams.

SailPoint certification program staff had been aware that IT certification programs can have their exams stolen within days of being released. Those exams end up on braindump sites, making test scores from those exams useless for certification decisions. SailPoint wanted to ensure that any certification exams bearing its imprimatur would be as secure as the available testing technology would permit. Therefore, in its first certification program ever, SailPoint decided to use Caveon SmartItem technology for 100% of their first two exams.

The SmartItem: Invulnerable to Cheating and Theft

The SmartItem is unique in that each one renders in thousands or even millions of ways, providing a different but fair experience for each test taker. The SmartItem is invulnerable to theft and cheating, relieving SailPoint of the same security concerns that have plagued every other IT certification program for the past three decades. Utilizing SmartItem technology brought at least two other bonus advantages to SailPoint as well. The first is that SmartItems motivate proper preparation. Braindump material is unavailable, and multiple-choice-test-taking skills are irrelevant; to pass the exam, test takers need to study and gain experience on SailPoint technologies. The second advantage is in cost savings. Because SmartItems can’t be compromised, they will only need to be replaced because of legitimate performance issues or inevitable content updates, not because a question or even an entire form was stolen.

Of course, the biggest benefit of SmartItem use is that test scores can be trusted, forever, for certification decisions. With each test administered, SailPoint can be confident that candidates who pass the test are competent in the skills required by SailPoint. As long as the certification program is in place, the exams’ test scores can be used to make reliable certification decisions.

Lessons Learned from Utilizing SmartItem Technology

The process of creating and publishing certification exams is an extensive one, and one that needed some modification for SmartItems. Since the first two exams have been published and made available to candidates, what has SailPoint learned?

Development

  1. We learned that SmartItems can be congruent with and built to very complex SailPoint objectives.
  2. We learned that the most important SME task isn’t to write items, but to design SmartItems to completely cover the objectives.
  3. We learned that SailPoint SMEs, working in small teams, were able to utilize Caveon coders and technology to create SmartItems efficiently.
  4. We learned that SailPoint SMEs, who were all novice item writers, enjoyed writing SmartItems. Because SmartItems afforded more creative options for covering objectives, for instance, enabling as many correct options and incorrect options as needed to cover a given item, item authors could follow the subject matter wherever it took them. This was fun!
  5. We learned that the SmartItem authoring tools would need correspondingly functional item review capability, to enable authors to more quickly yet fully check item content and design.
  6. We learned that, despite the newness of the SmartItem approach, it took about the same amount of overall time and effort to create SmartItems as it would have taken to create traditional items.

Field Testing

A field test of the SmartItems (also sometimes called a “beta test”) was conducted to see if SmartItems performed well psychometrically ahead of exam release.

  1. We learned that, using established psychometric standards, SmartItems perform as well as or better than traditional items.
  2. We learned that SmartItems contribute to the high reliability of SailPoint exams.
  3. We learned that SmartItems contribute to the gathering of evidence for the valid use of test scores for the SailPoint exams.
  4. We learned that we could create pass/fail cut scores easily for tests using SmartItems.
  5. We learned, in general, that there is no psychometric reason to not use SmartItems in exams.

Conclusion

In conclusion, we learned that the promised benefits of SmartItem technology—including stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation—are confirmed.

Because of the very positive experiences from the first two certification exams, a third exam using SmartItems has since been developed.

 

 

 

About the Authors

 

Arthur Altman

Arthur Altman is a Program Manager in the Global Education Services business at SailPoint Technologies Holdings, where, among other duties, he stood up and now oversees the SailPoint Certified Identity Professional Program, including the launch of the first-ever certification exams at SailPoint in 2018, with more coming in 2019.

 

David Foster, Ph.D.

A psychologist and psychometrician, David has spent 36 years in the measurement industry. Over the course of his career, Dave’s background in research and innovation has motivated him to apply a scientific approach to addressing problems in testing while continually searching for new and innovative solutions for improvement.

 

Arthur Altman & David Foster

View all articles

About Caveon

For more than 18 years, Caveon Test Security has driven the discussion and practice of exam security in the testing industry. Today, as the recognized leader in the field, we have expanded our offerings to encompass innovative solutions and technologies that provide comprehensive protection: Solutions designed to detect, deter, and even prevent test fraud.

Topics from this blog: Exam Development Prevention Measures SmartItem™

Posts by Topic

See all

Recent Posts

    Learn more about SmartItem technology in our white paper:

    THE FUTURE OF MULTIPLE-CHOICE

    View the White Paper