The World's Only Test Security Blog

Pull up a chair among Caveon's experts in psychometrics, psychology, data science, test security, law, education, and oh-so-many other fields and join in the conversation about all things test security.

Testwiseness can Put Your Testing Program in Jeopardy!

Posted by Alison Foster

updated over a week ago


If you are a fan of the TV game show Jeopardy! (and even if you are not) you have probably heard of the recent contestant phenom, James Holzhauer. With over 32 consecutive wins, he amassed over $2.4 million dollars in winnings, only second to Jeopardy! winner, Ken Jennings with $2.5 million dollars. His reign of winning came to a close as he finally lost a match with an incorrect response in the final round.

Nobody questioned the intelligence of this candidate or his depth of knowledge on a wide array of challenging topics. However, what was surprising was how Mr. Holzhauer changed the answering strategy of Jeopardy!. You might say he had “gamewiseness.” His answering strategy was to answer the hard questions first and generate a sizable lead over the other contestants early. He then sought out the elusive "Daily Doubles" quickly. This drove his winnings even higher and allowed him to effortlessly win each match.

I got to thinking what the showrunners of Jeopardy! would do after James Holzhauer had ‘cracked the code’ with his gamewiseness strategy. Would the game makers and question creators change their strategy for question asking?

Gamewiseness vs Testwiseness

Now let’s turn our thoughts over to testing. Because as you see, the same problem of gamewiseness exists with test-taking. It’s called “testwiseness”. Testwiseness is the ability for test-takers to use test-taking strategies to achieve a successful outcome. E.g., a test-taker might use the testing strategy of answer option elimination to deduce a correct response.

For many years, test takers have learned how to obtain successful testing outcomes using testwiseness. Testwiseness is also perpetuated by test prep companies who teach test-taking strategies to help test-takers increase their likelihood of achieving a higher test score. Like the game of Jeopardy!, there is a need to rethink our item development strategies.

Recent innovations in item development are proving successful; they not only reduce testwiseness, but they are efficacious in providing valid measures of performance. Studies (Willing et al. 2014; Papenburg, et al., 2017) show that Discrete Option Multiple Choice (DOMC) items reduce testwiseness because answer options cannot be compared in order to determine a correct response. Additionally, DOMC items on average only disclose 2.5 answer options each time an item is rendered. This reduces item exposure, a long-time nemesis of psychometricians and testing program managers alike. Lastly, because of the random correct response attribute of the DOMC item type, sharing test content becomes irrelevant; one test-taker’s correct response will likely not be the same for next test-taker.

According to Caveon CEO and founder, David Foster, “It is time for us to embrace innovation and technology and bring the multiple-choice question into the 21st century where it can evolve to address our current needs and proactively tackle the problems of the future. “

Taking tests certainly is not the same as playing a TV game show. However, testing programs must begin thinking about the use of other item strategies that negate the use of game-like strategies to gain a positive outcome. Otherwise, they are putting their testing program in Jeopardy!

Alison Foster

View all articles

About Caveon

For more than 18 years, Caveon Test Security has driven the discussion and practice of exam security in the testing industry. Today, as the recognized leader in the field, we have expanded our offerings to encompass innovative solutions and technologies that provide comprehensive protection: Solutions designed to detect, deter, and even prevent test fraud.

Topics from this blog: Exam Development Test Security Basics DOMC™