The Discrete Option Multiple Choice™ item (DOMC™) is a secure format of item design that increases fairness by eliminating the ability for examinees to use testwiseness and other test-taking skills. DOMC also promotes fairness in testing by reducing the effectiveness of item harvesting and cheating. It does both of these things by presenting answer options one at a time (rather than all at once, as is done in multiple-choice test questions). It is extremely difficult for test takers to steal test questions, share content online, or use stolen questions to cheat when only a small percentage of the options are visible to each person. You can learn more about how the DOMC item type works and presents itself in this section.
When taking a test that uses multiple-choice questions, have you ever gotten bogged down by the option choices? Another way to ask that might be—do you look at questions, read the options, and suddenly become confused by the juxtaposition of the varying information? Perhaps you think, “Wait, I thought I knew the answer to this question, but why would they include this option which has potential to be correct? Did I misread the question?”
If this doesn’t sound familiar, it’s because this feeling of overwhelm or confusion doesn’t happen for everyone. In fact, there’s a group of people (perhaps you’re one of them) who enjoy the multiple-choice item format for a key benefit it provides: the ability to use testwiseness.
Over time (often through school environments, test prep courses, and tutoring), many folks learn a set of test-taking skills. These skills—such as skipping difficult questions and coming back to them or using the process of elimination to reduce distractors and find the correct answer—don’t translate to the content of the tests themselves. This advantage is called testwiseness.
Using testwiseness is a perfectly rule-abiding and rational way to gain a better score on an exam. Testwiseness has been shown to increase test scores on an average of 5-10% (Foster, 2018). There isn’t necessarily a problem with test takers who use these skills; the problem lies in the fact that not all test takers have these skills.
This disparity occurs for a number of reasons, including economic inequalities (some people can afford test prep, others cannot), educational differences (some schools teach these skills in depth, others do not), neurological differences (neurodiversity research shows us that many of our brains don’t process these cues in the same way), and others.
So how can we reduce the confounding variables of testwiseness from our test scores and interpret our test takers’ abilities more accurately?
We give the multiple-choice question an accessible makeover.
The Discrete Option Multiple Choice question format (DOMC) represents a relatively simple but very useful change in the delivery of multiple-choice item content on computerized tests. Instead of providing all of the answer options at one time to the test taker, as is usually done, options are randomly presented one at a time along with “Yes” and “No” buttons. (Please see the example below.) For each presented option, the test taker chooses “yes” or “no” as to whether the option is perceived as correct or incorrect, respectively. When the question is answered correctly or incorrectly, additional presentations of options are rendered unnecessary.
The simplest version of the DOMC item shown above can be modified in many ways for psychometric and security purposes, just as the typical multiple-choice item can be varied for similar reasons. The following significant variation examples are discussed below.
DOMC removes the cueing problems plaguing multiple-choice items, and thus the unfair advantages associated with test takers who can “pay to prep.” This levels the playing field for young children, older persons, those with test anxiety, those who lack test-taking prowess, those who are testing in a non-native language, and more. These populations have found DOMC to be less complex, making assessment a simpler and more straightforward task.
DOMC items expose options at a much lower rate compared to traditional multiple choice (by as much as 50 percent!). This makes DOMC items ineffective to steal and share with other examinees, and it allows testing programs to get the most out of their content. The more options an item has, the less the exposure rate. With the DOMC design, fewer eyes view your test content, and your test's security and usable lifespan is therefore expanded.
Because of reduced exposure and improved security, DOMC items have a longer useful lifespan. This reduces test development costs related to replacing items and redeveloping full exams. Also, because of the shortened testing length required by DOMC and the compacted presentation of options one at a time, DOMC makes testing cheaper and easier to implement using mobile devices.
It is very difficult to steal test questions, share content online, or use stolen questions to cheat when only a small percentage of the options are visible to each test taker. Additionally, DOMC items are naturally watermarked, meaning every test taker sees a unique set of items and options. When a test taker does manage to steal and disclose questions, it is relatively easy to trace the thief to their item presentation profile using DOMC.
DOMC has been the subject of multiple research studies, conducted both internally (see, for example, this article and this case study) and by neutral third-party universities and organizations. The results have been overwhelmingly positive, proving that DOMC is a psychometrically sound item type that out-performs multiple-choice items in all major item and test statistics. With DOMC, tests do not have to sacrifice quality while boosting fairness and test security.
DOMC isn’t just a thought exercise or an academic inquiry; it is a proven item design that assessment programs just like yours are already using. From educational exams to medical and IT certifications (like this one, which won an award), programs are reaping the benefits of DOMC.
The combination of SmartItem technology and the DOMC item type is a true power combo. We actually call this “SuperDOMC,” but we’re still waiting on Webster’s dictionary to get back to us on that one. SuperDOMC items are impossible to harvest. With them, you can build an un-stealable exam. You can learn more about SuperDOMC items in this book.
It is natural to wonder whether the DOMC format will perform as well in a test as the time-honored, century-old traditional format. There are several ways to look at this concern. The first is whether items in the DOMC format can combine to produce a reliable test score. Confirming the reliability of test scores is a strong professional standard for any exam, and score reliability is a strong feature of traditional multiple-choice tests. An important study by Kingston et al. (2012) using directly comparable conditions demonstrated that reliability was not diminished by the use of DOMC and may even have been slightly enhanced. Furthermore, the researchers went on to show that using the DOMC item did not add additional complexity and was not measuring a newly introduced construct. Those results are very encouraging for using DOMC at the test level. But what is happening at the item level?
One consistent fact emerging from studies on DOMC (Foster & Miller, 2009; Kingston et al., 2012; Willing et al., 2014) is that the test becomes more difficult, reflecting the combined increase in difficulty of most of the items. The most likely explanation for this increased difficulty is that by removing the influence of test-taking skills, the items and test will demonstrate a heightened, but realistic, degree of difficulty.
A review of the item discrimination statistics, such as the point biserial correlation, provides a bit more of a mystery. Overall, the studies show that average item discrimination does not change much between DOMC and the traditional form; however, looking at individual items reveals idiosyncratic differences (Foster & Miller, 2009). That is, for a particular item, the discrimination value would be higher for DOMC, but lower for the traditional item, or the reverse might be seen. This effect is grist for the research mill and may lead to new ways of viewing how we write items, both traditional multiple choice and DOMC.
One research result interesting to psychometricians is that tests with DOMC items take less time to complete. The time savings have ranged from 10% (Foster & Miller, 2009) to almost 40% (Willing et al., 2014). While the reasons for these time savings are not yet well understood, the savings are intriguing, have obvious practical benefits, and may provide new insight into how test takers process test questions and provide responses.
DOMC is a patented technology, and current item banking, test publication, test administration, and test results storage systems need modification to accommodate it. You can learn more about it on this web page. If you’d like to explore linking DOMC to your systems, contact Caveon’s technology team—there are a few different test development and delivery systems that have DOMC functionality, including Caveon's testing platform, Scorpion™. If you’d like to get started playing around with creating your own DOMC items, you’re invited to create a free Scorpion account and try it yourself.
Here are some helpful heads ups for you and your team as you get started with DOMC:
The Discrete Option Multiple Choice format is a relatively new item design that has been created to address some of the problems with the traditional multiple-choice item type, including testwiseness and other current and emerging threats to test security. Initial studies on the DOMC item type have shown that DOMC is appropriate in many testing situations and can enhance test security while reducing long-term testing costs. Like any format, DOMC has some limitations (e.g., it is yet to be preferred by test takers); however, overall, it is found to be a promising alternative to traditional multiple-choice testing. You can view live examples of the Discrete Option Multiple Choice item here.
WE'RE HERE TO HELP.
