Caveon Security Insights Blog

What Is DOMC? — The Discrete Option Multiple Choice Format — Caveon

Written by Caveon | December 28, 2021 at 2:45 PM

Table of Contents

  1. What Is DOMC?
    1. How Cheating and Testwiseness Impact Multiple-Choice Items
  2. How Does the DOMC Format Work?
  3. The Benefits of DOMC
    1. Increased Fairness
    2. Decreased Exposure
    3. Decreased Development and Delivery Costs
    4. Increased Security
  4. Exploring the Evidence
    1. SuperDOMC Items
    2. Long-term Performance
    3. Item Difficulty
    4. Item Discrimination Statistics
    5. Time Savings
  5. How Can I Implement the DOMC Format?
    1. What to Know Beforehand
  6. Conclusion

What Is DOMC?

The Discrete Option Multiple Choice™ item (DOMC™) is a secure format of item design that increases fairness by eliminating the ability for examinees to use testwiseness and other test-taking skills. DOMC also promotes fairness in testing by reducing the effectiveness of item harvesting and cheating. It does both of these things by presenting answer options one at a time (rather than all at once, as is done in multiple-choice test questions). It is extremely difficult for test takers to steal test questions, share content online, or use stolen questions to cheat when only a small percentage of the options are visible to each person. You can learn more about how the DOMC item type works and presents itself in this section.

How Testwiseness Impacts Multiple-Choice Items

When taking a test that uses multiple-choice questions, have you ever gotten bogged down by the option choices? Another way to ask that might be—do you look at questions, read the options, and suddenly become confused by the juxtaposition of the varying information? Perhaps you think, “Wait, I thought I knew the answer to this question, but why would they include this option which has potential to be correct? Did I misread the question?”

If this doesn’t sound familiar, it’s because this feeling of overwhelm or confusion doesn’t happen for everyone. In fact, there’s a group of people (perhaps you’re one of them) who enjoy the multiple-choice item format for a key benefit it provides: the ability to use testwiseness

Over time (often through school environments, test prep courses, and tutoring), many folks learn a set of test-taking skills. These skills—such as skipping difficult questions and coming back to them or using the process of elimination to reduce distractors and find the correct answer—don’t translate to the content of the tests themselves. This advantage is called testwiseness.

Using testwiseness is a perfectly rule-abiding and rational way to gain a better score on an exam. Testwiseness has been shown to increase test scores on an average of 5-10% (Foster, 2018). There isn’t necessarily a problem with test takers who use these skills; the problem lies in the fact that not all test takers have these skills.

This disparity occurs for a number of reasons, including economic inequalities (some people can afford test prep, others cannot), educational differences (some schools teach these skills in depth, others do not), neurological differences (neurodiversity research shows us that many of our brains don’t process these cues in the same way), and others. 

So how can we reduce the confounding variables of testwiseness from our test scores and interpret our test takers’ abilities more accurately?

We give the multiple-choice question an accessible makeover.

How Does the DOMC Format Work?

The Discrete Option Multiple Choice question format (DOMC) represents a relatively simple but very useful change in the delivery of multiple-choice item content on computerized tests. Instead of providing all of the answer options at one time to the test taker, as is usually done, options are randomly presented one at a time along with “Yes” and “No” buttons. (Please see the example below.) For each presented option, the test taker chooses “yes” or “no” as to whether the option is perceived as correct or incorrect, respectively. When the question is answered correctly or incorrectly, additional presentations of options are rendered unnecessary.

The simplest version of the DOMC item shown above can be modified in many ways for psychometric and security purposes, just as the typical multiple-choice item can be varied for similar reasons. The following significant variation examples are discussed below.

  1. Requiring more than one “Yes” response to correct options
  2. Scoring the test item as correct by using “No” responses to incorrect options
  3. Presenting only a single option, correct or incorrect, and then scoring the test item
  4. Increasing the number of correct and incorrect options, and
  5. Presenting one or more non-scored options after the item has been scored.

The Benefits of DOMC

Fairness

DOMC removes the cueing problems plaguing multiple-choice items, and thus the unfair advantages associated with test takers who can “pay to prep.” This levels the playing field for young children, older persons, those with test anxiety, those who lack test-taking prowess, those who are testing in a non-native language, and more. These populations have found DOMC to be less complex, making assessment a simpler and more straightforward task.

Decreased Exposure

DOMC items expose options at a much lower rate compared to traditional multiple choice (by as much as 50 percent!). This makes DOMC items ineffective to steal and share with other examinees, and it allows testing programs to get the most out of their content. The more options an item has, the less the exposure rate. With the DOMC design, fewer eyes view your test content, and your test's security and usable lifespan is therefore expanded.

Development and Delivery Advantages & Costs

Because of reduced exposure and improved security, DOMC items have a longer useful lifespan. This reduces test development costs related to replacing items and redeveloping full exams. Also, because of the shortened testing length required by DOMC and the compacted presentation of options one at a time, DOMC makes testing cheaper and easier to implement using mobile devices.

Test Security

It is very difficult to steal test questions, share content online, or use stolen questions to cheat when only a small percentage of the options are visible to each test taker. Additionally, DOMC items are naturally watermarked, meaning every test taker sees a unique set of items and options. When a test taker does manage to steal and disclose questions, it is relatively easy to trace the thief to their item presentation profile using DOMC.

Exploring the Evidence

DOMC has been the subject of multiple research studies, conducted both internally (see, for example, this article and this case study) and by neutral third-party universities and organizations. The results have been overwhelmingly positive, proving that DOMC is a psychometrically sound item type that out-performs multiple-choice items in all major item and test statistics. With DOMC, tests do not have to sacrifice quality while boosting fairness and test security.

DOMC isn’t just a thought exercise or an academic inquiry; it is a proven item design that assessment programs just like yours are already using. From educational exams to medical and IT certifications (like this one, which won an award), programs are reaping the benefits of DOMC.

SuperDOMC Items

The combination of SmartItem technology and the DOMC item type is a true power combo. We actually call this “SuperDOMC,” but we’re still waiting on Webster’s dictionary to get back to us on that one. SuperDOMC items are impossible to harvest. With them, you can build an un-stealable exam. You can learn more about SuperDOMC items in this book.

Long-term Performance

It is natural to wonder whether the DOMC format will perform as well in a test as the time-honored, century-old traditional format. There are several ways to look at this concern. The first is whether items in the DOMC format can combine to produce a reliable test score. Confirming the reliability of test scores is a strong professional standard for any exam, and score reliability is a strong feature of traditional multiple-choice tests. An important study by Kingston et al. (2012) using directly comparable conditions demonstrated that reliability was not diminished by the use of DOMC and may even have been slightly enhanced. Furthermore, the researchers went on to show that using the DOMC item did not add additional complexity and was not measuring a newly introduced construct. Those results are very encouraging for using DOMC at the test level. But what is happening at the item level?

Item Difficulty

One consistent fact emerging from studies on DOMC (Foster & Miller, 2009; Kingston et al., 2012; Willing et al., 2014) is that the test becomes more difficult, reflecting the combined increase in difficulty of most of the items. The most likely explanation for this increased difficulty is that by removing the influence of test-taking skills, the items and test will demonstrate a heightened, but realistic, degree of difficulty.

Item Discrimination Statistics

A review of the item discrimination statistics, such as the point biserial correlation, provides a bit more of a mystery. Overall, the studies show that average item discrimination does not change much between DOMC and the traditional form; however, looking at individual items reveals idiosyncratic differences (Foster & Miller, 2009). That is, for a particular item, the discrimination value would be higher for DOMC, but lower for the traditional item, or the reverse might be seen. This effect is grist for the research mill and may lead to new ways of viewing how we write items, both traditional multiple choice and DOMC.

Time Savings

One research result interesting to psychometricians is that tests with DOMC items take less time to complete. The time savings have ranged from 10% (Foster & Miller, 2009) to almost 40% (Willing et al., 2014). While the reasons for these time savings are not yet well understood, the savings are intriguing, have obvious practical benefits, and may provide new insight into how test takers process test questions and provide responses.

How Can I Implement the DOMC Format?

DOMC is a patented technology, and current item banking, test publication, test administration, and test results storage systems need modification to accommodate it. You can learn more about it on this web page. If you’d like to explore linking DOMC to your systems, contact Caveon’s technology team—there are a few different test development and delivery systems that have DOMC functionality, including Caveon's testing platform, Scorpion™. If you’d like to get started playing around with creating your own DOMC items, you’re invited to create a free Scorpion account and try it yourself.

Helpful Tips Before Getting Started

Here are some helpful heads ups for you and your team as you get started with DOMC:

    • The DOMC format is unfamiliar at first. The DOMC item type is different enough from the usual multiple-choice format that it takes a bit of practice to gain familiarity for people who have seen a large number of multiple-choice test items.

    • DOMC items may not be appealing to most test takers at first, especially those who believe they have strong multiple-choice test-taking skills (Foster & Miller, 2009). Providing education on the benefits of DOMC, most of which apply directly to the test taker, can help to overcome this objection.

    • DOMC options must be clearly correct or incorrect. The DOMC format places a greater demand on the test developer to create items with options that are clearly correct or clearly incorrect (which is, of course, a standard practice even for traditional multiple-choice test items). Poor item writing is never acceptable.

    • DOMC items, like items delivered in adaptive tests and through other test designs, cannot be reviewed again during an exam. However, there is an exception to this rule; if the test taker does not respond to the item, they can mark the item for later review. It can then be revisited and answered at a later time.

Conclusion

The Discrete Option Multiple Choice format is a relatively new item design that has been created to address some of the problems with the traditional multiple-choice item type, including testwiseness and other current and emerging threats to test security. Initial studies on the DOMC item type have shown that DOMC is appropriate in many testing situations and can enhance test security while reducing long-term testing costs. Like any format, DOMC has some limitations (e.g., it is yet to be preferred by test takers); however, overall, it is found to be a promising alternative to traditional multiple-choice testing. You can view live examples of the Discrete Option Multiple Choice item here.