Table of Contents
- Definition of Automated Item Generation
- How AIG Works
- The Benefits of AIG
- How AIG Impacts the Security of Exams
- How AIG Saves Time and Money
- How to Implement and Utilize AIG
- Conclusion
Definition of Automated Item Generation (AIG)
Automated Item Generation (AIG) is a process that leverages test item templates and computer algorithms to quickly create a large variety of item permutations, or test items. The result is hundreds or thousands of new test questions based on one single item model.
As the name suggests, AIG automates much of the effort involved in item creation—which is one the most time-intensive and costly aspects of test development. AIG helps to maintain and improve test quality in a world where online braindumps and content sharing platforms disclose test content and threaten test validity. With AIG, stolen test questions can be replaced quickly, and the damage that shared test content can inflict on assessment programs is palliated.
How It Works
While many of the negative effects of cheating and content theft can be mitigated by a large enough item bank, it isn’t feasible for testing programs to dedicate teams of item writers, psychometricians, and subject matter experts to writing thousands of items. AIG technology is a great alternative for producing large numbers of high-quality test items. For more information, you can view a large body of scholarly research on the topic of AIG, including in this article by Gierl, Lai, and Turner (2012). While there are variations, the steps of automated item generation are generally as follows:
- Build a Template: Thousands of unique items start with the construction of a strong item template or model. These can be easily created with app-based graphical user interfaces.
- Add Data: Next, populate your item template with the necessary data and variables.
- Generate: Once you're done designing your item, select the number of items you want to generate. An algorithm then turns this template into a family of related items, often by producing all possible permutations. The system should do the hard work for you.
- Review and Test: Next, subject matter experts review and test the generated items through the exact same process as they would with manually-written items. This review process is done to ensure the psychometric quality and cohesiveness of the items.
- Import: Next, upload your approved items to your own item banking or delivery systems, and deploy.
The Benefits
AIG has many short- and long-term benefits for testing organizations. From a bird’s eye view, the overall benefits of AIG include:
- Security: AIG helps rapidly increase the size of your item pool, reducing item exposure and keeping you ahead of content thieves and aging exams.
- Cost Savings: AIG is the most cost-effective way to rapidly expand your items—saving development costs and security costs while making your item pools last longer. You can learn more in this section.
- Optimize your SME’s Expertise: AIG can make your best resources (your subject matter experts and item writers) more effective, so you can get the most out of your most valuable people.
- Increased Creativity: AIG will allow your team the freedom to get creative with your item development process by automating the redundant tasks, leaving your team with more hours to put toward item design.
- Accuracy: Reduce inevitable human error by using AIG to generate your new items. Save your brainpower for the review process to cut down on mistakes.
- Increased Options for Your Item and Test Design: Certain test designs like CATs (Computerized Adaptive Tests), multiple forms, & LOFTs require large item banks. AIG makes these possible.
How Does AIG Boost Security?
If security is your testing program’s priority (and it should be), then AIG has a lot to offer. An expanded item pool has many added benefits that impact overall test security endeavors:
- It limits item exposure: More items in your item pool means fewer eyes see each item. Limit item exposure by rapidly expanding your item pool through AIG.
- It reduces the effect of item theft: To combat item theft, it is important to continually republish new item content. AIG makes it possible to refresh your item pool as often as necessary.
- It stops pre-knowledge: A larger pool of items can prevent test takers from using pre-knowledge by making it harder for them to predict which test question they'll see.
- It limits the number of people who see and access your item bank: Maintain an extra layer of security by involving fewer individuals in the item writing process. AIG gives smaller teams the tools to be more efficient.
- It reduces human error: You can maintain an extra layer of security by involving fewer individuals in the item writing process.
How Does AIG Save Time and Money?
AIG reduces development costs, increases the productivity of subject matter experts, and eliminates the expense required from redevelopment and republishing after content exposure. And those ripple into other benefits:
- It Reduces Development Costs: The boost in productivity when using AIG affords many programs the luxury of reducing the human resources necessary to create a set amount of items.
- It Eliminates expense required by redeveloping and republishing exams after they have been exposed: The enhanced security benefits of a larger item pool mean fewer security and redevelopment costs in the long run.
- It Increases productivity of your SMEs: How long would it take a team of item writers to craft 400 unique items? Using AIG, that team could potentially develop twice that many at the push of a button. This will allow you to use your experts’ time toward more pressing and important tasks.
Here is an infographic that illustrates the lifetime cost of AIG compared to the cost of traditional item development (click to enlarge):
How Can I Implement Automated Item Generation?
AIG has been used by large testing programs for many years. However, the plausibility of AIG has been out of reach for smaller testing programs with limited budgets and slimmer teams. Caveon AIG aims to change that. Available through Scorpion™, Caveon’s state-of-the-art AIG tools are built right into the software. Caveon AIG also utilizes SmartItem™ technology for even greater test security and item protection. Since AIG tools aren't all created equal, it’s important you put time and effort into researching the best AIG tool for your specific program. Look for AIG tools that include the following features:
- An easy-to-use GUI: Many AIG tools are difficult to use and require coding and time-consuming technical tasks to generate items. Look for a drag-and-drop item building software with easy-to-use navigation that makes item generation a breeze. Few tools offer automated item generation at the push of a button, but they do exist. Find a software your team will have an easier time implementing. Then, they can utilize their valuable time toward more innovative item design and ensuring the psychometric quality of your items.
- Integration with any development and delivery system you use: It’s difficult and time-consuming to accommodate a different development or delivery system than your team is used to. Look for an AIG system with items that are Question Test Interoperability (QTI) 2.2 compliant, meaning you can easily transport items into your pre-existing systems. Having an AIG tool that works with your existing development and delivery system enables your team to spend more time generating sound items and less bouncing around how-to articles.
- Compatibility with any item design: What kinds of items do you want to generate? Make sure the AIG tool you purchase enables you to choose from all kinds of item types. Item types to look out for include drag-and-drop, multiple-choice, DOMC™, matching, SmartItem, and others.
- Psychometric transparency features: Make sure the automated item generation tool allows you to easily psychometrically analyze your items. You should be able to review the complete batch, test them, and then make adjustments to your item design as needed. The information should be in your hands at all stages.
Conclusion
Automated item generation leverages test item templates and computer algorithms to quickly create hundreds or thousands of new item permutations based on one single item model.
AIG has many short- and long-term benefits for testing organizations. These benefits include heightened security by increasing item pool size and reducing item exposure; cost savings through reduced redevelopment costs and increased item pool lifespan; time savings for subject matter experts and item creators; access to certain test designs including CATs and LOFTs that require large item banks; and more.
AIG is becoming more accessible for smaller testing programs and programs with limited resources, but not all AIG tools are created equal. It is important to find an AIG tool that is easy to use and offers psychometric transparency, increased item design capability, and ample software integration options. Choose the software that best aligns with the needs of your test program
