From the Item Bank

The Professional Testing Blog

Continuous, On-Demand Testing vs. Testing Windows

One of the first questions an organization must answer when establishing a computer-based examination program is whether they want to offer their exam continuously, on-demand or during testing windows.  Many organizations automatically default to choosing continuous, on-demand testing because of its availability and the belief that it will provide the greatest benefit to their candidate population.  However, there are many factors that must be considered when making this decision.

Definitions

The difference between testing continuously, on-demand and testing in windows is that continuously, on-demand means the candidate can schedule their computer examination any day that the computer testing center is open and has availability.  Testing windows is when the exam is only available during specific periods of times and the candidate may only schedule their examination during those windows of time.

Retake Policies

When testing in windows, the retake policy might state that a candidate who fails the exam must wait until the next “window” to sit for the exam again.  When testing continuously, on-demand it is more critical that the organization establishes a policy outlining the number of days/months a candidate must wait between exam administrations and how many times the candidate may retake the exam.  Otherwise there could be the possibility of a candidate testing over and over almost every day.

Exam Form Development

When testing continuously, on-demand the organization should have at least two forms in rotation so that a candidate who fails one form is then exposed to a second form (although this can be mitigated with a longer “sit out” period as part of the retake policy).  When testing continuously, on-demand, more items will need to be generated to minimize high item exposure rates.

Exam Authorizations

When a candidate has met all eligibility criteria, and has been approved to sit for the exam, he or she should be given the authorization to test.  When testing in windows, it’s easy for an organization to state a candidate is approved for the next “window”.  However, when testing continuously, on-demand, there should be a time frame for which someone is authorized to test.  For example, an organization might grant a candidate the authorization to test for the next 90 days.  This is to encourage candidates to actually sit for their certification exam.  Without time frames around authorizations to test, some candidates will delay taking their test indefinitely.

Item Analyses

When testing in windows, all data from each window can be analyzed at the close of the window.  Decisions about promoting items from pretest to active, decisions related to key changes, and even post-equating can all be completed at the close of a window.  When testing continuously, on-demand, a schedule must be established for analyzing item- and form-level data since exam administrations don’t stop and start at specific points in time.

Exam Applications

When testing in windows, exam applications tend to come in waves (a lot of applicants in the few weeks prior to an exam window).  However, when testing continuously, on-demand, exam applications are more likely to be spread out across time.

Availability

When testing in windows, there are less days of availability at computer test centers and during some peak times such as summer months, seats may be limited.  Windows testing might require the windows to be selected during less popular times depending on the candidate volume.  However, when testing continuously, on-demand this ceases to be a problem since there are more days available.

Conclusions

The decision of whether to offer computer-based examinations continuously, on-demand or in windows is a decision that should be made in conjunction with your psychometric consultant and your computer testing vendor.  By working closely with both of these experts, the pros and cons of both modes of computer administration can be discussed (including many of the issues mentioned above) and a solution that best meets the candidates’ and organization’s needs can be selected.

Additional Posts

For other more articles on options for your examination program check out Key Considerations for Translating Examinations and Pitfalls to Avoid and Is it Time to Drop Option D? Considerations for Choosing the Three-Option Multiple Choice Item.

Categorized in: Test Development