Test Topics
What are important aspects of converting an exam to computer-based testing?
Introduction
Many established certification and licensure programs are choosing to convert their exams from paper-and-pencil test programs to computer-based test (CBT) programs. However, it should be kept in mind that computer-based testing is not right for all exam programs, and converting to a CBT program will require substantial resources. For example, considerable effort is usually needed to prepare the item bank. To ensure a successful conversion, any certification or licensure program that decides to convert to CBT is well advised to develop a conversion plan. This plan should include decisions for the CBT program about such matters as the optimal method for test delivery, the preferred approach to scheduling test administrations, the essential requirements for CBT sites and software, and the most effective methods for addressing test security. Another important step in the conversion will be contracting with a CBT vendor. Finally, preparations for informing stakeholders about the conversion will be needed, to ensure a smooth transition. Due to these and other critical aspects of converting to computer-based testing, many programs find it worthwhile to first conduct a feasibility analysis, to determine if a CBT is right for them.
- CBT Resource Needs
- CBT Item Banks
- CBT Delivery Methods
- CBT Scheduling Approaches
- CBT Sites and Software
- CBT Security Considerations
- Selecting a CBT Vendor
- Informing CBT Stakeholders
- CBT Feasibility Analysis
- Summary
When developing a plan for converting an exam program to computer-based testing, it is important to recognize that a CBT program usually costs more than a paper-and-pencil exam program. While certain costs associated with paper-and-pencil testing are removed, such as shipping costs for test booklets and the rental of test administration locations, the overall CBT expenses will still typically be higher. The additional expenses for computer-based testing include specific costs associated with the conversion itself, as well as additional continuing costs associated with maintaining a CBT program. [more]
The conversion of a paper-and-pencil test to a CBT often requires a substantial effort in order to prepare the item bank. First of all, CBT programs usually require a larger number of test items, due to the greater number of test administration dates typical of CBT programs. The item bank must usually be supplemented so that a sufficient number of items, both within specific sub-content areas and overall, is available. A second aspect of preparing the item bank for computer-based testing relates to the need to transfer the items from the current storage system into appropriate CBT software. Depending on specific characteristics of the exam program, these two tasks can be considerably time consuming and expensive to accomplish. Nevertheless, they are essential for the successful functioning of the CBT. [more]
A test that is administered on computer might be delivered through any of several methods. The most common delivery method for CBT programs is linear, or fixed CBT. This approach to test delivery is the method that is most similar to paper-and-pencil testing; a fixed set of items is administered to examinees, in a fixed order. A variation on the fixed CBT method is the random CBT method. The random CBT method also administers a fixed set of items; however, in this method the order of the items is randomized, to provide a modest test security benefit. A very different approach to test delivery is offered in the adaptive CBT method, also known as the computer adaptive test, or CAT approach. In a CAT, items are individually selected for the examinee, based on his or her performance on earlier items in the test. While an adaptive CBT can result in a shorter test, it is more challenging to develop and can only be used by exam programs that have a large number of examinees. [more]
When an exam program is being converted from paper-and-pencil to computer-based testing, one decision that needs to be made is whether to maintain the current test administration schedule, or to change it in some fashion. Some of the options for CBT scheduling include fixed dates, testing windows, continuous testing, and event testing. The scheduling approach selected will impact various aspects of the exam program, including: the frequency of examinee access; the need for additional items; and the structure of organizational administration. Two of these approaches, fixed dates and event testing, may provide an exam program with an administration schedule that remains similar to the former paper-and-pencil schedule. The other approaches to scheduling, testing windows and continuous testing, usually provide increased numbers of administration dates. The greater frequency of administration dates typically provides examinees with increased access and scheduling flexibility, which most examinees find to be highly beneficial. However, an increased number of test administration dates also results in additional test security challenges for the exam program, which must be addressed. [more]
A CBT must be transmitted to the examinee's computer at a specific location, and then displayed through a specific software application. The physical location of the computer on which an examinee will take the exam, in most cases, will be a CBT center or site. Other physical locations for CBT administration are possible, including unsecured locations such as an examinee's home. The software application that is used to transmit the electronic test data to the examinee's computer may also be used to display the test items on the screen. Alternatively, a browser application may be used for displaying the test items. Both the physical location of the CBT administration and the software used to display the items have considerable implications for the important test administration issues of standardization and security. [more]
Test security is an important element of any exam program, whether the exam is administered as a paper-and-pencil test or as a CBT. However, the relative risks of test security violations differ for paper-and-pencil exam programs and CBT programs. Furthermore, the ways in which certain aspects of security are addressed also differ across these two administration modes. The primary aspects of security which are different for the two administration modes include: the procedures used to identify examinees, the risks of examinees cheating on an exam, the methods employed for delivering tests to examinees, and the number of test administration dates offered. [more]
Most credentialing exams that are administered as CBTs are published and delivered in conjunction with a CBT vendor. These vendors are companies that specialize in the delivery of computer-based tests. In most cases, the vendors maintain CBT software as well as a network of CBT administration sites. Exam program staff might decide which CBT vendor to select by evaluating them in terms of relevant criteria. One consideration is the number and location of the CBT sites maintained by the vendor, as well as the availability of those sites within exam program time frames. Another consideration is the CBT software itself. CBT software varies in terms of characteristics such as: the item types supported, the security provisions included, the item banking functionality, and the overall ease of use of the software. Finally, the fee structure and contract requirements established by the vendor should also be considered. In one type of fee structure, a vendor might require an exam program to guarantee a minimum number of examinees or to guarantee a minimum revenue. In another fee structure, a vendor might specify higher set-up fees, but lower per-examinee charges. The relative offerings of each CBT vendor should be considered in light of the specific exam program, in order to identify the best match to program needs.
One component of preparing to convert an exam program to computer-based testing is the development of a plan for informing stakeholders about the relevant issues. A variety of forums can be used for communicating with stakeholders. These include press releases, articles or ads in journals for the profession, educational presentations at conferences, and informational materials on the test sponsor's website. The content of these informational materials can range from technical details about the CBT methods in use to testimonials from early CBT users. For a given exam program important stakeholders might include test sponsors, test users, the regulatory community, and the training community. A primary audience for informational material will be the exam program's candidates. Candidates should be informed about any aspects of CBT they might find beneficial, such as increased flexibility in test scheduling and quicker score reporting. An emphasis on these advantages may help offset examinee objections to any increased test fees. Candidates should also be kept informed about transition dates leading to the full conversion. Finally, a practice test should be made available, either over the internet or at the CBT administration sites, so that candidates will have the opportunity to practice with the actual CBT software prior to taking the test. The goal in all of these communication efforts is to promote a smooth transition by keeping stakeholders informed about the conversion.
Exam program staff may elect to undertake a feasibility analysis, prior to making the decision to convert to computer-based testing. This type of preliminary study can provide useful information about whether or not converting to a CBT program is the right choice, and whether or not the exam program is ready to make that move at the present time. A feasibility analysis might include surveys to obtain perspectives from stakeholder groups such as candidates, Board members, training centers, and practitioners. The survey of exam program candidates could address whether the candidates appear to be ready for computer-based testing, and whether they value the advantages of a CBT program enough to willingly pay an increased fee. A feasibility study might also include a careful analysis of the impact that converting to computer-based testing is likely to have on exam program finances. This costing analysis might address both the direct impact on expenses for the exam program, as well as a potential loss of revenue should the number of examinees decrease after the conversion. The feasibility analysis might also consider psychometric issues, such as whether the item bank can easily be made ready for computer-based testing. The level of psychometric challenges for an exam program may be related to decisions about delivery method, scheduling approach, and security needs. If the feasibility analysis should indicate that CBT is a good choice for the exam program, information collected during the study might also be useful when developing the conversion plan. For example, information collected during the feasibility analysis could help in establishing an optimal time frame for the conversion process, by ensuring that test security is fully maintained, and by identifying key organizational changes that will be need to provide program support.
Computer-based testing has many advantages, both for the credentialing program and for the examinees. However, it also has certain disadvantages, including greater demands on program resources. Prior to making the decision to convert an exam program to computer-based testing, it may be useful to evaluate the potential match between characteristics of CBTs and the exam program. This evaluation might be conducted through a feasibility analysis. A smooth and successful transition to a CBT program will be further aided by a conversion plan that emphasizes thorough preparation and careful decisions. Preparations that are likely to be needed include supplementing the item bank, revising the organizational structure, and training internal staff. Decisions that will need to be made include the test delivery method to apply, the scheduling approach to establish, and the CBT vendor to use. Even with careful planning, a CBT program is likely to be more expensive than a paper-and-pencil test program. However, for some exam programs the advantages provided by computer-based testing sufficiently outweigh the additional resources required to be worth the effort of conversion and the costs of maintenance.