Co-authored by: Reed Castle, Ph.D. and Vincent Lima

Certification examination programs often have more than one examination form (set of questions, or items) in use at a time, and they require routine updates. The exam forms represent the same content, but have different items. As new items come into use, they may be easier or harder than the first set of items. As a result, exam forms can have varying degrees of difficulty. These variations need to be taken into account when reporting scores and pass/fail decisions over time and exam forms.

Reed Castle, Ph.D.

The primary goal of any testing program is to treat test takers in a fair and equitable manner and to assure that inferences made from an exam score are appropriate. To do so, one component is to maintain proper scoring and reporting procedures. 

There are many philosophical debates as to what constitutes validity or the validity of test score inferences. Within certification we hear things like “a job analysis is typically conducted every five years unless the content changes more or less frequently.” You may have also heard that more than eight SMEs are required for test development activities, while others suggest six or more is sufficient. Some suggest that SMEs should not overlap with separate test development meetings.  So in essence psychometricians will typically say “it depends” in response to questions about the appropriate way to design a certification examination program.

Computer Adaptive Testing (CAT) is a testing methodology that weds two processes—adaptive testing and computer administration—for efficient measurement and administration. When compared to fixed length exams delivered linearly on a computer, CAT exams measure a candidate’s ability with fewer delivered exam questions and higher precision. By precision, it is suggested that error is reduced and reliability is increased.