From the Item Bank
The Professional Testing Blog
Tools of the TradeFebruary 15, 2017 | | Leave a comment
What is the role of a psychometrician and software in the development and maintenance of a certification examination program?
As another Psychometric trade conference approaches, we receive many advertisements for revolutionary software that will assist certification bodies, reduce their costs and run the necessary reports to assist with making exam program validation claims. These ads are intriguing and some of the software is very good.
With that said, and given the high-stakes nature of our business, psychometric results should be reviewed by someone with experience. Such interpretations are critical to the constant improvement of a testing program. Interpretations also help identify threats to a testing program. A psychometrician will help a certification body understand the data, interpret results, and apply findings in a way that benefits the examination within the context of the overall program.
A good example of this is the interpretation of item and test statistics. Many of us are familiar with basic item statistics such as p-values (% correctly answering an item correctly) and item discrimination indices (a statistic indicating an item’s ability to differentiate low ability candidates from higher ability candidates). There are many factors that affect these statistics and these statistics affect other facets of an exam’s performance. Below is just a sampling of factors to consider when reviewing item and exam statistics:
Much of the value in data and statistics is how it is used within the parameters of a program—the pool of candidates, scoring decisions, volume of test-takers, programmatic goals etc. The value of a psychometrician is understanding the characteristics of a certification body and its program elements, and will guide their clients in the right direction. Using data and statistical analyses in a meaningful way is a useful tool in mitigating threats to validity and strengthening exam programs.
The Standards for educational and psychological testing (2014) developed by the American Educational Research Association, American Psychological Association and The National Council on Measurement in Education provide this general definition: “Validity refers to the degree to which evidence and theory support the interpretation of test scores for proposed uses of test.” The standards have over a dozen chapters providing guidance on process and empirical evidence that should be collected to help make validity claims. Empirical evidence must be interpreted correctly. Empirical evidence is only part of the validation process. Correctly interpreting the data in the validation process is what makes a validation strategy meaningful.
Tags: conference, psychometrician, software, statistics, validity
Categorized in: Test Development