From the Item Bank

The Professional Testing Blog

Selecting the Right Rating Scales for Evaluating Job Analysis Tasks

The first step in developing a credentialing exam is to conduct a job/task analysis (JTA).  A job/task analysis refers to the systematic process of defining all job duties and tasks, knowledge, skills, and abilities (collectively referred to as KSAs), and other characteristics (e.g., working conditions, job responsibilities) associated with a job or position.  This is done by a panel of individuals who have intimate knowledge of the job. Typically, a job/task analysis is followed by a large-scale validation survey, the purpose of which is to validate the job tasks identified during the JTA.

The validation survey can be thought of as having two sections.  The first is a demographic section to ensure that those who have responded to the survey are representative of the population of interest.  The second section asks participants to rate job tasks on one or more rating scales.  According to many job analysts, the most commonly used rating scales for evaluating job tasks for a credentialing exam are:

·         Need at Entry

·         Task Criticality or Consequence of Error

·         Task Frequency

·         Task Importance

It is extremely important that the right rating scales are chosen, as the choice in rating scale will greatly affect the results.  If survey respondents do not clearly understand the rating scale they will not provide accurate responses to the survey.  Additionally, the more rating scales used, the longer it will take a participant to respond to your survey, and the more likely that participant is to not finish the survey.  In order to choose the right rating scale for a validation survey, one should have a fairly good understanding of the types of rating scales typically used.

The Need at Entry rating scale refers to the extent to which an entry-level individual should be able to perform the task.  It is argued that this may be one of the most important rating scales to use when evaluating a job task for inclusion on a credentialing exam, as only tasks (and underlying knowledge, skills, and abilities) that are required at entry-level should be included on a credentialing exam.

The Task Criticality or Consequence of Error rating scale refers to the risk or adverse consequence of performing a task incorrectly or not performing the task at all.  The typical purpose of licensure is to protect public health, safety, and welfare. When developing a credentialing exam for licensure, tasks that are critical to public health, safety, and welfare should have the greatest emphasis.

The Task Frequency rating scale refers to the frequency with which a task is performed.  Kane (1982) stated that in a credentialing exam, a greater emphasis should be given to tasks that are performed most frequently.  He uses the example of creating a credentialing exam for General Practitioners (GPs).  If one were to develop a credentialing exam for GPs, one would want to put a greater emphasis on treating those diseases that are encountered more frequently (e.g., heart disease, diabetes, and cancer) than those encountered less frequently (e.g., tropical diseases – outside the tropics).

Finally, the Task Importance rating scale refers to the relative importance of knowing how to or being able to perform a task.  It has historically been considered the most crucial scale when evaluating job tasks.  Even the Standards for Educational and Psychological Testing specifically state “the content domain to be covered by a credentialing test should be defined clearly and justified in terms of the importance [emphasis added] of the content for credential-worth performance in an occupation or profession” (AERA, NCME, APA, 2014, p.181).

When determining which of these tasks may be most appropriate for your validation study, the profession or job for which you conducted the initial JTA may be the biggest determining factor.  For example, if one were conducting a JTA for food safety managers, Task Criticality or Consequence of Error may be the most suitable rating scale.  Food safety managers should be able to easily evaluate the consequence of error for performing some food safety tasks incorrectly or not at all.  Conversely, if one were conducting a JTA for digital media salespeople, asking survey participants to rate the consequence of error for performing a task incorrectly or not all wouldn’t have the same meaning as asking participants to rate the overall importance of each task to his or her job.

For more information on the rating scales used to evaluate job tasks, please refer to the following articles:

Cadle, A.W.  (2012). The relationship between rating scales used to evaluate tasks from task inventories for licensure and certification examinations (Doctoral dissertation).  Retrieved from ProQuest database.  (UMI No. 3547749)

Knapp, J., & Knapp, L. (1995). Practice analysis: Building the foundation for validity. In J.C. Impara (Ed.), Licensure Testing: Purposes, procedures, and practices (pp. 93-116).

Manson, T.M., Levine, E.L., & Brannick, M.T. (2000). The construct validity of task inventory ratings: A multitrait-multimethod analysis. Human Performance, 13(1), 1-22.

Newman, L.S., Slaughter, R.C., & Taranath, S.N. (1999, April). The selection and use of rating scales in task surveys: A review of current job analysis practice. Paper presented at the annual meeting of the National Council of Measurement in Education, Montreal, Canada.

Raymond, M.R. (2002). A practical guide to practice analysis for credentialing examinations. Educational Measurement: Issues and Practice, 21, 25-37.

Raymond, M.R. (2005). An NCME instructional module on developing and administering practice analysis questionnaires. Educational Measurement: Issues and Practice, 24, 29-42. doi: 10.1111/j.1745-3992.2005.00009

Categorized in: Industry News