{"id":155,"date":"2016-01-14T11:01:27","date_gmt":"2016-01-14T11:01:27","guid":{"rendered":"http:\/\/proftesting.com\/blog\/?p=155"},"modified":"2016-01-29T13:42:56","modified_gmt":"2016-01-29T13:42:56","slug":"selecting-the-right-rating-scales-for-evaluating-job-analysis-tasks","status":"publish","type":"post","link":"https:\/\/www.proftesting.com\/blog\/2016\/01\/14\/selecting-the-right-rating-scales-for-evaluating-job-analysis-tasks\/","title":{"rendered":"Selecting the Right Rating Scales for Evaluating Job Analysis Tasks"},"content":{"rendered":"

The first step in developing a credentialing exam is to conduct a job\/task analysis (JTA<\/a>).\u00a0\u00a0A job\/task analysis refers to the systematic process of defining all job duties and tasks, knowledge, skills, and abilities (collectively referred to as KSAs), and other characteristics (e.g., working conditions, job responsibilities) associated with a job or position.\u00a0 This is done by a panel of individuals who have intimate knowledge of the job. Typically, a job\/task analysis is followed by a large-scale validation survey, the purpose of which is to validate the job tasks identified during the JTA.<\/p>\n

\"Selecting<\/p>\n

 <\/p>\n

The validation survey can be thought of as having two sections.\u00a0 The first is a demographic section to ensure that those who have responded to the survey are representative of the population of interest.\u00a0 The second section asks participants to rate job tasks on one or more rating scales.\u00a0 According to many job analysts<\/a>,\u00a0the most commonly used rating scales for evaluating job tasks for a credentialing exam are:<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Need at Entry<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Task Criticality or Consequence of Error<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Task Frequency<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Task Importance<\/p>\n

 <\/p>\n

It is extremely important that the right<\/em> rating scales are chosen, as the choice in rating scale will greatly affect the results.\u00a0 If survey respondents do not clearly understand the rating scale they will not provide accurate responses to the survey.\u00a0 Additionally, the more rating scales used, the longer it will take a participant to respond to your survey, and the more likely that participant is to not finish the survey.\u00a0 In order to choose the right rating scale for a validation survey, one should have a fairly good understanding of the types of rating scales typically used.<\/p>\n

The Need at Entry rating scale refers to the extent to which an entry-level individual should be able to perform the task.\u00a0 It is argued that this may be one of the most important rating scales to use when evaluating a job task for inclusion on a credentialing exam<\/a>, as only tasks (and underlying knowledge, skills, and abilities) that are required at entry-level should be included on a credentialing exam.<\/p>\n

The Task Criticality or Consequence of Error rating scale refers to the risk or adverse consequence of performing a task incorrectly or not performing the task at all.\u00a0 The typical purpose of licensure is to protect public health, safety, and welfare. When developing a credentialing exam for licensure, tasks that are critical <\/em>to public health, safety, and welfare should have the greatest emphasis.<\/p>\n

The Task Frequency rating scale refers to the frequency with which a task is performed.\u00a0 Kane (1982) stated that in a credentialing exam, a greater emphasis should be given to tasks that are performed most frequently.\u00a0 He uses the example of creating a credentialing exam for General Practitioners (GPs).\u00a0 If one were to develop a credentialing exam for GPs, one would want to put a greater emphasis on treating those diseases that are encountered more frequently (e.g., heart disease, diabetes, and cancer) than those encountered less frequently (e.g., tropical diseases \u2013 outside the tropics).<\/p>\n

Finally, the Task Importance rating scale refers to the relative importance of knowing how to or being able to perform a task.\u00a0 It has historically been considered the most crucial scale when evaluating job tasks.\u00a0 Even the Standards for Educational and Psychological Testing <\/em>specifically state \u201cthe content domain to be covered by a credentialing test should be defined clearly and justified in terms of the importance<\/em> [emphasis added] of the content for credential-worth performance in an occupation or profession\u201d (AERA, NCME, APA, 2014, p.181).<\/p>\n

When determining which of these tasks may be most appropriate for your validation study, the profession or job for which you conducted the initial JTA may be the biggest determining factor.\u00a0 For example, if one were conducting a JTA for food safety managers, Task Criticality or Consequence of Error may be the most suitable rating scale.\u00a0 Food safety managers should be able to easily evaluate the consequence of error for performing some food safety tasks incorrectly or not at all.\u00a0 Conversely, if one were conducting a JTA for digital media salespeople, asking survey participants to rate the consequence of error for performing a task incorrectly or not all wouldn\u2019t have the same meaning as asking participants to rate the overall importance of each task to his or her job.<\/p>\n

For more information on the rating scales used to evaluate job tasks, please refer to the following articles:<\/p>\n

Cadle, A.W.\u00a0 (2012). The relationship between rating scales used to evaluate tasks from task inventories for licensure and certification examinations<\/em> (Doctoral dissertation).\u00a0 Retrieved from ProQuest database<\/a>.\u00a0 (UMI No. 3547749)<\/p>\n

Knapp, J., & Knapp, L. (1995). Practice analysis: Building the foundation for validity. In J.C. Impara (Ed.), Licensure Testing: Purposes, procedures, and practices<\/em> (pp. 93-116).<\/a><\/p>\n

Manson, T.M., Levine, E.L., & Brannick, M.T. (2000). The construct validity of task inventory ratings: A multitrait-multimethod analysis. Human Performance<\/em>, 13(1), 1-22.<\/a><\/p>\n

Newman, L.S., Slaughter, R.C., & Taranath, S.N. (1999, April). The selection and use of rating scales in task surveys: A review of current job analysis practice<\/em>. Paper presented at the annual meeting of the National Council of Measurement in Education, Montreal, Canada.<\/p>\n

Raymond, M.R. (2002). A practical guide to practice analysis for credentialing examinations. Educational Measurement: Issues and Practice<\/em>, 21, 25-37.<\/a><\/p>\n

Raymond, M.R. (2005). An NCME instructional module on developing and administering practice analysis questionnaires. Educational Measurement: Issues and Practice<\/em>, 24, 29-42. doi: 10.1111\/j.1745-3992.2005.00009<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"The first step in developing a credentialing exam is to conduct a job\/task analysis (JTA).\u00a0\u00a0A job\/task analysis refers to the systematic process of defining all job duties and tasks, knowledge, skills, and abilities (collectively referred to as KSAs), and other characteristics (e.g., working conditions, job responsibilities) associated with a job or position.\u00a0 This is done… View Article<\/a>","protected":false},"author":4,"featured_media":156,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[19,17,38,39],"class_list":["post-155","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-industry-news","tag-item-type","tag-job-task-analysis","tag-rating-scales","tag-task-frequency"],"_links":{"self":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/comments?post=155"}],"version-history":[{"count":1,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/155\/revisions"}],"predecessor-version":[{"id":157,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/155\/revisions\/157"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/media\/156"}],"wp:attachment":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/media?parent=155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/categories?post=155"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/tags?post=155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}