{"id":55,"date":"2015-10-07T18:32:10","date_gmt":"2015-10-07T18:32:10","guid":{"rendered":"http:\/\/proftesting.com\/blog\/?p=55"},"modified":"2016-11-16T02:20:29","modified_gmt":"2016-11-16T02:20:29","slug":"2015107when-to-use-a-performance-test-or-an-alternative-assessment-method-part-ii","status":"publish","type":"post","link":"https:\/\/www.proftesting.com\/blog\/2015\/10\/07\/2015107when-to-use-a-performance-test-or-an-alternative-assessment-method-part-ii\/","title":{"rendered":"When to Use a Performance Test or an Alternative Assessment Method, Part II"},"content":{"rendered":"

This is a continuation of the previous blog article<\/a> that discussed the use of performance examinations for certification and licensure. \u00a0To summarize the previous blog, performance examinations (i.e., \u201chands-on\u201d) are expensive to use and maintain. Many certification and licensure authorities have abandoned hands-on performance examinations for some alternative measure of a person\u2019s skills. However, there are situations where the expense and effort are worth the expenditure because of consequences to clients.\u00a0 Situations where a performance examination may be worth the expense are those where failing to measure an examinee\u2019s performance skills (fine motor) could result in pain or suffering for other people (e.g., patients).<\/p>\n

What about situations where fine motor skills are not critical, but complex cognitive processes need to be assessed by another human?\u00a0 There are still several options to \u201chands-on\u201d performance tests.\u00a0 Some alternatives might include essays, short answers, portfolios, exhibitions, and structured oral examinations.\u00a0 The largest group of rater\/grader scored standardized examinations in the United States, and perhaps the world, are essays.\u00a0 Essays are used to measure a variety of complex cognitive skills and are routinely used in multistate bar exams, national K-12 student achievement exams, national teacher credentialing exams, college entrance examinations (ACT\/SAT), and certification examinations.<\/p>\n

Most people have written an essay at one point during their education, and writing skills are commonly evaluated on criteria such as organization of thought, grammar, and syntax. This type of essay is an ideal instrument for measuring writing skills, however, essays can be used for much more The standardized essay is different from the essay format most people have taken while progressing through K-12 grades and postsecondary levels.\u00a0 They are a handy instrument for measuring the highest levels of cognitive functions (evaluation) for any professional domain.\u00a0 For example, they can be used to evaluate how well the principles of law are incorporated into a complex legal scenario or evaluate the examinee\u2019s approach to a complex hazardous materials spill.\u00a0 For the purpose of this blog we\u2019ll reference this type of essay as a \u201cstandardized essay\u201d.<\/p>\n

Standardized essays can be useful in credentialing when a job\/practice analysis has been performed and the content domain has been clearly defined.\u00a0 To make a good measure, validated writing prompts, scoring rubrics, scoring anchors, examiner\/ grader training, and evaluation of rater reliability are needed. \u00a0In other words, a significant amount of organized research and development will need to occur in order to provide the highest level of reliability and validity of test scores as possible. The cost for producing a standardized essay is not inexpensive, however, this type of examination is typically less than a \u201chands-on\u201d performance examination. \u00a0The reason why an essay costs less than a performance measure is that \u201chands on\u201d performance examinations frequently require simulators (e.g., flight), laboratories, or complex computer equipment for test administration and the standardized essay just requires a proctor, table, chair and a comfortable room.<\/p>\n

The development of the essay is an intuitive process because it is familiar to most people, including most subject matter experts (SMEs).\u00a0 The primary drawback of the essay approach is that the amount of the examinee\u2019s knowledge and abilities that can be measured with a few writing prompts may be limited.\u00a0 If the essay prompt happens to ask (i.e., samples) something that an otherwise competent person does not know, the assessment might misclassify the examinee as failing when the examinee is actually competent.<\/p>\n

A solution to the limited sampling of the examinees knowledge and abilities is to simply increase the number of essay prompts and limit the examinee\u2019s responses (e.g., less than 100 words). It is possible to ask 50 short essay questions in a 2.5 hour period.\u00a0 Classical measurement theory tells us that this increase in the number of items increases the reliability of the examination.\u00a0 Also, the content validity of the measure is increased because the test covers more of the examinee\u2019s knowledge and abilities, \u00a0\u00a0although the depth of coverage may not be as large as a three page essay answer.\u00a0 Lee J. Cronbach (deceased \u2013 Professor Emeritus at Stanford) called this the bandwidth-fidelity relationship.<\/p>\n

Writing assessments are not without costs because raters still have to be recruited, trained, and in many cases raters have to be paid for their services. Scoring anchors (sample essays) and scoring rubrics (scoring guides) must be produced to standardize and guide the raters\/scorers.\u00a0 Rater performance (reliability) must still be psychometrically evaluated and reported. \u00a0Just like \u201chands-on\u201d performance tests, a writing assessment can provide substantial challenges to achieving consistent reliability from one examinee to the next.<\/p>\n

Despite the draw backs to essay type assessments, there are good reasons that this form of assessment is still administered in high volumes across the nation. For one, written essay assessments do not require complex computer technology to develop or administer in a standardized manner. Also, in most cases, scoring anchors and scoring rubrics can be developed in a reasonably cost effective manner using subject matter experts. Furthermore, in some cases, the raters (non SMEs) can be trained to use the grading rubric which in turn saves money related to using only SMEs.\u00a0 The next blog will discuss some alternative item types that may avoid the costs associated with examiner scored performance examinations.<\/p>\n

Please stay tuned for: When to Use a Performance Test or an Alternative Assessment Method, Part III.<\/p>\n","protected":false},"excerpt":{"rendered":"

This is a continuation of the previous blog article that discussed the use of performance examinations for certification and licensure.  To summarize the previous blog, performance examinations (i.e., \u201chands-on\u201d) are expensive to use and maintain. Many certification and licensure authorities have abandoned hands-on performance examinations for some alternative measure of a person\u2019s skills. However, there are situations where the expense and effort are worth the expenditure because of consequences to clients.  Situations where a performance examination may be worth the expense are those where failing to measure an examinee\u2019s performance skills (fine motor) could result in pain or suffering for other people (e.g., patients). <\/p>","protected":false},"author":14,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[29,28],"class_list":["post-55","post","type-post","status-publish","format-standard","hentry","category-industry-news","tag-alternative-assessment-method","tag-performance-test"],"_links":{"self":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/55","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/comments?post=55"}],"version-history":[{"count":1,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/55\/revisions"}],"predecessor-version":[{"id":121,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/55\/revisions\/121"}],"wp:attachment":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/media?parent=55"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/categories?post=55"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/tags?post=55"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}