From the Item Bank

The Professional Testing Blog

Scenarios in Testing: Five Tips to Improve Your Mileage

If this image from the New Amsterdam Market in New York City is used in an exam, candidates may remember it as “the one with the guy with the colorful fish tattoo.” One solution? Use his images in several items. Photo by istolethetv licensed under Creative Commons Attribution 2.0 Generic. Some rights reserved.

Photo: If this image from the New Amsterdam Market in New York City is used in an exam, candidates may remember it as “the one with the guy with the colorful fish tattoo.” One solution? Use his images in several items. Photo by istolethetv licensed under Creative Commons Attribution 2.0 Generic. Some rights reserved.

In licensing and certification tests, brevity is considered a virtue. Here’s the stem of a raw item that lacks this virtue.

The driver of a midsized sedan is pleased with the number of miles per gallon of gasoline the car consumes in highway conditions, but is unhappy with the amount of gasoline consumed in city driving. After changing the car’s oil and checking the tire pressure, the driver decides to look at the octane rating of the gasoline. Which of the following grades of gasoline is likely to provide the driver with the most economical gasoline use in city driving conditions?

Bring this item to a review meeting, and it may well leave nicely shorn and blow dried:

Which of the following grades of gasoline is the most economical for a midsized sedan seeking the best gas mileage?

The purpose of the item is to determine whether the candidate understands that higher octane ratings don’t get you better mileage. By eliminating information that is not essential to making that determination, the reviewers lighten the candidate’s reading and information-processing burden. Since the item isn’t meant to evaluate reading skills, this is an important consideration. It also allows for a more efficient use of time and candidate energy. A test taker who has just read fifty-six 80-word items is probably more spent on reading Item 57 than the candidate who had 1/4 of the reading burden.

So, by all means, keep your items short and to the point. And yet there is a strong case to be made for more time-consuming items in licensing and certification tests.

Sometimes the job requires sifting through a bunch of facts and identifying which ones are salient. Very brief items might ask the candidate to recall a fact in isolation; in asking the candidate to apply knowledge, skills, and experience to a more complex set of circumstances, a vignette-style question may test higher-level abilities and allow for a fairer assessment of the candidate in the domain.

In 1996, Susan M. Case, David B. Swanson, and Douglas F. Becker reported on a pair of studies conducted by the National Board of Medical Examiners (“Verbosity, Window Dressing, and Red Herrings: Do They Make a Better Test Item?” Academic Medicine 71, no. 10, pp. S28–S30). In the studies, they took the same situation and presented it in three different ways:

1. a long-vignette version, where a client’s symptoms, history, and examination results are presented;
2. a short-vignette version, where some of this information is presented in interpreted form;
3. a minimalist version, where everything is interpreted.

They found that longer vignettes tended to be harder without sacrificing discrimination. (This means the added difficulty likely didn’t come from irrelevant sources. See my earlier blog post on “good” discrimination.)

Although their title promises to talk about “window dressing” and “red herrings,” the longer vignettes did not, in fact, have window dressing and red herrings – distractions like the oil change and the difference between city and highway mileage in my verbose example above. Rather, the longer vignettes presented relevant challenges to the candidate. The added challenge of the longer vignettes was to interpret the patient history and examination results before getting to the question.

Here are some guidelines for writing vignette- or scenario-style questions:

1. The vignette can be in a form other than words: use images, audio, or video, if possible.
2. Consider developing a template. This makes it easier to write the questions, and the candidate knows where to look for information. It also allows for some not-strictly-relevant information to be presented without deliberate obfuscation: A [y]-year-old [male/female] [carnivorous/vegetarian/vegan] client with [f] percent body fat, who does [frequency and intensity of physical exercise]… For example, if scenarios in a nutrition exam generally start with the information presented above, then the candidate is not left wondering why you mention age or gender in a particular case. 
3. If your scenario is more than, say, 75 words long, consider using it for more than one item (with a header like, “This scenario applies to this question and the next two.”) This way, the reading burden is shared over multiple items, giving you more miles per gallon, as it were. Just make sure each item is independent of the others. (If you have five independent items, your team can pick different subsets of three items for different tests. You can eliminate a defective one without losing the whole set. And, more importantly, your assessment of the candidate’s knowledge of one question’s subject matter won’t depend on his or her knowledge of another question’s material.)
4. With vignettes – especially multimedia ones – memorability can be an issue. For test security, you don’t want word getting out about “that one question with the guy with the colorful fish tattoo.” A solution is to have that guy in several questions.
5. Finally, make sure you really are testing higher-level skills and not just taking a long-winded road to asking a simple question.

Categorized in: Industry News