From the Item Bank

The Professional Testing Blog

Conducting a Revalidation Job Analysis: How the Delphi Method May Prove Useful

Let us begin by defining the Delphi Method for collecting data.  The Delphi Method involves creating a questionnaire, administering it to a panel of stakeholders, soliciting feedback, updating the questionnaire based on the feedback, and sending it out again, as illustrated below.  This iterative process may go for 2, 3, or more rounds until consensus is obtained.

For example, let’s pretend we were trying to identify the Top 10 best burger places in a specific city.  In our Round 1 questionnaire we might ask individuals to name their Top 10 favorite burger places.  We would consolidate that feedback into a list of the most common places given by our stakeholders (let’s pretend that there were 15 places common across all stakeholder responses).  Then, our Round 2 questionnaire might ask these same individuals to rank order the 15 burger places from best to worst.  For Round 2, we would look at the response options from all stakeholders and rank the burger places based on a consolidation of all responses, hopefully eliminating the bottom 5 restaurants.  In our Round 3 questionnaire we would provide the Top 10 list of burger places ranked in order and ask individual to adjust the ranking, as necessary.  For example, the majority of our respondents might be fine with our top two burger places, but might feel strongly that 3 and 4 need to be swapped.  At this point we should be nearing consensus, but if not, we would continue administering questionnaires until we have reached consensus.

Now, how does this apply to a Job Analysis?  One could use this method to revalidate an existing job/task analysis when a face-to-face focus group isn’t feasible (either due to financial or geographical constraints).  In this instance, the existing job/task analysis could be turned into a questionnaire.  For Round 1, we would present the existing job/task analysis and ask our stakeholders if the current list of domains and subdomains is complete as-is, or if each should be updated (i.e., should content be added, removed, or changed).  If anything should be updated, how should it be updated?  The job analyst would consolidate all feedback and administer a Round 2 questionnaire asking the same questions.  Are the domains and subdomains fine as-is, or should they be updated?  And if so, how?  This process would continue until consensus is reached, as illustrated below.

Should you decide to utilize the Delphi Method for a job/task analysis revalidation, there are some additional considerations:

• Timing between rounds.  This will depend on many factors, including, but not limited to the overall job/task analysis revalidation timeline, how quickly your stakeholders respond to your questionnaire, and how quickly the results can be consolidated and turned into an updated questionnaire for the next round.
• Number of rounds.  You should have some idea of the number of rounds you are willing to go through before you determine you are finished.  For most organizations, somewhere between 2-3 rounds should be enough.  However, it’s possible that you may need 4 or 5 rounds to reach consensus.  This should be determined in advance (or at the very least, you should put a cap on the total number of rounds you are willing to entertain in advance of the process).
• Number of stakeholders.  The number of stakeholders who should respond to your questionnaire should be based on the number of subject matter experts you would need to obtain a representative sample of your candidate population.  This should be somewhere between 8-15 people, depending on how many demographic buckets you’re trying to fill.

While we would argue that a face-to-face focus group is always the best way to conduct the revalidation of a job/task analysis, we hope this alternative methodology proves useful.

Categorized in: Test Development