{"id":1148,"date":"2019-09-27T09:49:01","date_gmt":"2019-09-27T09:49:01","guid":{"rendered":"http:\/\/www.proftesting.com\/blog\/?p=1148"},"modified":"2024-05-30T16:07:17","modified_gmt":"2024-05-30T16:07:17","slug":"expectations-when-developing-a-new-certification-examination-be-aware-so-you-can-manage-your-expectations-and-those-of-subject-matter-experts","status":"publish","type":"post","link":"https:\/\/www.proftesting.com\/blog\/2019\/09\/27\/expectations-when-developing-a-new-certification-examination-be-aware-so-you-can-manage-your-expectations-and-those-of-subject-matter-experts\/","title":{"rendered":"Expectations When Developing a New Certification Examination\u2014Be Aware So You Can Manage Your Expectations and Those of Subject-Matter-Experts"},"content":{"rendered":"\n

Co-authored by\u00a0Reed Castle, PhD<\/a> and Christine D. Niero, PhD<\/a><\/h4>\n\n\n\n

If you are new to developing a certification program<\/a>, this post outlines some of the key steps needed to develop a psychometrically sound exam. No matter what approach you take in exam development<\/a>, subject-matter-experts (SMEs) are critical contributors to achieving a valid and reliable product.  While many SMEs are anxious to participate, they are sometimes unaware of the expectations made of them, and the demands of the process.   The following five key examination development activities describe the process and the role of the Certifying Body (CB) or sponsoring organization and SMEs during initial exam development. They are: job analysis, item and form development, administration, standard setting (Cut-score), and statistical review.<\/p>\n\n\n\n

Expectations Associated with the Job Analysis<\/h3>\n\n\n\n

A job analysis is the foundation of a certification examination\u2014it is the first step in exam development. During this process, the role of the certified professional is delineated which subsequently leads to the test blueprint.  The use of SMEs is essential as they are the persons either doing the job, or very knowledgeable about the competencies required to perform the job.  No matter what methodology is selected, SMEs need to participate.  Professional Testing has achieved efficient and satisfactory results conducting one in-person meeting with subject matter experts (SMEs), followed by a post meeting after the validation survey results administered to existing practitioners and job incumbents have been analyzed. The CB is responsible for recruiting eight to twelve SMEs for the job analysis meeting, establishing a plan for SME travel and housing, and securing a host site for the two to three-day meeting. When creating a budget, considerations should be given to SME travel expenses, (e.g., food, travel and lodging), geographic location, employment setting, and job title, in addition to other relevant demographic data.  A SME panel that collectively represents the breadth, depth and scope of the certification will yield the best outcomes.  SMEs need to understand why they are selected, and why they were not.  <\/p>\n\n\n\n

After the in-person meeting is completed, a survey will be drafted for the validation of the blueprint. The CB is responsible for identifying a cohort of survey respondents. Typically, the surveys are electronic and the CB will be responsible for gathering email addresses and sending out the invitation to survey participants. After the survey is administered, a follow-up webinar will be scheduled with SMEs to finalize the examination blueprint.  At the outset, SMEs should understand their roles and responsibilities, time commitments, and the length of time it takes to conduct both steps of the JTA.  It is also useful to inform SMEs that they may be asked to participate in subsequent exam development activities.  <\/p>\n\n\n\n

Expectations Associated with the Item and Form Development<\/h3>\n\n\n\n

After the exam blueprint is finalized by SMEs, exam items (questions) and exam forms are developed. For new programs, there may be two SME in-person meetings. The first is to write new items to populate the item bank, and the second SME meeting is to review newly written items and approve them. So, expect two, three-day meetings with eight to ten SMEs to populate the item bank of approximately 250 items for a 100-question exam. After the item bank is populated with SME approved items, a psychometrician<\/a> or test developer will create an exam form adhering to the exam specifications and weightings derived from the validation survey step of the job analysis. At this stage, the CB may be able to convene a conference call\/webinar with three to five SMEs to review the newly created exam form. The purpose of the review is to assure there are not duplicative items or items that provide clues to other items on the form, and to perform one last check of the form prior to being published.  It is optimal if some of the SMEs who participated in the job analysis can write and review items to carry over \u201cinstitutional memory,\u201d while also including SMEs new to the exam development process.  Typically, a group of 8 \u2013 10 SMEs are required for item writing and review.  Again, the CB will need to consider costs associated with travel, meals, accommodations, meeting space and psychometric expertise. <\/p>\n\n\n\n

Expectations Associated with Exam Administration<\/h3>\n\n\n\n

Working closely with the psychometric team the CB will make\ndecisions on how the exam will be administered. Believe it or not, there are\nstill organizations using paper administrations but increasingly most CBs have\nmoved to computer administration. Consideration of exam administration length\n(also known as \u201cseat time\u201d), testing within specific date ranges versus open\ntesting, matching test access with CB professionals, how the candidates will\nschedule their exam and costs are a few of the decisions that must be made. A\ncouple of costs for the CB are the initial cost to set-up with a CBT provider\nand the direct costs associated with each exam administration. CBs should also\nconsider the characteristics of their test-takers\u2014how much can they afford to\npay to take the test; are costs of submitting the application in addition to,\nor bundled with, test administration fees; do they reside in geographic areas\nthat are easily accessible to a CBT test center, or will air travel be required\nfor some; would they prefer to test at an industry conference, etc.  Knowing the expectations of test-takers will\nguide the CB in making the right choice for test administration.  SMEs are invaluable in providing CBs with\nthis type of information.<\/p>\n\n\n\n

Expectations Associated with Standard Setting <\/h3>\n\n\n\n

After the exam is administered to the first cohort or pilot\ngroup of test-takers, a passing score (also known as a cut score) must be\nestablished.  Typically, a SME panel is\ncreated for a one or two-day in-person meeting. During this meeting, a\npsychometrician will facilitate the process of defining the minimally qualified\nprofessional. SMEs will subsequently provide ratings or make judgements as to\nwhere the passing score should be. This information will be used to determine\nhow many items need to be correctly answered on the exam to pass and receive\ncertification. The CB is responsible for recruiting eight to twelve SMEs for the\nin-person standard setting meeting, establishing a plan for SME travel and\nhousing, and securing a host site for the one to two-day meeting.  Again, it is optimal if a few SMEs involved\nin previous activities participate in standard setting, along with new\nSMEs.  <\/p>\n\n\n\n

Expectations Associated with Statistical Review of the Exam<\/h3>\n\n\n\n

After the exam form is administered and enough candidates\nhave completed the exam to provide sufficient data, exam questions with poorly\nperforming item statistics will need to be reviewed and decisions about current\nand future scoring choices will need to be made. This process can be done\neither via webinar or at a future item development meeting, and depending on\nthe number of items to be reviewed, can involve as few as three to four SMEs,\nor more.  Your psychometrician will\nadvise the best number of SMEs to recruit.  As the program matures, there may also be the\nneed to develop additional examination forms, and this may be accomplished via\na psychometrician facilitated webinar to evaluate the results of equating\nstudies (e.g., what the cut score should be for a new exam form). This is\nbecause not all exam forms are equal with respect to difficulty. <\/p>\n\n\n\n

As can be seen, there are many steps in developing a\ncertification exam, and understanding the expectations of the CB, SMEs and the\nprocess will help to ensure a smooth, well-articulated and communicated process.\nA CB can expect to have up to four in-person SME meetings and a number of\nwebinars just to launch the first administration. While there may be an\ninclination to give thought to doing some of these steps remotely, starting a\nprogram takes a great deal of time and effort and relying on remotely working\nSMEs can extend timelines and create bottlenecks to the process. Some CBs have\nalso had experiences where the quality of the product declines with SMEs\nworking remotely. It is much better (and our experience) to do remote work\nafter the program is mature and SMEs are well trained and experienced. <\/p>\n\n\n\n

This is a very high overview of what to expect during the initial test development process. A CB can expect the process to take anywhere from four to twelve months. It is largely dependent on the scheduling of SME meetings, SME availability, resources, and the expectations of the CB and its leadership. <\/p>\n\n\n\n

For more related blogs check out Careful Selection of Subject Matter Experts is the Key to a Successful JTA Meeting<\/a> and The Use of Certification and Licensure Programs in Developing Career Path Models<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Co-authored by\u00a0Reed Castle, PhD and Christine D. Niero, PhD If you are new to developing a certification program, this post outlines some of the key steps needed to develop a psychometrically sound exam. No matter what approach you take in exam development, subject-matter-experts (SMEs) are critical contributors to achieving a valid and reliable product.  While… View Article<\/a>","protected":false},"author":5,"featured_media":1155,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1148","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-industry-news"],"_links":{"self":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/1148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/comments?post=1148"}],"version-history":[{"count":7,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/1148\/revisions"}],"predecessor-version":[{"id":1157,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/posts\/1148\/revisions\/1157"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/media\/1155"}],"wp:attachment":[{"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/media?parent=1148"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/categories?post=1148"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.proftesting.com\/blog\/wp-json\/wp\/v2\/tags?post=1148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}