From the Item Bank

The Professional Testing Blog

Does Meeting Modality Matter in Exam Development?

An Evaluation of In-Person, Virtual, and Hybrid Modalities for Exam Development Meetings in Certification and Licensure Programs

Credentialing organizations must consider a range of factors when selecting a format for exam development meetings, including cost, scheduling, accessibility, participant engagement, and last but not least, the experience and satisfaction of the volunteer subject matter experts (SMEs) who make the work possible. Item development, form review, and passing score studies may be held in person, virtually, or through a hybrid format that includes both in-person and virtual participants.

Each option offers both advantages and limitations. Virtual meetings can significantly reduce travel costs and make it easier to include participants from across the country or even the world. Hybrid meetings can provide flexibility when key participants are unable to travel. In-person meetings may require more time at a higher cost to the organization, but they may also create an environment that better supports discussion, collaboration, and engagement and can be much more efficient than hybrid or virtual modalities.

Some organizations select virtual or hybrid meeting formats primarily based on cost, scheduling, and SME availability. Those are important considerations, but they are not the only ones. The meeting format can also affect how participants communicate, contribute, build relationships, and perceive their experience throughout the process.

An exploratory survey of professionals involved in exam development provides additional insight into how participants experience these different formats. The findings suggest that meeting modality should be treated as part of the overall meeting design, rather than only as a logistical decision made after the work has already been planned.

Note: This blog is informed by an exploratory survey of professionals involved in certification and licensure exam development, conducted as part of a graduate research course. The findings reflect the experiences of this group and are not to be interpreted as generalizable conclusions applicable to every credentialing program.

Study Background

The survey included 51 professionals with experience participating in exam development meetings across more than 15 unique examination programs from various credentialing organizations. Participants included subject matter experts, credentialing staff, psychometricians, test developers, and program managers. To participate in the survey, respondents were required to have experience with at least two of the three meeting formats.

The survey focused on three common exam development activities:

• Item development and review

• Examination form review

• Passing score studies

Participants rated their experiences across seven constructs, each operationalized through multiple Likert-type items that were combined to create composite construct scores:

• Perceived meeting effectiveness

• Engagement and attention

• Logistical burden

• Communication and contribution

• Networking and interactions

• Technology considerations

• Overall experience and enjoyment

Participants also ranked each meeting format and provided extensive qualitative feedback about the strengths and challenges associated with each modality.

Results

In-person meetings received more positive feedback throughout most of the survey. Participants rated in-person meetings more favorably than virtual and hybrid meetings across every construct except logistical burden, and the qualitative findings strongly supported those statistically significant quantitative results. This pattern was consistent across all three exam development meeting types.

In contrast, there was no statistically significant difference between virtual and hybrid meetings.  Participants generally viewed both formats less favorably than in-person meetings, although their qualitative feedback suggested that the two formats presented somewhat different challenges and limitations.

Participants’ rankings showed a similar pattern. In-person meetings were selected as the first choice by almost 98% of respondents for item development, approximately 90% for form review, and approximately 91% for passing score studies.

Results from the qualitative feedback portion of the survey overwhelmingly supported the selection of in-person meetings. Participants felt that in-person meetings were more enjoyable, more efficient, contributed to better relationships amongst the groups, and ultimately produced higher-quality content and outcomes.

Key Insights

1. Communication: Unsurprisingly, communication and collaboration were easier in person. Exam development work depends on discussion and collaboration. SMEs must explain their reasoning, question the content, provide feedback, and work toward consensus. This can be especially important during item development and review, when a relatively small wording change may affect the meaning or defensibility of an item.

Participants frequently described in-person meetings as more collaborative. They reported that discussion occurred more naturally and that it was easier to respond to questions, clarify misunderstandings, and build on the ideas of others. The ability to read body language also repeatedly came up within the results as an important factor. In an in-person setting, facilitators and participants can often recognize when someone is confused, hesitant, ready to contribute, or not fully in agreement. Those cues are more difficult to discern in a virtual meeting, especially when cameras are off.

Virtual discussion may become more structured because participants must wait for an opening, use a raised-hand feature, or avoid interrupting. Structure is not necessarily negative, but it can make interactions less natural and may limit spontaneous discussion.

The survey results indicated that hybrid meetings were especially challenging because of the disparity between in-person and virtual participants. Beyond technological issues, which often occur, respondents reported that virtual participants could miss parts of the conversation and were not always on an equal footing with those attending in person.

2. Relationships and Networking: Participants emphasized the value of relationship building and networking as a major part of their reason for participating in exam development. In-person meetings create opportunities for committee members to interact before the meeting begins, during breaks, over meals, and after the formal work has ended. These interactions may seem separate from the primary purpose of the meeting, whether that is writing test questions, finalizing a form, or establishing a cut score, but they can greatly affect how the group works together and the level of commitment volunteer SMEs feel toward the project. For many SMEs, the relationships, networking, and “fun” associated with in-person meetings are a big part of why they continue to serve on committees, and that is not something that often translates to the virtual environment.

Additionally, committee members are being asked to share ideas, receive criticism, question one another, and sometimes reconsider positions based on the group’s feedback. That work requires a certain level of comfort and trust, which comes much more easily when committees are able to develop relationships outside of the meeting.  Respondents described in-person meetings as being much better suited to developing a rapport with other SMEs and facilitators. Opportunities to establish or build relationships are of particular importance for groups that are newly formed or have new members who are unfamiliar with the committee’s processes and norms.

Virtual meetings tend to focus more narrowly on the agenda. Participants join the call, complete the scheduled work, and disconnect. While this may be efficient, it provides fewer opportunities for informal interactions that help committee members become comfortable working together, and when efforts are made to provide virtual networking opportunities, they can often feel forced or artificial.

3. Focus and Sustainability: Participants reported that they were better able to remain focused during in-person meetings. Being physically present in a meeting room created separation from normal work responsibilities and everyday distractions.

Virtual participants have significantly more competing distractions and demands. This becomes particularly important during full-day or multi-day exam development meetings. Item review, form review, and passing score studies require sustained attention and can be cognitively demanding. Respondents described virtual meetings as more tiring and more difficult to sustain over long periods due to “Zoom Fatigue”.

Shorter virtual sessions may help reduce fatigue, but dividing the work across several days can create other concerns. Participants must repeatedly shift in and out of the work, and the organization may need to manage more scheduling, preparation, and follow-up. Realistically, it is incredibly difficult to maintain a consistent group of participants across multi-day scheduled calls, especially with conflicting time zones when including SMEs from around the country or even the world.

4. Accessibility: Although in-person meetings received stronger overall ratings, participants recognized several important advantages of virtual meetings.

Virtual formats reduce or eliminate travel. This can substantially lower costs, reduce time commitments, and make it possible to include SMEs who might not otherwise be able to attend. This is a significant benefit for credentialing programs. Committee composition can be increased and further diversified by including individuals who would otherwise not be able to attend due to employer restrictions, personal responsibilities, mobility limitations, or other circumstances that make traveling for an in-person meeting prohibitive.

5. Extenuating Circumstances: One interesting insight from the survey was that many SMEs felt virtual and hybrid meetings for exam development were understandable and well-accepted during the COVID-19 pandemic, when there were limited alternatives. But now that no such limitations exist, participants reported less acceptance of virtual or hybrid formats. This suggests that SMEs may be more open to these formats when they are made aware of legitimate operational constraints but would certainly prefer to have opportunities to meet and work in person while on committee.

Selecting the Right Modality

There may not be one meeting format that is appropriate for every committee, every time. A more useful approach is to first consider the purpose of the meeting and the existing committee dynamics (or lack thereof) before selecting the meeting modality.

In-person meetings may be appropriate when:

• The committee is newly formed, and participants have no prior relationships.

• Several members are new to the exam development process.

• Work requires extensive discussion, debate, or consensus.

• Participants will be substantially revising content.

• The meeting will last a full day or several consecutive days.

Virtual meetings may be appropriate when:

• The committee has a strong, existing rapport.

• Travel would prevent key participants from attending.

• The meeting has a narrow and clearly defined purpose.

• Participants have experience in and understanding of the process.

• The work can be completed in shorter sessions.

• Some of the participant’s contributions can be completed asynchronously.

• The meeting is primarily informational or procedural.

• Accessibility and scheduling flexibility are the primary considerations.

• Costs are a major limiting factor.

Hybrid meetings may be appropriate when:

• The committee, especially the virtual participants, has a strong, existing rapport.

• The meeting room includes appropriate audio and video equipment.

• Virtual attendees can hear and see the full discussion.

• An additional facilitator can help monitor and support the virtual environment.

• Materials and activities are equally accessible to both groups.

• Virtual participants are critical contributors (bringing unique, diverse perspectives that are not represented in the room).

Summary

Cost, travel, scheduling, and accessibility will continue to influence how certification and licensure programs conduct exam development meetings. These factors cannot be ignored. At the same time, the least expensive or easiest format may not always be the best format for the work. Exam development depends on participants’ ability to communicate clearly, remain engaged, provide critical feedback, and work toward shared decisions. The meeting environment can either support or complicate those activities.

The survey findings suggest that participants viewed in-person meetings as the strongest format for collaboration, communication, engagement, and relationship building. Virtual meetings offered meaningful accessibility benefits but were more often associated with distractions and participant fatigue. Hybrid meetings provided flexibility but introduced additional concerns related to facilitation and equal participation, along with many of the same challenges as virtual meetings.

The takeaway from this narrowly focused exploratory research is not that every exam development meeting must be held in person. The meeting format should be selected intentionally.

Before choosing a modality, credentialing staff should consider what the committee is being asked to accomplish, how much discussion the work requires, whether participants already have established relationships, and what support will be needed for everyone to contribute as effectively and efficiently as possible.

The meeting modality is part of the exam development process. Selecting a format that supports the SMEs can strengthen the quality of the work, improve the volunteer experience, and help sustain engagement over time. At the same time, no two committees are exactly alike. Credentialing organizations should consider the individual preferences, needs, and prior experiences of their own volunteers when determining which format is most appropriate and revisit those preferences as committee composition evolves over time.

Study Limitations

As with any exploratory project, interpretation should be done with caution. The sample was relatively small and was drawn through purposive sampling from professional networks, so the results do not represent the experiences of exam development volunteers for all certification and licensure programs. Participants also did not have experience with every possible combination of meeting type and modality, which resulted in different sample sizes across comparisons. In addition, the sample was not large enough to support meaningful subgroup analyses, such as comparing the perspectives of SMEs, credentialing staff, and facilitators. Finally, the survey measured participants’ perceptions and preferences; it did not independently evaluate outcomes from each modality.

As a reminder, this study was conducted as part of a graduate research project, and results are not intended to be generalizable conclusions applicable to every credentialing program.

Categorized in: Item Writing, Test Development