From the Item Bank

The Professional Testing Blog

Virtual Meetings—Good for the Interim But Less Than Ideal? – Part 1

By Christine Niero, Ph.D., Vice President, Professional Certification and Cynthia D. Woodley, Ed.D., Chief Operations Officer & Psychometrician

Introduction

As a result of the COVID 19 pandemic Professional Testing staff needed to quickly adapt to hosting examination development and maintenance meetings virtually. While some of us were conducting virtual meetings pre-COVID, all meetings rapidly transitioned to a virtual environment. As we have navigated through this change, we’ve had the opportunity to look at the pros and cons of virtual meetings and identify what makes virtual meetings different from in-person meetings. We are learning what we can do to make them a better experience for our facilitators, certification organizations, and subject-matter-experts (SMEs). Collectively, our experiences fall into a few categories: engaging and facilitating subject-matter-experts (SMEs); working with the right number of SMEs; communicating with credentialing staff; and handling challenges dealing with technology and IT support. In this series of blogs, Professional Testing’s experts share what they have learned moving from in-person to virtual meetings, and considerations going forward to assure a productive and satisfying virtual meeting experience. This first blog addresses engaging subject-matter experts.

Subject-Matter-Expert Engagement

One of the vulnerabilities of a virtual meeting as compared to in-person meetings is losing the interest of SMEs due to lack of engagement or SMEs multi-tasking. We have found there are strategies that seem to improve SME engagement. This post will explore some of the issues surrounding SME engagement we have experienced and some of the strategies we have found that help keep SMEs focused and contributing.

Establish and Communicate Purpose: One means of reinforcing SME engagement is to clearly outline the purposes, objectives and activities to be accomplished at the outset of each meeting, with an emphasis on the role and responsibility of the SMEs. Whether SMEs are regular contributors, or new recruits, reminding them of their valuable contribution interacting as partners with psychometricians and test developers can reinforce their comfort and contribute to their engagement.

Utilize Video: Almost everyone we have worked with has shared stories about experiencing “Zoom burnout” and fatigue at seeing themselves on camera. Despite this, we have found that keeping cameras on results in improved SME engagement and better contribution to meeting outcomes. Seeing each other’s faces helps to unite a group distanced by miles. By looking at each other’s faces and seeing each other’s eyes, we tend to stay more engaged than if we are only using our sense of hearing. We understand that internet and bandwidth issues sometimes prohibit the use of video, but where possible it helps to encourage the SMEs to use their cameras and keep them on. At a minimum, even if the cameras must be turned off, encouraging the SMEs to turn them on while they are speaking helps. Plus, it helps the facilitator better identify SMEs who are not engaged by seeing who has not turned their camera on or contributed during the meeting.

Focus on Continuity of Meeting Activities: Another challenge to virtual meeting dynamics is changing the typical stages of meetings—progressing from meeting preparation, to conducting the meeting, to meeting close-out, thanking and dismissing SMEs. Face-to-face meetings that previously ran for several, consecutive, 8-hour days, for example, a job-task analysis (JTA) meeting, have now been split across days and weeks with virtual webinars scheduled for a few hours a day. This draws out the meeting and contributes to SMEs becoming disengaged, multi-tasking, and removing themselves from the meeting altogether. And with meetings being spread out over weeks and months, the time to product is increased exponentially. One strategy to keep SMEs on task is to issue “homework” assignments between meetings, for example, creating several new test questions, filling out forms, or reviewing work completed by other SMEs and providing comments at the beginning of each meeting. Another strategy is to ask certification personnel to routinely communicate with SMEs between meetings with updates on their progress. A third strategy is to recap what was accomplished in previous meetings, so SMEs see their work in progress. A fourth strategy is to provide rewards for service such as points that count toward meeting recertification requirements (if appropriate), discounts in certification fees, and public acknowledgement of the role of SMEs in keeping certification programs current and relevant, even during disruption.

Communicate the Value of Active Participation and Attendance: For SMEs familiar with face-to-face meetings, the change in meeting dynamics can affect their orientation to the meeting, and in some cases, their engagement. Most SMEs report enjoying meeting at a destination, the camaraderie experienced with fellow SMEs, and a group process that results in a tangible product—in other words, feelings of a job well done by the end of the meeting. SMEs will schedule time dedicated to the destination meeting and focus on the task at hand (for the most part—there are times when SMEs need to step out to tend to business or life.) During virtual meetings, facilitators have noted that some SMEs are not as committed to the meetings. They seem to be less hesitant to cancel their participation at the last minute for a virtual meeting more so than for a meeting in which airline tickets, hotel rooms and other transportation has been arranged. And when they do participate, they tend to disengage more, turn off their cameras, multi-task, and deal with interruptions in their homes or offices. It is incumbent upon the meeting organizer and meeting facilitator to stress that the virtual meeting is of no less importance than the in-person meeting, the criticality for the SMEs maintaining their commitment to participate, and the valuable role the SMEs are playing by active participation.

Energize and Involve New SMEs: The loss of camaraderie expressed by SMEs when participating in virtual meetings is especially worth noting. During in-person meetings, facilitators have “command of the room” and can encourage quieter SMEs to speak up or manage SMEs who are disruptive. The physical presence of the facilitator also sets a different meeting tone as SMEs are always in sight. There is often energy felt in the room and the excitement of being part of a project. It is also easier to introduce and engage new SMEs to the process through conversations during meeting breaks and meals in an in-person meeting, but this is lost with a virtual meeting. For new SMEs this can be especially problematic. In circumstances in which SMEs do not know each other, for example, in a start-up program, or on a new initiative, SMEs may be hesitant to speak up in a virtual meeting if they do not know other SMEs. SMEs who know each other and have previously met in-person tend to engage faster possibly due to a comfort level of knowing other participants.

To adjust for groups of SMEs who are unfamiliar with each other it is important when conducting virtual meetings to take time to make introductions and to provide the group opportunities to build a sense of camaraderie. One suggestion is to create a “warm up” activity to help the group get to know one another. Asking each participant to talk about themselves, their jobs, their interests, their hobbies, etc. “personalizes” the meeting and helps to introduce the SMEs to one another. Often through these introductions, SMEs find commonalities. Even if the entire first meeting is nothing but introductions and one simple activity, it is a step that should not be skipped. This also helps to reinforce the areas of expertise each SME brings to the initiative and the value of the collective whole.

Carefully Consider and Schedule Virtual Meeting Lengths: The amount of time SMEs will remain engaged also depends on the type of project activities and the meeting lengths. For more intensive work such as editing items or reviewing items for standard-setting, shorter meetings (for example two hours) scheduled at a set time can keep SMEs on schedule and help to reduce fatigue. For longer meetings (for example 4 or 5 hours, or all day), scheduling bio and meal breaks can off-set fatigue and permit SMEs to tend to business. Be sure to provide enough time for meal breaks, as SMEs and facilitators are probably getting their own meals unlike in a face-to-face meeting where meals are catered and working lunches can occur.

For some certification boards, scheduling meetings in the evening after business hours is a necessity, so facilitators need to keep fatigue top of mind for the SMEs and for themselves. It’s important for facilitators to discuss the personality of their SMEs with certification personnel in order to determine what will work best for everyone, for example, frequency of meeting breaks and time of meetings—workdays, weekends, or evenings.

Add to the Overall Project Timeline: When asked what did not work well, introducing new SMEs to a project in a virtual environment presented a unique set of challenges. One example of this is item writing. While a training workshop is conducted and check-in points are provided to allow for feedback and assistance to item writers, there is less one-on-one interaction with item writers than what is provided in a face-to-face session. Item writers who are self-motivated and dedicated are successful in submitting items remotely; however “life-happens” scenarios can derail well-intended item writers, compromise the quality of the item, and delay progress. Certification boards may need to plan for additional item writing and review sessions if this occurs, which will add to the time and cost of the project.

Another example is with JTAs. With some certification boards, conducting a JTA virtually has resulted in a product less rich and in depth from what is typically produced in face-to-face meetings. This can result in the validation stage of the project requiring more time, more extensive revisions to the examination blueprint, and certification boards conducting a revalidation study at an earlier interval than otherwise required.

Closing Thoughts: The experiences and strategies for conducting virtual meetings shared in this post are resulting in better meetings and outcomes. While not applicable to every project, our practices will continue to evolve to meet the needs of certification organizations, their personnel, and SMEs. We start the New Year with the knowledge that virtual meetings will continue for the unforeseeable future, and some certification organizations benefiting from cost savings may wish to continue working in a virtual environment. It is important for psychometricians and certification staff to monitor the experiences of all involved as well as the quality of the work to make informed decisions moving forward.

Additional posts

Read Part 2 of Virtual Meetings—Good for the Interim But Less Than Ideal?

Categorized in: Industry News, Item Writing, Test Development