From the Item Bank

The Professional Testing Blog

Virtual Meetings—Good for the Interim But Less Than Ideal? – Part 2

Introduction

As a result of the COVID 19 pandemic Professional Testing staff needed to quickly adapt to hosting examination development and maintenance meetings virtually.  While some of us were conducting virtual meetings pre-COVID, all meetings rapidly transitioned to a virtual environment.  As we have navigated through this change over the last year, we’ve had the opportunity to look at the pros and cons of virtual meetings and identify what makes virtual meetings different from in-person meetings.  We are learning what we can do to make them a better experience for our facilitators, certification organizations, and subject-matter-experts (SMEs).  Collectively, our experiences fall into a few categories: engaging and facilitating subject-matter-experts (SMEs); working with the right number of SMEs; communicating with credentialing staff; and handling challenges dealing with technology and IT support.  In this series of blogs, Professional Testing’s experts share what they have learned moving from in-person to virtual meetings, and considerations going forward to assure a productive and satisfying virtual meeting experience.  In this second blog, we discuss finding the right number of SMEs to complete examination activities.

Working with the Right Number of SMEs for Virtual Meetings

One important consideration for conducting virtual meetings is working with the right number of SMEs.  Determining the right number of SMEs depends on several factors, including the type of meeting (single vs series); meeting objectives (new initiative vs routine activity such as item writing); experience of the SMEs; and expectations of the certification organization in terms of time and end-product. 

One outcome from the transition to all virtual meetings is a savings in direct costs associated with air-travel, meeting space, hotel accommodations, meals, etc.  Given these savings, the involvement of more SMEs and their commitment to participating seems to be a reasonable assumption.  But what is often experienced can be the opposite, since a commitment to travel entails resources, making travel plans, and booking time off makes SMEs more likely to show up.  With virtual meetings, however, SMEs may assume there are no direct costs and may feel comfortable cancelling a meeting resulting in fewer participants and perhaps the absence of the requisite level of expertise to produce quality work.  This has a cost—full involvement of SMEs representing the breadth, depth and scope of experience required for the project (for example JTA, passing score study, item writing) has an impact on the quality of the end-product.  It is important, therefore, for certification boards to reiterate the importance of involvement of a diverse group of SMEs, make participation a selection requirement, and recruit more SMEs than they have for in-person meetings.  It is equally important for facilitators to remind SMEs of their value at the outset of each meeting—we can’t do our jobs well without them.  One strategy is during introductions; have SMEs provide a brief statement about their experiences, work environment, years on the job, and other demographic information that highlights the importance of full representation of expertise to achieve a successful meeting outcome and a quality product.

The number of SMEs to recruit depends on the type of meeting and the level of involvement of the SMEs.  However, experience has shown us that due to attrition and cancellations by SMEs, it is always better to involve more SMEs for virtual meetings than one would for in-person meetings.  Even doubling or tripling the number of SMEs required is not unrealistic for virtual meetings.  If 10 SMEs are required to participate in item writing, inviting 20 – 30 is suggested. Over time, attrition will occur.

Certification organizations and facilitators alike need to plan for attrition.  For example, in developing a new item bank 25 SMEs may start with the task of writing 150 items.  The project may be initially scheduled for one month of item writing conducted remotely, post training session, with two weeks of remote meetings scheduled for item review, totaling six weeks of virtual meetings.  As attrition occurs, the project may stretch to three months, with the group of 25 SMEs dwindling to a core group of eight SMEs who consistently attended webinars and write items remotely to populate the item bank.  While a valuable outcome from this typical experience is the identification of 8 stellar SMEs to include in future program development activities, whether in person or on-line, certification organizations need to understand that the timeline and number of contributing SMEs will change, and more demands may be placed on SMEs than originally proposed.

In planning item writing and review activities, experience shows that facilitators will assume half of the SMEs may not be as productive as required; some will drop-out; and a core group of SMEs will likely complete the project as presented in the example above.  This is fairly typical of all meetings, but unlike in-person sessions conducted over two days, facilitators have less interaction with SMEs and are unable to work one-on-one with SMEs.  The opportunity for peer-to-peer support also diminishes, which can result in fewer than anticipated, lower quality items.  In remote item writing sessions, certification organizations may need to adjust their expectations, for example, if 200 items are need, 120 may be what organizations actually work with.  Similarly, with new SMEs who are often excited to be involved in exam development activities, inexperienced SMEs may contribute more items than requested, but they might not always be quality items.  When thinking about the right number of SMEs, also consider their experience, and be sure to populate exam development activities with experienced SMEs.  This will require more deliberation from certification organizations in selecting SMEs.

Unfortunately, some attrition is occurring with SMEs who are not skilled in the required technology to participate in remote meetings.  The frustration resulting from technical issues and connectivity interruptions can cause SMEs to drop out of the process.  Frustration also occurs with SMEs unaccustomed to working from home, juggling family, work, and volunteer responsibilities.  SMEs working remotely pre-COVID tend to be better at self-monitoring their work, managing job and family commitments, and trouble-shooting technology issues.  If SMEs have these experiences, select them.  Taking the time to find the right SMEs can reduce the number needed to complete a project and may reduce attrition. 

Resource Considerations

The “behind-the scenes” resources should go into budgeting and planning on-line meetings.  The support facilitators provide to SMEs struggling with technology, answering calls and messages from SMEs with questions, and conducting more meetings than initially anticipated, adds to the cost of the meeting and the number of personnel assigned to the project.  For example, in a two-day item review meeting, eight SMEs were able to review 150 items.  When converted to an online meeting, less than 30 items were reviewed in the first two hours.  In another example of converting to an in-person item writing meeting to a virtual meeting, 10 – 12 SMEs completed the task in a two-day in-person meeting, while the online meeting involving 16 item-writers with a timeline of one month of virtual meetings, stretched to three months with 50% SME attrition rate, resulting in the loss of eight SMEs for future work.

Larger meetings, particularly large virtual meetings, have their own sets of challenges.  For example, if each of 30 SMEs participating in a virtual meeting contribute their thoughts for only one minute, it would take half an hour for each SME to speak only once.  This clearly would slow down a virtual meeting.  Thus, for large meetings involving 30 – 50 SMEs, meeting rules such as only speaking if you have a new idea or thought to contribute; not supporting a previous person or repeating a thought; and efficient processes for voting on consensus items, can result in finishing the project in a reasonable amount of time.  This information should be shared with SMEs in writing prior to the meeting and reinforced at the outset of each meeting so SMEs are clear about the rules of engagement.  Additionally, certification organizations will need to provide additional resources for monitoring chat and audio functions, and for solving any technical issues.

For all types and sizes of virtual meetings, it is important for the facilitator and/or the certification organization to explain to SMEs how vital their participation is in the success and outcome of the initiative.  It is also important to explain how important their commitment to participate is, and that their absence will affect other SMEs as well as the project.  Once meeting schedules have been confirmed, cancelling meeting should be avoided at all costs, since cancellations affect multiple schedules for multiple days. 

When asked what they would do differently now that many virtual meetings have been conducted, our psychometricians suggest the following.  Double and triple the number of SMEs for the project to avoid cancelling meetings due to lack of participation; cancel meetings when the minimum required fail to attend; and double the amount of time you think it will take to complete a project.  Also consider the learning curve of SMEs.  For example, in a two-day face-to-face meeting SMEs move more quickly from a novice writer to a good writer, and typically write usable items on the second day.  This is due to the one-on-one attention facilitators can provide, and the ability of experienced writers to assist new writers.  With virtual meetings, this type of interaction is not possible, so facilitators will need to manage the expectations of the certification organization regarding quality and number of items.  In face-to-face meetings for JTAs, SMEs are well into the rhythm of the meeting and contributing well by the afternoon of the first day, and resume the pace throughout the duration of a two and a half – three day meeting involving 12 – 15 SMEs.  Conducted virtually over a period of weeks can interrupt the pace and level of detail of JTA participants, and with attrition, the end-product may not be as robust.

While there are many challenges to conducting examination development and maintenance activities virtually, many of them can be managed with the careful selection and number of SMEs, establishing realistic timelines, and involving the certification organization in providing meeting support.  Candid discussions need to occur with certification organizations so that the right SMEs are involved; while SMEs may be willing to volunteer, they may not be right for the virtually accomplished job.  During this time when virtual meetings are necessary, it might not be the best time to involve inexperienced SMEs if it can be avoided.  If new SMEs are involved, it is critical to help certification organizations understand the importance of facilitating the peer-to-peer interaction, and to provide opportunities for this to occur off-line, for example, an experienced SME is paired with an inexperienced SME for assistance in item writing pre-virtual session.  The same applies to checking in with SMEs ahead of the scheduled meetings to make sure SMEs are comfortable with the technology and have some skills to trouble-shoot during meetings should interruptions occur.  This will help the meetings run more smoothly, with fewer interruptions, and help to mitigate attrition resulting from frustration.

Closing thoughts

With a year of adjustments and achievements accomplished, certification organizations may wish to consider conducting their meetings virtually once pandemic restrictions are lifted.  While the cost-savings may be a perceived advantage, there are also costs associated with stretching out project timelines and additional project management and supporting resources that need to be factored into planning.  There may also be some SME “burnout” with more frequent participation, and a loss of interest with the absence of opportunities to network and interact, a great benefit of the face-to-face meetings.  While these benefits of meetings are hard to monetize, keep them in mind as you consider the future of your meetings. 

Additional posts

Read Part 1 of Virtual Meetings—Good for the Interim But Less Than Ideal?

Categorized in: Industry News