CONTINUOUS IMPROVEMENT OF PROGRAM DESIGN USING THE KIRKPATRICK EVALUATION MODEL
By Sieglinde Simon, CPTD, Senior Instructional Designer, Security First Insurance
In March 2020, just as COVID-19 was quickly spreading into the U.S., the Learning and Development (L&D) team at Security First Insurance (SFI) was wrapping up its third delivery of its 120-Day Claims Examiner Trainee program. This program offers a comprehensive curriculum that provides trainees with no prior insurance experience the knowledge, skills, and abilities needed to be a successful claims examiner. We hire people for this program who have the right attitude and aptitude and then we train them to help homeowners put their homes and lives back together after disaster strikes.
Facing state-mandated lockdowns, the L&D team did as many other Training departments did: We gathered our wits, pulled up our bootstraps, and transitioned most, if not all, trainings to virtual instructor-led and/or self-directed online learning via a learning management system (LMS). This was no easy feat and countless blogs and articles have been devoted to the success stories of what I’ll coin as the “2020 Training Pioneers” to boldly go where few L&D folks have gone before, navigating and conquering a foreign world to show what can be done when will and motivation make a pact.
But this Best Practice article is not about that endeavor. This article is about what happened in 2021, and how the SFI L&D team leveraged the Kirkpatrick Evaluation surveys of the fourth and fifth classes to spark a focus group to capture more specific information to further improve the four-month program.
Evaluation Feedback
Let me back up a bit and explain that the Kirkpatrick Level 1 (learner reaction) and Level 3 (behavior change) survey results for the third class revealed conclusions that the L&D team was able to incorporate into the fourth class. The third class ended March 2020; the Level 1 survey was distributed immediately to the trainees, and the Level 3 survey was distributed in June 2020 to both the trainees and their managers. Findings from both surveys included the need for incorporating more job shadowing, hands-on claim handling using case studies, and additional construction training. Refreshed instructional methods and events were built into the design and were ready for the fourth delivery of the program.
In a textbook scenario, recurring training programs are appropriately spaced, allowing resources to make design and sequencing improvements, as well as content revisions/updates based on evaluation surveys and interviews (group or individual). This iterative process is how training programs are fine-tuned. However, in the real world, L&D teams must adjust and adapt when employee positions must be filled and jobs must get done. Therefore, in 2021, the SFI L&D team delivered two 120-Day Claims Examiner Trainee programs back-to-back. The fourth class ran from February to July and the fifth class from August to December.
Unfortunately, even after adding more construction-related training activities and events, we still missed the construction target as noted in the fourth and fifth classes’ Level 1 and 3 survey results. (Note: The Level 3 survey for the fifth class was distributed in March 2022 and too late to include its results for this publication.) Survey results revealed that, overall, the trainees were either fully proficient or competent in targeted knowledge, skills, and abilities (KSAs); however, all respondents (trainees and managers) indicated that still more construction training was needed.
Let’s Go to the Focus Group
Truth be told, the L&D team was stumped. What were we missing? How could we discover the gap and address it in the program? We agreed the surveys were not giving us enough information and decided to conduct a focus group.
To prepare for the focus group, the L&D team met with the Claims leadership to determine the expected audience of the group. We also met among ourselves to determine the set of questions, our individual roles, and how we would conduct the focus group. To have a variety of participants with different skill levels, we asked for a blend of experienced claims examiners and recently graduated trainees.
Three weeks before the day, invites to the 90-minute focus group were sent to 17 claims examiners; 11 accepted and participated. There was one moderator and two note-takers. On the day of the focus group, six questions were written on large Post-it notes and strategically placed on the training rooms’ walls. Each participant received a questionnaire that anonymously asked what construction training they had within the last 12 months, how many years they served in the role of claims examiner, and to self-assess their level of construction knowledge (novice, expert, apprentice, master).
These characteristics gave us the information we needed to gauge our audience. We prepared the room in a U-shape to give the participants the sense that all of their voices/opinions are equal. And borrowing from Maslow’s Hierarchy of Needs, we provided a variety of juices and donuts.
After the introduction and focus group ground rules were explained, the participants had 20 minutes to go to each posted question and write their response/answer. Answers poured onto the pages. If participants liked someone else’s response/answer, they were to give it a checkmark (similar to a “Like” in social media). After 20 minutes, the moderator reviewed each question and answer, allowing for deeper discussion while the note-takers documented these reflections.
To wrap up the session, the participants were asked one final question: “If you had the power to change one thing about construction training, what would it be?” The verbal responses to this question overwhelmingly reinforced the individual answers on the posted questions: more hands-on training.
Resulting Program Changes
The L&D team evaluated the focus group data and concluded that several interventions to the course design would be necessary to adequately address the construction training aspect of the 120-Day Claims Examiner Trainee program:
- We lengthened the online learning portion to include more time to take and retain the construction self-directed online learning modules, as well as added more time at the end of the day to reinforce the material learned that day.
- We modified the design to include discussing the construction aspect of the home whenever a claim was reviewed, thereby repetitiously reinforcing construction topics.
- We are partnering with Habitat for Humanity to have trainees volunteer (paid and on company time) to help build area homes. The L&D team believes this direct, hands-on experience of building a home from the ground up will not only expose trainees to construction terms but anchor and connect those terms to real construction know-how, allowing it to take root and build—one cement block at a time.