As we close in on the end of 2020, we’ve also reached the end of our review of the “basics.” We have traveled through Assessment and Analysis (https://trainingmag.com/trgmag-article/you-still-need-do-work/), Design planning (https://trainingmag.com/trgmag-article/design-moreskin-deep), and Develop and Implement (https://trainingmag.com/trgmag-article/magic-developimplement). Now let’s talk through Evaluation. A quick reminder: The evaluation plan is part of the design planning and is included in the needs analysis (report) to leaders and stakeholders.
The performance objectives inform the plan for what gets evaluated, in what timeframe, and what intervals. The more complex the business need—and the learning required to address it—the more robust the evaluation plan must be. The way ADDIE is the basis for planning the instructional design work, Kirkpatrick’s Four-Level Evaluation Model is the basis for planning the evaluation strategy.
LEVEL 1: REACTION
Administer Level 1 evaluations right after the learning is complete. Evaluations should be written to elicit learners’ feedback about their experience while completing the learning content. Level 1 evaluations also might measure the degree to which learners find the experience of learning favorable, engaging, and relevant to their work. Ask learners to provide feedback on items such as:
- Were the instructions clear and easy to follow?
- Were you easily able to use the system/tools?
- Which tools and practices will you put into action immediately?
- Which aspects of the learning content will you apply on the job immediately?
- Are the content and materials what you need to excel on the job?
- Was the program a good use of your time?
- How will you apply what you’ve learned at work?
- Would you recommend the learning program to your peers? Why or why not?
When strategically crafted, Level 1 evaluations provide feedback results that reveal unmet learning needs, barriers to learning, and where the content could be adjusted to improve its impact.
LEVEL 2: LEARNING
Learners complete Level 2 evaluations shortly after the learning is complete. This might be done in phases—in most instances, one to two weeks afterward. Level 2 evaluations often include post-training tests of knowledge and use of skills.
LEVEL 3: BEHAVIOR
Level 3 evaluations are completed by the learners’ direct supervisor and possibly a few peers, depending on the need. The intent for Level 3 evaluations is to determine the extent to which the learners apply the learning in their work.
Craft evaluation items in a manner that accounts for any variables that may impede application of the learning on the job. (i.e., delays). Equip direct supervisors with useful items such as: a set of standards for observing employees consistently, how best to formulate feedback, where to report results, etc. Level 3 evaluations might take the form of on-the-job observation, skill drills, quizzes, simulations, or scenarios. Surveys work, too, but likely provide more limited results.
LEVEL 4: RESULTS
Not all learning programs will be evaluated through Level 4. However, don’t miss the opportunity to evaluate the more complex learning projects, those with significant exposure to the organization, involving many employees. Such projects require an evaluation plan that measures the impact on business operation through Level 4. Not doing this is a mistake with significant opportunity cost to the Learning team and affiliated groups who participated in the training.
Be creative, invested, and involved in the process of evaluating your learning content. Think beyond just surveys. Possible data collection methods for Levels 1, 2, and 3 include: interviews, focus groups, surveys, rating scale (i.e., Likert), open-ended and multiple-choice questions, pre- and post-questionnaires, and formal and informal reactions to the learning experience through meetings. These meetings might target specific areas, such as engagement, relevance, and learner satisfaction.
Whatever you do to celebrate the arrival of a new year, be safe! I look forward with you to 2021!
Dawn J. Mahoney, CPTD, owns Learning in The White Space LLC, a freelance talent development (“training”) and instructional design consultancy. She is passionate about developing people through better training, better instructional design, and better dialog. Mahoney asks the tough questions to ensure the training content is relevant to the work and performance expectations. She does this work because she loves to see the moment when the learning “dawns” on her learners. If you need help, get in touch with her at: dawnjmahoney@gmail.com.