Organizations put huge amounts of time, money, and energy into delivering learning. Yet real insight into the impact learning has on a company remains somewhat elusive. Most companies do a good job of tracking who completed what training, but that isn’t really telling the whole story, let alone helping determine the ROI of learning.
Brandon Hall Group’s 2020 Learning Measurement Study found that fewer than 16 percent of organizations are very effectively able to identify and track a series of metrics, including participation, satisfaction, knowledge transfer, behavior change, and business impact, for any of their learning—formal, informal, or experiential.
This lack of measurement effectiveness is evidenced by the study results, which show that companies are not very good at measuring learning that is not formal in nature, and rely far too heavily on basic metrics such as completion rates and smile sheets. This shortsightedness prevents these companies from transforming learning into a strategic influencer of the business. The study also finds that organizations should be less concerned with the specific ROI of learning, but instead focus on learning’s impact on business outcomes.
Why Measure
Organizations measure their learning for many reasons. The most common is to improve the effectiveness of the learning programs. But fewer are measuring for more strategic reasons, such as creating a better link between learning and individual or organizational performance. This means that most learning measurement tends to be inward-looking, rather than focusing on the learner or the business.
Improving performance is arguably the sole purpose of L&D, but to understand learning’s impact, companies should gather the right data—and a lot of it—and do the analysis that shows what is happening.
This lack of strategic thinking is reflected in the depth that learning is measured. Using the Kirkpatrick model of learning measurement, we find few companies measuring much of their learning programs at Level 3, the degree to which participants apply what they learned during training when they are back on the job; or Level 4, the degree to which targeted outcomes occur as a result of the training event and subsequent reinforcement. In fact, 43 percent of companies say they do no Level 4 measurement at all.
Smile Sheets
Most learning measurement is relatively basic, focusing on completion rates, test scores, and smile sheets. That is because this type of measurement is easy. It takes almost no effort to track these elements, as the processes are essentially automated. But companies aren’t failing to measure more meaningful metrics because they don’t think they should. Only 9 percent of companies say they don’t see a need to measure learning at Kirkpatrick Levels 3 and 4. Beyond that, there are many reasons there isn’t enough measurement at these levels:
Unfortunately, companies that rely too heavily on completions and smile sheets are making a dangerous assumption about their learning programs. They assume all their learning programs are effective and the act of getting learners to complete them equals success. This is often not the case.
The problem doesn’t begin with measurement, however. It starts before the program is even created. Less than 4 percent of companies say the design of their learning programs is based on a set of specifically defined metrics, and fewer than 20 percent say that even most programs are designed this way. This means the Learning function has no way to measure their efforts other than the very basics.
Flying Blind
By failing to establish outcome-based metrics before developing programs, most organizations are flying blind when it comes to determining the effectiveness of their learning. They are in a constant cycle of putting content out there and hoping for the best.
The lack of meaningful measurement can be detrimental. For starters, it makes it nearly impossible to ensure the learning strategy is aligned with the overall business strategy. Metrics that demonstrate alignment are required for that to happen. The top challenge to achieving critical learning outcomes, according to Brandon Hall Group’s 2020 Learning Strategy Survey, is the inability to measure learning’s impact, cited by 69 percent of companies.
Another consequence of an immature measurement model is the inability to provide learners with either a connection between the learning and their jobs/roles, or the expected outcomes of the learning. These two items can have a huge impact on learning outcomes. According to Brandon Hall Group’s 2019 Learner Experience Survey, companies with learning programs that have a strong positive impact on things such as individual performance and time to productivity are far more likely to provide these elements than those with less effective learning.
Key Questions
To improve learning measurement, organizations must answer several key questions, including:
- Do we have a framework for measuring learning effectiveness?
- What metrics should we use to measure learning?
- How do we move beyond Level 1 and Level 2 measurements?
- What data is available to us outside of learning?
- Are we including the right stakeholders to help determine metrics?
- What technology, if any, do we need to properly measure and analyze?
- What new measurement models do we need to look at as learning continues to evolve?
To download a complimentary copy of Brandon Hall Group’s eBook “Converting COVID-19 Crisis Management to Leading-Edge People Management,” click here.
David Wentworth is principal learning analyst at Brandon Hall Group. The firm’s vision is to inspire a better workplace experience, and its mission is to empower excellence in organizations around the world through its research and tools. Brandon Hall Group has five HCM practices and produces the Brandon Hall Group HCM Excellence Awards and the annual HCM Excellence Conference, in West Palm Beach, FL.
