Will your next program deliver results that matter most to the business? If not, why not? If so, how do you know? A talent development program’s failure to deliver expected business results is often due to ineffective design and strategy implementation. Programs successfully delivering value were designed to do so. Pressure by senior leaders and other funders is compelling program owners to deliver on their promise. The promise? That their programs will add value that matters and do so at a reasonable cost
The Concept
Most professionals would agree that participant manager involvement is a key to program success. They likely also will agree that by beginning with the end in mind with precise impact data, there is a greater likelihood of ending with improvement in that impact data. Another likely belief is that if program objectives focus on impact, there is a greater chance the program will deliver that impact. Just how much difference do these factors make?
Enter the ROI Estimator. The ROI Estimator helps program owners address common challenges head-on by offering invaluable insights into program effectiveness and actionable suggestions for enhancement. Whether you’re embarking on a new program or refining an existing one, this tool serves as an indispensable resource for maximizing outcomes and optimizing return on investment (ROI).
The idea for an ROI Estimator originated during the pandemic and represents significant research into understanding the effects of design and implementation decisions on the outcome of programs. The study was conducted by ROI Institute and funded by Warwick Conferences in the United Kingdom.
Back Story
Warwick Conferences helps clients organize, connect, and conduct learning programs in their conference centers. Their clients had been discussing the concept of ROI. Warwick Conferences decided to conduct research to see if the need was real. Their research brought some surprising results:
- ROI is important.
- Executives and CFOs expect ROI.
- Learning and development should address the issue. (Warwick Conferences and ROI Institute. The ROI Conundrum: Attitudes and Barriers Towards Providing the ROI of L&D. Research Report, 2020)
In today’s dynamic and competitive business landscape, investing in talent development is essential. Given the resource requirements to do so, optimizing the ROI in those investments has never been more crucial. With that in mind, Warwick Conferences approached ROI Institute with a request to develop a tool their clients could use to determine if their programs would deliver planned results.
Collectively, ROI Institute and Warwick Conferences decided to focus on the program design perspective, analyzing how the programs are designed to deliver results, the implementation process, and the nature of the program itself.
The Research
The ROI Estimator was meticulously crafted through extensive research. Drawing on ROI Institute’s comprehensive database of more than 9,000 impact studies, members of the ROI Institute team selected 2,000 ROI studies focusing on talent development for analysis. The team comprised senior staff members, associates, and partners who have extensive experience conducting ROI studies. Combining data with experience, the result was 14 critical success factors that directly contribute to positive return on investment in learning programs.
The team systematically compared ROI studies where each factor was present against those where it was absent. The working hypothesis was that programs incorporating these factors would demonstrate higher returns on investment. To validate these findings, we conducted statistical tests to assess the reliability of the outcomes, adjusting for the number of case studies and determining an error range for each factor. This refinement process was applied to the 14 factors, allowing us to quantify the incremental differences in ROI attributable to each factor.
Ultimately, we identified the eight most important factors that, when properly addressed, guarantee program success and deliver a favorable ROI. The resulting data powers a robust and credible assessment of the eight key elements influencing a program’s ROI. It provides users with a reliable estimator to evaluate and enhance their investment strategies based on empirical evidence.
The eight factors include:
- Direct Reports: Involve participants with direct reports.
- Business Alignment: Align the program to business measures in the beginning.
- Appropriate Solution: Ensure that the proposed program is the right solution to improve the business measures.
- Focus on Success: Design for success at each outcome level.
- Appropriate Audience: Involve the right audience for the program.
- Perceived Value: Ensure that the program is valuable to participants.
- Manager Support: Establish supportive partnerships with key managers.
- Delivery: Minimize distraction and interruption with offsite facilities.
The Algorithm
The research boils down to this important algorithm:
|
R = Base + XA + XB + XC + XD + XE + XF + XG + XH Where R = Expected return on investment (ROI). The X is the ratings from your proposed design and implementation. The A, B, C . . . are the factor weights. |
The base is negative 80 percent, the expected ROI if none of the factors are present, which is obviously a disappointing result. Usually, a negative 100 percent is the worst-case scenario. This is when the program delivers no benefits despite the program’s cost. A negative 80 percent indicates that the program delivers some value and that the factors not included in the estimator are likely contributing to that value.
The Weights
Obviously, some factors are stronger influencers than others, and the weight is based on the amount of that influence.
A = Direct Reports
B = Business Alignment
C = Appropriate Solution
D = Focus on Success
E = Appropriate Audience
F = Perceived Value
G = Manager Support
H = Delivery
The Ratings
The X values are the ratings program owners give each factor based on their presence in the program design.
Rating of 1, no presence at all (X = 0)
Rating of 2, some presence (X = 1)
Rating of 3, moderate amount of presence (X = 2)
Rating of 4, significant amount of presence (X = 3)
Rating of 5, very significant amount of presence (X = 4)
The options have enough detail to decide which rating is best for the planned design and implementation. The weight of the factors is the secret sauce for the algorithm.
An Example
As an example, we selected a program from a popular off-the-shelf provider, The Leadership Challenge, offered in a major car rental organization. Follow along in the ROI Estimator at https://roiinstitute.net/roi-estimator/ to experience the process.
Inputs
As mentioned above, there are eight factors that proved to be most influential in driving a positive return on investment in learning programs.
Direct Reports
In this case, the participants are first-level managers, and 20 will attend as a group. Each participant has about 12 direct reports.
Business Alignment
The auto rental business is competitive and price sensitive. Customer service is critical. Executives were interested in addressing key business measures in operations, customer service, service and support, sales, administration, finance and accounting, and information technology. While they had no clear business measures in mind, they knew developing leadership skills would drive business measures when applied in the work units.
Appropriate Solution
The Leadership Challenge was identified as the right solution because it focused on competencies executives believed were most important to the organization. The decision was not made based on a needs assessment or predefined competency model determined to support the business.
Focus on Results
While there was an underlying intent for participants to apply what they learn and improve business measures, the program objectives focused on participant reaction and learning. There were no application or impact objectives per se to direct the design, delivery, and ultimate program outcomes.
Appropriate Audience
The program targeted first-line supervisors, which was the target audience most closely connected with performance in the functional areas of interest.
Perceived Value
Based on the conversation with those participating, it was believed they would perceive the content as valuable once they began the program. Successful programs ensure they get buy-in from participants early and throughout the program.
Manager Support
Managers of the firstline supervisors participating in the program fully supported the program. They knew they would be coaching the supervisors and taking on an active role in their development.
Delivery
The off-the-shelf program is a five-day program delivered in-person at the company facility. This was important to the leaders in that they knew they would get better content and would have the opportunity to better engage with colleagues.
Interpretation
Here are the potential ranges for the estimated ROI:
-26 percent to -100 percent: Very unsuccessful; program needs significant improvement.
-6 percent to -25 percent: Unsuccessful; some elements are present for success; multiple opportunities for improvement exist.
-5 percent to 25 percent: Average success; need more focus on improvement.
26 percent to 100 percent: Above average success; refine current processes for even more improvement.
100 percent to 200 percent: Excellent program; should yield a positive business contribution. There is still potential for more results.
Results
The actual results are shown in the next screenshot. The resulting 139 percent ROI means that for every $1 invested, the program should return the dollar invested plus an additional $1.39, or a return of 139 percent. Given the estimator is providing an indicator of potential success, the output includes a range. As shown in the figure below, there is a range of +/- 22 percent. This means the ROI could be as low as 108.42 percent or as high as 169.58 percent.
Improving the ROI
In the next screenshot, you can see that the ratings for each factor are either a 2, 3, or 4. Anything less than 5 is an opportunity for improvement. Those factors with the lowest ratings offer the greatest opportunities for improvement. The beauty of this process is that if the output of the estimator is a negative ROI, you can see what to change in terms of design and implementation to improve the potential impact and ROI.
Using the Tool
The ROI Estimator is a valuable tool for anyone seeking to evaluate the effectiveness of their important, expensive, and strategic program by understanding their financial returns. ROI is vital because it provides a clear picture of how well resources are being utilized to achieve desired outcomes, helping to ensure that every dollar spent contributes to meaningful results. As a follow-up, a clearly developed, reliable, and credible ROI study is also important for executives so they see the actual results and so program owners can make further adjustments in their programs.
When using the ROI Estimator, it is crucial to be candid with your input—if exact data are unknown, make the best estimate. If questions arise, an ROI Institute team member can work through the scoring together to interpret the results, identify areas for improvement, and help you develop strategies to enhance future performance. By mastering this tool, talent development professionals can refine their approach to measuring ROI and learn how to achieve better results through more informed decision-making.
You Need to Follow Up
If a program is implemented based on the estimator, it is imperative to follow up with an actual ROI study. This ROI value then is compared to the estimator. This is a credibility issue. Ideally, it will be larger than the estimator. The executives will be interested in the actual follow-up data. If not, it is best to review to see if some of the ratings were overstated. Also, from ROI Institute’s perspective, we would like to see your follow-up study results to see how they compare with the actual ROI estimator. We will collect studies from many users to check the validity of this process.
To access and learn more about the ROI Estimator, please visit:
https://roiinstitute.net/roi-estimator/ or e-mail: info@roiinstitute.net
