What if you could take a peek at the interaction other Learning and Development (L&D) professionals build into their training sessions? In the last year alone, more than 3 million interactive PowerPoint questions were created in audience response app Poll Everywhere. Seventeen percent of those were created by learning designers, chief learning officers (CLOs), and subject matter experts (SMEs) to facilitate quick formative assessments, evaluations, and learning discussions.
If you’ve ever used a clicker system to vote in a class or meeting, you’ll have some idea of what new polling systems do in the training session. Device-agnostic systems such as Poll Everywhere are the descendents of clickers. Participants use their phones or laptops to respond to live questions on the PowerPoint screen, and the results appear in a real-time display on the slide. But rather than simple ABC voting, the deviceagnostic system can handle open-ended questions, plus response upvoting for lively Reddit-style discussions, and compelling visualizations such as living word clouds. Corporate Training departments have been quick to pick up on the applications for interactive learning.
When the Poll Everywhere data science team anonymized and analyzed millions of L&D questions, they discovered some standout trends in interactive learning for the workplace. Some of them were entirely predictable (such as the prevalence of Kirkpatrick’s Level 1 evaluation), and some were a little more surprising.
ANALYSIS METHODOLOGY
Poll Everywhere data scientists ran a Latent Dirichlet Allocation (LDA) analysis to identify key topics among millions of questions and poll titles, categorized by the type of usage. In this case, they looked at questions from accounts that were self-identified as corporate training or continuing education presenters. An LDA analysis quantifies how often words are used, removes “noise” from unhelpful words, and groups them into topics. This makes it easier to organize a giant mass of questions into a few prevalent topics that carry meaning.
TREND #1: SALES AND MARKETING WIN AT INTERACTIVE LEARNING.
The No. 1, most-used topic for Poll Everywhere questions asked by learning pros was the sales and marketing topic. This topic is characterized by question words such as: product, customer, business, company, market, and brand.
Why is a training module for sales and marketing more likely to include interactive polling questions? Interaction is the name of the game for this crowd. Sales staff members are on the phones, at the expo hall, on the front lines with customers. And marketing teams live and breathe people insight. For both of these groups, it makes sense that their training would include lots of lively discussion, quick answers, and hands-on engagement with both the instructor and their co-learners.
It isn’t strictly academic interaction, however. From interviews with Poll Everywhere customers, the team learned that sales reps tend to love a good competition. Their case studies cite sales trainers at KNEIP and HP, for example, running live contests and quizzes through the polling system to foster teambuilding and employee engagement. With the combination of live training assessments, discussions, evaluations, and department-wide games, too, it’s no wonder sales and marketing land in first place among interactive question topics.
TREND #2: KIRKPATRICK’S LEVEL 1 EVALUATION IS STILL KING.
Like it or not, the data team found tell-tale signs of Level 1 (Reaction) evaluations among the second most common group of L&D question words: rate, level, scale, highest, lowest, score.
Other studies have found the same results. Bersin/Deloitte reported that 90 percent of training evaluations are Level 1 Reaction evaluations (compared to only 5 percent at Level 4, which measures impact on business metrics). This often appears at the close of a module, as an exit slip, “On a scale of 1-5, how would you rate the training you received today?”
Now that audience response apps have begun to replace paper evaluations, corporate learning pros often use a multiple choice poll to make it easy for learners to instantly choose a score on the Likert scale, porting the results directly to a learning management system.
But a few are beginning to use new options such as a clickable smile sheet image instead. In more advanced cases, they run a before-and-after analysis, asking a suite of questions about prior knowledge before training, and again after training to measure progress. Perhaps someday we’ll see a rise in questions for Level 2 (Learning) and beyond, maybe to the point where the CLO routinely uses a mobile phone survey to check in with managers on the long-term results of training initiatives (Level 4 at last).
TREND #3: “GOOD LEADERSHIP” IS A HOT TOPIC AT EVERY LEVEL.
It shows itself as the third most prevalent topic, in a cluster of frequently used words: good, leader, change, conflict, characteristics, make, style. You see it in questions such as:
- What characteristics do good leaders have in common?
- How do good leaders handle conflict?
- How do good leaders manage change?
Because of the high frequency of leadership words in training questions, it’s unlikely that these are limited to management training. Rather, the volume of questions in this topic indicates that leadership discussions are becoming a staple of the learning that happens on a day-to-day basis, at nearly every level of the company.
It’s possible that this free and frequent discussion of leadership is in part because of the new capabilities of live mobile phone polling. The Poll Everywhere team has interviewed more than one corporate customer using the app to level the playing field in discussions between management and employees. From Ask-Me-Anything CEO events at Blinds.com to diversity and inclusion workshops, it’s easier for entry-level employees to feel safe speaking honestly when they have that layer of anonymity provided by their phones, along with the confidence of seeing other responses on the big screen mirror their own experiences and opinions.
TREND #4: LEARNING DISCUSSIONS GET PERSONAL.
The amount of language surrounding home, family, and personal life in corporate learning environments might surprise you. Poll Everywhere question data shows the home/family topic came in fourth place among training questions with words such as life, good, family, time, feel, friends, give, love.
This might be another case of technology making discussions more accessible. Personal questions work well in the form of anonymous (or even non-anonymous) audience response system questions. “How do you make room for family time?” Or “What’s your idea of a good time with your friends?”
It’s a safe way to build camaraderie among new recruits without fear of embarrassment. Once one person starts to get honest, and his or her response appears on the screen, instructors report that a “snowballing vulnerability” effect takes hold. Suddenly, everyone is willing to jump in and discuss sensitive topics with total candor. It’s a powerful moment for a new group of coworkers to see how much they have in common.
When used well, an anonymous discussion about a personal topic can even promote goodwill toward the company. Employees intuit that the company recognizes the importance of “real” life, not just work life. That’s the kind of engagement that produces tangible results in the long run.
Rebecca Kerr is a Marketing and Communications manager for Poll Everywhere, a learner engagement application used in more than 60 percent of the Fortune 500, and by 300,000-plus educators worldwide.
