Smile sheets! Yech! In the crime thriller that might be the Training industry, smile sheets are the darkest of back alleys, oily puddles of water, strewn trash, potholes, broken fragments of macadam, desolate footsteps, a music backdrop thick with tension. We’d rather not be cornered defending our smile sheets. In truth, we’d rather not have to think about them at all. “They’re not my job! I’m an artist, a freedom fighter. I help my learners rise against mediocrity. I bring knowledge to light and skills to my brother and sister citizens. I am an instructional design superhero. Smile sheets are for Voldemort, Hannibal Lecter, Gordon Gecko. I am transcendent. Smile sheets are common, mundane, secretarial. Give me a storyline, not a survey. Give me a simulation, not feedback. Give me magic, not necessity. I am who I want to be, not what they tell me I am!”
OK, let’s stop daydreaming. In truth, we know we have to use smile sheets. Learners expect them; our organization needs some sort of feedback; and darn it all, they’re traditional. True, we have serious doubts about their benefits, but hey, they’re pretty benign, aren’t they?
No. No. No. No. No! Here’s the truth: Smile sheets are the most dangerous tool in the instructional design toolbox! They are well-coiffed punks who hide Glocks in their waistbands. They give us bad information, which we treat like sacred scrolls. Research looking at more than 150 scientific studies has demonstrated that traditional smile sheets are virtually unrelated to learning results—really! And yet, you and I and our organizational overlords still look at our smile sheet results and evaluate our training as good or bad.
But enough of darkness! The truth is that we can do better. No, we won’t create a perfect smile sheet, but we can create one that gives us more valid feedback. Later, I’ll describe how this can be done. First, let me advocate for the singular importance of smile sheets.
YOUR SMILE SHEET CAN BE YOUR MOST IMPORTANT FRIEND
Smile sheets will never be omniscient and we can’t rely on them alone, but what if we could get good feedback? What if we could make instructional design decisions based on our smile-sheet results? Wouldn’t that help us improve our learning designs? Of course, it would! Feedback begets improvement. Lack of feedback gets us lost.
Let me digress a minute to talk about famed psychologist B.F. Skinner’s pigeons. Skinner decided to find out what would happen if he randomly rewarded his pigeons. Instead of giving his pigeons a reward every time they pecked a lever—or through some variation of intermittent reinforcement whereby they earned a pellet after some number of pecks—he decided to give them a food pellet at random times regardless of what they did. What do you think happened? What did the pigeons learn to do? What behavior changed?
I’ll report the finding in a minute, but first let me tighten the analogy. We are like Skinner’s pigeons. Our smile sheets, because they are virtually uncorrelated with learning, reward us randomly. Sometimes they give us valid feedback and sometimes they give us invalid feedback— even dangerous feedback. We can’t tell when we’re getting good feedback or bad, so we make decisions that don’t make sense. Sometimes we do more of the good stuff. Sometimes we do more of the bad stuff. Chickens with our heads cut off? Or pigeons?
One of Skinner’s pigeons started noticing that it got rewarded when it pecked its shoulder, so it pecked its shoulder again. As it was pecking its shoulder, it got another random reward—so it got verification that shoulder-pecking was good, and it pecked its shoulder more and more and more until it pecked itself raw! Another pigeon noticed that it got a pellet—a reward—when it circled to the right. Afterward, as it circled to the right, it got another random pellet, confirming its hypothesis. So, being a smart pigeon, it kept circling to the right. While nothing happened for a while, eventually another pellet appeared (remember these pellets are randomly delivered), and the pigeon learned again that circling right was rewarded. Poor pigeon! It lived its remaining days circling to the right, evermore—mindlessly caught in a dance of misinformation.
Although you might want me to highlight the analogous state of U.S. politics, what I really want to emphasize is our behavior as workplace Learning professionals. Without valid feedback, are we not like pigeons dancing in circles, leaving our learners unfulfilled?
We are! We are getting poor feedback and we’re making poor decisions in learning design and delivery.
THE PERFORMANCE-FOCUSED SMILE SHEET
Traditional smile sheets may give us a good sense of learners’ satisfaction, but they don’t tell us anything about learning. If we could create a smile sheet that did both—tracked satisfaction and learning—wouldn’t that be better?
In my new book, “Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form,” I show in some detail how traditional smile sheets have let us down. Likert-like scales are one of our biggest problems. The truth is that smile-sheet decision-making gets distorted when we rely on the fuzzy response terms—such as “strongly agree” and “agree”—in our Likert-like scales. Bias increases when learners can’t see real distinctions between options. It’s better to use more concrete answer choices. For example, in asking about whether learners are likely to use what they learned in their jobs, we can do better than “strongly agree” versus “agree” by offering choices such as the following:
A. I don’t have sufficient knowledge of key concepts.
B. I have awareness of key concepts, but I’m not ready to put them into practice.
C. I have awareness of key concepts, but I need more practice.
D. I have awareness of key concepts, and I can put them into practice.
E. I have awareness of key concepts, and I can use them with complete success.
By creating stronger distinctions between our answer choices, we create three critical benefits. First, we help our learners make better smile-sheet decisions and we avoid the garbage-in, garbage-out problem. When learners have to choose between “disagree” and “strongly disagree,” they simply can’t be precise. Second, we motivate our learners to engage in our smile sheet questions. When they see there is more clarity between alternatives, they know the data will have more meaning and be more beneficial. Third, by having more granularity between our answer choices, we create much better data. Previously, we’d be stuck contemplating the difference between a 4.1 and a 4.3—the numbers we generated with our fuzzy smile-sheet mathematics. Such averages had no inherent meaning, so many times, we found ourselves in a state of paralysis, not knowing what to do about our learning designs.
With Performance-Focused Smile Sheets— and specifically with our more granular answer choices—we can designate standards before we deploy our learning. For example, in the question to the left, Answers A, B, and C might be deemed “unacceptable,” Answer D might be “acceptable,” and Answer E might be labeled “superior.”
The lack of distinctiveness between answer choices is the first major weakness of traditional smile sheets. The second major weakness is that smile sheets too often ask questions that are not directly relevant to learning effectiveness. We ask about satisfaction, whether people liked the trainers, whether the room setup was acceptable, etc. We’ve tended NOT to ask questions aligned with findings from the science of learning. For example, we fail to ascertain whether learners got sufficient practice, whether they became motivated to apply what they’ve learned, or whether there are supports in place that will help them apply what they learned to their jobs.
THE SINGLE MOST IMPORTANT THING YOU CAN DO THIS YEAR
Our current smile sheets are poorly designed. They give us invalid feedback. They push us to focus on the wrong things. They don’t support learning or learning improvement. Obviously, we can do better.
Improving your smile sheets this year might be the single most important thing you do—because with better smile sheets, you’ll get better data. With better data, you’ll make better learning design decisions. With better learning designs, you’ll create better learning and better on-the-job performance. With better on-the-job performance, your organization will thrive, your fellow employees will be more likely to reach their goals, and you’ll burnish the image of the Learning department. Perhaps as you build trust in this way, you and your Learning professional colleagues will be consulted more often, you’ll be enabled to do more potent work, and you’ll all become rich and famous.
Well, shucks! Who needs those extrinsic reinforcers when we’re helping people learn?
Will Thalheimer, Ph.D., is a consultant and research translator, providing organizations with learning audits, research benchmarking, workshops, and strategic guidance. Thalheimer is one of the authors of the Serious eLearning Manifesto (eLearningManifesto.org), founder of The Debunker Club (Debunker. Club), and author of “Performance- Focused Smile Sheets” (SmileSheets.com). Thalheimer blogs at WillAtWorkLearning.com, tweets as @WillWorkLearn, and consults through Work-Learning Research, Inc. (Work-Learning.com).