Lessons Learned in Implementing Performance Support Platforms

“Day One Performance” was a vision developed in the 1990s by Gery (1991) and other EPSS (Electronic Performance Support Systems) practitioners. This vision portrays a seemingly perfect solution for corporate IT competency challenges. The solution was based on the most basic form of workplace learning: apprenticeship. The innovation this solution brought forth was that in the modern workplace the expert was replaced by an expert system, which, in turn, became scalable throughout the IT systems. EPSSs were regarded as a combination of both performance and online learning solutions. The use of relevant information while performing a specified task should improve performance while reducing the need for pre-task formal learning.

Though fostering great promise, sometime in the last decade the term, “EPSS,” faded into other more established domains, such as Knowledge Management (KM) and e-learning.

As an e-learning director at a telco company that implements two types of performance support platforms (EPSSs), I was amazed to learn about implementation success stories in some call centers and total failures in others. I called upon colleagues from the Tel Aviv University and together we conducted empirical research, in which we investigated the success factors in performance support platform implementation.

Research and Findings

The research was conducted in a telco company that implements two types of performance support solutions:

1. KMS (Knowledge Management System—also known as an External EPSS). All professional information is stored in knowledge items accessible via manual search. During task performance, the user refers to the KMS, searches for information, reads the specified item, and incorporates it accordingly in the task.
2. Interaction Management System (Intrinsic EPSS). The system displays information and data within the IT system screens according to preprogrammed rules. Support content is “pushed” automatically on the screen. Users refer to the information nuggets while on task.

The scarce research and literature about this topic view the intrinsic approach as more effective, due to the immediacy of knowledge retrieval while on task.

Procedure

Some 276 call center agents participated in the research. Approximately half were in the initial agents’ course during the research and the others were veteran agents working in call centers. Each participant performed three common service call scenarios in a subject matter area unfamiliar to him. Half performed the scenarios using the KMS (external EPSS) and half using the Interaction Management System (Intrinsic EPSS). The performance and learning effectiveness variables that were measured are:

• Time on task
• Quality of performance
• Task comprehension level
• Confidence level

A pre-experience questionnaire was used to collect information about the users’ attitude toward the EPSS, their e-learning experience, and their willingness to use an EPSS solution. A post-experience questionnaire was used to collect information about satisfaction levels and the users’ willingness to reuse the EPSS. The findings from both EPSSs were compared to one another and to an absolute scale indicating the desired performance level set by the company.

Findings

Performance and Learning Achievements

Figure 1 (please download the file below) displays findings according to EPSS type (External vs. Intrinsic) and organizational environment (Learning vs. Work). It shows the following:

1. EPSS type has a high influence in the learning environment. Once participants enter the work environment, the specific type of EPSS affects only the quality of performance.
2. Participants using an intrinsic EPSS while in preliminary training attain a high organizational competency level at a very early stage.
3. Self-confidence is not influenced by the type of EPSS or by the attained performance level.

Attitude and Satisfaction Levels

Correlation (Pearson) measurements were conducted between attitudes and the performance/learning variables findings. The learning environment participants’ attitude toward learning and working with an EPSS was found to be significantly high compared to work environment participants. As exhibited in Table 1, the strongest correlations were found when the research population was divided according to organizational environments. The learning environment participants’ attitude toward the EPSS was found to be strongly connected to their willingness to reuse the EPSS and their satisfaction levels from learning with the EPSS.

Table 1 – Pearson Correlation between Attitude towards EPSS and User’s Satisfaction from Learning with an EPSS and Willingness to Reuse it

*P<0.05 **P<0.01

Lessons Learned

EPSS effectiveness—Learning While Performing
When considering IT skills, “Day One Performance” can be put to practice with an intrinsic EPSS solution. However, the current research shows two restrictions that must be considered:

1. As opposed to what is stated in professional literature, EPSS does not enhance confidence levels in novice employees. Sending them “out there” with IT skills they do not yet recognize most likely will result in poor performance.
2. Once the user gains experience with the external solution (KMS), he can perform tasks just as quickly as a user of the intrinsic solution. On the other hand, the performance of users utilizing intrinsic solutions is still of better quality.

We concluded that the seeming inferiority of external solutions arises from designated EPSS skills, which novice workers lack and veteran workers partly exhibit, due to their daily experience with the EPSS.

These skills are:

1. Locating and processing information while working (Bastiaens et al., 1997; Bastiaens, 1999; Nguyen, 2006).
2. Converting declarative knowledge into procedural knowledge (van Schaik, 2010).

We, therefore, suggest that EPSS implementation efforts start during the initial training courses. Intrinsic EPSS potential can be realized there while crucial external EPSS skills can be gained, supporting effective implementation later on.

Learning with an EPSS
The current research provides substantial evidence as to the ability of EPSS to allow effective real-time learning. The research further indicates that users perceive the EPSS as an effective learning tool and tend to preserve this notion regardless of their initial experience with it. We, therefore, suggest that EPSS should be incorporated in all formal and informal learning efforts. We strongly recommend creating a positive attitude toward the solution before introducing it in order to support successful implementation.

References

Bastiaens, T.J. (1999). “Assessing an electronic performance support system for the analysis of job and tasks.” International Journal of Training and Development, 3(1), 54-61.

Bastiaens, T.J., Nijhof, W.J., Streumer, J.N., & Abma, H.J. (1997b). Working and learning with electronic performance support systems: an effectiveness study. International Journal of Training and Development, 1(1), 72-78.

Gery, G. (1991). “Electronic performance support systems: how and why to remake the workplace through the strategic application of technology.” Tolland, MA: Gery Performance Press

Nguyen, F. (2006). “What We Already Know Does Matter, Expertise and Electronic Performance Support Systems.” Performance Improvement Journal, 45(4), 9-12.

van Schaik, P. (2010). Psychological perspective. In P. Barker & P. van Schaik (Eds.), Electronic performance support: using technology to enhance human performance. Aldershot, Hants: Gower.

Eran Gal is an e-learning and performance support consultant. Until recently, he held the position of E-Learning & Instructional Technologies Director in a large telecommunications firm in Israel. He has more than 14 years of experience as an instructional designer and training manager in large corporations, mainly in implementing technology based cross-organizational learning solutions. Since 2007 he has been leading a large-scale implementation project of an advanced electronic performance support systems platform, supporting approximately 4,000 users nationwide. He is a doctoral student in the School of Education, Knowledge and Technology Lab at the Tel Aviv University. The research is focused on the effectiveness of performance support technology in corporate settings. For more information, e-mail Gal.pbt@gmail.com.

Rafi Nachmias is the head of Tel Aviv University’s School of Education. Since 2000, he has headed the Science and Technology Education Centre (SATEC), and the Virtual TAU initiative in Tel Aviv University. Professor Nachmias was the National Research Coordinator of the IEA’s Second International Technology in Education Study (SITES) and the TIMSS study. He participates in EU’s FP7 project on the implementation of innovative IT tools in academic instruction. He has published five books on ICT in education, Web-based learning, and science education. His major research areas are: Web-based learning, educational data mining, Web-based academic instruction, and innovative pedagogical practices using ICT and science education. For more information, e-mail nachmias@post.tau.ac.il.