By Melissa Fein, Ph.D.
Standard setting describes the process used to determine the thresholds of minimally acceptable performance levels. Standard setting is a critical component of the evaluation of learning events (Level 2 evaluation in the Kirkpatrick framework), as well as the evaluation of how effectively learning is subsequently being applied on the job, as evidenced by job performance behaviors (Kirkpatrick evaluation Level 3). Attending to fundamental standard setting, process issues can provide support for the validity of using and interpreting a test score or on-the-job behavioral measure for a specific decision-making purpose. Regardless of the particular method used to set standards, the main process issues revolve around establishing a standard-setting team, facilitating the standard-setting team in working together to create a vision of a minimally competent person, motivating the team members to agree to personally take the assessment, managing standard setting activities so closure is made on a defensible decision within the time constraints faced by the team, and managing team member absences and personnel changes.
A standard-setting team should comprise subject matter experts (SMEs), a planner, and a facilitator. All participants on the standard-setting team need to sign confidentiality agreements. The SMEs are the team members who are making the standard-setting decision, with the guidance of a facilitator; the planner handles logistics related to the standard setting activities. The credentials of the SMEs on the standard-setting team need to be documented as a part of the standard-setting process. This provides evidence supporting content validity. The credentials that need to be documented include: relevant positions held; the amount of time served in those positions; relevant education, training, licenses, registrations, certifications and certificates held; general contact information; and references, such as in the following job aid insert:
Job Aid Insert: SME Credentials Form
Name:
Work Telephone:
E-mail Address:
Work Address:
Current Job Title:
Time Served in Position:
Previous Positions:
1. Dates
a. Job Title
b. Organization
c. Address
d. Supervisor
2. Dates
a. Job Title
b. Organization
c. Address
d. Supervisor
3. Dates
a. Job Title
b. Organization
c. Address
d. Supervisor
Education:
Licenses/registrations/certifications/certificates:
Additional relevant training:
Professional references:
1.
2.
3.
The planner should have some general understanding of the standard-setting process but does not need to be an expert; he also should have the ability to communicate effectively with critical decision-making personnel within the organization. The facilitator should have expertise in the standard-setting process and be able to manage the group dynamics of a standard-setting session in person or electronically. The planner and facilitator functions can be accomplished by the same person, but they do not have to be.
The first order of business in any standard-setting session, after giving team members a general orientation regarding the purpose of the session and expectations, is to have the SMEs define a hypothetical minimally competent person. There is almost always a humorous backlash to this request. The SMEs perceive that the whole reason they are involved in exam development is to ensure that the highest standards are being met, and they often find it ironic that they are being asked to think in terms of minimal competence. The issue is that most SMEs have optimal or ideal competence foremost in their minds and they need to make a shift from the idea of optimal performance to envisioning the lowest performance level a person can have and still be considered competent. Facilitating a productive discussion on the concept of minimal competence can enable the SMEs to (hopefully) come to a shared vision of what this means operationally, in terms of applying this to standard-setting activities. Success of the standard-setting session depends on some type of consensus, agreement, or at least a manageable degree of disagreement on this concept.
It is valuable to have the SMEs personally take the exam as if they were examinees. Even though items have (hopefully) been reviewed prior to the standard-setting session, additional item flaws or scoring problems can become startlingly obvious when the exam is actually administered to SMEs on the standard-setting team. Some SMEs will not be happy about being asked to take the exam; they may suggest that they are the experts and should not be required to submit to this activity, while others will worry that they will do poorly and be embarrassed. SMEs can be reassured. Scores should not be shared. Each SME will know her score, and this will inform her participation in the standard-setting session.
Management of standard-setting sessions will vary depending on the method used, the session format (for example, in-person versus electronic), and the assessment format. In-person standard-setting sessions are the most desirable format for many reasons, one being that when teleconferencing, nonverbal communication can be missed in coming to a shared vision of minimal competency, which is a core theme of the session. However, Web conference sessions, which provide screen sharing or video, are feasible in many contexts. The reality of the resources available for standard-setting activities cannot be ignored, but lack of resources is not an excuse to cut corners. If an in-person standard setting session is not going well, and team members find themselves spinning their wheels, the facilitator should urge them not to make a rushed decision for the sake of having closure, but should get agreement on the feasibility of having some type of follow-up session, either in-person or via a teleconference or Web conference.
Ideally, members of the standard-setting team will be present for all sessions and none will drop out and need to be replaced. Realistically, people will miss sessions and a member or even several members might need to be replaced. When attrition happens, it is particularly important to have a process ensuring that absent or new members are updated on what transpired so they understand the process and rationale that went into the shared vision of the hypothetical minimally competent person.
Adapted from “Test Development: Fundamentals for Certification and Evaluation” by Melissa Fein (ASTD, September 2012). For more information, visit http://www.astd.org/Publications/Books/Test-Development
Melissa Fein, Ph.D., has graduate degrees in statistics, psychology, and economics, along with coaching certification, and first started working in the field of training evaluation in 1988. She began private consulting on industrial psychology/evaluation issues in 1996, and for several years was affiliated with the Maryland Assessment Research Center for Education Success (MARCES), at the University of Maryland, College Park, where she has taught a variety of courses in measurement, statistics, and evaluation.
