As we know evaluation is a key part of Instructional Design — it is the only way to know whether our training program actually produced learning. Unfortunately, evaluation is also one of the parts of ID most frequently left out or short-changed.
This week in IPT 536 we learned about Kirkpatrick’s 4 Levels of Evaluation and created a Level 1 smiles sheet utilizing ARCS. Donald Kirkpatrick developed the Four Levels Evaluation Model
Level 1: Reaction
- To what degree participants react favorably to the training
CRUCIAL QUESTION: Were the participants pleased with the training program? Did they like it? Will they come for another program? Will they recommend it to others?
IMPORTANCE: According to Kirkpatrick learner reactions can make or break a training program. First because what learners say about training filters up to higher-level managers who make decisions about the future of training programs. Second because negative reactions inhibit learning (positive ones do not ensure learning, but they do not prevent it). Measuring reaction can also provide IPTers with important information about how to improve training to make it more effective
Level 2: Learning
- To what degree participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in a training event
CRUCIAL QUESTION: What did they learn in the program? Did they really learn what they were supposed to learn?
IMPORTANCE: Kirkpatrick defines this as the extent to which attitudes are changed, knowledge is increased, and/or skills are improved by the training.
Level 3: Behavior
- To what degree participants apply what they learned during training when they are back on the job
CRITICAL QUESTION: Did they change their work behavior based on what they learned? Are they using the new skills?
IMPORTANCE: According to Kirkpatrick measuring behavior without measuring reaction and learning is not sufficient to tell you whether or not your training has been successful. He describes four conditions necessary for behavior change:
- the learner must have the desire to change
- the learner must know what to do and how to do it
- the learner must work in the right climate
- the learner must be rewarded for changing
Level 4: Results
- To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement
CRUCIAL QUESTION: Did the change in behavior positively affect the organization?
IMPORTANCE: Kirkpatrick defines results as
- increased production
- improved quality
- decreased costs
- reduced frequence and/or severity of accidents
- increased sales
- higher profits
- return on investment (ROI)*
These are the ultimate reason for training programs so the final results need to be stated in these terms
Points to ponder…
- I wanted to find a chart that shows how evaluation is embedded within the ADDIE model. I found a great table through Learning Solutions Mag in an article entitled, A.D.D.I.E meets the Kirkpatrick Four
|LEVEL 3 On-the-Job||LEVEL 2
|Analysis||What are the business challenges, financial and competitive goals?||What performance supports overall goals on the job? What performance problems or obstacles make it difficult to support the goals?||What exactly do top performers do that supports over-all goals and which can be observed and measured?||How can employees best learn how to perform in ways that support overall business goals?|
|Design||How to show connection with business goals throughout the program?||If employees forget details, what job aid or other resources can they use as a reminder?||What is an effec-tive sequence for teaching the concepts, information, skills, and attitudes needed in order to achieve performance goals?||What are some alternatives to abstract, some-times boring, slide lectures?|
|Development||Is there a clear connection between content and the need to support department and organizational goals?||Does the training program have the look and feel of real world challenges so participants learn what they need to do on the job?||Are key concepts and skills presented, demonstrated, practiced and reviewed … not just lectured about?||Are the program activities and materials what partici-pants want as well as need?|
|Implemen-tation||Is there organiza-tional support for the training at the executive level?||Is it job-related?||Are performance objectives and learning content in sync?||Is it user friendly?|
|Evaluation||Return on invest-ment in training program , based on business needs/goals||360 interviews or on-the-job assign-ments, etc. to check achievement of performance goals||Paper and pencil tests or observed or scored activities (role play, simula-tion, presentations)||Reaction and Feedback Questionnaire with ratings and comments (Smile sheets)|
2. Kirkpatrick discusses that even though a participant can 1) have the desire to change and 2) know what to do and how to do it (job transfer), a training program can still be unsuccessful if the participant job climate does not promote change or if they don’t have support system that encourages applying the new knowledge and skills. To me this echoes the importance of getting the backing of managers and getting them invested in the training because ultimately managerial support directly affect the job climate. If we can have support from management, then this will increase the success of the program and produce organizational results. Ultimately, this is both the goals of IPTers and management
3. In class, our instructor Carol Porter provided an interesting statistic. Workplace learning and performance improvement practitioner were asked about what percentage of organizations conducting different levels of training evaluation using the Kirkpatrick model. The results were quite telling, level 1 Reactions were given in 75 percent of program, learning in 41 percent of programs, transfer to the job 21 percent of programs, and impact in 11 percent of programs.
Question: Why are IPT practitioners stopping at Level 2? Is it access to data or participants? Time constraints? Is it complicated mathematics? Are there other evaluation methodologies out there that help IPTers answer the question of job transfer and organizations results better/easier?
Update: I found a great article on other evaluation methodology through a Learning Solutions article entitled Nuts and Bolts: How to Evaluate E-learning. I look forward to learning more about conducting and evaluation in IPT 530 Evaluation Methodology