In IPT 536 Foundations, we were introduced to Kirkpatrick’s 4 Levels of Evaluation. His taxonomy is quite revered in the IPT world (kind of like a magical buddha). Donald Kirkpatrick published a taxonomy of criteria for evaluating instruction that is widely regarded as the standard for evaluating training
- Level 1: Learner satisfaction
- Level 2: Learner demonstration of understanding
- Level 3: Learner demonstration of skills or behaviors on the job; and
- Level 4: Impact of those new behaviors or skills on the job
What I found a little disconcerting about the taxonomy was that to me it seems to focus on “improving the training after the training is over”. It seems to focus too heavily on end results while gathering little data that will help inform training program improvement efforts. (Discovering after training that financial advisors still did not have increased sales tells us that the sales training program didn’t “work”; it tells us little about how to improve it.)
Secondly, as a novice in the field, I admit that I did assume a linearity and causality between each levels. In our Foundations class, we used the book Foundations in Instructional and Performance Technology there was a nice graphic (Chyung, 2008) that showed how increases in each level of evaluation correlated with increases with in next level. I know Professor Chyung was only trying to make a point of aligning goals with evaluations, however this graphic does perpetuate the common misconception of linearity and causality.
It was eyeopening this week to realize that I held these 3 problematic assumptions of the Kirkpatrick model: 1) the levels are arranged in ascending order, 2) the levels are causally linked, and 3) the levels are positively inter-correlated (Alliger and Janak, 1989).
For instance before I read this paper, I would assume that a participant who passed a test at Level 2 would result in improved job performance at Level 3 (but after having taken more class I realize that if the training didn’t provide on the job based activities then then the training won’t transfer into measurable results.
Ultimately, the main drawback in my opinion of Kirkpatrick’s evaluation is that it view “training” as an isolated event. The implied assumption is that that the“training” is the ONLY cause of business impact. As Jane Bozarth wrote in her article, Nuts and Bolts: How to evaluate Elearning
“many factors enable — or hinder — the transfer of training to on-the-job behavior change, including support from supervisors, rewards for improved performance, culture of the work unit, issues with procedures and paperwork, and political concerns. Learners work within a system, and the Kirkpatrick taxonomy essentially attempts to isolate training efforts from the systems, context, and culture in which the worker operates.
This week’s readings highlighted that the success of a training program depends mostly on the performance management (managers must provide coaching, support, and a supportive climate).
BOTTOM LINE: ASTD reports: “Employee learning and development is taking center stage as business leaders increasingly understand that a highly skilled, knowledgeable workforce is critical to achieving growth and success”. Thus, we as IPT professionals must spend more time showing how we do indeed improve performance and how our activities add value. We must prove that our activities DO add value to the organization. We have to “talk the talk” to the audience that is funding us. They don’t care about reaction levels or how “learner-centered” it was . For the most part, the business units’ main concern is the IMPACT — did the resources spent on the learning process contribute to the overall health and prosperity of the enterprise? Thus, we have to spend more time showing how training, development, and educational programs provide a real difference to the business.
points to ponder..
- Big Dog, Little Dog Juxtaposition
- Big Dog, Little Dog Blog
- Barzathzone: Alternatives to Kirkpatrick
- Learning and Training: Statistics and Myth
Alliger, G. M., Janak, E. A. (1989). Kirkpatrick’s levels of training criteria: Thirty years later. Personnel Psychology, 42 (2), 331-342.
Chyung, S. Y. (2008). Foundations of instructional and performance technology. Amherst, MA: HRD Press.