In my humble opinion, one of the most useful (a.k.a. easy to both understand and implement) contributions to the discipline of training evaluation ever came from Mary Broad and John Newstrom. They developed something called The Transfer Matrix which was introduced in their book entitled “Transfer of Training,” published in the early 1990s. The book told the story of their decade-long research project, and I believe told it in a way that still has relevance for anyone in the training industry with an interest in impact analysis.
Like most compelling research, theirs initiated with an irrefutable problem that, for a variety of different reasons, nobody seemed to be doing much about. The problem, in “Kirkpatrick terms,” was correlations. The Kirkpatrick Levels Model had been in active use since the early 1960s and featured (as we know!) four levels of evaluation for a learning experience:
- Level I (Reaction) – How did the learners feel about the learning event or experience?
- Level 2 (Learning) – Did the learners learn anything during the event or experience?
- Level 3 (Behavior Change) – Could the learning be tied to behavior change(s) post training?
- Level 4 (Results) – Did those behavior changes produce desirable results?
Research suggested, beyond reasonable doubt, that Level 1 outcomes were highly correlated with Level 2 outcomes (if you liked a training program there was a high degree of probability that you learned something, and vice versa). That research also demonstrated that Level 3 outcomes were highly correlated with Level 4 outcomes (if you changed behavior, you would indeed get different results; and if you wanted different results, you needed to change behavior). Unfortunately, there was no such correlation between Level 2 and Level 3 outcomes (just because you learned something, did not mean you were going to run back to your workplace and change your behavior).
So, in the absence of Level 3 and 4 evidence, Broad and Newstrom set out to find out why! The essence of their research revolved around something they referred to as “role-time combinations.” When they asked themselves about the roles that were critical in determining the transfer of training, they landed on these three:
- Manager – The direct supervisor of the individual attending training
- Trainer – The person (representing the entire department) that was delivering the training
- Trainee – The individual attending training
When they asked themselves about the relevant time periods that were critical, they quite reasonably came up with these three:
- Pretraining – The 4 to 6 weeks that precede the training event
- During training – The training event or experience itself
- Post-training – The 4 to 6 weeks following training
They put those parameters together and formed a nine-box matrix they used to ask the two questions that formed the bulk of their research:
- What role-time combination(s) most accurately identify responsibility for the transfer of training (i.e., which stakeholders in which time periods have traditional responsibility for the transfer of training)?
The answer (quite overwhelmingly) was “the trainer during training.” And, in one way or another, we’ve all been there (i.e., where the trainer is begging learners at the end of an event to “use this stuff back on the job.”
The second question produced results that are a blinding flash of the obvious, but still hold true today:
- What role-time combination(s) should be responsible (i.e., which stakeholders in which time periods truly drive Level 3 and Level 4 outcomes)?
Which is the common sense that unfortunately is not common practice!
The implications here drove the development of The Center for Leadership Studies’ sustainment system: The Four Moments of Truth™. Net-net:
- Trainees need to value the skills being taught in training and see themselves being successful developing those skills and producing desired results
- The training needs to be good (both the design and the delivery)!
- The Next-Level Manager of the Trainee needs to drive training transfer before and after the event (which means the Training Department needs to get those managers on board and prepared to fulfill those roles!)
