Investments in learning - why is it so hard to measure
The world has two kinds of people - those who like to measure and analyze everything that they are doing and those who believe that measuring something is a surefire way to turn something fun into a deadly serious pursuit. I wonder which of these two groups would be attracted to the world of learning and development. When did you last LEARN something that was a human skill eg becoming a better manager or even a better parent? How do you measure effectiveness of that?
Training is easy to measure ... or is it?
The world of training has lived by Kirkpatrick's model of measurement of training. He believes that Reaction, Learning, Behavior, and Results are the four ways the impact of training can be measured.
Reaction: This is what the participants share as soon as the training gets over. I was part of a group that offered Management Development for a steel company. A 4-point rating system was used where the participants rated if they found something (1) Poor (2) Satisfactory (3) Good (4) Excellent. The same scale was used to rate the program, the quality of lunch, the reading material and the trainer ("I am a facilitator - NOT a trainer," some protested). Every trainer knew that the quality of lunch impacted the rating of the program as much as their own skills as a trainer.
Learning: Kirkpatrick referred to this as the next level of evaluation. the training has developed the necessary knowledge, skills and attitudes needed to improve on the job. Learning has a before and after version of improvement. The motivation of the learner and the confidence exuded are seen as evidence of learning. That is where the measure becomes fuzzy because there are too many variables at play. I may have the knowledge but not the skills. I may have knowledge and skills but no confidence or the motivation to build a new habit.
Behavior: Behavior change can be a powerful measure. Behavior change is easily observed by others around. In skills that involve involving other humans, behavior change is evidence of success. Behavior change is dependent on several factors that are beyond the control of the individual. For example, the role may offer limited opportunity to practice, and the culture of the organization may not reward the new behavior. Some skills are built on the strength of other adjacent skills. If the change in behavior is not enough, the reason for that may not lie in the training program.
Results: This is the holy grail of measuring training effectiveness and the messiest to calculate. Business results are driven by a mix of dozens and dozens of factors. If the leaders of the team attend a workshop and the company gets great financial results that year, it may be hard to claim that the trainer or the training were directly causing the impact. The sector may have faced economic headwinds, or a talent shortage or high attrition and a change a change in the government policies. Correlation and causation are not the same. Training is no exception to the rule.
Most examples of measuring ROI of training are about hard skills. They are rarely about soft skills. Learning is even harder to measure.
Read more: The Kirkpatrick Model
What makes it hard to measure learning
For a long time, the saying in advertising was that half of the ad budget was a waste, but no one knew which half to fix. In training, the jobs that depend on a learnable skill are often used to demonstrate Kirkpatrick's model. When a coffee shop trains its baristas to make specialty coffee, it is easy to measure the impact on the business (unless there is a recession and customers get frugal!).
Knowledge work is complex - the dependencies make it harder to isolate the cause and effect. The Nobel Prize committee grapples with this complexity every year. While the prize for Economics was awarded in 1969 in the era of neo-classical economics. It was built on the assumption that people make rational choices. That is not true you would say. People do not exercise even though they know that it is good for them. That is why Behavioral Economics combines elements of economics and psychology to understand how and why people behave the way they do in the real world. It differs from neoclassical economics, which assumes that most people have well-defined preferences and make well-informed, self-interested decisions based on those preferences. Fortune 500 companies lose their sheen for a variety of reasons. The current CEO may be reaping the benefits or paying the price for decisions that were taken by predecessors.
Continuous processes are hard to measure - Schools and colleges have a definitive date of starting and ending. The graduation ceremony, the examinations and grades are all signaling devices marking the end of the association. Human beings work well with clear bookends. Despite such clear indicators, people often struggle to see the value of subjects that do not have clear rules and outcomes. Humanities and Liberal Arts are often accused of being fuzzy. People get hired based on the hard skills, but it is "soft skills" that gets people promoted. While everyone who went to college with you learned the same content, their career trajectory is shaped by multiple factors especially if they have been in jobs that depend on working and persuading others.
Skills exist in combinations - The ability to be a powerful storyteller depends on a number of skills - the ability to sense the needs of the audience to judge how much information can be shared. The ability to sequence the information needed and to create the presentation slides is a different skill that storytelling needs. The ability to narrate the content spontaneously and to sense the engagement level of the audience are among the many skills a storyteller needs. The audience that is exposed to fabulous storytellers has higher standards that speakers have to live up to.
The dropping half-life of skills: The half-life of skills is about four years. That means the value of our skills will be halved after four years. For technical skills, the half-life is closer to two years.
69% of the world’s most admired companies value learning agility and curiosity over career history and experience when it comes to hiring." (Read more: Future of Work Trends).
Not investing heavily in learning and development means giving away the firm's competitive advantage. Only 22% of organizations monitor the use and progress of training.
Measurement is in itself a great way to motivate people to change. The constant nudge of smart watches that remind you to walk 10,000 steps every day has made more people move. Ever since I started tracking my sleep, I realized that I needed to increase my sleeping hours. Measuring technical training may be the place to start. Measuring leadership development and learning agility may require us to think of more creative measures to gauge its effectiveness.
Measuring hard skills may be easier than agreeing on measures to evaluate "soft skills" or people skills. That should not prevent organizations from investing more in learning and development. After all it is way cheaper to invest in the existing employees than to bring in talent from outside.
----------------
Have a question? Email me at abhijitbhaduri@live.com
