Metrics (excerpt)

Metrics: Measurements that Matter

These are excerpts from Metrics, an eBook available in the store.

Four years ago I attended a how-to-ROI presentation at a major eLearning event and found it so misleading that I began writing about how companies really evaluate project potential and after-the-fact results. Requests for my old articles and white papers still pour in.

Learning about ROI seems to be enjoying a renaissance in the training industry. Workshops and certificate programs abound. However, the courses I've looked at teach things that no business manager would buy. Here, let me tell you why I feel that way.

Metrics are broader than ROI

Metrics are measurements that matter. The Industrial Age is over. Measures that fail to account for intangibles are misleading.

Decision-makers use metrics to

• choose the best course of action
• supplement gut feel with a framework of logic
• assess project failure or success
• monitor progress
• uncover ways to make improvements
• divine ways to do better next time
• focus attention on profitable activities

Metrics are in the eye of the beholder. They are not simply the application of a rote formula or accounting rule. They are subject to interpretation. This is what makes metrics worthy of discussion.

Training jargon doesn't play well in the executive suite, so you need to express yourself and position what you bring to the table in business terms.

'Proof'

If only I had $10 for every time I've heard training managers lament that they can't separate out the impact of the training from everything else that was going on. Some suggest that certain employees go untrained to provide a control group. (Forget it; the Hawthorne effect* would skew the results.)

*In a classic experiment in the 30s at Western Electric's Hawthorne Works, researchers found that workers were more productive when they cut the lights up. Also, when they cut the lights down! Conclusion: Workers are more productive if you pay attention to them. Placebos work.

Why would you want a control group anyway? Business is not precise. Deciding whether to invest in more training or increasing bonuses is not some physics experiment requiring 6-place accuracy. Consider John Wanamaker's regret, "I know only half my advertising is effective. If I only knew which half." Wanamaker didn't become a department store mogul by cutting his ads; he did what his gut told him to do.

Who decides whether an iffy investment, like Wanamaker's ads or your training program, is worthwhile? Your sponsor. The sponsor is the person who most strongly influences the decision on how to spend the money. The sponsor is your client. The sponsor decides what markers constitute proof.

The Performance Agreement

You've got to describe the linkage of your initiative and business results quantitatively, using assumptions your sponsor will buy into. You must be explicit about the what-if's. Do this in writing, as a "Performance Agreement" that:

• Provides a shared understanding of the problem to be solved
• Describes what you intend to provide in its solution
• Estimates the expected increase in profit and the step to get there
• Sets out a way to assess whether the goal was accomplished or not
• Lays the foundation for solving the next problem

The Agreement also shows that you understand the business and that you're on the same page as your sponsor.

Fortune Favors the Bold

Before you get too far into metrics, ask yourself, "Does it matter?"

One of the few aspects of accounting that I like is The Principle of Materiality. This principle says that if it doesn't matter, don't worry about it.

For example, if Chevron-Texaco?s accountants uncover a $32,000 error in the sales department?s expense budget, they don't make Chevron-Texaco note the error in its annual report. Chevron rakes in $100 billion a year. $32,000 is a drop in the bucket; it's immaterial. Now then, if the accountants find a $32,000 discrepancy in your personal expense report, that's material. Send us a postcard from the slammer.

You can?t measure everything. Therefore, you should seek to measure important things. Let everything else coast. Don't fritter away time on the small stuff.

Time Matters

While training directors may have different objectives from CEOs, everyone in today's business world shares one need: they want it all now. Benefits you don't see for two years are hardly benefits at all. Given enough time, a million monkeys at a million terminals could develop your entire curriculum, with Flash animations and a repository of SCORM-compliant objects. Nobody's got time to wait.

An appropriate metric for most eLearning is time-to-proficiency. How long will it take until your people are performing competently? By competent, I mean able to meet or exceed the expectations of customers, be they internal or external to the organization.

Traditional ROI

ROI is often a mask for uncertainty or an attempt to quantify cost/benefit with accounting principles that don't count people as assets. The business return on eLearning investment should be so obvious that you can figure it out on the back of a napkin.

Traditionally, executives assume training has little or no impact on revenue, so they measure training benefits in terms of cost savings. This works against eLearning, where increases in top-line revenue generally exceed reduced expenses by a wide margin.

ROI is relative

ROI or cost/benefit analysis is relative, not some absolute value like the speed of light used to be. Where you stand depends upon where you sit. CEOs don't care about learning objects or LMS. Line managers focus on the performance of their unit, not the overall corporation. Training directors don't allocate resources to business transformation. One size does not fit all.

Beware of bad numbers

Present-day accounting is an anachronism. Invented half a millennium ago to maintain accurate shipping records, double-entry bookkeeping helped Venice dominate its part of the world. Formal accounting worked well when you could go out to the warehouse to count your assets. In the information age, it's an inappropriate yardstick for measuring anything. Most assets drive home every night.

In a nutshell, the basic problem is that accounting recognizes nothing but physical entities. Intangibles are valued at zero. Vast areas of human productivity -- ideas, abilities, experience, insight, esprit de corps, and motivation -- lie outside the auditor's field of vision.

The largest cost of all is foregone opportunity.

Again and again, I've found the largest overall cost of any corporate learning endeavor is the cost of people's time. I'm not talking about salaries and benefits; I refer to the value they would have created had they not been tied up in training. Opportunity cost per hour is not a fixed amount. A salesperson's time during working hours in peak buying season is worth much more than the same individual's time after closing time in non-peak season. eLearning often enables the employee to shift learning to those non-peak hours.

There's more.

I could go on for another ninety pages. In fact, I do just that in a newly published eBook titled Metrics.

Posted by Jay Cross at December 27, 2003 05:42 PM | TrackBack