Metrics (excerpt)

Metrics: Measurements that Matter

These are excerpts from Metrics, an eBook available in the store.
Four years ago I attended a how-to-ROI presentation at a major eLearning event and found it so misleading that I began writing about how companies really evaluate project potential and after-the-fact results. Requests for my old articles and white papers still pour in.

Learning about ROI seems to be enjoying a renaissance in the training industry. Workshops and certificate programs abound. However, the courses I've looked at teach things that no business manager would buy. Here, let me tell you why I feel that way.

Metrics are broader than ROI

Metrics are measurements that matter. The Industrial Age is over. Measures that fail to account for intangibles are misleading.

Decision-makers use metrics to

  • choose the best course of action
  • supplement gut feel with a framework of logic
  • assess project failure or success
  • monitor progress
  • uncover ways to make improvements
  • divine ways to do better next time
  • focus attention on profitable activities

Metrics are in the eye of the beholder. They are not simply the application of a rote formula or accounting rule. They are subject to interpretation. This is what makes metrics worthy of discussion.

Training jargon doesn't play well in the executive suite, so you need to express yourself and position what you bring to the table in business terms.


If only I had $10 for every time I've heard training managers lament that they can't separate out the impact of the training from everything else that was going on. Some suggest that certain employees go untrained to provide a control group. (Forget it; the Hawthorne effect* would skew the results.)

*In a classic experiment in the 30s at Western Electric's Hawthorne Works, researchers found that workers were more productive when they cut the lights up. Also, when they cut the lights down! Conclusion: Workers are more productive if you pay attention to them. Placebos work.

Why would you want a control group anyway? Business is not precise. Deciding whether to invest in more training or increasing bonuses is not some physics experiment requiring 6-place accuracy. Consider John Wanamaker's regret, "I know only half my advertising is effective. If I only knew which half." Wanamaker didn't become a department store mogul by cutting his ads; he did what his gut told him to do.

Who decides whether an iffy investment, like Wanamaker's ads or your training program, is worthwhile? Your sponsor. The sponsor is the person who most strongly influences the decision on how to spend the money. The sponsor is your client. The sponsor decides what markers constitute proof.

The Performance Agreement

You've got to describe the linkage of your initiative and business results quantitatively, using assumptions your sponsor will buy into. You must be explicit about the what-if's. Do this in writing, as a "Performance Agreement" that:

  • Provides a shared understanding of the problem to be solved
  • Describes what you intend to provide in its solution
  • Estimates the expected increase in profit and the step to get there
  • Sets out a way to assess whether the goal was accomplished or not
  • Lays the foundation for solving the next problem

The Agreement also shows that you understand the business and that you're on the same page as your sponsor.

Fortune Favors the Bold

Before you get too far into metrics, ask yourself, "Does it matter?"

One of the few aspects of accounting that I like is The Principle of Materiality. This principle says that if it doesn't matter, don't worry about it.

For example, if Chevron-Texaco?s accountants uncover a $32,000 error in the sales department?s expense budget, they don't make Chevron-Texaco note the error in its annual report. Chevron rakes in $100 billion a year. $32,000 is a drop in the bucket; it's immaterial. Now then, if the accountants find a $32,000 discrepancy in your personal expense report, that's material. Send us a postcard from the slammer.

You can?t measure everything. Therefore, you should seek to measure important things. Let everything else coast. Don't fritter away time on the small stuff.

Time Matters

While training directors may have different objectives from CEOs, everyone in today's business world shares one need: they want it all now. Benefits you don't see for two years are hardly benefits at all. Given enough time, a million monkeys at a million terminals could develop your entire curriculum, with Flash animations and a repository of SCORM-compliant objects. Nobody's got time to wait.

An appropriate metric for most eLearning is time-to-proficiency. How long will it take until your people are performing competently? By competent, I mean able to meet or exceed the expectations of customers, be they internal or external to the organization.

Traditional ROI

ROI is often a mask for uncertainty or an attempt to quantify cost/benefit with accounting principles that don't count people as assets. The business return on eLearning investment should be so obvious that you can figure it out on the back of a napkin.

Traditionally, executives assume training has little or no impact on revenue, so they measure training benefits in terms of cost savings. This works against eLearning, where increases in top-line revenue generally exceed reduced expenses by a wide margin.

ROI is relative

ROI or cost/benefit analysis is relative, not some absolute value like the speed of light used to be. Where you stand depends upon where you sit. CEOs don't care about learning objects or LMS. Line managers focus on the performance of their unit, not the overall corporation. Training directors don't allocate resources to business transformation. One size does not fit all.

Beware of bad numbers

Present-day accounting is an anachronism. Invented half a millennium ago to maintain accurate shipping records, double-entry bookkeeping helped Venice dominate its part of the world. Formal accounting worked well when you could go out to the warehouse to count your assets. In the information age, it's an inappropriate yardstick for measuring anything. Most assets drive home every night.

In a nutshell, the basic problem is that accounting recognizes nothing but physical entities. Intangibles are valued at zero. Vast areas of human productivity -- ideas, abilities, experience, insight, esprit de corps, and motivation -- lie outside the auditor's field of vision.

The largest cost of all is foregone opportunity.

Again and again, I've found the largest overall cost of any corporate learning endeavor is the cost of people's time. I'm not talking about salaries and benefits; I refer to the value they would have created had they not been tied up in training. Opportunity cost per hour is not a fixed amount. A salesperson's time during working hours in peak buying season is worth much more than the same individual's time after closing time in non-peak season. eLearning often enables the employee to shift learning to those non-peak hours.

There's more.

I could go on for another ninety pages. In fact, I do just that in a newly published eBook titled Metrics.

Posted by Jay Cross at December 27, 2003 05:42 PM | TrackBack

cool stuff

Posted by: nicky and paris hilton at June 30, 2004 04:17 AM

30 Poppy Lane
Berkeley, California 94708

1.510.528.3105 (office & cell)

Subscribe to this Blog

Enter your email address to subscribe. We vow never to share your information with anyone. No Spam.

Subscribe Unsubscribe

Reference Pages

Building Community
CSS, Semantic Mark-Up, and codes
First Principles
How People Learn
Knowledge Management
Learning Links
Learning Standards
Making It Work (Implementing)
Metrics & ROI
Social Software
String theory
The eLearning Museum
Visual Learning


Our Infrequent Newsletter
Sign up for our sporadic newsletter.

Entries by category...

Customer care
Emergent Learning
Just Jay
Recycled from Blogger
The Industry
Workflow-based eLearning


Internet Time Group

© 2004 Internet Time Group

Click for Berkeley, California Forecast
Berkeley, California

Recent entries

New Blog
Blogger Experience, Housekeeping, Something New
Loosely Coupled
Above all
Demographics is destiny
Are you setting the bar high enough?
Virtual Apps
Aerobic Learning
Work as Video Game
Oracle and Macromedia, Sitting in a Tree
The Blogosphere
ASTD Silicon Valley
Performance Support
Kingsbridge Conference Center
First Post by Email
Inactive Blog
RSS Feed for New Site
Comment Spam
Testing ... testing ... 1...2..3
IT Doesn't Matter - Learning Does.
All blogging is political
Mutlimedia Learning
Damn, damn, double damn
Nonverbal impact?
The New Religion
Wolf! Wolf! Wolf! Wolf! Wolf! Wolf!
Business Process Management (2)
Really Big
Business Process Management Conference
Don't Lose a Common Sense: LISTEN
It's only natural
Go with the flow
Time Out for the Fair
Informal get-together in SF this Wednesday
Repetition, reverb, and echoes
Who Knows?
Cognitive Mapping
Push vs pull
The Big Picture on ROI
Art Break
TDF Finale
New Community of Practice Forming
More TDF04
Training Directors Forum 2004
A Rare One-Liner
PlaNetwork LIVE 2
PlaNetwork LIVE
ASTD 2004 Leftovers
Worker Effectiveness Improvement, not KM
Upcoming Events
eLearning Effectiveness?
Jay's Talk at ASTD
Mintzberg & Cooperider
Lest ye forget
ASTD International Conference & Exposition 2004
Knowledge Tips
What is Workflow Learning?
ASTD msg 1 of n
Look out, it's Outlook
Collaboration at ASTD Next Week
Tell me a story
User indifference
The shortest presentation on metrics you will ever hear
Back to Blogger
Windows fixes
The Alchemy of Growth
Grab bag
Very loosely coupled
E-Learning from Practice to Profit
Robin Good kicks off Competitive Edge
China Bloggers
Sonoma Dreaming
Upcoming Events
Emergent Learning Forum: Simulations
The Best Things in Life Are Free
Metrics and Web Services
OpEd: ROI vs. Metrics
e-Merging e-Learning
Loosely Coupled
Search me