OpEd: ROI vs. Metrics

Learning Circuits just published my article, ROI vs. Metrics, which I thought was just telling it like it is, but editor Ryann Ellis considered sufficiently controversial to make it the first OpEd piece they've ever run.

What's controversial with this?

I don't understand how anyone could disagree when I write

A friend of mine read the article, came to this site, and plunked down $250 for the latest version of Metrics. If his $250 subscription to Metrics helps him justify his $500,000 eLearning budget, it's cheap at the price. If his newfound attitude improves his career options, that's icing on the cake.

The next version, Metrics 2.0, will be a great improvement, but it's not out yet. Here's a piece from the introduction:

Origins

Last year at TechLearn I promised to send several people my thoughts on measuring the value of learning investments. On the flight home, I assembled a batch of white papers, interviews, and articles I’d written over the past few years and was surprised to find I’d come up with 90 pages! This was more like a book than a sheaf of white papers.

Why had I written so much on learning metrics and ROI? Frustration. I’ve been in the training business for nearly thirty years, but before that I had been a systems analyst and market research director and I’d earned an MBA. I cut my teeth selling loan officer training to major banks. That requires real what's-in-it-for-me ROI. It pained me to go to conferences like TechLearn and Training and ASTD, only to hear the same worthless claptrap about Level 1 and Level 2 and ROI. Many of the “experts” got their expertise from textbooks rather than the real world. This made me angry. I wrote to expose the charlatans.

Metrics 1.0 & 1.1

Metrics 1.0 was little more than the white papers and articles put in a logical sequence. I slapped on a table of contents and inserted transitions. I sent copies to two dozen friends in the business for advice and comment.

I love books. Several rooms of my house are lined with bookcases. Last year I donated eight cartons of books to the library to clear some shelf space. Already, the gaps have filled in, and the shelves are packed. Nevertheless, I chose to make Metrics an eBook rather than a printed one. Our understanding of how to measure value is in flux; printed books become dated so quickly. I also figured that if I responded to readers’ questions and advice, the work would always be getting better. So Metrics 1.1 was issued as an eBook – and I promised purchasers they would receive the next version as well as this one.

Metrics 1.2 & 1.3

Metrics 1.2 was considerably tighter. I incorporated a mind map to introduce the subject and a simple flowchart for carrying out the full cycle of the Metrics process. I corrected glitches readers had pointed out. Version 1.3 was a minor upgrade, mainly error correction. Several readers pointed out weak spots in the manuscript. They wanted a more direct, forceful, organized presentation. They also asked me to spend less time trashing the current situation and more time on what to do next. To get Metrics into distribution, I offered it to charter subscribers for a mere $25.

Few readers actually gave much feedback but one individual made up for the silence of the rest. The GAP’s Dave Lee went beyond the call of duty. Dave’s background in publishing, experience in accounting, and current work in eLearning make him the ideal critic. Dave is a major influence on the improvements you’ll see in version 2.0. He offered suggestions on overall organization. He convinced me to take a more positive attitude, for example, telling me, “Jay, I’m not sure what accountant ran over your puppy when you were young, but you really don’t need the strawman of 'accountants are out to get us' to make your argument.”

Metrics 1.3 is the current version. People who purchase it will receive Metrics 2.0 when it is ready. The charter subscription period is over. The price of Metrics 1.3+2.0 is $250.

Metrics 2.0

Metrics 2.0 is a total reorganization and rewrite of the original material. It focuses on what to do and why in lieu of me bitching about the ramblings of false prophets. I’ve chopped superfluous material and added more explanatory text. I expect the final Metrics 2.0 to be ready in a couple of months.

People tell me they buy Metrics because they have an immediate need. They’re in budget trouble; their management just doesn’t get it; the big boss wants to see the numbers. In all likelihood they already know part of the material and have come to me to fill gaps or help them polish their approach. This latest version of Metrics begins with a roadmap of what’s to come. The map will guide you to the chapters that contain what you most need to know. I’ve relegated more philosophical issues to the back of the book.

Making the business case involves a lot more than doing the math. You have to understand the business. This takes credibility with managers outside of the training function. You must, as SumTotal president Kevin Oakes recently wrote in T+D (2004), “earn a seat at the table.” My goal is to help you get there and to be invited back again and again. It's probably a better job than the one you have now.

April 2004