CivieBlog

It's been a while since I looked at: ROI and Metrics in eLearning and in this case, I was more looking at traditional content delivery and their impact.  I've recently been more interested in measures around social learning.  And I think I need real help here understanding how to measure this stuff.

And, I just saw a post Measuring Networked (or Social) Learning that discusses how we could go about measuring social learning.  In this, Eric suggests the following minimum measures:

• Networking patterns - the relationship between people and content categories, the network make up or profile (business unit, job, level, etc), key brokers and influencers by content category, and the degree of networking across silos.
• Learning efficiency - time lag between posting content and when content is viewed,  amount of time spent producing content for others to view, amount of redundant or significantly overlapping content, the degree to which “informal” content is reused in “formal” content (and perhaps reducing formal content development costs and effort).
• Learning needs - differences between the learning needs or demand between “formal” and “social” learning (are some skills best learnt formally?), most popular learning needs by job, level, business unit, etc.
• Contribution patterns - most active contributors and methods of contribution, busiest days and times for contributing, frequency and amount of contributions by job, level, business unit, etc. 
• Content usage patterns - preferred ways to consume various content topics, busiest days and times for viewing content,  amount of time spent viewing content and participating in discussion threads and blogs, and preferred way to “find” content.
• Content quality - ratings by content category, contributor, and medium, amount of “inappropriate” or “wrong” content reported by users, and the amount and type of content with very few or a lot of hits or views.

I'm not sure I buy how much real impact any of this will have on bottom line measures.  The recent MIT Study that showed that more highly networked individuals were more productive (see Workplace Productivity).  So, size and access of networks might be a proxy, but what about all the rest of these suggested measures.  Do we really believe they will be proxies for effectiveness?

I like what Kevin Jones talked about in Objection #13: How Do You Measure ROI?

How do you know that it is producing bottom line results? …

Rachel Happe suggested some measurements of ROI. A lot of them are for environments that face the customer, but some are for internal. Among those were:

# Number of new product ideas
# Idea to development initiation cycle time
# Retention/Employee turn over
# Time to hire
# Prospect identification cost
# Prospect to hire conversion rate
# Hiring cost
# Training cost
# Time to acclimation for new employees

Remember, we are looking at the final outcome, not necessarily “did they learn”. Because, honestly, we don’t care if they learn if they don’t use it for the benefit of the company. So the benefit is what we measure.  Other’s measurements might be:

# How large one’s network is
# Number of meetings taking place (or, more intuitively, are NOT taking place)
# Number of travel arrangements made (or, again, NOT made).

I always like to work on project where there are clear metrics that are the focus at the end of day (see Data Driven).  But in many cases, we need to define Intermediate Factors in Learning (see also Intermediate Factors).  As an example, we might be focusing on Customer Loyalty.  However, this metric is far too hard to directly measure and impact and thus we might say that there are intermediate factors such as customer satisfaction (based on surveys), recent contact, staff knowledge, etc.  We often know that these factors are contributors to the end measure.  Thus, we can define these as the actual goal.

What is particularly challenging about social learning measurement is understanding what these intermediate factors are going to be?  If you have really good support for networks, communities, social learning, etc. – how would we expect that to impact (in a measurable way) the above factors?  Why?  Why would this impact something like engagement or turnover?  Is there any proof?  What does that suggest about the intermediate measures?  Any pointers to good resources on this?