Defining the problem: The first step to successful learning analytics

06 February 2020 by Ian Blackburn

While it’s fundamentally true that the LRS is a tool for storing activity data, it doesn’t mean you should store everything possible in the vague hope that the data might give you the answer you seek.

Having supported many organizations in getting to grips with learning analytics, I find myself asking “how did you get here?” and “why did you want to do it in the first place?” Many folks can find themselves dealing with seemingly endless buckets of data coming from all kinds of sources.

You might be thinking that it’s the LRS’ job to handle all this data. And I wish it was, but we aren’t doing this to be in the data storage game (we’d be rich with over a billion data points in storage) we’re doing it to be in the problem solving one.

What problem are you trying to solve?

I find myself asking  “what problem are we trying to solve?” of all sorts of people in L&D time and time again. Typically I respond to two big problems; I don’t know what I don’t know and I don’t know if my training is working. Let’s break these questions down into something that we can make useful:

The “I don’t know what I don’t know” problem 

This is super common and it’s understandable why it’s such a go-to option for an LRS. Years of solutionizing and trying to find a platform that gives you what you need has left you with a setup that looks a bit like this:

You’ve got lots of systems, made by lots of vendors and none of them talk to each other. You’ve been tasked with understanding what happens in your learner journey, but the idea of doing so is almost impossible as the record of the activity is spread around across these platforms with no understanding of each other’s structure. Simple solution, point everything at the LRS – job done. But wait, don’t I just have two copies of the data and still I’m none the wiser on what any of it means? Yep, just another platform to maintain. 

So what do I store? 

There are a few ways to answer that, but we have to think about it in the form of a problem.

Problem: I don’t know what’s happening with my content as there is no data for it.

Explanation: Content doesn’t store the interactions in a database, so it’s difficult to know a great level of detail about it. We don’t know much about drop-offs or assessment interactions without a method to capture that information. 

Solution: xAPI is great for this. It gives a detailed look at the interactions with the content and stores that in the LRS. This is a sensible use case as we aren’t duplicating data and we can start to understand what’s happening.

So that’s great, we have a sensible step. Next is to make sure we have some sort of hypothesis to prove, there is no logic in simply going through each platform one by and building integrations until you’re lost in a sea of data, weary from months of custom development. 

The “I don’t know if my training is working’ problem

Another common thing we hear is when a customer comes to us tasked with demonstrating the return on investment from a financial perspective. What you often find though is that many of the courses and workshops that are delivered aren’t necessarily suited to measuring the impact of the training against a monetary value. Also like many people have pointed out, ROI doesn’t have to mean making more money, it could be reducing churn or measuring an increase in employee confidence. The key here is to really think about what data is actually available to you, what is possible to link together, and what hypothesis you could look to prove before you start chucking every bit of data into the LRS. There is an argument for data duplication here, but only if it can be justified by a test that you want to carry out. 

Problem: I need to prove what the impact of my training is on other areas of the business.

Explanation: There is a task to demonstrate that training delivered has a positive impact on a particular area of the business. It is likely that the data does not all come from a single source so it is important to link this together. 

Solution: Decide on something that you want to test like “we want to see if employee confidence in a subject improves in the period of time after the course.” You’ll need to take a dataset from before you implemented the training which might be pre-course surveys or current result data that the business uses to measure success, ensure that the course is designed to have an impact on the area that you’re looking to test and then decide on the completion measurement and whether there are periods of time you wish to test for. 

At this point, you can decide on the storage method. It might be that the xAPI is suitable as a tool to standardize the shape of the data for easier analysis, or it might be that you are looking to make use of the LRS simply as a validation tool for your content before handing it off to your existing reporting tools which should be possible with any modern LRS. My message here is to use the tools in the best way, not just because you can. 

What’s next?

Now, I’ve only provided a couple of examples of problems in L&D that I’ve experienced, I know there are many more. I’m also certain that there are many that I’ve yet to come across. One thing I’m sure about though is that we have to really think about learning analytics as a series of problems that we are trying to solve, not just as a place to regurgitate the same old boring reports that we always have done. What’s the point in measuring completions if there is no measurable impact? We might as well not bother. Get creative, experiment and find answers to important questions. Trust me it’s much more fun that way. 

So the next time you’re thinking about learning analytics, maybe come up with a little checklist like the below before you commit lots of time and energy to the journey. 

Checklist 

  • What is the problem I’m trying to solve?
  • Does it have a positive impact? 
  • Is the training I’m delivering designed around solving this specific issue?
  • Do I have the data to demonstrate the existing state before the training takes place?
  • Can I get the data to test whether the training makes a difference?
  • Where is the best place to store my data to carry out my analysis?
Ian Blackburn
Product Manager

Ian first joined HT2 Labs in the summer of 2017 as a Solutions Architect, working between their in-house and client technical teams to design and implement effective learning programmes.

Taking on additional Product Management responsibilities later that year, Ian is now the Product Manager for Learning Locker.

Away from work, Ian is a Liverpool FC supporter and a keen amateur footballer for his local team.

View more from Ian Blackburn
Read more about Learning Pool
Visit our Learn and Connect section
Make an enquiry
Get in touch to find out how we can help

Find the best solution

  • Your details
  • Organisation
  • Enquiry




Do you currently use an LMS?
YesNo

How many employees require elearning?










Learning Pool needs the contact information you provide to us to contact you about our products and services. You may unsubscribe from these communications at any time. For information on how to unsubscribe, as well as our privacy practices and commitment to protecting your privacy, check out our Privacy Policy.


Success!
One of our team will respond back to you in the next few hours.
Read how we helped A.G. Barr create an award-winning LMS here.

Speak to our sales team


Our team of skilled learning professionals will respond to you as soon as possible.


Contact Information
Kevin Keating
UK Sales Manager
Nicole Kastner
US Sales Manager