Harnessing the power of AI to provide personalized feedback at scale

When Benjamin Bloom conducted his ‘2 Sigma’ study in 1984, he evidenced what many had suspected; people learn much more effectively in a personalized environment, such as one-to-one tutoring with an expert than in the classroom or in a group setting.

Ever since this study was published, trying to replicate the positive impacts of one-to-one tutoring has been a mission for many involved in education and organizational development, including here at Learning Pool, and a lot of progress has been made in terms of adaptive pathways and tailored content. But the real value in one-to-one tutoring is personalized feedback which has been really hard to replicate at scale – until now!

The new wave of artificial intelligence models – known as generative AI due to their ability to generate content, whether textual, voice, or visual – opens up a whole world of possibilities. 

These AI Large Language Models are incredibly clever. For example, you could ask an LLM to write you a bedtime story personalized to your child. It could include their interests, it could be done in the style of their favorite author and it could even create some amazing illustrations to go alongside it. But there are also some challenges. Generative AI can’t be entirely trusted to always give quite the right response or even to be factually correct. So if you’re using it for learning or sharing information the output needs “to be carefully double-checked as the software seems to generate inaccurate content, based on inaccurately reported sources of ideas.” (International Journal of Information Management Volume 71, August 2023, 102642)

And this fact-checking is hard, if not impossible, to do at scale.

This is why the Learning Pool and Mind Tools teams decided to collaborate to look at how we can use the power of generative AI combined with highly trusted content to create unique, trustable, scalable and effective learning interactions that not only impact the learning experience but also improve learning outcomes. 

And for our first foray into this, we decided to focus on feedback so the AI initially isn’t generating content or training but instead analyzing the inputs of the learners against a trusted framework.

Why is feedback so critical to learning? 

Feedback is a critical component of the learning process, as it provides learners with valuable information about their performance, progress, and areas for improvement. Academic research has shown that effective feedback can enhance learning outcomes, motivate learners, and improve their self-regulation and metacognitive skills.

One study by Hattie and Timperley (2007) examined the effects of feedback on learning and found that task-specific feedback that focused on how to conduct the task more effectively had one of the largest effect sizes on student achievement. They identified three key elements of effective feedback: 

• Clarity
• Specificity
• Goal orientation

The study also emphasized the importance of providing timely feedback, as learners tend to benefit more from feedback when it is received immediately after the task is completed.

From the self-regulation model and the research literature on formative assessment it is possible to identify some additional principles of good feedback practice. 

• Promotes reflection
• Encourages dialogue 
• Improves self-esteem
• Provides opportunities to close the skills gap 

Additionally, research has shown that learners who receive feedback that is personalized and aligned with their individual goals and needs tend to be more motivated and engaged in the learning process (Nicol & Macfarlane-Dick, 2006).

Achieving this type of specific, timely and personalized feedback at scale has long been a challenge since it is so dependent on human interactions. It is exactly this type of feedback achievable at scale that we set out to provide when we designed Converse.

What is Converse?

Converse is a prototype that uses AI to allow managers to build their communication and management skills by practicing the types of difficult conversations they may need to have with people that report to them. They not only get feedback from the AI in the form of realistic, conversational responses, but they also receive detailed personalized feedback on how they performed down to the specifics of the exact words they used and how they met certain pre-set criteria. 

How does it work?

Converse allows learners to enter their name and select an avatar. Once this is done, they are presented with the scenario. The scenario is the same each time in the prototype but the characters are selected at random and each has different character traits, such as how stubborn they are and how good a work ethic they have, that build their overall personality.

In the prototype scenario, one of the learner’s employees is having some issues. We don’t know what’s causing the issues, but they’re not performing to their usual standard and they’re also coming in late and leaving early. The learner is about to have a first meeting to approach the issues. They are given advice on the key points to consider when having this type of conversation and also informed that the interaction is time limited to encourage learners to give instant responses, just as they would in the real world. 

Here is an example of a conversation between a learner and their ‘AI employee’. Learners can choose whether to use Converse in text-only mode or whether to turn on audio and audio transcription, allowing them to speak their responses and hear their employees, really elevating the experience. As well as feeling more realistic it also becomes more powerful as users can often say much more than they can type in the same time bringing depth to the interactions that isn’t possible through text alone. 

With multi-language support, Converse ensures that employees can engage in high-quality learning experiences in their preferred language, enhancing global accessibility and engagement.

Once the learner has completed the conversation, they can get detailed personalized feedback against each of the set criteria based on their conversation responses and choose whether they want to try again. 

We are jumping into this Generative AI world because we believe in the positive impact it can have in many situations, But we are critically testing our prototype along the way to ensure it is effective and trustable. Converse is at an early stage but feedback so far has been incredibly positive – there is real excitement about the possibilities. If you are interested in finding out more about Converse please email us at [email protected].

References

• “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy International Journal of Information Management Volume 71, August 2023, 102642
• The Two Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring Benjamin S Bloom Vol. 13 No. 6 (Jun-Jul 1984) pp 4-16
• Review of Educational Research March 2007, Vol. 77, No. 1, pp. 81-112 DOI: 10.3102/003465430298487 The Power of Feedback John Hattie and Helen Timperley
• David J. Nicol & Debra Macfarlane‐Dick (2006): Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice, Studies in Higher Education, 31:2, 199-218

*This blog is a MindTools and Learning Pool collaboration.