While stress has always been an issue in education, there have been differing opinions on its role. However, recent studies indicate that stress negatively impacts learners while preparing and revising (Theobald Et al.) and hinders memory recall (Vogel Et al.). Therefore the challenge is to create low-stress environments to encourage knowledge acquisition and recall.

At least, that was the hypothesis.

With that in mind, it was natural that Memrise would want to put their new language practice chatbot, MemBot, to the test via an independent study by Kouo on 60 language learners using real-time emotion analytics.

What is MemBot?

MemBot is an AI language partner built using GPT-3 to create human-like conversations. Learners can use it for practice by accomplishing ‘missions’ where they talk to MemBot in the language they want to learn.

Memrise designed MemBot to be low-stress to help learners. So for them, understanding how the experience affects people and whether stress had a negative or positive impact also meant having a clear idea of what to improve next.

Why use emotion analytics?

Emotion analytics allowed Memrise to see the emotions felt by learners while in practice sessions and their in-the-moment reactions to every step of the MemBot user journey.

It was a superpower that helped them understand their user and the drivers of user behaviour.

From a product perspective, this superpower showed them if MemBot was performing as intended. While strategically, Memrise, using insights into emotions, could identify the experiences that drove retention and positive habit formation.

But how was emotion analytics better than regular analytics?

Memrise had two options to understand how MemBot affected learners.

Option 1 — The usual method

The usual method looks a little like this ⤵

Look at data on user actions and behaviour → Identify key problem points → Run user interviews and surveys → Organise feedback → Make sense of all the feedback → Come up with hypotheses → Run some more user interviews and surveys → Look for behaviour changes in data → Come up with new hypotheses → Build → Test. And repeat the learning cycles until they found what worked.

In the context of MemBot, it is easy to see that the (🚨JARGON ALERT🚨) search space, the area they have to look at to understand where MemBot might be introducing too much or too little stress, was huge.

Additionally, they would need to account for recency bias from user feedback, spend time separating UX stress from conversation missions, and would not have the ability to measure real-time cognitive stress.

Post-facto information gathered using surveys, interviews and feedback forms can’t provide reliable data on in-the-moment stress levels.

Option 2 — Emotion Analytics

Emotion analytics enabled them to:

• Minimise the search space → They were able to separate emotional reactions to UX elements from feelings of stress induced by the practice missions.
• Reduce the number of learning cycles → Separating UX from practice sessions and knowing how user cohorts reacted to stress allows them to reduce the number of iterations needed to make valuable changes.
• Understand how each part of the experience affected people → They were able to focus on the actual practice sessions and test their hypotheses on stress in the learner’s natural environments.
• Understand what emotions drove behaviour → They could get a clear picture of what emotional states helped learners in habit formation, increasing retention rates for MemBot.

In addition, they could connect their existing analytics data to emotions making Option 2 far superior to Option 1.

What was the outcome of the study?

Kouo helped validate the hypothesis and proved MemBot to be a superior language practice companion.

With MemBot, learners experienced 45% lower stress levels than when speaking with a tutor, and their motivation to practice more than doubled.

But crucially, emotion analytics helped Memrise uncover new insights to improve habit formation and user retention. Kouo found that the learners' perception of overall stress in practice sessions was identical to the amount of stress they experienced at the end of the session, and learners who experienced lower levels of stress were more likely to return. This meant that Memrise now had a way to effectively calibrate stress for its learners and a mechanism to encourage habit formation, a crucial part of every language learner’s journey to mastery.

To learn more about the results, see our case study here.

Or get in touch with us to understand how Kouo could help your product teams.

‍