Prediction of Learning Outcomes Based on User Interaction with the Recommender System

Abstract

Learning indicators serve as objective measures of students' activity, enabling the assessment of learning outcomes [1]. These outcomes represent the development of competences [2] and have long been utilized to measure and communicate students' progress [3].

With the possibility of self-assessment through recommender systems, employing algorithmic approaches based on concept mapping to personalise the learning process, students can study more time and outcome efficiency. Moreover, improved student engagement over time is expected [1].

Interested by the insights offered by learning indicators using recommender systems, we aimed to investigate, whether it would be possible to predict the learning outcome based on the selected learning indicator using machine learning.

During each session, 932 students responded to five questions related to matrices and linear algebra, selecting one option from a list of possible answers [1]. The selected learning indicator for predicting learning outcomes is the time spent on a question, as a shorter duration may suggest a better understanding of the topic. For each subject, we first computed time spent on each question, including instances when they revisited a question. Additionally, we determined the percentage of correctly answered questions. Subsequently, we categorized learning outcomes into three grading classes: "low, "medium," and "high", after generating a histogram to display their distribution.

For classification purposes, we will apply the Support Vector Machine (SVM) algorithm, which separates data into predefined classes. We will evaluate the model's performance and discrimination ability using Accuracy and Area Under the Curve (AUC) metrics. An AUC of 0.5 represents a random classifier, indicating no better performance than random guessing. Thus, our null hypothesis (H0) is AUC = 0.5.

The results aim to enhance our understanding of the effectiveness of the selected learning indicator in predicting learning outcomes through machine learning. Should our analysis reveal an inability to accurately predict outcomes, it suggests that relying solely on this indicator may be insufficient for this purpose. In such a scenario, we could explore alternative indicators to improve our predictive capabilities.

References

[1] A. Košir et al., “Student Activity Recommender System,” ERK 2023, 2023.

[2] R. Ogawa, and E. Collom, “Educational Indicators: What Are They? How Can Schools and School Districts Use Them?” California Educational Research Cooperative, Riverside, 1998.

[3] J. Caspersen, J. C. Smeby, and P. Olaf Aamodt, “Measuring learning outcomes,” European Journal of Education, vol. 52, no. 1, pp. 20-30, 2017.