Evaluating User Experience of the Mobile Application for ESM Studies
Abstract
Experience Sampling Methodology (ESM) is a research approach offering insights into people's daily experiences through self-reports, usually completed multiple times a day [1]. Mobile application Curious (about) consciousness was developed by the Center for Cognitive Science at the Faculty of Education, University of Ljubljana, and is specifically tailored for conducting ESM studies. It allows researchers to create various questionnaires for their studies, while participants can join these studies and complete questionnaires whenever they receive notification from the app. The main goal of this study was to evaluate the user experience of the application. We aimed to assess both the application’s functionalities and participants’ experiences of reporting their experiential moments through the app’s questionnaires.
We conducted usability testing with 8 participants. With this method, we can evaluate a product’s usefulness, accessibility, and overall value by observing participants’ interactions with it [2]. Testing tasks included creating a new account, joining research study, managing application settings, and filling out a questionnaire about a lived experience before receiving an alarm. The questionnaire comprised qualitative prompts for describing the experience, as well as quantitative measures: a Likert scale for rating the typicality of the day, a valence and arousal plane for assessing mood and emotional state, and multiple-choice questions for selecting factors affecting mood and emotional state. Participants also reported on the proportions of different experiential categories using a split linear scale.
After testing, we conducted interviews to understand how natural it was for participants to report their experiential moments using the app’s questionnaire. Specifically, we focused on questions related to reporting mood and emotional state, as well as the proportions of experiential categories, since this type of question is novel in ESM applications.
We transcribed and coded the usability tests and interviews. Overall, participants found the application intuitive and easy to use, though some encountered usability issues when joining research study or managing application settings. Participants expressed comfort with describing their experiences through open-ended questions. However, differences emerged when reporting proportions of experiential categories: while most reported experiencing multiple categories, some struggled to quantify them as percentages. Additionally, some participants faced challenges in determining the valence or arousal of their mood or emotional state, possibly due to varying levels of experience with self-reporting. Preferences also varied in reporting format: some favored describing their experiential moment in their own words, while others preferred quantitative questions.
The Curious (about) consciousness mobile application was generally well-received by participants. To better support different reporting preferences, the input format could be refined to provide a more natural and effortless way of expressing experiential dimensions through the app’s questionnaires.
References
[1] I. Myin-Germeys and P. Kuppens, The Open Handbook of Experience Sampling Methodology: A Step-by-Step Guide to Designing, Conducting, and Analyzing ESM Studies, 2nd ed. Leuven, Belgium: The Center for Research on Experience Sampling and Ambulatory Methods Leuven (REAL), 2021.
[2] C. M. Barnum, Usability Testing Essentials: Ready, Set-- Test! Boston, MA, USA: Morgan Kaufman, 2011.
