CHS: Medium: Adapting to Affect in Multimodal Dialogue-Rich Interaction with Middle School Students

Affect, or emotion, profoundly shapes human experience. It influences how we perform tasks, how we build relationships with one another, and how we navigate the complexities of our daily lives. Affect is shaped and influenced by communication with other humans, experiences with the natural world, and interactions with machines. Affect plays a particularly prominent role in learning. During learning, a recurring subset of the broad range of human emotions such as confusion, frustration, boredom, anxiety, engagement, surprise, and delight appear regularly. Different emotions are best responded to in different ways. For example, task-based feedback and guidance is a helpful response to emotions of confusion and frustration, while empathetic feedback is more helpful for emotions of anger or excitement. Prior research has not answered the question of how affective adaptation can maximize the benefit to students as they interact with interactive computer-based learning environments. And yet the investigators on this project are now well positioned to address a central, unanswered question of how learning environments can adaptively respond to students' affect to create the most effective, engaging learning experiences while simultaneously promoting improved attitudes toward learning.

The project will provide important societal benefits by generating theoretical and practical advances across multiple disciplines. The project will lead to a deeper understanding of affect-rich learning; a set of broadly applicable affect adaptation principles; and a computational model of affective adaptation and dialogue that will be incorporated into a learning environment for science learning. The resulting affect-modeling technologies can serve as a foundation for the next generation of adaptive educational software that will promote learning through affect-rich adaptation. This will be broadly useful throughout education. The project will address issues of diversity by partnering with the highly diverse Dunn Middle School and Harnett Central Middle School, and through ongoing collaboration with the STARS Alliance for Broadening Participation in Computing. To ensure societal impact, the results will be disseminated to the public through middle school outreach programs, and to the scientific community through publication at scientific venues.

The three major scientific goals of the project are to: (1) Capture rich multimodal data of students' affective experiences while interacting with a fully instrumented learning environment with spoken dialogue. Observational studies will be conducted by having middle school students interact with an existing learning environment for science education called 'Crystal Island.' Crystal Island was developed by the investigators on this project and has already been used by thousands of students in middle school classrooms to learn microbiology, but it does not currently support rich multimodal interaction or natural language dialogue. Crystal Island will be fully instrumented to collect rich, multimodal data including speech, facial expression, gaze, posture, skin conductance response, heart rate, and problem-solving actions. (2) Design, develop, and refine an affect-understanding model that integrates students' natural language, nonverbal behavior, physiological response, and task-action phenomena into a rich multi-dimensional stream of affective data. By utilizing this data collected from the observational studies, an affect-understanding model will be constructed using machine learned including hidden Markov modeling. This will be the first affect-understanding model for learning environments that integrates the full complement of affect signals of spoken language (including prosody, syntax, and semantics), nonverbal behavior (including gaze and posture), physiological data (including skin conductance response and heart rate), and task actions (including navigation and manipulation actions in the learning environment). (3) Design, develop, and refine an integrated affect and dialogue management model that adaptively responds to students' affective states in the course of their learning interactions. By utilizing the learning-interaction data collected in the observational studies, a Partially Observable Markov Decision Process (POMDP) affect adaptation policy will be acquired with reinforcement learning, integrating affect and dialogue management. The resulting adaptation policy will govern both when and how the system responds to students' affect as they solve problems. The computer-based mentor will provide problem-solving advice, encouragement, empathetic responses, and other support as is needed to improve the educational experience and outcome.