Our project is a collaborative effort between the University of Massachusetts Amherst (PI Andrew Lan), the University of Pennsylvania (Co-PI Ryan Baker), and WPI (Co-PI Neil Heffernan).

This material is based upon work supported by the National Science Foundation under Grants No. IIS-1917713, IIS-1917545, and IIS-1917808: "Collaborative Research: Student Affect Detection and Intervention with Teachers in the Loop," 09/19-08/22 (expected). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Please contact Andrew Lan, for inquiries related to the project.

Last updated: 07/20

PROJECT GOAL: a human-in-the-loop system

This project aims at creating a “human-in-the-loop” system for building high-quality student affect detectors and interventions for classroom use.

Though teachers often participate in the initial design of adaptive learning systems, and learning analytics dashboards bring information on students back to teachers, the partnership between teachers and adaptive systems remains limited. Adaptive intervention often fails to leverage the insights on learning that teachers have accumulated through years of practice. Instead, we envision the future of teaching with adaptive learning technology as one in which technology actively seeks and incorporates teacher feedback into a human-in-the-loop system, a symbiosis between teachers and technology that leverages the strengths of both.

Our system i) suggests the teacher to conduct observations of the affective state of students ii) collect self-reported confidence ratings from teachers to improve affect detection, and iii) show teachers the detected affective states and observe how they intervene with students.

We focus on “sensor-free” detectors of affect that are generated when students use computer-based learning platforms. These detectors take the form of machine learning-based classifiers trained on student affective state labels provided by human observers in real classrooms.

In the first stage of the project, we will employ active learning methods to suggest to teachers that they observe students whose affective states are more informative to improving the classifier, rather than the current method of observing students in round-robin fashion. We will explore whether these more informative observations can improve affect detection with less teacher effort and no disruption to classes. In the second stage of the project, we will incorporate richer data on how certain teachers judge student affect into the classifiers to improve their quality and degree of alignment to teacher judgements, increasing their usefulness to teachers. In the third stage of the project, we will use crowdsourcing to solicit ideas from teachers as to when specific affect interventions will be appropriate for specific students, and will build these ideas into a comprehensive intervention system. We will test this intervention system in an experimental study in real classrooms.

The teacher interface to identify student affective states.


L-MMSE-based active learning

We develop a new active learning method leverages the linear minimum mean squared error (L-MMSE) estimation framework. Experiments on a real-world student affect dataset shows that our method outperforms existing active learning methods and can reduce the number of labels needed to build high-quality, sensor-free affect detectors.

Attentive knowledge tracing

We develop attentive knowledge tracing (AKT), a method for knowledge tracing which couples flexible attention-based neural network models with a series of novel, interpretable model components inspired by cognitive and psychometric models. Experiments on multiple real-world datasets show that AKT (sometimes significantly) outperform existing KT methods while offering excellent interpretability.



  • Aritra Ghosh, Graduate Research Assistant, UMass Amherst

  • Shamya Karumbaiah, Graduate Research Assistant, University of Pennsylvania

  • Anthony Botelho, Research Scientist, WPI

  • Sami Baral, Graduate Research Assistant, WPI

  • ASSISTments Staff that assist in integration of this work into the ASSISTments infrastructure & WPI