NSF Contribution

ASSISTments is one of a handful of proven interventions shown to cause better student learning. The platform currently reaches hundreds of thousands of students so here is a summary of NSF support and funding of ASSISTments.

NSF Support Toward ASSISTments Development

This document outlines the seventeen NSF grants that have contributed to the success of ASSISTments as a learning tool for teachers and students, and hence, led to the efficacy result showing that ASSISTments almost doubles student learning when regularly used for homework.

IES funded the first version of ASSISTments, but NSF funded Heffernan to add system capabilities for embedded short-cycle randomized controlled trials (RCT). NSF support came from the NSF CAREER award, Learning about Learning ($646,000; 2005-2010). The NSF funding allowed for randomization of content at the student level, leading Heffernan to conduct and publish on 24 RCTs throughout the duration of the grant and beyond. These RCTs compared experimental conditions such as when to give worked examples or step-by-step assistance, or what types of motivational messages are most effective. After Heffernan used this system effectively, he opened up the research capabilities to outside scientists to propose their own ideas and conduct similar work using the platform.

Why is it that physicists can share particle accelerators and astronomers can share telescopes, and yet education scientists have no shared instruments? NSF SI2:SSE &SSI Adding Research Accounts to the ASSISTments Platform: Helping Researchers Do Randomized Controlled Studies with Thousands of Students ($486,000; 2014-2016) funded Heffernan to create that shared tool. Since leveraging the platform through the ASSISTments TestBed, a dozen researchers have conducted studies. More about this project can be found at www.assistmentstestbed.org, and the ideas behind it are discussed in this Brookings Institute Article.

Other NSF grants had specific impacts on how ASSISTments works in the classroom with teachers and students. An NSF GK12 Award Partnership Implementing Math and Science Education: Assisting Middle School Use of Tutoring Technology ($2,000,000; 2008-2013) allowed for our graduate student developers to work directly with teachers and respond to their needs. This led to innovations for essay grading, parent notification, and the PLACEments system, all of which are now integrated features in ASSISTments.

Similarly, the very popular Automatic Reassessment and Relearning System (ARRS), used by most teachers in the efficacy trial, was funded by an NSF REESE grant, Taking Advantage of Cognitive Science Principles: Adding to a Computer-Based tutor an Automatic Reassessment and Relearning System (ARRS) ($749,600; 2011-2014). This feature allows teachers to automatically re-assess students’ comprehension and supply additional practice as necessary. Research has also explored the practice spacing schedules used by ARRS. Additionally, students’ performance on reassessment tests is often used as a dependent measure in the analysis of randomized controlled trial data.

Currently, the NSF Collaborative Research grant The Downside of Perseverance - Investigating and Moving Students Beyond Unproductive Persistence ($499,892; 2015-2018) is allowing for the observation and understanding of how to support students who work continuously but still fail to learn, or those showing the downside of grit.

Another area of research has been in the exploration of student affect and the ability to predict it from students’ clickstream data gathered as they work in ASSISTments. Ryan Baker and Neil Heffernan’s first grant in this area was through the NSF: ITEST Program, Predicting STEM Career Choice from Computational Indicators of Student Engagement within Middle School Mathematics Classes. ($700,000; 2011-2014). Three papers were written showing how the data could be used to predict test scores, college attendance, and majoring in STEM fields. These affect detectors were employed again in the NSF REAL award Making Math Tutors More Engaging and Effective through Interaction Design Patterns and Educational Data Mining ($372,125; 2013-2017). In this project, the influence of content on student affect was explored and patterns were isolated to highlight boring content. Randomized controlled trials were then conducted to test ways to make boring topics more interesting.

The grant, NSF BD Spokes: Spoke: NorthEast: Collaborative: Grand Challenges for Data-Driven Education. ($429,110; 2017-2019) funded Dr. Heffernan to help use the research-driven aspects of ASSISTments to teach data science and to raise interest in data science as applied to educational topics. The skills characteristic of working with big data are growing increasingly important, and Heffernan plans to use his position to help the NorthEast Community improve upon data science in education.

The NSF CIF21 DIBBs: PD: Enhancing and Personalizing Educational Resources through Tools for Experimentation ($500,000; 2017-2021) enabled Heffernan to continue his work with an open-science agenda. Over 40 papers were published by others and launched E-TRIALS as a platform to help other research proposed by students in ASSISTments.

This was followed by the NSF CSSI Collaborative Research: CyberInfrastructure for Shared Algorithmic and Experimental Research in Online Learning ($3.5 million; 2019-2024) which funded the building of the E-TRIALS infrastructure to run studies in the UPENN MOOC. One of the researchers who used E-TRIALS to run her own study was Candace Walkington, and after publishing that work she invited Heffernan and ASSISTments to participate in an NSF grant that lets students write their own questions. ASSISTments readily agreed which led to the NSF grant, Collaborative Research: Personalizing Mathematics to Maximize Relevance and Skill for Tomorrow's STEM Workforce ($1 million; 2018-2021) in which ASSISTments was funded to build out the extra supports necessary for students to write questions.

Around this point in time, Cristina and Neil Heffernan began to realize that their platform and their user-base could help more researchers run studies, and they have subsequently partnered with others on IES grants to test different research ideas. In this role, ASSISTments does the pre- and post-testing as well as randomization, and the running of some conditions--for example in Erin Ottman's IES Efficacy Trial testing the "From Here to There" intervention. Since that time, IES has decided to dedicate funding to other platforms to add research capacity to their own resources, and ASSISTments is one for 5 winners of a research competition to do just that.

While most of the work in ASSISTments is computer gradeable--in which students receive correct/incorrect responses from the computer--over half of the problems that students are asked are open-ended, often in the form of "Explain how you got that number"--which computers do not do well in grading. To address this issue, we proposed to NSF a research project to experiment with better ways to grade open-ended questions, but also to suggest comments for teachers to send back to a student on ways they could improve their answer. The first was for middle school math teachers called NSF-Cyberlearning Putting Teachers in the Driver’s Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT) ($750,000; 2018-2022) followed by a similar project for undergraduate algorithms classes called NSF: IUSE EHR: Improving Undergraduate Algorithms Instructing with Online Feedback ADVISOR ($250,000; 2019-2022).

In 2019 Heffernan and Baker and Lan were funded by NSF Collaborative Research: Student Affect Detection and Intervention with Teachers in the Loop ($750,000; 2019- 2022) to do active-learning research (active learning is a type reinforcement learning paradigm where the learner is allowed to select actions). In a related grant that used reinforcement learning, using contextual bandits as its dominant learning paradigm, NSF Collaborative Research: Precision Learning: Data-Driven Experimentation of Learning Theories using Internet-of-Videos ($1,400,000; 2019-2022) funded ASSISTments to add a new feature to see if we could figure out which YouTube video to give out to help learners.

Finally, two other current projects are a NSF REU Site: Leveraging The Learning Sciences & Technologies to Enhance Education and Learning in Secondary Schools ($320,639; 2020-2024) and NSF Collaborative Research: Common Error Diagnostics and Support in Short-answer Math Questions ($849,775; 2021-2024). Because both projects are relatively new, there is nothing to report.