In October of 2016 I was asked by NSF to define how NSF grants contributed to ASSISTments. This has not been updated since.
ASSISTments Development supported by NSF
An NSF supported project, ASSISTments, was recently shown in an IES Efficacy trial to significantly “move the needle” in improving student achievement on a standardized measure. Detailed results will be published in late October 2016 through AERA Open.
The present document outlines the eight NSF grants (totaling $8.5 million of which WPI received $5.8 million) that contributed to the success of ASSISTments as a learning tool for teachers and students, and hence, led to the efficacy result showing that when used for homework, ASSISTments almost doubles student learning.
While IES funded the first version of ASSISTments, NSF funded Heffernan to add system capabilities for embedded short cycle randomized controlled trials (RCT). This support came from the NSF CAREER (#0448319) award, Learning about learning ($646,000 2005-2010). This innovation allowed for randomization of content at the student level, leading Heffernan to conduct and publish on 24 RCTs throughout the duration of the grant and beyond. These RCTs compared experimental conditions such as when to give worked examples or step by step assistance, or what types of motivational messages are most effective. After Heffernan used this system effectively, he decided that he should let other researchers propose their own ideas and conduct similar work using the platform. Why is it that physicists can share particle accelerators and astronomers can share telescopes, and yet education scientists have no shared instruments? NSF SI2-SSE&SSI (#1440753) Adding Research Accounts to the ASSISTments Platform: Helping Researchers Do Randomized Controlled Studies with Thousands of Students ($486,000 2014-2016) funded Heffernan to create that shared tool. Since leveraging the platform through the ASSISTments TestBed, a dozen researchers have conducted studies. More about this project can be found at www.assistmentstestbed.org while the ideas are discussed in this Brookings Institute Article. More recently, NSF has further supported bringing in more researchers with NSF DIBBS (#1724889) CIF21 DIBBs: PD: Enhancing and Personalizing Educational Research Through Tools for Experimentation.
Other NSF grants had specific impacts on how ASSISTments works in the classroom with teachers and students. An NSF GK12 (#0742503) award Partnership Implementing Math and Science Education: Assisting Middle School Use of Tutoring Technology ($2,000,000 2008-2013) allowed for our graduate student developers to work directly with teachers and respond to their needs. This led to innovations for essay grading, parent notification, and the PLACEments system, all of which are now integrated features in ASSISTments. The very popular Automatic Reassessment and Relearning System (ARRS), used by most teachers in the efficacy trial, was funded by an NSF REESE (#1109483) grant, Taking advantage of Cognitive Science Principles: Adding to a Computer-Based tutor an Automatic Reassessment and Relearning System (ARRS) ($750,000. 2011-2014). This feature allows teachers to automatically re-assess students’ comprehension and supply additional practice as necessary. Research has also explored the practice spacing schedules used by ARRS. Additionally, students’ performance on reassessment tests is often used as a dependent measure in the analysis of randomized controlled trial data. Currently, the NSF Collaborating Research (#1535428) grant The Downside of Perseverance - Investigating and Moving Students Beyond Unproductive Persistence ($499,892 2015-2018) is allowing for the observation and understanding of how to support students that work continuously but still fail to learn, or those showing the downside of grit.
A final area of research has been in the exploration of student affect and the ability to predict it from students’ clickstream data gathered as they work in ASSISTments. Ryan Baker and Neil Heffernan’s first grant in this area was through the NSF: ITEST Program (#1031398), Predicting STEM career choice from computational indicators of student engagement within middle school mathematics classes. ($700,000 2011-2014). Three papers were written showing how the data could be used to predict test scores, college attendance, and majoring in STEM fields. These affect detectors were employed again in the NSF REAL (#1252297) award Making Math Tutors More Engaging and Effective through Interaction Design Patterns and Educational Data Mining ($372,125 2013-2017). In this project, the influence of content on student affect was explored and patterns were isolated to highlight boring content. Randomized controlled trials were then conducted to test ways to make boring topics more intriguing.
Finally, the most recent grant, NSF BD Spokes (#1636782): Spoke: NorthEast: Collaborative: Grand Challenges for Data-Driven Education. ($429,110. 2016-2019) is funding Heffernan to help use the research driven aspects of ASSISTments to teach data science and to raise interest in data science as applied to educational topics. The skills characteristic of working with big data are growing increasingly important, and Heffernan plans to use his position to help the NorthEast Community improve upon data science in education.