IES Contribution

ASSISTments Development supported by IES / DOE

In 2003, an IES grant (R305K030140: 2003-2007) funded the creation of ASSISTments, an online tool to help teachers and students. With this tool, WPI personnel and Worcester Public School teachers could input questions with associated feedback and deliver the content to students. The first problems in the system were simply Massachusetts’ released high-stakes state math test questions. WPI students entered these questions and wrote detailed feedback for each question following the advice of the experienced teachers. Teachers were able to choose what to assign from the library of premade content or they could write their own questions. Reports provided teachers with information on 1) which questions were hardest for their students and 2) what common wrong answers were being submitted by students. This information gave the teachers data to use to initiate class discussions around the completed assignments. 

To give something for teachers to assign for repetitive practice Heffernan won his second IES grant (R305A070440: 2007-2011) to create and research sets of  “skill builders.” WPI created 200 skill builder problem sets for middle school math topics. Each skill builder gives students practice until they reach some initial threshold of performance. Students who do well on a skill builder finish quickly, but students who are not proficient in the skill will be given practice and hints until they get three correct in a row. Grants from NSF allowed Heffernan to extend the idea of mastery learning to a system called Automatic Reassessment and Relearning (ARRS). After a student completes questions to show skill proficiency,  one week later the students are automatically reassessed on the skills assigned by their teacher. If they have forgotten the skill, they received remediation. In addition to this extension grant on the idea of skill builders initially funded by IES, NSF has funded ASSISTments eight times to support the development of and research with ASSISTments.

All of this work combined into what was ASSISTments in 2012 was tested as an effective online homework intervention by the IES Efficacy grant (R305A120125: 2012-2016). In order to conduct the study, the team took every commonly used 7th grade textbook in the State of Maine and added all the answers to textbook homework problems into ASSISTments. The system was also made to work offline so a student could work on the school bus, for example, away from wifi. The project assessed the role of immediate feedback to both teachers and students with a particular focus on the need for low-income students to get smarter homework. In 2016 the results of that Efficacy Trial showed that 1) teachers changed their homework reviewing behavior, 2) students learned reliably more and that difference was an extra three quarters of a year of schooling, and 3) Using ASSISTments reliably closed achievement gaps contrary to current research that “draws attentions to the issue of whether computerized learning might contribute to the achievement gap between students with different achievement levels” (Steenbergen-Hu & Cooper, 2013).  

The study does not tease out why the tool may have worked so well, but Heffernan and his wife Cristina have a hypothesis. “Cristina and I were both school teachers and we think our system works well because we designed it to be used by teachers, rather than to replace teachers. Many education technology products put the computer in charge, and try to promote the students through the curriculum at just the right pace, fast for some students and slow for others. When this happens the teacher who is an important factor in motivating students, is no longer able to track the data because each student is in a different place. We let teachers give out short cycle adaptive assignments while remaining in charge of the pacing. Each morning the teacher checks in and uses the data to support decisions about the next lesson. We recognize that this is a different mindset from the push to highly personalized products. We have created a product that is highly personalized to the teacher not the student and we let the teacher use the data to  individualize their feedback to their students and to inform the social part of learning that is their daily class.” 

Now that the Efficacy Trial proves that ASSISTments works, where is ASSISTments going next? He points out that of the 80+ teachers that were in the Efficacy Trial, one teacher wrote a feedback message for every problem he assigned from his textbook. Heffernan wants to make it possible for other teachers to use that feedback. In other words he would like to crowdsource one or more teachers' feedback and give it to other teachers using the same textbook. Heffernan says, “My life’s goal is to be like Jimmy Wales who created Wikipedia that allows people to write encyclopedia entries, but I want to do that for feedback on every math and science questions given to students in K12 in America, that means letting teachers write the feedback and letting other teachers find it and use it.”

In 2023, the results of the IES Replication grant found the same effect but in North Carolina. 

When asked why ASSISTments is free, he responds, “One day I  had a seizure and was diagnosed with a brain tumor and I was told I had 2-3 years to live. I did chemo and radiation and surgery for a year. Luckily the doctors were wrong and I have beaten my cancer. The experience changed me, my wife and I decided that our goal in life had to be to make a difference. I am glad Worcester Polytechnic Institute has just donated two staff members to the project for the next two years to keep it growing. Last year alone 12 million problems were solved by 50,000 students. Next year looks to be even bigger. And like Wikipedia, we have to be free if we are going to ask teachers to share in this endeavor in our crowdsourcing project.”

Here is a partial listing of grants that have supported the growth and development of ASSISTments. 

U.S. Department of Education has funded Heffernan from the Institute for Education Sciences 

1. (2003). U.S. Department of Education: Institute of Education Sciences. Using Web-Based Cognitive Assessment Systems for Predicting Student Performance on State Exams. (R305K030140). The PI is Ken Koedinger (CMU). Co-PIs Neil Heffernan, Brian Junker and Ritter. $1.4 million. 2003 - 2007.

2. (2007). U.S. Department of Education. Making Longitudinal Web-Based Assessments Give Cognitively Diagnostic Reports to Teachers, Parents & Students While Employing Mastery Learning. (R305A070440). The PI is Neil Heffernan (WPI). Co-PIs Ken Koedinger, Brian Junker, George T. Heineman, Murali Mani & Cristina Heffernan. $2 million. 2007 - 2011.

3. (2010). U.S. Department of Education: Institute for Education Sciences. National Center for Cognition and Mathematics Instruction (NCCMI) (R305C100024). WestEd is the project lead. Co-PI is Neil Heffernan's role is this. WPI receive $500,000 of $10 million. 2010 - 2015.

4. (2012). U.S. Department of Education. “An Efficacy Study of Online Mathematics Homework Support: An Evaluation of the ASSISTments Formative Assessment and Tutoring Platform.” (R305A120125). SRI’s Jeremy Rochelle is the PI on this efficacy trial. Read more here and here. The results are published here. Of the $3.5 million award, WPI will get $1,200,000. 2012-2016. 

5. (2017). US Department of Education: Institute for Education Sciences. Efficacy of ASSISTments Online Homework Support for Middle School Mathematics Learning: A Replication Study. (R305A170137). The PI is SRI's Mingyu Feng. $3.2 million (WPI's share is $1.2 million). 2017-2021. 

6. (2017). U.S. Department of Education: Institute for Education Sciences. Evaluating the Effectiveness of the ASSISTments Feedback System. (R305A170243). The PI is AIR's Kirk Walters. $3.8 million total (WPI's share is $1.5 million). 2017-2022. 

7. (2018). U.S. Department of Education: Institute for Education Sciences. "From Here to There" Efficacy Trial. (R305A180401). The PI is Erin Ottmar. ($3.2 million). 2018-2021. 

8. (2021). U.S. Department of Education: Institute for Education Sciences. Fully Latent Principal Stratification: A New Framework for Big, Complex Implementation Data from Education RCTs. (#R305D210036). The PI is Tiffany Whittaker (UT-Autsin). Co-PI at WPI is Adam Sales. $891,895 for 3 years. 2021-2024. 

9. (2021). U.S. Department of Education: Institute for Education Sciences. Revisions to the ASSISTments Digital Learning Platform to Expand Its Support for Rigorous Education Research. (R305N210049). The PI is Neil Heffernan (WPI). $2 million. This is a grant to allow us to add extra research infrastructure to the ASSISTments platform. 2021-2024.   

10. (2022). U.S. Department of Education: Institute for Education Sciences. A Systemic Replication Study of Interleaved Mathematics Practice. (R305R220012.) The Co-PIs are Bryan Matlen of WestEd and Doug Rohrer of University of South Florida. This grant will conduct an efficacy replication study of the mathematics learning intervention known as "Interleaved Practice." $3,986,368. (WPI's share is $250,000.) 2022-2027. 

Heffernan received eight grants from the NSF that are described here

Heffernan has been funded three times by the US Dept of Education's GAANN program

1. GAANN Fellowships in Computer Science to Support Data-Driven Computing Research. U.S. Department of Education (P200A150306) PI Elke Rundensteiner. $885,834 09/01/2015 - 08/31/2018.

2. US Department of Education: GAANN. Fellowships in CS to Support the Learning Sciences and Security. PI Heffernan. Awarded approximately $800,000 to support five PhD students per year. 2012-2016. 

3. US Department of Education: GAANN. Fellowships in CS to Support the Learning Sciences and Security. PI Matt Ward. Co-PIs Heffernan, Agu and Mani. Awarded approximately $800,000 to support five PhD per year. 2006 - 2009.

ONR has funded Heffernan five times to build better tools to author intelligent tutoring systems

1. “Integration of Intelligent Tutoring Systems for Electronics” Office of Naval Research. $166,649 5/1/15 – 12/31/15. 

2. Office of Naval Research. Two computer tutors will be combined to together. PI Bev Woolf of UMass. Of the $1.5 million award, WPI will receive $700,000. 2012 - 2014. 

3. Office of Naval Research (ONR). Demonstrating Affordable Behavior Modeling with CTAT through Machine Learning and Human Computer Interaction Techniques. PI Neil Heffernan. $275,000. 2005 - 2008.

4. Office of Naval Research (ONR). Affordable Cognitive Modeling Authoring Tools using HCI Methods. PI Heffernan. $203,304. 2003 - 2006.

5. Office of Naval Research (ONR). Cognitive Tutor Tools for Advanced Instructional Strategies. PI Ken Koedinger. Co-PIs Heffernan & Aleven. $200,000. 2002.

More Recently A different arm of the US Dept of Ed called EIR  has funded ASSISTments. 

• 1. (2022). Recovering from COVID-Learning-Loss with a Platform to Support Human Tutoring. My spouse, Cristina Heffernan, wrote a great grant to allow us to study human tutors, and to try to assist them with an AI agent (my part) that uses ASSISTments data from their classwork/homework to suggest which problem the 

  2. (2019). EIR Midphase: Scaling Teachers' Professional Development for ASSISTments. U.S. Department of Education. In collaboration with the ASSISTments Foundation, WestEd and Lesley University. $8 million  The abstract and full grant and others we competed with are listed here.

  3. (2023). Scaling ASSISTments for Algebra Readiness (ASSISTments4AR) with Curriculum-based Professional Learning and Coaching. PDF

New Notes