The Matlen-Rohrer Replication Project

A Systemic Replication Study of Interleaved Mathematics Practice

Replication Project Type and Approach: Efficacy Replication, Systemic Replications Using Digital Platforms

WestEd with the support of University of South Florida, Worcester Polytechnic Institute, and the ASSISTments Foundation propose a systematic replication study of a highly promising mathematics learning intervention, interleaved practice, in 7th grade classrooms. Though psychologists have long known that interleaving and spacing improves long-term learning, the practice problems in most mathematics are arranged so that the majority of problems relating to the same skill or concept are blocked together. With the intervention, some of the practice problems are rearranged so that 1) problems of different kinds are mixed together, which improves learning, and 2) problems of the same kind are distributed across multiple assignments, which improves retention. Numerous studies in the laboratory and classroom have demonstrated that merely rearranging practice problems so that the students receive a higher dose of interleaved practice can dramatically boost scores on researcher-developed measures of learning. The systematic replication study will determine whether this promising intervention can improve scores on externally-developed outcome measures and whether these intervention can scale to a widely-used online intervention that currently reaches tens of thousands of students in diverse settings.

We will specifically conduct the study with a representative sample of students in the U.S. east coast with a diverse representation of mathematical proficiency, socioeconomic background, gender, and ethnicity. Our proposed study replicates and extends prior work by; including educationally relevant outcome measures, including students at a variety of proficiency levels, testing long term retention, and leveraging the ASSISTments digital mathematics learning platform. As interleaved practice appears to be highly effective yet scarcely used intervention that can be inexpensively implemented in nearly any mathematics course, the findings from our replication will be of interest to researchers, practitioners and policymakers.

The study will involve a cluster-randomized trial, randomizing classrooms within teachers to implement interleaved practice problems or blocked practice problems in the grade 7 mathematics school year. Seventy grade 7 teachers across the U.S. east coast will participate, with approximately 189 classrooms, and 3780 students. Student outcomes will be assessed at the end of the school year on proximal and distal tests as well as at the beginning of the 8^th grade year. The study will collect fidelity and implementation data and will conduct a cost analysis to assess its scalability.

Setting: The setting will be 13 U.S. east coast states who have adopted Common Core.

Population / Sample: The sample will be representative of the population with balanced representation along free/reduced lunch status, underrepresented in STEM minority status, gender, and baseline mathematics achievement.

Intervention: Interleaved practice can be added to any curriculum by rearranging a portion of the practice problems. The intervention is applied by rearranging existing practice problems along the following two criteria: (1) The practice problems in a course are arranged so that most problems follow a problem relating to a different skill or concept so that problems of different kinds are interleaved within an assignment, and (2) Most of the practice problems relating to a particular skill or concept are distributed across many assignments.The systematic replication will apply the interleaved strategy by rearranging problems from existing Grade 7 math curricula that are freely available within ASSISTments, a widely-used digital learning platform.

Research Design and Methods: The specific design we propose is a blocked, randomized controlled trial, with random assignment occurring at the classroom-level. The teacher will serve as the block, and we will measure impact on student mathematics achievement both immediately and after a six-month delay. The study will consist of a year-long implementation to take place across two-cohorts (in the second and third years of the project).

Key Measures: Student outcome measures will include a) a Proximal measure adapted from Rohrer et al. (2020), b) an externally-developed 8th grade readiness test, and c) a State Standardized Test for 7th grade.

Data Analytic Strategy: The primary analytic approach will use Hierarchical linear modeling, regressing the student outcome on condition assignment and controlling for student, classroom, and teacher/school covariates, and including random effects for classroom (level of assignment) and teachers/school (block). The analysis will be repeated for each outcome and post hoc adjustments will be applied to account for multiple comparisons.

Cost Analysis: Costs will be gathered using the “ingredients method” as described in Levin, McEwan, Belfield, et al. (2017), and they will include all expenditures on personnel, facilities, equipment, materials, and training. Upon completion of the cost analysis and impact analysis, we will compute cost-effectiveness ratios to determine the amount of additional money spent on the treatment (as compared to the control group) in relation to the amount of additional achievement of the treatment group, where achievement is measured in standard deviation units.

Related IES Projects: Co-PI Doug Rohrer was the lead investigator on several previous projects that were designed to develop and evaluate interleaved practice:

• R305A160263, An Efficacy Study of Interleaved Mathematics Practice https://ies.ed.gov/funding/grantsearch/details.asp?ID=1790

• R305A110517; Interleaved Mathematics Practice https://ies.ed.gov/funding/grantsearch/details.asp?ID=1184