Gagnon & Sales IES grant

In 2016 Dr Heffernan's lab published upon, and released, the first data set that had the data from 22 experiments. Dr Heffernan was interested in strong methods that could allow us better ways to measure the effects of these experiments.

2018 at the Atlantic Causal Inference conference Anthony and Adam met Johann Gagnon-Bartsch and Luke Miratrix who were interested in this issue and we're already doing great work in this area. We decided to try to collaborate. That collaboration has resulted in a working paper (see below) under submission and an IES grant:

1. We have a working paper Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data. A. Gagnon-Bartsch*, A. C. Sales*, E. Wu, A. E. Botelho, L. W. Miratrix, N. T. Heffernan

2. This has now resulted in this IES grant that is what IES calls a "Methodological Innovation" grant line.

Open Data

Side note: Dr Heffernan is proud of the fact that he anonymized the data and shares it with others. Many others have used these released datasets. In particular, for this data set of experiments, Rafferty, Ying & Williams (2019) JEDM paper, used this data set.

• Rafferty, A., Ying, H., & Williams, J. (2019). Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments. JEDM | Journal of Educational Data Mining, 11(1), 47-79. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/357 Used the 22 Experiments dataset.

2019

• Sales, A.C.,Botelho, A.F., Gagnon-Bartsch, J., Heffernan, N.T., Miratrix, L., Patikorn, T., & Wu,E.Using Machine Learning and Auxiliary Data for Precise, Unbiased, Causal Inference., Contributed Presentation at: The Society for Research in Educational Effectiveness Spring Conference, Washington, DC, March 2019; The Atlantic Causal Inference Conference, Montreal, Canada, May, 2019.

• Botelho, A., Sales, A., Patikorn, T. & Heffernan, N. (2019) The ASSISTments TestBed: Opportunities and Challenges of Experimentation in Online Learning Platforms. Workshop on Learning Anaystic Services to Support Persilzied Learning & Assessment at Scale.

2018

• SP24 Sales, A. C., Botelho, A. F., Wu, E., Gagnon-Bartsch, J., Miratrix, L., Patikorn, T. & Heffernan, N. T. (2018) Residualization Methods to Better Estimate Treatment Effects in Randomized Controlled Trials. Presented at the Conference on Digital Experimentation (CODE) held at MIT. View recorded talk here. Here in youtube.

• CP88 Sales, A., Botelho, A. F., Patikorn, T., & Heffernan, N. T. (2018, July). Using Big Data to Sharpen Design-Based Inference in A/B Tests. In Proceedings of the Eleventh International Conference on Educational Data Mining, 479-485. Retrieved from https://files.eric.ed.gov/fulltext/ED593197.pdf Corrected Version is here

• Sales, A., Botelho, A. F., Patikorn, T., & Heffernan, N. T. (2018, May). Deep Learning, Auxiliary Data, and Randomization: Analyzing Experiments Run Within a Computerized Math Tutor. In The 2018 Atlantic Causal Inference Conference, Pittsburgh, PA.[abstract]

2017

• S. Zhao and N. Heffernan. Estimating individual treatment effects from educational studies with residual counterfactual networks. In 10th International Conference on Educational Data Mining, 2017.

Early 2016

• Patikorn, T., Selent, D., Heffernan, N. T., Yin, B., Botelho, A. (2016) ASSISTments Dataset for a Data Mining Competition to Improve Personalized Learning. Poster at MIT Conference on Digital Experimentation. CODE 2016.

• Selent, D., Patikorn, T. & Heffernan, N. T. (2016) ASSISTments Dataset from Multiple Randomized Controlled Experiments. A “Work in progress” category presented at Learning at Scale 2016. At ACM Digital Library, 181-184.