Assessing the Efficacy and Implementation of a Technology-based Mathematics Intervention for Middle School Students

Middle school is a critical juncture for mathematics learning and a point where many U.S. students get left behind and turn away from mathematics. This has a lasting effect, as these students enter high school unprepared for the college preparatory courses that would set them up for more advanced academic and career opportunities. This project will investigate the potential of a novel approach to mathematics intervention that leverages the affordances of technology and evidence-based pedagogy to improve mathematics learning for middle school students. The mathematics intervention entitled EMPIRES is a collaborative activity set in Ancient Mesopotamia that offers a multifaceted approach in which (1) rich, narrative math problems increase conceptual comprehension; (2) animated representations of mathematics concepts support mathematical understanding; (3) multiplayer collaboration leads to peer instruction and modeling; (4) simulations offer exciting challenges that increase mathematics resiliency; and (5) a bridge curriculum aids transfer of learning to multiple contexts, including traditional standardized tests. This project will contribute to the research base on middle school math interventions; the application of technology to mathematics education more broadly; and the potential of educational technologies to support mathematics motivation, resiliency, and achievement for students, especially English learners. EMPIRES augments teachers’ ability to facilitate learning for a range of levels and provides timely feedback, allowing mathematics intervention teachers from a range of backgrounds to provide higher-quality learning experiences. This model could help English learners and other student groups underrepresented in STEM to stay motivated, persist, and pass the gateway of algebra into STEM career pathways.

The core research design will employ a one-year cluster randomized controlled trial. Schools will be randomly assigned to implement EMPIRES in grade 7 mathematics intervention classes during the 2024-25 school year or continue with business as usual. The primary purpose of the study is to determine whether EMPIRES improves mathematics achievement on average, whether student-level characteristics (e.g. English learner status) moderate the effect of EMPIRES, and whether student motivation, mathematics resiliency, and peer teaching mediate the effects of EMPIRES. Student scores from the state mathematics assessment serve as the primary outcome measure. Other key measures include (1) electronic usage records automatically collected by the EMPIRES platform; (2) validated student survey scales measuring motivation, mathematics resiliency, and peer teaching mediators; and (3) teacher implementation interviews and surveys. Researchers will estimate the main effect of EMPIRES using hierarchical linear modeling, controlling for prior achievement and other covariates. Moderator analyses will also employ hierarchical linear modeling to examine whether the impact of EMPIRES varies for students across policy-relevant subgroups, including English learners. Structural equation modeling will be used to assess whether student motivation, mathematics resiliency, and peer teaching mediate the effects of EMPIRES on student learning. Researchers will also examine the relationship between teacher and student implementation fidelity and student achievement. Findings from this study will help researchers and educators better understand design characteristics of effective games and best practices for implementing them in educational settings, particularly in settings with higher percentages of English learners.