Systems Reform

Systemic Formative Assessment to Promote Mathematics Learning in Urban Elementary Schools

This project builds on the study of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students and combines the intervention with research-based understandings of systemic reform. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders.

Lead Organization(s): 
Award Number: 
1621333
Funding Period: 
Thu, 09/15/2016 to Sat, 02/29/2020
Full Description: 

Districts have long struggled to implement instructional programming in ways that meaningfully and sustainably impact teaching and learning. Systemic education reform is based on the hypothesis that prevailing patterns of incoherence and misalignment in an educational system can send mixed messages to local implementers as they try to respond to various cues and incentives in the environment. Systemic reform seeks to bring alignment to education systems in multiple ways, including consistency across instructional philosophies, alignment across grade levels, and vertical coherence from district to schools to classrooms. This project builds on the Consortium for Policy Research in Education's (CPRE) ongoing, NSF-funded experimental study of the impacts of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students in Philadelphia-area schools. The project will combine the OGAP math intervention with research-based understandings of systemic reform. OGAP is based upon established theory and research demonstrating the impact of teachers' use of ongoing short- and medium-cycle formative assessment on student learning. It combines these understandings with recent research on learning trajectories within mathematics content domains. By bringing to bear the strengths of all three of these areas of research - formative assessment, learning trajectories, and systemic reform - the project promises a significant contribution to the knowledge base about the application of math learning research to classroom instruction on a large scale. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders. This project is funded by the Discovery Research PreK-12 (DRK-12) and EHR Core Research (ECR) Programs. The DRK-12 program supports research and development on STEM education innovations and approaches to teaching, learning, and assessment. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field.

CPRE and the School District of Philadelphia (SDP) will establish a research-practice partnership focused on developing, implementing, refining, and testing a systemic support model to strengthen implementation of the OGAP math intervention in elementary schools. CPRE's current experimental study of OGAP's impacts reveals, preliminarily, statistically significant positive effects on teacher knowledge and student learning. As a result, SDP has decided to expand OGAP into an additional 60 schools in 2016-17. However, the current OGAP study has also revealed weak implementation stemming from a lack of consistent leadership support for the intervention. The project will address this implementation challenge by developing, refining, supporting, and documenting a systemic support component that will accompany OGAP's classroom-level implementation. The systemic supports will be developed by a research-practice partnership between CPRE; SDP; OGAP; the Graduate School of Education at the University of Pennsylvania (PennGSE); and the Philadelphia Education Research Consortium (PERC). The team will use principles of design-based implementation research to iteratively refine and improve the systemic support model. Along with the design and development of the systemic support model, the project will conduct a mixed-methods study of its impacts and roll-out. A three-armed quasi-experimental study will examine the differential impacts of OGAP, with and without systemic supports, and business-as-usual math programming on teacher and student outcomes. A mixed-methods study will examine teacher and administrator experiences in both treatment groups, and will provide feedback to inform the iterative development of the systemic support model.

Improving the Implementation of Rigorous Instructional Materials in Middle Grades Mathematics: Developing a System of Practical Measures and Routines (Collaborative Research: Ahn)

The goal of this project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale through a system of practical measures and routines for collecting and using data that both assesses and supports implementation.

Lead Organization(s): 
Award Number: 
1620900
Funding Period: 
Sat, 10/01/2016 to Thu, 09/30/2021
Full Description: 

The goal of this 5-year research project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale. Many projects seek to improve mathematics instruction, but are not able to easily track their efforts at improvement. The primary product of this project will be a system of practical measures and routines for collecting and using data that both assesses and supports the implementation of rigorous instructional materials in middle-grades mathematics. In contrast to research and accountability measures, practical measures are assessments that require little time to administer and can thus be used frequently. The data can be analyzed rapidly so that teachers can receive prompt feedback on their progress, and instructional leaders can use the data to decide where to target resources to support improvement in the quality of instruction and student learning. The system of practical measures and routines will include 1) measures of high-leverage aspects of teachers' instructional practices that have been linked to student learning (e.g., rigor of tasks, quality of students' discourse) and attend to equitable student participation; and 2) measures of high-leverage aspects of key supports for improving the quality of teachers' practice (e.g., quality of professional development; coaching); and 3) a set of routines regarding how to use the resulting data to engage in rapid, improvement efforts. A key principle of the proposed project is that the system of measures and routines can be adapted to a wide range of school and district contexts. This project is supported by the Discovery Research preK-12 (DRK-12) program. The DRK-12 program supports research and development of STEM education innovations and approaches in assessment, learning, and teaching.

The project will establish three research-practice partnerships with five districts, in three different states, that are currently implementing rigorous instructional materials in middle-grades mathematics. Year 1 will focus on the development of a set of practical measures of classroom instruction. Year 2 will focus on testing the use of the classroom measures in the context of supports for teachers' learning, and the development of practical measures of key supports for teachers' learning. Years 3-4 will focus on how the project can "learn our way to scale" (Bryk et al., 2015), which requires strategically implementing measures and routines in increasingly diverse conditions. The project will engage in rapid improvement cycles in which researchers will work alongside district leaders and professional development (PD) facilitators to analyze the data from the measures of both classroom instruction and the quality of support for teacher learning to test the effectiveness of improvements in intended supports for teacher learning and to adjust the design of the support based on data. Across Years 1-4, the project will use recent developments in technology and information visualization to test and improve 1) the collection of practical measures in situ and 2) the design of data representations (or visualizations) that support teachers and leaders to make instructional improvement decisions. In Year 5, the project will conduct formal analyses of the relations between supports for teachers' learning; teachers' knowledge and classroom practices; and student learning.

Improving the Implementation of Rigorous Instructional Materials in Middle-Grades Mathematics: Developing a System of Practical Measures and Routines (Collaborative Research: Jackson)

The goal of this project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale through a system of practical measures and routines for collecting and using data that both assesses and supports implementation.

Lead Organization(s): 
Award Number: 
1620851
Funding Period: 
Sat, 10/01/2016 to Thu, 09/30/2021
Full Description: 

The goal of this 5-year research project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale. Many projects seek to improve mathematics instruction, but are not able to easily track their efforts at improvement. The primary product of this project will be a system of practical measures and routines for collecting and using data that both assesses and supports the implementation of rigorous instructional materials in middle-grades mathematics. In contrast to research and accountability measures, practical measures are assessments that require little time to administer and can thus be used frequently. The data can be analyzed rapidly so that teachers can receive prompt feedback on their progress, and instructional leaders can use the data to decide where to target resources to support improvement in the quality of instruction and student learning. The system of practical measures and routines will include 1) measures of high-leverage aspects of teachers' instructional practices that have been linked to student learning (e.g., rigor of tasks, quality of students' discourse) and attend to equitable student participation; and 2) measures of high-leverage aspects of key supports for improving the quality of teachers' practice (e.g., quality of professional development; coaching); and 3) a set of routines regarding how to use the resulting data to engage in rapid, improvement efforts. A key principle of the proposed project is that the system of measures and routines can be adapted to a wide range of school and district contexts. This project is supported by the Discovery Research preK-12 (DRK-12) program. The DRK-12 program supports research and development of STEM education innovations and approaches in assessment, learning, and teaching.

The project will establish three research-practice partnerships with five districts, in three different states, that are currently implementing rigorous instructional materials in middle-grades mathematics. Year 1 will focus on the development of a set of practical measures of classroom instruction. Year 2 will focus on testing the use of the classroom measures in the context of supports for teachers' learning, and the development of practical measures of key supports for teachers' learning. Years 3-4 will focus on how the project can "learn our way to scale" (Bryk et al., 2015), which requires strategically implementing measures and routines in increasingly diverse conditions. The project will engage in rapid improvement cycles in which researchers will work alongside district leaders and professional development (PD) facilitators to analyze the data from the measures of both classroom instruction and the quality of support for teacher learning to test the effectiveness of improvements in intended supports for teacher learning and to adjust the design of the support based on data. Across Years 1-4, the project will use recent developments in technology and information visualization to test and improve 1) the collection of practical measures in situ and 2) the design of data representations (or visualizations) that support teachers and leaders to make instructional improvement decisions. In Year 5, the project will conduct formal analyses of the relations between supports for teachers' learning; teachers' knowledge and classroom practices; and student learning.

CAREER: A Study of Factors that Affect Middle School Levels of Readiness for Implementing STEM Programs

This project will investigate whether six urban middle schools are implementing highly effective science, technology, engineering and mathematics (STEM) programs based on factors identified through relevant research and national reports on what constitutes exemplary practices in 21st century-focused schools.

Lead Organization(s): 
Award Number: 
1553098
Funding Period: 
Mon, 02/15/2016 to Sun, 01/31/2021
Full Description: 

This is a Faculty Early Career Development Program (CAREER) proposal responsive to Program Solicitation NSF 15-555. The CAREER program is a National Science Foundation-wide activity that offers the most prestigious awards in support of junior faculty who exemplify the role of teacher-scholars through outstanding research, excellent education and the integration of education and research. This project will investigate whether six urban middle schools are implementing highly effective science, technology, engineering and mathematics (STEM) programs based on factors identified through relevant research and national reports on what constitutes exemplary practices in 21st century-focused schools. The project will make this determination through the use of a STEM level of readiness rubric developed through a previous award that will be further revised through this study. The rubric will document the participating schools' level of readiness at the principal, teacher, and student levels using 15 criteria that include a combination of essential supports, core elements, attributes, and characteristics about STEM through: (1) school leadership as the driver of change in education; (2) professional capacity among teachers and staff in all academic areas; (3) student-centered learning climate reflective of high-quality teaching and learning practices; and (4) investment of resources (e.g. staffing, time, space, materials and supplies, partnerships) that support exemplary school-based programs.

The project will use surveys, focus groups, and face-to-face interviews to collect data from 18 principals; classroom observations and a survey to collect data from 380 teachers, and a survey to collect data from 3700 students. These data collections, augmented by other intermittent research activities, will provide insights about extant programs in participating schools regarding effective school leadership, state-of-the art teaching and learning practices, and the impact on students' interest, motivation, and self-efficacy about STEM education. The primary outcome from this project will be a field-tested jointly refined STEM level of readiness rubric based on input from principals, teachers, and students with guidance from the project's advisory board and the Center for Research in Educational Policy at the University of Memphis. The rubric will be instrumental in informing district-level education stakeholders and university-partner decision-makers' choices about where and when to invest resources to further support the development of higher quality STEM programs and schools. It will also be useful in identifying ways to improve students' overall perceptions about future courses of study and careers and the development of professional development modules for teacher training. Beyond these key school district-level outcomes, results will be used to enhance teacher preparation efforts through further refinement of methods courses and the STEM Teacher Leadership Certificate Program at the University.

Developing Teachers' Capacity to Promote Argumentation in Secondary Science

This project will produce insights into the challenges teachers face in modifying their teaching in the substantial and complex ways demanded by the Next Generation Science Standards. This project will develop and study a program of professional development to help middle and high school science teachers support their students to learn to argue scientifically. 

Award Number: 
1503511
Funding Period: 
Wed, 07/01/2015 to Sun, 06/30/2019
Full Description: 

This project will produce insights into the challenges teachers face in modifying their teaching in the substantial and complex ways demanded by the Next Generation Science Standards. This project will develop and study a program of professional development to help middle and high school science teachers support their students to learn to argue scientifically. The program includes strategies for organizing science activities to create contexts where students have something to argue about and teaching practices that promote sustained, productive argumentation among students. Results will document what aspects of these new practices teachers find easier and more difficult to implement, and how challenges are influenced by the urban schooling contexts in which project teachers work. The project will also further our understanding of how site-based professional development can be structured to support teacher learning and improvement.

The project is a longitudinal study of a cohort of 30 secondary science teachers from an urban school district in California. The professional development (PD) program will be organized around intensive summer institutes followed by 2 school-based lesson study cycles each year, facilitated by trained coaches. The PD work will be carried out over three years. All PD sessions will be recorded for interaction analysis to identify variations in coaching and teacher participation and the influences of such variation on teacher learning. Repeated measures of teachers' conceptions of argumentation will be given over 3 years as a measure of teacher learning. An observation protocol will be developed and used to measure teacher talk and its change over time. A sub-sample of teachers' classrooms will be video recorded to produce a longitudinal record for interaction analyses to link teacher talk to patterns of student argumentation. The third year of the project will add measures of student learning and link them to variations in teacher practice. The final year of the project will produce retrospective analyses that link pathways in teacher learning to features of the PD program and teachers' participation. The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

CAREER: Exploring Beginning Mathematics Teachers' Career Patterns

Research increasingly provides insights into the magnitude of mathematics teacher turnover, but has identified only a limited number of factors that influence teachers' career decisions and often fails to capture the complexity of the teacher labor market. This project will address these issues, building evidence-based theories of ways to improve the quality and equity of the distribution of the mathematics teaching workforce. 

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1506494
Funding Period: 
Fri, 08/15/2014 to Wed, 07/31/2019
Full Description: 

Recruiting and retaining effective mathematics teachers has been emphasized in national reports as a top priority in educational policy initiatives. Research indicates that the average turnover rate is nearly 23% for beginning teachers (compared to 15% for veteran instructors); turnover rates for beginning mathematics teachers are even higher. Many mathematics teachers with three or fewer years' experience begin their careers in high-needs schools and often transfer to low-need schools at their first opportunity. This reshuffling, as effective teachers move from high- to low-need schools, exacerbates the unequal distribution of teacher quality, with important implications for disparities in student achievement. Research increasingly provides insights into the magnitude of mathematics teacher turnover, but has identified only a limited number of factors that influence teachers' career decisions and often fails to capture the complexity of the teacher labor market. Thus, it is essential to understand the features, practices, and local contexts that are relevant to beginning teachers' career decisions in order to identify relevant strategies for retention. This project will address these issues, building evidence-based theories of ways to improve the quality and equity of the distribution of the mathematics teaching workforce. This support for an early CAREER scholar in mathematics policy will enhance capacity to address issues in the future.

This work will be guided by three research objectives, to: (1) explore patterns in mathematics teachers' career movements, comparing patterns between elementary and middle school teachers, and between high- and low-need schools; (2) compare qualifications and effectiveness of teachers on different career paths (e.g., movement in/out of school, district, field); and (3) test a conceptual model of how policy-malleable factors influence beginning math teachers' performance improvement and career movements. The PI will use large-scale federal and state longitudinal data on a cohort of teachers who were first-year teachers in 2007-08 and taught mathematics in grades 3-8. Three samples will be analyzed separately and then collectively: a nationally representative sample from the Beginning Teacher Longitudinal Study (about 870 teachers who represent a national population of nearly 85,970); about 4,220 Florida teachers; and about 2,410 North Carolina teachers. In addition, the PI will collaborate with Education Policy Initiative at Carolina (EPIC) at UNC-Chapel Hill to collect new data from the 2015-16 cohort of first-year teachers in NC (about 800 teachers) and follow them for 2 years. The new data collection will provide detailed and reliable measures on the quality of both pre- and in-service teacher supports in order to understand how they may be linked to teachers' career movements and performance.

The original award # of this project was 1350158.

Centers for Learning and Teaching: Research to Identify Changes in Mathematics Education Doctoral Preparation and the Production of New Doctorates

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs.

Lead Organization(s): 
Award Number: 
1434442
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

The quality of the mathematical education provided to teachers and ultimately to their students depends on the quality of teacher educators at the colleges and universities. For several decades, there has been a shortage of well-prepared mathematics teacher educators. Doctoral programs in mathematics education are the primary ways that these teacher educators learn the content and methods that they need to prepare teachers, but the quality of these programs varies and the number of qualified graduates has been insufficient to meet the demand.

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs. It will also gather data on the number of doctorates in mathematics education from the CLT institutions prior to the establishment of the CLT and after their CLT ended. A comparison group of Doctoral granting institutions will be studied over the same time frame to determine the number of doctoral students graduated during similar time frames as the CLTs. Follow-up data from graduates of the CLTs will be gathered to identify programmatic strengths and weaknesses as graduates will be asked to reflect on how their doctoral preparation aligned with their current career path. The research questions are: What were the effects of CLTs on the production of new doctorates in mathematics education? What changes were made to doctoral programs in mathematics education by the CLT institutions? How well prepared were the CLT graduates for various career paths?

Human Subjects Protection in the Digital Age

This project will convene a panel of experts in government, industry and academia to raise and discuss emerging concerns for human subjects' protections in the digital age. This project will support scholarly discussion on human subjects' protections in the digital age with implications for funding agencies, schools, and those who work with human subjects in a variety of environments.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1419055
Funding Period: 
Sat, 03/15/2014 to Sat, 02/28/2015
Full Description: 

This project will convene a panel of experts in government, industry and academia to raise and discuss emerging concerns for human subjects' protections in the digital age. Learners taking part in formal education, informal education, and out-of-school settings are subject to a ubiquitous tracking of their activities: locally, using the internet of things (e.g., smart phones, smart sensors and other cyberphysical devices), and globally, via the internet. This tracking may include data tracked passively (e.g., online purchases) or data made available on social media websites by the learners themselves. In addition, the use of the longitudinal data collected by local educational agencies for research is an increasingly political concern. Decisions about the use of these data by university researchers and scholars are typically made by Institutional Review Board (IRB) offices. New guidelines on IRB practices are being considered by a number of bodies, including the National Research Council, which issued a report in early 2014.

This project will support scholarly discussion on human subjects' protections in the digital age with implications for funding agencies, schools, and those who work with human subjects in a variety of environments. The issues discussed are of national import, including, but not limited to FERPA privacy concerns. To the extent that US researchers work with data from other countries (e.g., via massive open online courses or MOOCs), the impact of the reports that will be produced as part of this conference for education research is potentially global.

Testing a Professional Development Model for High School Science Reform and the Relationship of Key Variables to Student Achievement

This project tests the efficacy of an intensive, three year professional development program, the BSCS National Academy for Curriculum Leadership (NACL) on student science achievement in the state of Washington. The goal of the NACL is to develop the capacity of district-based secondary science leadership teams to sustain the implementation of research-based science instructional materials that promote improvement in teaching and learning.

Award Number: 
1316202
Funding Period: 
Sun, 09/15/2013 to Mon, 08/31/2015
Full Description: 

This project conducted by the Biological Sciences Curriculum Study(BSCS) tests the efficacy of an intensive, three year professional development program, the BSCS National Academy for Curriculum Leadership (NACL) on student science achievement in the state of Washington. The goal of the NACL is to develop the capacity of district-based secondary science leadership teams to sustain the implementation of research-based science instructional materials that promote improvement in teaching and learning. This study examines the influence of the program on student achievement after the schools and districts have had sufficient time for the effects to take hold.

The project uses existing data gathered from two cohorts of Washington-based NACL teams and archived student achievement data from Washington State?s Office of the Superintendent of Public Instruction (OSPI). Additional data are collected NACL-participating districts and personnel through surveys and interviews. The project compares student achievement between the 27 districts that participated in the NACL, and a minimum of 27 carefully matched, non-NACL districts using propensity-scoring methodology. Districts have experienced different dosages of the NACL, and the project examines the differential effects of being involved in the NACL over time.

This research study provides an opportunity for multiple stakeholders including NSF, other corporate foundations that have funded the development and implementation of the NACL, BSCS, and participating school districts to determine the extent to which professional development promotes the improvement student science achievement results. The broader impact of the research is testing the extent to which basic elements of teacher professional development models correlate with student achievement and to do so in a way that could be replicated by others in similar contexts. The proposed work would inform educators about the research-based approaches to professional development that has evidence of efficacy. Moreover, by determining the time-scales by which professional development programs might be shown to influence student achievement, the findings provide new information to policymakers and researchers regarding the amount of time that could be required to see a positive impact from new educational policies and programs.

Cross-National Comparison of School and District Supports for High-Quality Mathematics Instruction in the US and China

This RAPID project is a cross-national comparative study of U.S. and Chinese instructional support systems, building from earlier data about mathematics teaching and learning in large urban school districts of both the United States and the People's Republic of China. The study uses quantitative methods to compare and contrast the effectiveness of supports (e.g., professional development, teacher networks, school leadership) in improving teachers' instructional practices and student achievement using comparable instrumentation.

Lead Organization(s): 
Award Number: 
1321828
Funding Period: 
Sun, 09/15/2013 to Sun, 08/31/2014
Full Description: 

Since the publication of the result that students from Shanghai, China, outperformed students from all other participating countries on the 2009 Programme for International Student Assessment (PISA) in mathematics, researchers have sought to understand why Chinese mathematics education appears to be both more successful at boosting student learning and more equitably distributed. This RAPID project is a cross-national comparative study of U.S. and Chinese instructional support systems, building from earlier data about mathematics teaching and learning in large urban school districts of both the United States and the People's Republic of China. The work is being conducted by researchers from Vanderbilt University, Virginia Polytechnic Institute and State University and Beijing Normal University. The study uses quantitative methods to compare and contrast the effectiveness of supports (e.g., professional development, teacher networks, school leadership) in improving teachers' instructional practices and student achievement using comparable instrumentation.

The study contributes to research and policy in several ways. First, it is helping to identify supports that have been particularly effective in improving mathematics teaching and learning in China. This should inform current theories about how to best support mathematics education in the United States. Second, the cross-nationally validated instruments used to collect the data can be used by other researchers investigating curricular reform implementation cross-nationally. The findings of this study are especially relevant to district leaders as they develop support and accountability systems to effectively implement the content and practice standards of the Common Core State Standards for Mathematics.

This award is co-funded by NSF's International Science and Engineering Section, Office of International and Integrative Activities.

Pages

Subscribe to Systems Reform