This Spotlight features DRK-12 impact studies that are researching the efficacy or effectiveness of previous work. We hope this Spotlight provides insight into this research type.
In this Spotlight...
- Featured Projects
- An Efficacy Study of a Comprehensive, Middle School Science Curriculum that Integrates Disciplinary Core Ideas, Science and Engineering Practices, and Crosscutting Concepts (PI: Christopher Harris)
- An Efficacy Study of the Learning and Teaching Geometry Professional Development Materials: Examining Impact and Context-Based Adaptations (PI: Jennifer Jacobs)
- Anchoring High School Students in Real-life Issues that Integrate STEM Content and Literacy (PI: Amy Lannin)
- Developing Leaders, Transforming Practice in K-5 Mathematics: An Examination of Models for Elementary Mathematics Specialists (PIs: Chandra Lewis, Patrice Woods, Nicole Rigelman)
- Networking Urban Resources with Teachers and University to enRich Early Childhood Science (NURTURES) Phase II: Expansion and Evaluation (PI: Charlene Czerniak)
- PBS NewsHour Student Reporting Labs StoryMaker: STEM-Integrated Student Journalism (PI: Leah Clapman)
- Supporting Success in Algebra: A Study of the Implementation of Transition to Algebra (PI: Deborah Spencer)
- Related Resources
- Additional Projects
Featured Projects
An Efficacy Study of a Comprehensive, Middle School Science Curriculum that Integrates Disciplinary Core Ideas, Science and Engineering Practices, and Crosscutting Concepts
PI: Christopher Harris
Grade Level(s): 6-8
STEM Discipline(s): Science
About the Innovation Being Studied: This is a study of the impact of the Amplify Science Middle School (ASMS) curriculum on teaching and learning. Developed by the Lawrence Hall of Science in collaboration with Amplify Education Inc. ASMS is the Hall’s first comprehensive curriculum program that has been designed specifically to meet the vision of the National Research Council’s Framework for K-12 Science Education and address the performance expectations of the Next Generation Science Standards (NGSS).
The curriculum package includes a digital platform for students and teachers along with physical materials for hands-on activities. Students interact with physical materials and within a digital workspace with access to custom-written science articles, science simulations, and design tools. The lessons follow an instructional sequence meant to build students’ proficiencies with NGSS performance expectations over time. Teachers are provided with digital instructional guides and online monitoring and reporting tools that allow them to view summaries of student progress.
Why Did You Decide to Study the Impact of this Innovation?: ASMS is among the first generation of curriculum materials expressly designed for the NGSS and with an eye toward meeting today’s vision for science education. This vision emphasizes that all students must have the opportunity to learn and actively participate in science through using and applying disciplinary core ideas in concert with science and engineering practices and crosscutting concepts to make sense of phenomena or to solve problems. Along with ASMS, other new NGSS-designed materials have recently been made available to support science educators in realizing the vision. As all of these materials become more widely used across different geographic regions and with varying student populations, it will be important to conduct evidence-based research on their efficacy. This research is greatly needed for us to understand the promise of new curriculum materials for shifting classroom practice toward the vision of the Framework and the NGSS and ensuring that the vision is realized for all students.
How Are You Assessing the Impact of this Work?: We conducted a multisite cluster randomized controlled trial to examine the impact of the materials in 7th grade classrooms across three public school districts in two states. We randomly assigned 29 middle schools within districts to either a treatment condition with an immediate use of the NGSS-designed curriculum or to a control condition with the use of the district’s business-as-usual curriculum. Teachers in both groups targeted the same NGSS performance expectations in their science instruction.
Our measures included two assessments of student learning and teacher-self reports of classroom instruction and curriculum enactment. The assessments required students to use and apply their knowledge related to performance expectations. The implementation and instructional practice measures were in the form of weekly online instructional logs and teacher surveys. Weekly log items addressed, for example, coverage of NGSS science and engineering practices, coverage of crosscutting concepts, and prevalence of instructional opportunities among other implementation topics.
The study was run during the 2019-2020 school year and impacted by the COVID-19 pandemic. Please see below to learn how our study team made adjustments to overcome the challenges and conduct the study.
When Do You Expect to Have Results?: We have results! Our main analyses compared physical science learning outcomes for students in the treatment condition and the control condition. Results from the analysis of the estimated impact of assignment to the ASMS curriculum show that the curriculum had a significant positive impact on student learning. The estimated impact was statistically significant (p < 0.001) and corresponds to an effect size of 0.36 (Hedges’ g). This effect size is equivalent to the average student in the treatment schools improving 14 percentile points (moving from 50th to 64th percentile) relative to the average student in the control schools. Additional analyses focused on classroom instruction and curriculum enactment where we used data collected from weekly instructional logs to understand the nature of the science instruction provided by teachers in both conditions during the teaching of physical science. Results from these analyses highlight differences in teachers’ self-reported science instruction between conditions.
To learn more about our results, please refer to the dissemination activities that we describe under Product(s).
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: The data collection for the study was impacted by the school closures related to the COVID-19 pandemic. We had a successful launch of the study within all the participating districts but had to close our classroom-based research activities when the pandemic struck. Our study plans had originally included examining two science topic areas: physical science and life science. Just as schools were closing, we were able to collect learning outcome data in physical science along with implementation data from a subset of treatment and control schools. Unfortunately, we had school sites where teachers had completed their instructional sequence but were unable to administer the physical science assessment due to the shutdown. Also, unfortunately, we were not able to continue with the remaining portion of the study that focused on life science. In addition, we faced challenges in relation to work office closures which delayed access for scoring and analysis activities. In the end, we had outcome data from 15 of the participating 29 middle schools. The 15 schools represented all schools from 7 complete randomization blocks where 7th grade science teachers fully completed their physical science lessons and administered the study’s assessment prior to school closures.
Product(s): Our dissemination activities are currently underway. In 2022, we released initial results via the report, Curriculum materials designed for the Next Generation Science Standards show promise: Initial results from a randomized controlled trial in middle schools. In 2023, we have a full description of the study described in a recently submitted manuscript, Curriculum materials designed to support implementation of the Next Generation Science Standards: Results from a randomized trial in middle schools. In between, we have had posters and presentations at AERA, NARST, and SREE.
- Harris, C. J., Feng, M., Murphy, R., & Rutstein, D. (2022). Curriculum materials designed for the Next Generation Science Standards show promise: Initial results from a randomized controlled trial in middle schools. San Francisco, CA: WestEd.
- Murphy, R. F., Harris, C. J., Feng, M., Iveland, A., Rego, M., Rutstein, D., & Huang, K. (under review). Curriculum materials designed to support implementation of the Next Generation Science Standards: Results from a randomized trial in middle schools.
An Efficacy Study of the Learning and Teaching Geometry Professional Development Materials: Examining Impact and Context-Based Adaptations
PI: Jennifer Jacobs
Grade Level(s): 6-12
STEM Discipline(s): mathematics (geometry)
About the Innovation Being Studied: The project, An Efficacy Study of the Learning and Teaching Geometry PD Materials: Examining Impact and Context-Based Adaptations (LTG PD Efficacy Study, NSF Award #1503399) investigated the Learning and Teaching Geometry Professional Development (LTG PD) program’s impact on teacher learning, classroom practice and student achievement. The LTG PD program is a video-based mathematics professional development program for middle and high school teachers, focused on improving teaching and learning of similarity based on geometric transformations. The program is designed to be implemented by a knowledgeable facilitator, using a set of provided resources to engage teachers in a specified learning trajectory.
The LTG PD Efficacy Study examined the effectiveness of the LTG PD program using a group-randomized experimental design. Participants were 103 mathematics teachers serving grades 6-12 (47% middle school, 53% high school) from two diverse geographic locations. The most robust and pervasive impacts of the LTG PD were on classroom practice with instructional improvements seen across mathematics content areas, meaning that the program can support teachers’ ability to make widespread changes in practice across grades and topics. Moreover, teachers’ sustained their implementation of these improved practices for a year during which they received no further PD as part of this study.
Why Did You Decide to Study the Impact of this Innovation?: The LTG PD model incorporates a number of underlying design features that research suggests are critical to support mathematics teachers’ professional learning. These design features include foregrounding video, having well-specified goals and resources, building on a mathematical learning trajectory, ensuring applicability across school and district contexts, and supporting learning for diverse teacher populations. In particular the use of videocases with specified activities, extensive visual resources and discussion questions may have been critical to organizing productive, iterative sequences for the teachers to engage in. An important takeaway for educators may be the intentional use of this sequence in ongoing professional learning efforts, particularly if the primary goal is improving classroom practice.
How Are You Assessing the Impact of this Work?: We used a variety of approaches and methods to assess the impact of the LTG PD program, ranging from in-depth qualitative analyses of the professional development model and teachers’ uptake from the workshops to quantitative analyses of knowledge assessments, surveys and classroom video data. One approach to highlight has been our use of the Math in Common teacher observation protocol (Perry et al., 2015). We selected this protocol because of its potential to capture instructional practices intentionally targeted by the LTG PD. The protocol includes three dimensions: teacher work to support the richness of the mathematics, engaging students in mathematical practices, and actionable mathematics for students. We found significant effects for the participating teachers, from baseline to one year after the PD program, on all three of these dimensions.
When Do You Expect to Have Results?: Funding for this project has ended and we have published the results in a variety of outlets, which are described in more detail below. We do have a few more articles in progress and plan to continue our efforts to disseminate knowledge gained from this study.
Product(s):
- Basu, M., Koellner, K., Jacobs, J. & Seago, N. (2022). Understanding similarity through dilations of non-standard shapes. Mathematics Teacher: Learning and Teaching PreK-12, 115(9), 642-649.
This practitioner-focused article describes a sequence of mathematics tasks used in the LTG PD that progressively engage students in geometric reasoning. - Koellner, K., Seago, N. & Jacobs, J. (2022). Instructional change after participating in a mathematics professional development: An exploration of impact. The Teacher Educator. https://doi.org/10.1080/08878730.2022.2067277
This journal article examines the degree and nature of teachers’ pedagogical shifts after taking part in the LTG PD, finding that they made significant gains relative to a control group. - Jacobs, J.K., Koellner, K., Seago, N., Garnier, H. & Wang, C. (2020). Professional development to support the learning and teaching of geometry: Examining the impact on teacher knowledge, instructional practice and student learning in two contexts. In P. Jenlink (Ed.), The Language of mathematics: How the teacher’s knowledge of mathematics affects instruction (pp.143-173). Lanham, MD: Rowman and Littlefield.
This book chapter describes the impact of the LTG PD program on teachers’ knowledge, classroom practices, and students’ learning, as well as differences across the two geographic locations where the PD was held. - Koellner, K., Seago, N. & Jacobs, J. (2018). Representations of practice to support teacher instruction: Video case mathematics professional development. In O. Buchbinder & S. Kuntze (Eds.), Mathematics Teachers Engaging with Representations of Practice (pp. 9-22). ICME-13 Monographs. Springer International Publishing, Switzerland.
This monograph classifies teachers into four different user categories based on how they carried their experiences as participants in the LTG PD program back into their mathematics classrooms. - Seago, N., Koellner, K. & Jacobs, J. (2018). Video in the middle: Purposeful design of video-based mathematics professional development. Contemporary Issues in Technology and Teacher Education, 18(1), 29-49.
This journal article articulates a central design feature of the LTG PD materials, in which video is sandwiched between pre- and post-viewing activities to comprise a videocase. - Jacobs, J., Seago, N. & Koellner, K. (2017). Preparing facilitators to use and adapt professional development materials productively. International Journal of STEM Education, 4(1), 30.
This journal article describes the preparation process of a facilitator as she learned to use the LTG PD materials with fidelity and to make productive adaptations for particular audiences. - Seago, N., Jacobs, J., Driscoll, M., Callahan, P., Matassa, M. & Nikula, J. (2017). Learning and teaching geometry: Video cases for mathematics professional development, grades 5-10. San Francisco, CA: WestEd. https://www.wested.org/resources/learning-teaching-geometry-foundation-grades-6-12/
This resource kit for facilitators of the LTG PD program includes the Foundation Module (supporting 30 hours of professional learning), a facilitator’s guide, and all of the materials needed to conduct the workshops.
Anchoring High School Students in Real-life Issues that Integrate STEM Content and Literacy
PI: Amy Lannin
Grade Level(s): 5-12
STEM Discipline(s): science, mathematics
About the Innovation Being Studied: The STEM Literacy Project’s professional development model was originally developed through an Improving Teacher Quality Grant and revised through DRK-12 funding. The overarching goals of the DRK-12 project are 1) the creation of a community of practice that recognizes participating teachers as content experts, 2) the implementation of high-quality professional development for teachers, so as to experience STEM/Literacy integration, 3) the development of scenario-based assessments based on STEM and Literacy standards that inform instruction, and 4) the undertaking of rigorous research to understand the impact of professional development on teacher and student learning.
Why Did You Decide to Study the Impact of this Innovation?: This Impact Study (Learning Strand) presents evidence of the STEM Literacy Project’s professional development model that supports teachers’ integration of literacy practices to improve student learning. The effectiveness is based in the normal context of the teacher participants’ classroom and teaching. In addition, the use of Scenario-based Assessments (SBAs) aligns with the types of assessments used for statewide tests and that include reading and writing tasks within a problem-based/issue-based scenario.
Educators recognize a need for rich content and literacy integration (Moje et al., 2004). STEM and literacy connections are complex and often isolated. By blending content areas and high impact literacy practices for the tasks of reading, interpreting, and writing scientifically, students can gain in understanding how STEM learning applies to real-world problems (Zeidler & Sadler, 2008). The outcomes are measured through scenario-based assessments designed to integrate STEM and literacy practices.
Data and Findings:
- Surveys: Anonymous teacher survey data has been collected following each PD session (n=14) from 2020-2022. Evaluations provided positive feedback particularly in the teachers’ appreciation of opportunities to collaborate (forming a community of practice) and of the provision of literacy strategies that they could introduce in their own classrooms.
- Scenario-based Assessments - Year 1: We collected and scored scenario-based assessments from approximately 239 students across 12 teachers at a single time point. The purpose of this initial data collection provided measurement validity of the instruments we developed and to inspect student responses to generate a descriptive understanding of what the students were able to do without any intervention.
- Scenario-based Assessments (SBA) - Year 2: We collected and scored SBAs from 771 students across 14 teachers.
- Interviews: We interviewed five teachers and five students in Year 1 and five teachers and eight students in Year 2. All interviews have been transcribed and are currently being coded. The interview analysis has helped us understand how teachers have applied PD strategies into their classes. Student transcripts showed examples of how students approached reading complex texts, and how they integrated literacy strategies to support learning. Codes that were used included Integration of literacy practices, Relevance to students/real-world connections; Impact on student experience (engagement, motivation, stamina, and confidence); and Assessment of argumentation practices. Preliminary findings from the first year’s interviews included the following:
- Connecting to the Standards, Assessment, and Argumentation to Support Student Learning: Teachers found that SBAs provided valid assessments that connect to the standards: “The scenario-based assessments really play right into those standards to get the kids to be problem-solvers, to come up with solutions and to write it down, to be able to communicate that effectively to other people” (Teacher interview).
- Collaboratively assessing helped teachers make instructional decisions: Teachers engaged in collaborative assessment during the PD and during scoring conferences in summer 2021 and summer 2022 to review student SBA written responses. They found that when seeing student work across grade levels and from other classrooms, it helped them understand their own students’ work. By engaging in this assessment together, they were also able to share ways to adapt instruction. For instance, one teacher noticed that his students struggled with interpreting graphs and charts. He then planned lessons to help students read different representations of data.
- The SBAs connected argumentation and Cross-cutting Concepts: Students were asked to make a claim, use evidence, and connect the evidence to the claim through logical reasoning. Several interviewees noted how the SBA helped students identify a claim in the text and support it with evidence. One teacher explained that the SBA allowed her “to connect math to other classes and give students more of a voice to find the math and put it into real-world context.” After having students complete SBAs for a few years, one of the teachers described how her assessments moved from multiple-choice questions and few fill-in-the-blank ones, to instead have students do more writing as part of their assessment, especially to work on reasoning.
How Are You Assessing the Impact of this Work?:
Impact of Professional Development towards Teachers’ Community of Practice and Implementation of Literacy Practices
- Survey responses provided at the end of each PD session: Teachers consistently gave high ratings about their intention to incorporate strategies and resources offered in the PD and of the overall quality of the session.
- Resource development: We currently have eight SBAs developed, an analytic rubric, and an assessment system for involving teachers and other program leaders in the scoring of student writing.
- Literacy Strategies developed and implemented by teachers and program leaders: The program has curated numerous literacy strategies that teachers have experienced in the PD and used in their classrooms.
Impact of SBA Analysis on Methodological Approaches: In our assessment model, a claim-evidence-reasoning score was derived from the observation of six claim-evidence-reasoning competencies (writing, discipline-specific content, making a claim, providing evidence, using reasoning, and a holistic score) were measured for each student in the context of a specific socio-scientific issue of the teacher’s choosing (of 8 total available). Scores on each competency were given by one of 8 trained scorers, and 10% of the responses were double scored. We took a scoring approach which treated the 8-rater scoring team as a group of independent experts, and so the focus was on consistency as opposed to perfect agreement (i.e. we wanted to allow scorers to vary in difficulty, but asked that they be consistently easy or difficult across all of the SBA responses). Hence we used the 4-faceted Rasch model as a criterion for validity, where the facets were student, CER competency, scorer, and type of SSI. Rater consistency was inspected through fit of the rater’s scoring pattern with the model-expected scoring pattern and comparison of model-expected agreement with the agreement observed in the data.
Impact on Student Learning and Argumentation as Measured on SBA Scores: Through a mixed effects linear modeling approach (Teacher, Testing, and Teacher x Testing as fixed effects and Intercept as a random effect), we found that students made significant gains in their CER ability (F(1, 312.8) = 6.27, p = 0.013)). The students started with a mean CER of -0.53 logits (SD = 3.24) and ended up with a mean CER of 0.55 logits (SD = 3.55). The effect size (Cohen’s D = 0.32) suggests that the teacher’s implementation of a literacy-based science curriculum focused on argumentation around SSI resulted in a significant improvement in their ability to use Claim-Evidence-Reasoning.
When Do You Expect to Have Results?: By Fall 2023 all data analysis and follow-up will be complete.
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: Due to COVID restrictions when we started, the professional learning sessions were held virtually through zoom. To prevent zoom burn-out, we planned for monthly one-hour sessions on Wednesdays at 4:00. When restrictions were lessened, we held some full day in-person sessions. Because the teachers did like the regular quick sessions on zoom, during which we modeled and engaged in strategies that could be implemented immediately, we have continued with the monthly zoom meetings.
Product(s):
Website: http://www.stemliteracyproject.org/
Presentations:
- Apple, S., Swigert, M., & Cunningham, H. [3 of the STEM Literacy Project teachers.] (2020-2021; 2021-2022). Presented at the Content Literacy sessions for statewide professional development. Missouri Department of Elementary and Secondary Education.
- Harper, N & Taylor, B. (2022). The Next Chapter: Storylines and SBAs. 2022 Interface conference for Missouri Math and Science teachers.
- Johnson, T. (a teacher of the STEM Literacy Project) (2021). Selected as a teacher leader in the Place-based Institute, Walt Disney Hometown Museum, Marceline, MO. This is a multi-disciplinary year-long institute to develop and implement place-based instruction.
- Kendrick, M. (2022). Engineering in your Place. 2022 Interface conference.
- Lannin, A. & Kareem, M. (November, 2022). Issue-based argumentation to support literacy and learning in STEM. Presentation at Science Teachers of Missouri Conference (STOM). Columbia, MO.
- Romine, W., Agarwal, A., Burwell, E., Kareem, M., Lannin, A. (April, 2023). Measuring Claim-Evidence-Reasoning Using Scenario-based Assessments Grounded in Real-world Issues. Accepted to the 2023 NARST Annual Conference, Chicago.
- Romine, W., Lannin, A., Agarwal, A., Kareem, M., & Burwell, E. (June, 2021). How Do Students Express Their Claim-Evidence-Reasoning on Scenario-based Assessments Grounded in Real World Issues? Poster Presentation at the Discovery Research K12 Principal Investigator Meeting. National Science Foundation, Washington, DC.
Publications:
- Otten, S., Bemke, J., & Webb, J. (In press). Exploring the viral spread of disease and disinformation. In S. R. Harper & D. C. Cox (Eds.), Math in Action: Grades 9-12 (Chapter 2). National Council of Teachers of Mathematics.
- Romine, W., Lannin, A., Kareem, J., & Singer, N. Using Multi-faceted Rasch Models to Understand Middle School Students’ Use of Claim-Evidence-Reasoning around Scenarios Grounded in Socio-scientific Issues. (In Press). Advances in Applications of Rasch Measurement in Science Education.
- Evaluation Report: A team from the Assessment Resource Center at the University of Missouri has been conducting the program evaluation. This report is organized in response to the evaluation questions aligned to four program goals.
Developing Leaders, Transforming Practice in K-5 Mathematics: An Examination of Models for Elementary Mathematics Specialists
PIs: Chandra Lewis, Patrice Woods, Nicole Rigelman
Grade Level(s): K-5
STEM Discipline(s): mathematics
About the Innovation Being Studied: The Developing Leaders, Transforming Practice (DLTP) is a partnership between Portland Public Schools (PPS), Portland State University (PSU), and RMC Research Corporation (RMC). DLTP is a K-5 mathematics intervention focused on demonstrating the effectiveness of the DLTP professional development (PD) model in terms of improving teachers’ instructional practices (Goal 1), increasing student mathematics understanding and achievement (Goal 2), and evaluating the efficacy and use of Elementary Mathematics Specialists (EMSs) by testing four implementation models that take into account the various ways EMSs are integrated into schools (Goal 3).
Why Did You Decide to Study the Impact of this Innovation?: The DLTP partnership uses a research-proven model for professional learning of EMSs (Rigelman & Lewis, 2022) and is studying the impact of four different implementation models on student learning and teacher learning and practice. The implementation models leverage EMSs as elementary mathematics teachers (i.e., teaching mathematics to multiple groups of students), grade level mathematics teacher leaders, and mathematics coaches. We seek to understand the relative effectiveness of the EMS models and in which contexts.
How Are You Assessing the Impact of this Work?: This project addresses four research questions:
- What is the effect of the intervention on teacher leadership?;
- What is the effect of the intervention on teachers’ use of research-based instructional practices?;
- What is the effect of the intervention on a school’s ability to sustain ongoing professional learning for teachers?; and
- What is the effect of the intervention on student mathematics achievement?
The Research Team is conducting four major studies that include rigorous quasi-experimental designs and a multimethod approach to address the research questions. The instruments include:
- project developed surveys measuring teacher leadership and instructional practice as well as perceptions about school climate;
- an adaptation of the Center for Strengthening the Teaching Profession teacher leader rubrics;
- Instructional Quality Assessment rubrics for classroom observations and artifacts;
- Learning Mathematics for Teaching mathematics content and pedagogical knowledge assessment; and
- district- and state-level assessments of student mathematical knowledge and skills.
When Do You Expect to Have Results?: Fall 2024
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: The COVID-19 pandemic caused many shifts in our planned professional learning (i.e., EMS coursework, book studies, lesson studies, project wide events) and innovation implementation. The pandemic also impacted teacher participation. We launched the grant in January 2020 and schools closed their doors to face-to-face learning mid-March. Teachers and grant faculty alike had to divert from what we have done in the past to facilitating student and teacher learning virtually. This format was unfamiliar, both adding pressures and providing opportunities for growth.
Recognizing the numerous challenges facing teachers, administrators, district leaders, and grant faculty, we made adjustments to the planned professional learning. Some learning took place at a slower pace affecting the duration of the grant, now planned for five years. Other aspects, such as the lesson study cycles, had to be rethought due to substitute teacher shortages. Our hope, now in year 4 of the grant, is to engage teachers in collaborative planning cycles, where teachers will have the opportunity to use the Thinking Through a Lesson Protocol (Stein & Smith, 2011) to plan lessons from their curriculum materials.
Originally several of the research designs included a preassessment in the first year and post assessment in the final year. Due to the pandemic, the length of the grant increased which resulted in a significant amount of time between the pre- and post-assessment. This increase in time could be problematic in terms of obtaining a post assessment because of teacher staffing changes and attrition. To account for this change in timing, the research team shifted to annual data collection efforts.
Product(s):
Website: https://sites.google.com/pdx.edu/dltp
Networking Urban Resources with Teachers and University to enRich Early Childhood Science (NURTURES) Phase II: Expansion and Evaluation
PI: Charlene Czerniak
Grade Level(s): PreK-3
STEM Discipline(s): science, technology, engineering, mathematics
About the Innovation Being Studied: NURTURES was established with funding from the National Science Foundation starting in 2011 (Phase 1: grant #1102808, 2011-2017; Phase 2: grant #1721059, 2018-2020). Current programming is supported by the Department of Defense NDEP-STEM, award #HQ00342010035 (Phase 3: 2021-2023).
Building on successful prior work, University of Toledo's NURTURES project researches the impact of teacher professional development and family engagement on PreK-3 achievement, simultaneously targets young children's teachers and families/caregivers in an effort to build both parties' capacity to promote student interest in science, technology, engineering and mathematics (STEM) learning. Teachers participate in a two-week summer professional development program and receive support across the school year in the form of individualized coaching and participation in professional learning communities. Families receive science inquiry packets (sent home from school) four times a year and attend community STEM events throughout the year. Inquiry packets and community events encourage science inquiry, discourse, and further exploration of key science ideas.
Additional work is underway to make NURTURES teacher professional development and family engagement components available to the general public, school districts, and families across the nation.
Why Did You Decide to Study the Impact of this Innovation?: The NURTURES program differs dramatically from other teacher PD programs because it creates a strong partnership among educators, parents, and communities. Most PD programs focus only on teachers, which we believe is important but insufficient because PD-only programs ignore the role of families and community organizations in developing children’s interests and abilities in STEM. Further, many STEM education programs do not engage families beyond “sign-up,” “pay for,” “drop-off” or “stay and watch” experiences. NURTURES is markedly different from most outreach programs because by educating parents how to support their child’s STEM learning, we provide a lasting mechanism to support STEM learning across a child’s lifetime.
Findings from the program include:
- A longitudinal case study revealed improved pedagogical practices among teachers, increased science content knowledge and confidence to teach science, and increased use of technology in the classroom.
- With baseline equivalence established, children participating in NURTURES showed gains on STAR Assessments: 11.24 points on Early Literacy spring score (effect size 0.09), 21.75 points on Mathematics spring score (effect size 0.18), and 47.85 points on Reading spring score (effect size of 0.29) over those who did not participate,
- Standardized science tests are not given in grades K-4 in the schools studied, so a longitudinal study to follow students through grade 5 (when science testing is administered) was conducted. With baseline equivalence established, the longitudinal study found students’ science, reading, and mathematics scores continued to show greater gains for students who participated in NURTURES over those who did not. NURTURES students were found to have 5.86 more points in science, 1.55 more points in mathematics, and 2.14 more points in reading,
- Achievement gaps between non-minority and minority students in reading and mathematics were reduced for minority students participating NURTURES. In science, the intervention roughly compensated for the attainment gap between boys and girls and partially ameliorated the gap between minority and non-minority children’s scores, and
- Under a Department of Defense project, student achievement for children in NURTURES was significantly higher (Hedges' g of 0.52.) than students’ scores nationally on the Galileo tests.
References for Findings:
- Demir, K., Paprzycki, P., Czerniak, C., Kaderavek, J., Hapgood, S., & Molitor, S. (2023, January 11-14). A preliminary study of the impact of a science program on student achievement in early grades [Paper presentation]. Association for Science Teacher Education, Salt Lake City, UT. http://theaste.org/wp-content/uploads/2022/01/2023-ASTE-Conference-Program_12-23-22_2.pdf
- Hapgood, S., Michaelson, M.K., Kaderavek, J.N., Paprzycki, P., Czerniak, C. M., & Molitor, S. (Submitted). Longitudinal impact of an NGSS-aligned program on early literacy and mathematics growth curves. International Journal of Science Education.
- Heuring, J., Paprzycki, P, Gilbert, A., & Czerniak, C. M. (2020). Longitudinal impact of early childhood science instruction on literacy and mathematics. Manuscript under preparation.
- Kaderavek, J. N., Paprzycki, P., Czerniak, C. M., Hapgood, S., Mentzer, G., Molitor, S. & Mendenhall, R. (2020). Longitudinal impact of early childhood science instruction on 5th grade science achievement. International Journal of Science Education, 1-20. DOI: 10.1080/09500693.2020.1749908
- Mentzer, G.A., & Paprzycki, P. (2020). NURTURES: Longitudinal Summary of Project Impact on Students’ Mathematics, Reading, & Science Learning. Retrieved at http://nurtures.utoledo.edu/reports/NURTURES_LongitudinalStudy.pdf
- Tuttle, N., Gilbert, A., & Hapgood, S. (2016). Family practices around science at community Sci-FUN events. A paper presented at the American Educational Research Association Annual Conference, Washington, DC, April 2016.
- Paprzycki, P., Tuttle, N., Czerniak, C. M., Molitor, S., Kadervaek, J. and Mendenhall, R. (2017). The impact of a Framework-aligned science professional development program on literacy and mathematics achievement of K-3 students. Journal of Research in Science Teaching, 54(9), 1174-1196.
- Reinhart, M. A. (2019). A Longitudinal Study of an Urban Kindergarten Teacher’s Instructional Strategies for and Perceptions of Young Children’s STEM Inquiry. Unpublished doctoral dissertation, University of Toledo, Toledo, OH.
How Are You Assessing the Impact of this Work?: This project, NURTURES Phase 2, used a randomized control group research design (RCT) to compare student achievement outcomes among three groups: Children whose teachers received professional development and family engagement activities, children whose teachers received only professional development, and a control group.
Further research on impacts continues with the DoD funded NURTURES Phase 3, focusing on the delivery of program elements, examining methods of program delivery, multi-site facilitation, impacts on student learning, and the feasibility of offering NURTURES at scale.
When Do You Expect to Have Results?: Current research, funded through the DoD NDEP STEM program, is focused on the delivery of program elements, examining methods of program delivery, multi-site facilitation, impacts on student learning, and the feasibility of offering NURTURES at scale.
After the first year of implementation, our research findings on the impact of the NURTURES program on grades 1-3 student learning in science and mathematics are promising. The results from Galileo testing demonstrates that students whose teachers participated in the NURTURES program exhibited dramatic gains in mathematics and science scores and outperformed the national cohorts on these assessments, which are aligned with Common Core State Standards (CCSS) and Next Generation Science Standards (NGSS).
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: The COVID-19 pandemic most significantly impacted NURTURES Phase 2 in the winter/spring of 2020. The closing of schools, need to convert from face-to-face to online formats, and the extraordinary impact on our teacher participants, their students, and families had a significant impact on our ability to engage teachers, students, and families with the program content and greatly limited the ability to collect vital research data. Efforts were made to offer online family engagement elements and online assessments, however there was a sizeable loss of data during the final program months of the NURTURES Phase 2 project.
The impacts of COVID related school conditions (e.g., remote learning, limited access to school buildings, local health & safety rules) continued into NURTURE Phase 3, resulting in significant changes to program delivery and data collection. Programming was re-designed for online delivery with synchronous and asynchronous components. Teacher professional development programming offerings were expanded to include both a summer and fall Institute and assessments were adjusted to accommodate local conditions and facilitate quality data collection.
Product(s):
Websites:
Videos:
- NURTURING STEM (video) – University of Toledo, 2022.
- NURTURES: Engaging Teachers, Children & Families (video) – STEM FOR ALL Video Showcase, 2018.
- Family Science videos: http://nurtures.utoledo.edu/families.html
Publications:
- See the program website for full listing of NURTURES publications and presentations.
PBS NewsHour Student Reporting Labs StoryMaker: STEM-Integrated Student Journalism
PI: Leah Clapman
Grade Level(s): 6-12
STEM Discipline(s): All possible disciplines, examples include: climate change, neuroscience, and health
About the Innovation Being Studied: The STEM-Integrated Student Journalism project (#1908515) created StoryMaker, a learning platform for educators to lead in-demand interdisciplinary, multimodal, STEM-infused storytelling activities. PBS NewsHour Student Reporting Labs program went from serving 160 classrooms to over 3,000 sites registered in StoryMaker. The project explored evidence-based strategies for structuring co-learning and mentorship connections to best support student and teacher outcomes. Stories produced by students through the project include:
- The ON OUR MINDS podcast on youth mental health
- SRL Citizen Science Events
- How to identify and debunk science misinformation via MediaWise Toolkit
- Student-centered reporting on issues such as
Why Did You Decide to Study the Impact of this Innovation?: The original goal of this study was to study the impact of teacher professional development. Specifically, we had planned to implement this program with STEM and journalism teachers to answer the following research questions:
- What teacher affordances are necessary for using journalism practices to support STEM learning across academic disciplines?
- How do teacher perceptions of their school constraints influence their use of STEM-based learning activities? (How do teachers from different disciplines teach numerical reasoning, communicating with data, and the other essential STEM thinking skills?)
- How might an online support community be structured to encourage teacher-to-teacher scaffolding related to STEM content given variation in their pedagogical training?
Given changes to project design and to the timeline of the platform development, as well as effects of the pandemic and remote school, we focused most critically on the first question.
How Are You Assessing the Impact of this Work?: Despite the ubiquity of talk about "multi-disciplinary" (transdisciplinary, etc.) education, most teachers working today are trained within a single discipline. That training remains a formidable challenge to implementing cross-curricular projects and programs. By walking through example program resources with teachers from different curricular backgrounds (STEM and journalism), we've been able to gain a stronger understanding of the program's potential as well as some of the structural obstacles to implementation in a range of different contexts at a wide range of grade levels. This understanding will inform the continued development of the StoryMaker platform and the instructional design of support materials for teachers to implement cross-curricular STEM storytelling and students to report on topics that include scientific knowledge and data analysis.
While this qualitative and somewhat exploratory approach may seem prior to a true impact study, we have found that large-scale changes to the educational context (see final question) have made this approach critical at this point in time.
When Do You Expect to Have Results?: We have completed all of the qualitative thinkaloud interviews (N = 5 STEM, 10 journalism) and are in the process of analyzing them. We anticipate submitting results to peer review before Spring.
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: As of this writing [early 2023], students around the US are back in the physical classroom. At a surface level, the classroom looks much as it did before the COVID-19 pandemic. However, there are deeper changes beyond remote and hybrid education that have caused us to radically rethink our research approach:
- Many districts have retained a "catching up" mentality that has made them cautious about innovation. In some cases, this has meant cutting electives and extracurriculars entirely. But we have heard even from the best-resourced schools that there is concern about spending classroom time on activities that don't speak directly to the core curriculum, ESSA and NCLB accountability.
- Student Reporting Labs overcame this challenge by using the StoryMaker platform and social media to engage with students directly.
- Teachers and students are both experiencing considerable burnout and other mental health consequences of the ongoing pandemic. Whether the cause of these is large-scale death and illness, isolation, or a combination thereof (all of which have been claimed), the impacts still need to be taken seriously.
Journalism prompts that addressed the challenges posed pandemic and the effects on the lives of students were the most successful. Remote interviews with experts replaced traditional in-person reporting activities.
Product(s):
Website: StoryMaker Getting Started page, with a pathway that leads to Build STEM and Science Journalism Skills
STEM journalism instructional resources:
- Engineering: Our People, Our Planet - How We Shape Our World
- What does climate change look like in your community?
- Invention and Design Stories That Inspire Us
- Youth Climate Leaders
- Evidence & Experts
- Research and Understanding
Supporting Success in Algebra: A Study of the Implementation of Transition to Algebra
PI: Deborah Spencer
Grade Level(s): 9
STEM Discipline(s): mathematics
About the Innovation Being Studied: Our research team at Education Development Center (EDC) and partners at Boston College are studying the implementation and impact of Transition to Algebra (TTA), a coherent algebra support curriculum designed for courses run concurrently with first-year algebra. Rather than re-teaching or pre-teaching algebra content, TTA focuses on building students’ algebraic habits of mind, key ways of thinking mathematicians use in their work that are critical in supporting the transition from arithmetic to algebra. TTA was developed at EDC with support from NSF (grant # 0917958) and is published by Heinemann and distributed by Houghton Mifflin Harcourt.
The TTA materials include Teaching Guides and Student Worktexts for 12 units that cover standard algebra topics through three pervasive mathematical themes: Expecting mathematics to make sense; Using intuitive visual models to support number and algebra; and Building broad-use algebraic habits of mind and problem-solving stamina. The TTA curriculum employs several distinctive features: daily Mental Mathematics activities develop fluency with algebraic properties of operations; Puzzles build logic and increase perseverance; Explorations provide opportunities for sustained reasoning; and Student Dialogues model mathematical discourse and develop academic language. Designed for use in full-year algebra support courses, TTA represents a novel approach to algebra support. The typical emphasis of such courses is on remediation, review, and test preparation, and the materials used are often teacher-developed or pulled from online sources, which can limit coherence (Mark, et al., 2012).
Why Did You Decide to Study the Impact of this Innovation?: Passing Algebra 1 is an important milestone on the path to college and career readiness. Students who succeed in algebra are more likely to graduate from high school on time and gain access to advanced mathematics courses that are key for career and college readiness. Unfortunately, a large number of students enter high school underprepared for Algebra 1 and pandemic-related learning loss has only exacerbated this persistent challenge. Algebra support courses are a common intervention strategy used in high schools nationwide. These yearlong courses provide additional instructional time during the school day for students whose mathematical learning needs have not been met. Studies have shown that the extra time, when paired with strong curricula and teacher supports, hold promise for improving student outcomes. However, more research and guidance is needed to understand which curricular approaches may be effective and to help districts design and implement the courses effectively.
How Are You Assessing the Impact of this Work?: We are using a pre-post quasi-experimental research design with propensity score analyses to reduce selection bias threats to examine the effects of TTA on algebra achievement and attitudes towards mathematics, compared with outcomes for similar ninth graders receiving business-as-usual instruction. Our research sample includes 54 treatment schools and 36 comparison schools in 17 school districts across the country that offered a full-year algebra support course in the 2018–2019 school year. In addition, using mixed methods approaches, we are drawing on teacher and administrator interviews and observations conducted in 30 treatment schools to investigate how teachers use and adapt TTA, the relationships between TTA use and student outcomes, and the features of algebra support course models and how they varied by district.
When Do You Expect to Have Results?: Initial results will be included in two papers submitted to peer-reviewed journals, and a paper accepted for the 2023 AERA Annual Meeting. We anticipate student outcome results will be available later this year.
What Challenges, If Any, Did You Face in Conducting Your Study Due to Factors Associated with the COVID-19 Pandemic, How Did You Attempt to Overcome Them, and What Impact Did They Have on the Research?: Fortunately, the primary data collection for our study was not affected by the pandemic, however the secondary data collection from school districts and data analysis was delayed due to the additional demands placed on school districts when schools shifted to remote learning.
Product(s):
Transition to Algebra curriculum and teacher resources:
- Transition to Algebra website (publisher): https://www.heinemann.com/products/ebk07743.aspx) also https://www.heinemann.com/transitiontoalgebra
- Making Sense of Algebra: Developing Students’ Mathematical Habits of Mind (publisher): https://www.heinemann.com/products/e05301.aspx
- Transition to Algebra website (EDC): http://ttalgebra.edc.org
Supporting Success in Algebra Study presentations:
- DRK–12 PI meeting virtual poster, 2021: https://cadrek12.org/posters/supporting-success-algebra-study-implementation-transition-algebra
- DRK–12 Webinar on adapting DRK-12 research amid the pandemic, 2020: https://cadrek12.org/resources/webinar-resources-adapting-your-drk-12-research-amid-covid-19-pandemic
Related Resources
- NSF DRK-12 Solicitation 20-572 | NSF
See page 7 for a description of DRK-12 Impact Study requirements. - Common Guidelines for Education Research and Development | IES and NSF (2013)
See page 9 for a description of the three types of Impact Research: efficacy, effectiveness, and scale-up, as well as project examples in the appendix of the guidelines.
See our Related Spotlights section for other Spotlights on project design and implementation topics.
Additional Projects
In addition to those featured above, the following projects provide a sample of DRK-12 impact studies:
2021 Awards
- Measuring the Effectiveness of Middle School STEM Innovation and Engineering Design Curricula (PI: Meltem Alemdar)
2020 Awards
- Comparing the Efficacy of Collaborative Professional Development Formats for Improving Student Outcomes of a Student-Teacher-Scientist Partnership Program (PI: Catrina Adams)
- Broadening Participation in Mathematics for English Learners with Mathematics Difficulties: A Multi-Site Impact Study (PI: Christian Doabler)
2019 Awards
- Learning Trajectories as a Complete Early Mathematics Intervention: Achieving Efficacies of Economies at Scale (PI: Douglas Clements)
2018 Awards
- Extending and Investigating the Impact of the High School Model-Based Educational Resource (Collaborative Research) (PIs: Cynthia Passmore, Chris Wilson)
- Investigating Impact of Different Types of Professional Development on What Aspects Mathematics Teachers Take Up And Use in Their Classroom (PI: Karen Koellner)
- LabVenture - Revealing Systemic Impacts of a 12-Year Statewide Science Field Trip Program (PI: Leigh Peake)
2017 Awards
- Algebra Project Mathematics Content and Pedagogy Initiative (PI: Oliver Hill)
2016 Awards
- Systemic Formative Assessment to Promote Mathematics Learning in Urban Elementary Schools (PI: Jonathan Supovitz)
- Science, Technology, Engineering and Mathematics Scholars Teacher Academy Resident System (PI: William McHenry)
RELATED SPOTLIGHTS
The following Spotlights highlight project design & implementation topics:
- Adapting Research Amid COVID-19 (2023)
In this Spotlight, four projects share the changes they made to their questions and/or collection, measurement, and analysis of data in response to the COVID-19 pandemic. - Conferences and Workshops Organized by DRK-12 Projects (2021)
This Spotlight highlights the work of DRK-12 project focused on organizing conferences or workshops to advance their work and the STEM education field. - DRK-12 CAREER Awards (2022)
Hear from 27 DRK-12 CAREER Awardees about their work, plus advice on developing a proposal and managing an NSF grant. - Exploratory Research (2022)
This Spotlight features descriptions of DRK-12 funded Exploratory projects and projects that have built on a previously funded Exploratory project, and a blog with more information about this project type, including how it is different from Early-Concept Grants for Exploratory Research (EAGER). - Replication & Building Evidence (2020)
This Spotlight features perspectives, research, and resources related to building evidence in STEM education, particularly by means of replication.