Synthesis

Validity Evidence for Measurement in Mathematics Education (V-M2ED) (Collaborative Research: Krupa)

The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education.

Partner Organization(s): 
Award Number: 
1920619
Funding Period: 
Thu, 08/01/2019 to Wed, 07/31/2024
Full Description: 

As education has shifted more towards data-driven policy and research initiatives in the last several decades, data for policy-related aspects are often expected to be more quantitative in nature.  This has led to the increase in use of more quantitative measures in STEM education, including mathematics education. Unfortunately, evidence regarding the validity and reliability of mathematics education measures is lacking. Furthermore, the evidence for validity for quantitative tools and measures is not conceptualized or defined consistently by researchers in the field. The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education. Drawing on the results of the synthesis study, the researchers will design, curate, and disseminate a repository of quantitative assessments used in mathematics education teaching and research. The researchers will also create materials and online training for a variety of scholars and practitioners to use the repository.

The team will address two main research questions: 1) How might validity evidence related to quantitative assessments used in mathematics education research be categorized and described? and 2) What validity evidence exists for quantitative instruments used in mathematics education scholarship since 2000? Researchers will use a cross-comparative methodology which involves conducting a literature search and then analyzing and categorizing features of instruments. The research team will examine cases (i.e., assessments described in manuscripts) in which quantitative instruments have been used, alongside specific features such as the construct measured, evidence related to sources of validity, and study sample. The team will then design, develop, and deploy a free online digital repository for the categorization of instruments and describe their associated validity evidence.

Teaching Students to Reason about Variation and Covariation in Data: What Do We Know and What Do We Need to Find Out?

The purpose of this project is to gather, analyze, and synthesize mathematics and science education research studies published from 1988 to the present that have investigated different approaches to supporting students in grades 6-14 in learning to analyze, interpret, and reason about data.

Lead Organization(s): 
Award Number: 
1920119
Funding Period: 
Mon, 07/01/2019 to Wed, 06/30/2021
Full Description: 

Because data are so much a part of modern life, making sense of data is a skill that benefits all members of society. Reasoning about data has been described as one of the most important cognitive activities and making sense of data is essential for a public's informed civic participation. But the public's ability to make sense of data is not what it should be. There is an important role for educators to play in supporting students' ability (and ultimately the public's ability) to be savvy consumers of data. But education researchers lack a coherent vision of the current best practices for supporting students in analyzing, interpreting, and reasoning about data. Existing research focused on supporting students in learning to analyze, interpret, and reason about data tends to reside in silos by grade band and by math or science domain. The purpose of this project is to gather, analyze, and synthesize mathematics and science education research studies published from 1988 to the present that have investigated different approaches to supporting students in grades 6-14 in learning to analyze, interpret, and reason about data. The researchers will carefully examine the nature of each education intervention and what the researchers found in each case, looking for patterns across studies. The findings of this study can inform mathematics and science education developers in the production of instructional programs for teachers and students.

The researchers will gather, analyze, and synthesize studies in mathematics and science education from 1988 to the present that examine instruction related to variation and covariation in data. The team will first conduct a descriptive synthesis including a wide array of studies (qualitative, single group pre/post, and experimental/quasi-experimental) and examine the nature of interventions in the field. Next, researchers will conduct a statistical meta-regression of experiments and quasi-experiments using Robust Variance Estimation (RVE) to examine how effect size estimates from primary studies depend on intervention characteristics, study design, outcomes of interest, and demographic characteristics of participants in the studies. The project will help researchers across math and science education build on each other's work and ultimately develop and refine highly effective approaches for supporting students in the life-long skill of making sense of data in a complex world.

Advancing Methods and Synthesizing Research in STEM Education

This project will address two critical opportunities to improve the translation and connection of innovations and evidence across federally funded STEM education projects. First, the project will aim to build capacity and learning opportunities for STEM education research and development. Second, the project will synthesize evidence of discovery and innovation across NSF-funded work.

Award Number: 
1813777
Funding Period: 
Sat, 09/01/2018 to Tue, 08/31/2021
Full Description: 

Rigorous research and development methods are essential for developing and testing new approaches in STEM education. This project will address two critical opportunities to improve the translation and connection of innovations and evidence across federally funded STEM education projects. First, the project will aim to build capacity and learning opportunities for STEM education research and development. Second, the project will synthesize evidence of discovery and innovation across NSF-funded work.

This project will conduct four research syntheses. Final topics will be determined but may include early math education, early learning in science, and engineering in the elementary grades. As part of this work, the team will produce a brief report for each topical synthesis, designed to highlight and elevate evidence and contributions across a set of projects, in a nontechnical format that combines graphics with text. Content will include a rationale for the topic and its importance in STEM education; synopsis of the investments in this area and the projects sampled; synopsis of the cross-project contributions, with illustrative highlights or examples from projects; commentary on the quality of evidence; and discussion of the contributions and potential broader impacts of the investments. The project will also plan and conduct a series of methods-focused webinars which may include topics such as rigorous quasi-experimental designs; measuring implementation fidelity and adaptation; applying improvement science methods; cluster randomized controlled trials; design-based research; or developing and testing valid measures. The materials from the webinars will be procured and made publicly available for the education research community.

Supporting English Learners in STEM Subjects

This project will conduct a study to identify instructional practices and professional development approaches for teachers and the policies needed to support ELLs' accomplishments in science and math. The study will synthesize research relevant to improving ELLs' STEM learning, offer insight into how to support both English language development and science and math learning, and provide a framework for future research to help identify the most relevant and pressing questions for the field.

Lead Organization(s): 
Award Number: 
1636544
Funding Period: 
Thu, 09/01/2016 to Thu, 02/28/2019
Full Description: 

The expectation that all students, including English language learners (ELLs), achieve high academic standards has become even more evident and complex to date as a result of several key factors. First, as the school-aged population continues to grow more racially, ethnically, and linguistically diverse, ELLs can now be found in virtually every school in the nation. Second, the science and mathematics education landscape has changed significantly resulting from the new visions in these fields, and the challenges posed by the new academic standards for all students. Third, the need to integrate new knowledge and perspectives from the language arts with knowledge from science and mathematics learning, instruction, and assessment has surfaced as a critical component of the potential strategies to be employed in addressing ELLs' current science, technology, engineering, and mathematics (STEM) education situation from pre-K-12 grades. The key challenges today include both enabling educators to better support this student subpopulation, as well as increasing the number and quality of research activities focused on how best to support ELLs' success in these subjects. In response to this challenge, the Board on Science Education (BOSE) of the National Academies of Sciences will conduct a consensus study focused on identifying instructional practices and professional development approaches for teachers, as well as the policies that are needed to support ELLs' accomplishments in science and mathematics education. The study will synthesize a wide range of research literatures relevant to improving ELLs' STEM learning, and provide a comprehensive understanding of how best to simultaneously support English language development and deep learning in the context of new and more challenging standards in science and mathematics. The study will also provide a framework for future research that can help to identify the most relevant and pressing questions for the field, as well as increase the number and quality of proposed research activities focused on ELLs in STEM.

To conduct the consensus study, BOSE will convene a multidisciplinary committee of experts who will synthesize the most relevant research on related subjects. The committee will include professionals in the fields of science and mathematics education, curriculum development, learning and instruction, linguistics, and assessment to address key sets of research questions: (1) Based on research-informed and field-tested models, strategies, and approaches, what are promising approaches to support ELLs (including ELLs with disabilities) in learning STEM? Given the diversity within the ELLs' population, what has worked, for whom, and under what conditions? What can be learned from these models and what additional research is needed to understand what makes them effective? What commonly used approaches may be less effective?; (2) What is the role of teachers in supporting the success of ELLs in STEM? What is known about the biases teachers may bring to their classrooms with ELLs and how these can be effectively addressed? What kinds of curriculum, professional development experiences, and assessment are needed in order for STEM teachers to improve their support for ELLs in STEM?; (3) How can assessments in STEM (both formative and summative) be designed to reflect the new content standards and to be appropriate for ELLs? What assessment accommodations might need to be considered?; (4) How do policies and practices at the national, state, and local level constrain or facilitate efforts to better support ELLs in STEM (including policies related to identification of students)? What kinds of changes in policy and practice are needed?; and (5) What are the gaps in the current research base and what are the key directions for research, both short-term and long-term? The committee will work over a 30-month period to synthesize relevant research literature and prepare a final consensus report, including results, conclusions, and recommendations. The study will address an issue of national importance and will inform future research on challenges directly related to ELLs, diversity, and equity in STEM education. This issue is particularly relevant to programs such as Discovery Research K-12 that supports efforts that reflect the needs of the increasingly diverse population, and Innovative Technology Experiences for Students and Teachers, which supports strategies for recruiting and selecting participants from identified groups currently underrepresented in STEM professions, careers, and education pathways. The report will target a broad audience of stakeholders, including teachers, school district administrators, researchers, congressional staff, and federal agencies that fund educational research and set policies related to ELLs.

PBS NewsHour STEM Student Reporting Labs: Broad Expansion of Youth Journalism to Support Increased STEM Literacy Among Underserved Student Populations and Their Communities

The production of news stories and student-oriented instruction in the classroom are designed to increase student learning of STEM content through student-centered inquiry and reflections on metacognition. This project scales up the PBS NewsHour Student Reporting Labs (SRL), a model that trains teens to produce video reports on important STEM issues from a youth perspective.

Award Number: 
1503315
Funding Period: 
Sat, 08/01/2015 to Wed, 07/31/2019
Full Description: 

The Discovery Research K-12 program (DR-K12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects. This project scales up the PBS NewsHour Student Reporting Labs (SRL), a model that trains teens to produce video reports on important STEM issues from a youth perspective. Participating schools receive a SRL journalism and digital media literacy curriculum, a mentor for students from a local PBS affiliate, professional development for educators, and support from the PBS NewsHour team. The production of news stories and student-oriented instruction in the classroom are designed to increase student learning of STEM content through student-centered inquiry and reflections on metacognition. Students will develop a deep understanding of the material to choose the best strategy to teach or tell the STEM story to others through digital media. Over the 4 years of the project, the model will be expanded from the current 70 schools to 150 in 40 states targeting schools with high populations of underrepresented youth. New components will be added to the model including STEM professional mentors and a social media and media analytics component. Project partners include local PBS stations, Project Lead the Way, and Share My Lesson educators.

The research study conducted by New Knowledge, LLC will add new knowledge about the growing field of youth science journalism and digital media. Front-end evaluation will assess students' understanding of contemporary STEM issues by deploying a web-based survey to crowd-source youth reactions, interest, questions, and thoughts about current science issues. A subset of questions will explore students' tendencies to pass newly-acquired information to members of the larger social networks. Formative evaluation will include qualitative and quantitative studies of multiple stakeholders at the Student Reporting Labs to refine the implementation of the program. Summative evaluation will track learning outcomes/changes such as: How does student reporting on STEM news increase their STEM literacy competencies? How does it affect their interest in STEM careers? Which strategies are most effective with underrepresented students? How do youth communicate with each other about science content, informing news media best practices? The research team will use data from pre/post and post-delayed surveys taken by 1700 students in the STEM Student Reporting Labs and 1700 from control groups. In addition, interviews with teachers will assess the curriculum and impressions of student engagement.


Project Videos

2019 STEM for All Video Showcase

Title: How Video Storytelling Reengages Teenagers in STEM Learning

Presenter(s): Leah Clapman & William Swift

2018 STEM for All Video Showcase

Title: PBS NewsHour's STEM SRL Transforms Classrooms into Newsrooms

Presenter(s): Leah Clapman & William Swift

2017 STEM for All Video Showcase

Title: PBS is Building the Next Generation of STEM Communicators

Presenter(s): Leah Clapman, John Fraser, Su-Jen Roberts, & Bill Swift


SimScientists Games: Development of Simulation-Based Game Designs to Enhance Formative Assessment and Deep Science Learning in Middle School

This project will focus on understanding how educational games, designed according to research-based learning and assessment design principles, can better assess and promote students' science knowledge, application of science process skills, and motivation and engagement in learning.

Lead Organization(s): 
Award Number: 
1503481
Funding Period: 
Sat, 08/01/2015 to Wed, 07/31/2019
Full Description: 

The Discovery Research K-12 (DRK-12) program seeks to significantly enhance the learning and teaching of science, technology, engineering, and mathematics (STEM) by preK-12 students and teachers through research and development of innovative resources, models, and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects. This project is a four-year design and development study submitted to the assessment strand of the program. It will focus on understanding how educational games, designed according to research-based learning and assessment design principles, can better assess and promote students' science knowledge, application of science process skills, and motivation and engagement in learning. The project will develop a new genre of games to serve as formative assessment resources designed to collect evidence of science learning during gameplay, provide feedback and coaching in the form of hints, and reinforce middle grade (6th-8th) students' life science concepts and investigation practices about ecosystems described in the Next Generation Science Standards (NGSS) (Achieve, 2013). The games will build on the designs of the simulation-based, curriculum-embedded assessments developed in previous NSF-funded efforts, which include student progress reports and reflection activities that allow teachers to provide feedback to students and adjust instruction. The design of the games will draw from multiple lines of research, such as cognition, particularly model-based learning; principled assessment design; and motivation. Intended to provide engaging activities for understanding and investigating the system components, roles, interactions, and population dynamics of ecosystems, the project will produce two sets of comprehensive games: (1) Organisms and Interactions, and (2) Emergent Population Levels: Managing an Ecosystem. Each game will consist of progressively advanced mini-games. Twenty-four California Bay Area middle school teachers will participate in the study. Teacher professional development (PD) will include face-to-face sessions and an online platform that permits a wide range of interactions among participants and the facilitators. The PD will emphasize the alignment of the ecosystem simulation-based curriculum modules with their state standards, instructional materials, and the new games. 

The project will address six research questions: (1) How well do the games align with the ecosystem crosscutting concepts, core ideas, and inquiry practices in the NGSS?; (2) How well do game components meet quality standards?; (3) How well do the games integrate with the existing simulation-based curriculum modules and the teachers' existing instructional sequence?; (4) What effect does the use of the games have on students' understanding of the science concepts, scientific practices, and collaboration skills?; (5) How does success in gameplay relate to improved performance on the external outcome measures comprised of the simulation-based benchmark and the pre/posttest?; and (6) How does the use of the games affect students' engagement in science learning? In a Year 1 usability study, the project will test, analyze, and revise alpha versions of the games. In Year 2, a classroom feasibility study of beta versions will inform further revisions. In Year 3, six teachers will pilot-test the games. A second pilot test in Year 4 will examine the effectiveness of the games by comparing student performance in classes using the existing simulation-based curriculum-embedded assessments and reflection activities with classes using the curriculum-embedded assessments plus the new games. Data collection and analysis strategies include: (a) alignment reviews; (b) focus groups and usability testing; (c) cognitive labs for construct validity and usability; (d) game reports (badges); (e) pre/posttest of American Association for the Advancement of Science (AAAS) items; (f) benchmark assessment data; (g) student interest in the games and science; (h) teacher surveys; (i) case studies; (j) game quality analysis; (k) differential item functioning; (l) analysis of covariance; and (m) analysis of variance on posttest scores (outcome variable) to compare the means across student groups (by intervention mode) and their prior science achievement levels.

Fostering STEM Trajectories: Bridging ECE Research, Practice, and Policy

This project will convene stakeholders in STEM and early childhood education to discuss better integration of STEM in the early grades. PIs will begin with a phase of background research to surface critical issues in teaching and learning in early childhood education and STEM.  A number of reports will be produced including commissioned papers, vision papers, and a forum synthesis report.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1417878
Funding Period: 
Mon, 06/15/2015 to Tue, 05/31/2016
Full Description: 

Early childhood education is at the forefront of the minds of parents, teachers, policymakers as well as the general public. A strong early childhood foundation is critical for lifelong learning. The National Science Foundation has made a number of early childhood grants in science, technology, engineering and mathematics (STEM) over the years and the knowledge generated from this work has benefitted researchers. Early childhood teachers and administrators, however, have little awareness of this knowledge since there is little research that is translated and disseminated into practice, according to the National Research Council. In addition, policies for both STEM and early childhood education has shifted in the last decade. 

The Joan Ganz Cooney Center and the New America Foundation are working together to highlight early childhood STEM education initiatives. Specifically, the PIs will convene stakeholders in STEM and early childhood education to discuss better integration of STEM in the early grades. PIs will begin with a phase of background research to surface critical issues in teaching and learning in early childhood education and STEM. The papers will be used as anchor topics to organize a forum with a broad range of stakeholders including policymakers as well as early childhood researchers and practitioners. A number of reports will be produced including commissioned papers, vision papers, and a forum synthesis report. The synthesis report will be widely disseminated by the Joan Ganz Cooney Center and the New America Foundation.

The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed project.

Understanding the Role of Contextual Effects in STEM Pursuit and Persistence: A Synthesis Approach

This synthesis project will inform educators and policymakers about the cumulative evidence that exists on the impacts of a variety of contextual factors on a multitude of STEM outcomes (e.g., math and science achievement, self-efficacy, future goals). This project will provide new evidence regarding the significance of youth contexts on STEM outcomes that will assist policy makers and educators in evaluating productive educational environments.

Award Number: 
1417601
Funding Period: 
Mon, 09/01/2014 to Wed, 08/31/2016
Full Description: 

The percentage of U.S. high school graduates pursuing STEM majors has declined over the last three decades with the largest decline among the highest achieving students. American youth are ill-prepared relative to their international counterparts - U.S. 15 year olds rank 16 out of 26 developed countries in science literacy and 19 out of 26 developed counties in mathematical literacy. There is much research in the areas of how students learn STEM in formal settings, but there is little knowledge of the impact of youth contexts on STEM. Youth contexts are social groups in the lives of young people such as neighborhoods, communities, schools, classrooms or friends. Understanding the role of youth contexts is crucial to ensuring that all students have the opportunity to learn STEM content. This project will synthesize the research literature on youth context and assess whether and how a range of these contexts shape K-12 STEM outcomes and engagement - predictors critical for later educational and occupational attainment. The researchers will conduct two large-scale meta-analyses - one based on the quantitative research body and one based on the qualitative research body - in order to draw conclusions about which contextual factors relate to which STEM outcomes across the span of extant research. In doing so, this synthesis project will inform educators and policymakers about the cumulative evidence that exists on the impacts of a variety of contextual factors on a multitude of STEM outcomes (e.g., math and science achievement, self-efficacy, future goals). This project will provide new evidence regarding the significance of youth contexts on STEM outcomes that will assist policy makers and educators in evaluating productive educational environments.

Syntheses of the research in youth contexts and their impact in STEM will address the following four research questions: (1) How do contextual factors impact STEM learning?; (2) How do these factors vary by the specific type of context?; (3) How do these factors vary by gender and race within each context?; and, (4) Are these factors influenced by the methodological features of the research? The review will include electronic searches of educational, economics, sociology, psychology, and general science databases covering the years 1980-2014. Results will be narrowed by youth context area, and separate analysis will be conducted on gender and race/ethnicity differences in STEM outcomes. The data for the full review will be evaluated by a common set of guidelines to be published along with the findings, enabling the conclusions of the review to be transparent and allowing for detailed information to be easily accessible. The review will discuss each study that meets the inclusion requirements for a valid research design. With this methodology, this study will be the first to provide a clearinghouse of rigorous research related to contextual factors of STEM outcomes.

Knowledge Assets to Support the Science Instruction of Elementary Teachers (ASSET)

This project will address two obstacles that hinder elementary science instruction: (1) a lack of content-specific teaching knowledge (e.g., research on effective topic-specific instructional strategies); and (2) the knowledge that does exist is often not organized for use by teachers in their lesson planning and instruction. The project will collect existing empirical literature for two science topics and synthesize it with an often-overlooked resource -- practice-based knowledge. 

Lead Organization(s): 
Award Number: 
1417838
Funding Period: 
Tue, 07/01/2014 to Fri, 06/30/2017
Full Description: 

This project will address two obstacles that hinder elementary science instruction: (1) a lack of content-specific teaching knowledge (e.g., research on effective topic-specific instructional strategies); and (2) the knowledge that does exist is often not organized for use by teachers in their lesson planning and instruction. The problem is particularly acute at the elementary level, where many teachers have limited science background and many have not taught science before. The project will collect existing empirical literature for two science topics and synthesize it with an often-overlooked resource -- practice-based knowledge. The resulting knowledge resources will be made available to teachers on a website. The resource will support elementary teachers as they plan for science instruction, and to enable them to productively adapt their own science materials to improve student learning. The project will work with teachers in high minority schools.

The project will contribute to a developing theory of Collective Pedagogical Content Knowledge (C-PCK) which includes the research literature, practitioner literature and collective wisdom of practice. The researchers will seek to understand how C-PCK can be made more useful for teachers. The research questions are: (1) What are the strengths and weaknesses of the knowledge collection and synthesis method? (2) What factors must be taken into account in applying the knowledge collection and synthesis method across science topics? (3) What affordances and limitations does the web-based resource present for teachers primarily, and for teacher educators and instructional materials developers? (4) How does access to content-specific teaching knowledge affect teachers' planning and instruction? Content-specific teaching knowledge will be collected through literature reviews (for empirical knowledge) and a series of iterative, on-line expert panels (to gather practice-based knowledge). The two sources of knowledge will be synthesized for each of the science topics and organized in a web-based resource for teachers. A group of pilot teachers will use the resource as they plan for and teach a unit of instruction on the science topics. Project researchers will observe their instruction and interview the teachers to look for evidence of the resource facilitating their instruction. In addition, researchers will administer assessments to teachers and their students to gauge changes on content knowledge that might be attributable to the resource. Teacher feedback will be used to modify the web-based resource and maximize its usability.

GRIDS: Graphing Research on Inquiry with Data in Science

The Graphing Research on Inquiry with Data in Science (GRIDS) project will investigate strategies to improve middle school students' science learning by focusing on student ability to interpret and use graphs. GRIDS will undertake a comprehensive program to address the need for improved graph comprehension. The project will create, study, and disseminate technology-based assessments, technologies that aid graph interpretation, instructional designs, professional development, and learning materials.

Award Number: 
1418423
Funding Period: 
Mon, 09/01/2014 to Sat, 08/31/2019
Full Description: 

The Graphing Research on Inquiry with Data in Science (GRIDS) project is a four-year full design and development proposal, addressing the learning strand, submitted to the DR K-12 program at the NSF. GRIDS will investigate strategies to improve middle school students' science learning by focusing on student ability to interpret and use graphs. In middle school math, students typically graph only linear functions and rarely encounter features used in science, such as units, scientific notation, non-integer values, noise, cycles, and exponentials. Science teachers rarely teach about the graph features needed in science, so students are left to learn science without recourse to what is inarguably a key tool in learning and doing science. GRIDS will undertake a comprehensive program to address the need for improved graph comprehension. The project will create, study, and disseminate technology-based assessments, technologies that aid graph interpretation, instructional designs, professional development, and learning materials.

GRIDS will start by developing the GRIDS Graphing Inventory (GGI), an online, research-based measure of graphing skills that are relevant to middle school science. The project will address gaps revealed by the GGI by designing instructional activities that feature powerful digital technologies including automated guidance based on analysis of student generated graphs and student writing about graphs. These materials will be tested in classroom comparison studies using the GGI to assess both annual and longitudinal progress. Approximately 30 teachers selected from 10 public middle schools will participate in the project, along with approximately 4,000 students in their classrooms. A series of design studies will be conducted to create and test ten units of study and associated assessments, and a minimum of 30 comparison studies will be conducted to optimize instructional strategies. The comparison studies will include a minimum of 5 experiments per term, each with 6 teachers and their 600-800 students. The project will develop supports for teachers to guide students to use graphs and science knowledge to deepen understanding, and to develop agency and identity as science learners.

Pages

Subscribe to Synthesis