Assessment

Measuring Early Mathematical Reasoning Skills: Developing Tests of Numeric Relational Reasoning and Spatial Reasoning

The primary aim of this study is to develop mathematics screening assessment tools for Grades K-2 over the course of four years that measure students' abilities in numeric relational reasoning and spatial reasoning. The team of researchers will develop Measures of Mathematical Reasoning Skills system, which will contain Tests of Numeric Relational Reasoning (T-NRR) and Tests of Spatial Reasoning (T-SR).

Award Number: 
1721100
Funding Period: 
Fri, 09/15/2017 to Tue, 08/31/2021
Full Description: 

Numeric relational reasoning and spatial reasoning are critical to success in later mathematics coursework, including Algebra 1, a gatekeeper to success at the post-secondary level, and success in additional STEM domains, such as chemistry, geology, biology, and engineering. Given the importance of these skills for later success, it is imperative that there are high-quality screening tools available to identify students at-risk for difficulty in these areas. The primary aim of this study is to develop mathematics screening assessment tools for Grades K-2 over the course of four years that measure students' abilities in numeric relational reasoning and spatial reasoning. The team of researchers will develop Measures of Mathematical Reasoning Skills system, which will contain Tests of Numeric Relational Reasoning (T-NRR) and Tests of Spatial Reasoning (T-SR). The measures will be intended for use by teachers and school systems to screen students to determine who is at-risk for difficulty in early mathematics, including students with disabilities. The measures will help provide important information about the intensity of support that may be needed for a given student. Three forms per grade level will be developed for both the T-NRR and T-SR with accompanying validity and reliability evidence collected. The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

The development of the T-NRR and T-SR measures will follow an iterative process across five phases. The phases include (1) refining the construct; (2) developing test specifications and item models; (3) developing items; (4) field testing the items; and (5) conducting validity studies. The evidence collected and evaluated during each phase will contribute to the overall evaluation of the reliability of the measures and the validity of the interpretations made using the measures. Item models, test specifications, and item development will be continuously evaluated and refined based on data from cognitive interviews, field tests, and reviews by mathematics educators, teachers of struggling students, teachers of culturally and linguistically diverse populations, and a Technical Advisory Board. In the final phase of development of the T-NRR and T-SR, reliability of the results will be estimated and multiple sources of validity evidence will be collected to examine the concurrent and predictive relation with other criterion measures, classification accuracy, and sensitivity to growth. Approximately 4,500 students in Grades K-2 will be involved in all phases of the research including field tests and cognitive interviews. Data will be analyzed using a two-parameter IRT model to ensure item and test form comparability.

Project MAPLE: Makerspaces Promoting Learning and Engagement

The project plans to develop and study a series of metacognitive strategies that support learning and engagement for struggling middle school students during makerspace experiences. The study will focus narrowly on establishing a foundational understanding of how to ameliorate barriers to engaging in design learning through the use of metacognitive strategies.

Award Number: 
1721236
Funding Period: 
Fri, 09/01/2017 to Sat, 08/31/2019
Full Description: 

The project plans to develop and study a series of metacognitive strategies that support learning and engagement for struggling middle school students during makerspace experiences. The makerspace movement has gained recognition and momentum, which has resulted in many schools integrating makerspace technologies and related curricular practices into the classroom. The study will focus narrowly on establishing a foundational understanding of how to ameliorate barriers to engaging in design learning through the use of metacognitive strategies. The project plans to translate and apply research on the use of metacognitive strategies in supporting struggling learners to develop approaches that teachers can implement to increase opportunities for students who are the most difficult to reach academically. Project strategies, curricula, and other resources will be disseminated through existing outreach websites, research briefs, peer-reviewed publications for researchers and practitioners, and a webinar for those interested in middle-school makerspaces for diverse learners.

The research will address the paucity of studies to inform practitioners about what pedagogical supports help struggling learners engage in these makerspace experiences. The project will focus on two populations of struggling learners in middle schools, students with learning disabilities, and students at risk for academic failure. The rationale for focusing on metacognition within makerspace activities comes from the literature on students with learning disabilities and other struggling learners that suggests that they have difficulty with metacognitive thinking. Multiple instruments will be used to measure metacognitive processes found to be pertinent within the research process. The project will tentatively focus on persistence (attitudes about making), iteration (productive struggle) and intentionality (plan with incremental steps). The work will result in an evidence base around new instructional practices for middle school students who are struggling learners so that they can experience more success during maker learning experiences.

Mathematical Learning via Architectural Design and Modeling Using E-Rebuild

This project will explore the learning of mathematics through architectural tasks in an online simulation game, E-Rebuild. In the game-based architectural simulation, students will be able to complete tasks such as building and constructing structures while using mathematics and problem solving. The project will examine how to collect data about students' learning from data generated as they play the game, how students learn mathematics using the simulation, and how the simulation can be included in middle school mathematics learning.

Lead Organization(s): 
Award Number: 
1720533
Funding Period: 
Tue, 08/01/2017 to Sat, 07/31/2021
Full Description: 

This project will explore the learning of mathematics through architectural tasks in an online simulation game, E-Rebuild. There is a need to connect mathematics to real world contexts and problems. In the game-based architectural simulation, students will be able to complete tasks such as building and constructing structures while using mathematics and problem solving. The learning platform will be flexible so teachers can customize tasks for their students. The project will examine how to collect data about students' learning from data generated as they play the game. The project will explore how students learn mathematics using the simulation and how the simulation can be included in middle school mathematics learning.

The project includes two major research questions. First, how will the design of a scalable game-based, design-centered learning platform promote coordination and application of math representation for problem solving? Second, how and under what implementation circumstances will using a scalable architectural game-based learning platform improve students multi-stranded mathematical proficiency (i.e., understanding, problem solving and positive disposition)? A key feature of the project is stealth-assessment or data collected and logged as students use the architectural simulation activities that can be used to understand their mathematics learning. The project uses a design-based research approach to gather data from students and teachers that will inform the design of the learning environment. The qualitative and quantitative data will also be used to understand what students are learning as they play the game and how teachers are interacting with their students. The project will include a mixed methods study to compare classrooms using the architectural activities to classrooms that are using typical activities.

BioGraph 2.0: Online Professional Development for High School Biology Teachers for Teaching and Learning About Complex Systems

This proposal will develop and test an open-access, online system of professional development for high school biology teachers in order to build pedagogical competencies for teaching about complex systems and to support the application of those competencies in high school biology classrooms.

Lead Organization(s): 
Award Number: 
1721003
Funding Period: 
Fri, 09/01/2017 to Sat, 08/31/2019
Full Description: 

This proposal will develop and test an open-access, online system of professional development for high school biology teachers in order to build pedagogical competencies for teaching about complex systems and to support the application of those competencies in high school biology classrooms. The online teacher professional development (PD) will be delivered through the edX open access platform.

This research project will include two cycles of design and development of the professional development experience. It will include mixed methods and a longitudinal examination of teacher and student learning fostered by professional development. The research for the first phase will be qualitative in nature and will result in a series of case studies that highlight different facets of the interactions that influence teachers' learning. Following this qualitative phase, through a field study that employs multivariate analysis of covariance and hierarchical linear models analytical techniques, the effectiveness of the design and development stages will be compared to an alternative professional development experience that is similar to the project's professional development but does not include collaborative design. The broad aim is to develop and test an open-access, online system of professional development (PD) that includes solutions for known challenges in teacher online PD. The project builds on a prior NSF-funded exploratory project. The project will employ a randomized control trial to assess the effectiveness of PD on improving teacher content knowledge and skills, changes in classroom practices and instruction, curriculum engagement by students and student achievement outcomes with an end goal to understand better what facilitates online PD and to create a low cost scalable and online version of the original NSF-funded BioGraph. This research will produce insights and guidelines that can immediately be incorporated into the emerging field of online professional development, and online education in general. The content goals are to build pedagogical competencies for teaching about complex systems and to support the application of those competencies in high school biology classrooms.

Research on the Utility of Abstraction as a Guiding Principle for Learning about the Nature of Models in Science Education

This project will develop a short instructional sequence and new student learning assessments that are implemented in earth science classes. The findings will help the field to understand whether the process of abstracting from multiple phenomena during model construction supports students' understanding of scientific models in relation to earth science ideas and the cross-cutting concept of scale.

Partner Organization(s): 
Award Number: 
1720996
Funding Period: 
Mon, 05/15/2017 to Thu, 04/30/2020
Full Description: 

Contemporary science education reforms consider modeling as a means to understanding science ideas and as an essential scientific practice to be learned. Modeling is the practice of developing and refining representations (or "models") as analogs of scientific phenomena. Important to the practice of modeling is the idea that, as an analog, a model draws out (or "abstracts") some as opposed to all aspects of a phenomenon. However, a well-known problem in modeling instruction is that leaners have difficulty understanding this essential point. Learners often think of models as literal interpretations, or replicas, of what they represent. The investigators hypothesize that engaging students in a process of abstraction -- that is, drawing out common structures from multiple scientific phenomena -- during the creation (or "synthesis") of their own model will help students better understand the nature of scientific models. Importantly, this approach will help students discover that a scientific model is not simply a literal interpretation, or replica, of any single phenomenon. Sixteen teachers and their estimated 960 students from economically challenged communities in Georgia and Maine will participate in and benefit from the research study in the context of high school earth and environmental science classes. The project will develop a short instructional sequence and new student learning assessments that are implemented in earth science classes. The findings will help the field to understand whether the process of abstracting from multiple phenomena during model construction supports students' understanding of scientific models in relation to earth science ideas and the cross-cutting concept of scale. The project will provide professional development workshops to up to forty-six teachers over three years as means of recruiting research participants and to cultivate teacher leadership around the new approach to modeling. The developed products and the research findings will be shared with researchers, teacher educators, and teachers through science education journals and conferences.

This Exploratory Learning Strand research study builds upon prior work of investigators at University of Georgia and University of Maine by rigorously testing their hypothesis that that engaging students in the process of abstracting from multiple source phenomena during model synthesis supports more scientifically accurate understandings of the nature of models. The research has the potential to (1) generate new knowledge about the potential value of abstraction as a guiding principle of learning about models and modeling practice; (2) identify ways in which specific classroom conditions, including teacher talk and actions, enable or hinder student learning about abstraction in models and modeling practice; and (3) demonstrate how teachers translate the modeling approach to other science disciplines they teach. Teachers will be recruited through existing partnerships with schools and through professional development workshops offered to teachers nearby the two universities. To address the first two goals, the investigators will develop and test a two-part instructional sequence that addresses core ideas in earth science and the cross-cutting concept of scale. The first component of the instructional sequence is a typical model-based inquiry, and the second component requires that students abstract structures from multiple phenomena during the synthesis of their own models. Twelve teachers and their students will be randomly assigned to either the treatment or the control group. The treatment group will experience the two-part instructional sequence. The control group will initially not experience the second component, but will have an opportunity to do so at the conclusion of the study. Quantitative and qualitative analysis of classroom observations, interviews with teachers, student knowledge tests, student work, and teacher logs will be used to determine the effectiveness of abstracting during model synthesis and classroom conditions that enable or hinder students' learning when the approach is used. To address the third goal, investigators will document the experience of four teachers as they develop and implement a similar instructional sequence in other science disciplines, providing preliminary evidence on the broader utility of the synthesis-through-abstraction approach to modeling. A new research assessment for measuring students' understanding of the nature of models, core ideas of earth science, and the cross-cutting concept of scale may be broadly useful for future research on learning at the intersection of the three knowledge dimensions. Findings will be shared by traditional means, such as papers in peer-reviewed research and practitioner journals and conference presentations. Investigators will conduct professional development workshops for teachers in the third year to disseminate the products and findings of the research to practitioner audiences and to further cultivate participating teachers' leadership around this novel approach to modeling practice in science education.

Promoting Scientific Explorers Among Students with Learning Disabilities: The Design and Testing of a Grade 2 Science Program Focused on Earth's Systems

The purpose of this project is to design and empirically evaluate a second grade science program, Scientific Explorers, aimed at promoting an early foundation for learning science among all students, including students at risk for or with learning disabilities in reading and mathematics.

Lead Organization(s): 
Award Number: 
1720958
Funding Period: 
Thu, 06/01/2017 to Mon, 05/31/2021
Full Description: 

A robust understanding of core science concepts and practices is necessary for obtaining jobs in STEM (science, technology, engineering, and math) fields. Despite these occupational and practical affordances, few effective instructional tools exist for the elementary science classroom. Moreover, early elementary school teachers have limited materials at their disposal to promote a rich knowledge of science among the full range of learners. The purpose of this project is to address this need by designing and empirically evaluating a second grade science program, Scientific Explorers, aimed at promoting an early foundation for learning science among all students, including students at risk for or with learning disabilities in reading and mathematics. Scientific Explorers will be designed to improve students' knowledge and understanding of core science concepts. Recognizing the important role of early literacy and mathematics in science learning and teaching, this project will integrate core disciplinary ideas with critical mathematics and literacy standards. To support students as they engage in scientific tasks associated with Earth's Systems, this project will engineer the Scientific Explorers program around a guided inquiry framework. Another aim of this project is to develop and empirically validate a science assessment that measures students' knowledge and application of core science concepts and practices related to Earth's Systems.

Employing a mixed-method approach, this project will investigate the feasibility and efficacy of the Scientific Explorers program. Additional research activities will include establishing the reliability and validity of a second grade science assessment. Approximately 40 second grade classrooms from two different geographical regions will participate in the project. Using multilevel modeling and item response theory techniques, this project will address five primary research questions: (1) To what extent can teachers feasibly implement the Scientific Explorers program in authentic education settings? (2) What is the impact of Scientific Explorers on the science achievement of students in participating classrooms? (3) Do early literacy skills at the beginning of second grade predict differential response to the Scientific Explorers program? (4) Does responsiveness to the Scientific Explorers program differ as a function of reading disability, mathematics disability, or a learning disability in reading and mathematics (comorbid LD)?, and (5) To what extent does the early science achievement measure demonstrate technical adequacy (reliability and validity)?

Designing a Middle Grades Spatial Skills Curriculum

This project will create a portable training system that can be easily deployed in middle grades (5th-7th grade) as a prototype for increasing students' spatial reasoning skills. The project will study gender differences in spatial reasoning and examine how learning experiences can be designed to develop spatial skills using Minecraft as a platform.

Lead Organization(s): 
Award Number: 
1720801
Funding Period: 
Sat, 07/01/2017 to Tue, 06/30/2020
Full Description: 

The ability to make spatial judgements and visualize has been shown to be a strong indicator of students' future success in STEM-related courses. The project is innovative because it uses a widely available gaming environment, Minecraft, to examine spatial reasoning. Finding learning experiences which support students' spatial reasoning in an authentic and engaging way is a challenge in the field. This project will create a portable training system that can be easily deployed in middle grades (5th-7th grade) as a prototype for increasing students' spatial reasoning skills. The project will study gender differences in spatial reasoning and examine how learning experiences can be designed to develop spatial skills using Minecraft as a platform. The resources will incorporate hands-on learning and engage students in building virtual structures using spatial reasoning. The curriculum materials are being designed to be useful in other middle grades contexts.

The study is a design and development study that will design four training modules intended to improve spatial reasoning in the following areas: rotation, mental slicing, 2D to 3D transformation and perspective taking. The research questions are: (1) Does a Minecraft-based intervention that targets specific spatial reasoning tasks improve middle grade learners' spatial ability? (2) Does spatial skills growth differ by gender? The experimental design will compare the influence of the virtual spatial learning environment alone vs. the use of design challenges designed specifically for the spatial skills. The data collected will include assessments of spatial reasoning and feedback from teachers' who use the materials. The spatial skills measures will be administered as a pre-test, post-test, and six-month follow-up assessment to measure long term effects.

Development of the Electronic Test of Early Numeracy

The project will develop and refine an electronic Test of Early Numeracy (e-TEN) in English and Spanish that will assess informal and formal knowledge of number and operations in domains including verbal counting, numbering, numerical relationships, and mental addition/subtraction. The overarching goal of the assessment design is to create a measure that is more accurate, more accessible to a wider range of children, and easier to administer than existing measures.

Partner Organization(s): 
Award Number: 
1621470
Funding Period: 
Thu, 09/15/2016 to Tue, 08/31/2021
Full Description: 

The project will develop and refine an electronic Test of Early Numeracy (e-TEN) in English and Spanish, focused on number and operations. The assessment will incorporate a learning trajectory that describes students' development of the understanding of number. The electronic assessment will allow for the test to adapt to students' responses and incorporate games to increase children's engagement with the tasks. These features take advantage of the electronic format. The achievement test will be designed to be efficient, user-friendly, affordable, and accessible for a variety of learning environments and a broad age range (3 to 8 years old). The overarching goal of the assessment design is to create a measure that is more accurate, more accessible to a wider range of children, and easier to administer than existing measures. This project is funded by the Discovery Research Pre-K-12 Program, which funds research and development of STEM innovations and approaches in assessment, teaching and learning.

The e-TEN will assess informal and formal knowledge of number and operations in domains including verbal counting, numbering, numerical relationships, and mental addition/subtraction. The items will be designed using domain-based learning trajectories that describe students' development of understanding of the topics. The test will be designed with some key characteristics. First, it will be semi-adaptive over six-month age spans. Second, it will have an electronic format that allows for uniform implementation and an efficient, user-friendly administration. The test will also be accessible to Spanish speakers using an inclusive assessment model. Finally, the game-based aspect should increase children's engagement and present more meaningful questions. The user-friendly aspect includes simplifying the assessment process compared to other tests of numeracy in early-childhood. The first phase of the development will test a preliminary version of the e-TEN to test its functionality and feasibility. The second phase will focus on norming of the items, reliability and validity. Reliability will be assessed using Item Response Theory methods and test-retest reliability measures. Validity will be examined using criterion-prediction validity and construct validity. The final phase of the work will include creating a Spanish version of the test including collecting data from bilingual children using both versions of the e-TEN.

Systemic Formative Assessment to Promote Mathematics Learning in Urban Elementary Schools

This project builds on the study of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students and combines the intervention with research-based understandings of systemic reform. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders.

Lead Organization(s): 
Award Number: 
1621333
Funding Period: 
Thu, 09/15/2016 to Sat, 02/29/2020
Full Description: 

Districts have long struggled to implement instructional programming in ways that meaningfully and sustainably impact teaching and learning. Systemic education reform is based on the hypothesis that prevailing patterns of incoherence and misalignment in an educational system can send mixed messages to local implementers as they try to respond to various cues and incentives in the environment. Systemic reform seeks to bring alignment to education systems in multiple ways, including consistency across instructional philosophies, alignment across grade levels, and vertical coherence from district to schools to classrooms. This project builds on the Consortium for Policy Research in Education's (CPRE) ongoing, NSF-funded experimental study of the impacts of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students in Philadelphia-area schools. The project will combine the OGAP math intervention with research-based understandings of systemic reform. OGAP is based upon established theory and research demonstrating the impact of teachers' use of ongoing short- and medium-cycle formative assessment on student learning. It combines these understandings with recent research on learning trajectories within mathematics content domains. By bringing to bear the strengths of all three of these areas of research - formative assessment, learning trajectories, and systemic reform - the project promises a significant contribution to the knowledge base about the application of math learning research to classroom instruction on a large scale. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders. This project is funded by the Discovery Research PreK-12 (DRK-12) and EHR Core Research (ECR) Programs. The DRK-12 program supports research and development on STEM education innovations and approaches to teaching, learning, and assessment. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field.

CPRE and the School District of Philadelphia (SDP) will establish a research-practice partnership focused on developing, implementing, refining, and testing a systemic support model to strengthen implementation of the OGAP math intervention in elementary schools. CPRE's current experimental study of OGAP's impacts reveals, preliminarily, statistically significant positive effects on teacher knowledge and student learning. As a result, SDP has decided to expand OGAP into an additional 60 schools in 2016-17. However, the current OGAP study has also revealed weak implementation stemming from a lack of consistent leadership support for the intervention. The project will address this implementation challenge by developing, refining, supporting, and documenting a systemic support component that will accompany OGAP's classroom-level implementation. The systemic supports will be developed by a research-practice partnership between CPRE; SDP; OGAP; the Graduate School of Education at the University of Pennsylvania (PennGSE); and the Philadelphia Education Research Consortium (PERC). The team will use principles of design-based implementation research to iteratively refine and improve the systemic support model. Along with the design and development of the systemic support model, the project will conduct a mixed-methods study of its impacts and roll-out. A three-armed quasi-experimental study will examine the differential impacts of OGAP, with and without systemic supports, and business-as-usual math programming on teacher and student outcomes. A mixed-methods study will examine teacher and administrator experiences in both treatment groups, and will provide feedback to inform the iterative development of the systemic support model.

InquirySpace 2: Broadening Access to Integrated Science Practices

This project will create technology-enhanced classroom activities and resources that increase student learning of science practices in high school biology, chemistry, and physics. InquirySpace will incorporate several innovative technological and pedagogical features that will enable students to undertake scientific experimentation that closely mirrors current science research and learn what it means to be a scientist.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1621301
Funding Period: 
Thu, 09/15/2016 to Tue, 08/31/2021
Full Description: 

This project will create technology-enhanced classroom activities and resources that increase student learning of science practices in high school biology, chemistry, and physics courses. The project addresses the urgent national priority to improve science education as envisioned in the Next Generation Science Standards (NGSS) by focusing less on learning facts and equations and instead providing students with the time, skills, and resources to experience the conduct of science and what it means to be a scientist. This project builds on prior work that created a sequence of physics activities that significantly improved students' abilities to undertake data-based experiments and led to productive independent investigations. The goal of the InquirySpace project is to improve this physics sequence, extend the approach to biology and chemistry, and adapt the materials to the needs of diverse students by integrating tailored formative feedback in real time. The result will be student and teacher materials that any school can use to allow students to experience the excitement and essence of scientific investigations as an integral part of science instruction. The project plans to create and iteratively revise learning materials and technologies, and will be tested in 48 diverse classroom settings. The educational impact of the project's approach will be compared with that of business-as-usual approaches used by teachers to investigate to what extent it empowers students to undertake self-directed experiments. To facilitate the widest possible use of the project, a complete set of materials, software, teacher professional development resources, and curriculum design documents will be available online at the project website, an online teacher professional development course, and teacher community sites. The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

InquirySpace will incorporate several innovative technological and pedagogical features that will enable students to undertake scientific experimentation that closely mirrors current science research. These features will include (1) educational games to teach data analysis and interpretation skills needed in the approach, (2) reduced dependence on reading and writing through the use of screencast instructions and reports, (3) increased reliance on graphical analysis that can make equations unnecessary, and (4) extensive use of formative feedback generated from student logs. The project uses an overarching framework called Parameter Space Reasoning (PSR) to scaffold students through a type of experimentation applicable to a very large class of experiments. PSR involves an integrated set of science practices related to a question that can be answered with a series of data collection runs for different values of independent variables. Data can be collected from sensors attached to the computer, analysis of videos, scientific databases, or computational models. A variety of visual analytic tools will be provided to reveal patterns in the graphs. Research will be conducted in three phases: design and development of technology-enhanced learning materials through design-based research, estimation of educational impact using a quasi-experimental design, and feasibility testing across diverse classroom settings. The project will use two analytical algorithms to diagnose students' learning of data analysis and interpretation practices so that teachers and students can modify their actions based on formative feedback in real time. These algorithms use computationally optimized calculations to model the growth of student thinking and investigation patterns and provide actionable information to teachers and students almost instantly. Because formative feedback can improve instruction in any field, this is a major development that has wide potential.

Pages

Subscribe to Assessment