Assessment

Development of the Electronic Test of Early Numeracy

The project will develop and refine an electronic Test of Early Numeracy (e-TEN) in English and Spanish that will assess informal and formal knowledge of number and operations in domains including verbal counting, numbering, numerical relationships, and mental addition/subtraction. The overarching goal of the assessment design is to create a measure that is more accurate, more accessible to a wider range of children, and easier to administer than existing measures.

Partner Organization(s): 
Award Number: 
1621470
Funding Period: 
Thu, 09/15/2016 to Tue, 08/31/2021
Full Description: 

The project will develop and refine an electronic Test of Early Numeracy (e-TEN) in English and Spanish, focused on number and operations. The assessment will incorporate a learning trajectory that describes students' development of the understanding of number. The electronic assessment will allow for the test to adapt to students' responses and incorporate games to increase children's engagement with the tasks. These features take advantage of the electronic format. The achievement test will be designed to be efficient, user-friendly, affordable, and accessible for a variety of learning environments and a broad age range (3 to 8 years old). The overarching goal of the assessment design is to create a measure that is more accurate, more accessible to a wider range of children, and easier to administer than existing measures. This project is funded by the Discovery Research Pre-K-12 Program, which funds research and development of STEM innovations and approaches in assessment, teaching and learning.

The e-TEN will assess informal and formal knowledge of number and operations in domains including verbal counting, numbering, numerical relationships, and mental addition/subtraction. The items will be designed using domain-based learning trajectories that describe students' development of understanding of the topics. The test will be designed with some key characteristics. First, it will be semi-adaptive over six-month age spans. Second, it will have an electronic format that allows for uniform implementation and an efficient, user-friendly administration. The test will also be accessible to Spanish speakers using an inclusive assessment model. Finally, the game-based aspect should increase children's engagement and present more meaningful questions. The user-friendly aspect includes simplifying the assessment process compared to other tests of numeracy in early-childhood. The first phase of the development will test a preliminary version of the e-TEN to test its functionality and feasibility. The second phase will focus on norming of the items, reliability and validity. Reliability will be assessed using Item Response Theory methods and test-retest reliability measures. Validity will be examined using criterion-prediction validity and construct validity. The final phase of the work will include creating a Spanish version of the test including collecting data from bilingual children using both versions of the e-TEN.

Systemic Formative Assessment to Promote Mathematics Learning in Urban Elementary Schools

This project builds on the study of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students and combines the intervention with research-based understandings of systemic reform. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders.

Lead Organization(s): 
Award Number: 
1621333
Funding Period: 
Thu, 09/15/2016 to Sat, 02/29/2020
Full Description: 

Districts have long struggled to implement instructional programming in ways that meaningfully and sustainably impact teaching and learning. Systemic education reform is based on the hypothesis that prevailing patterns of incoherence and misalignment in an educational system can send mixed messages to local implementers as they try to respond to various cues and incentives in the environment. Systemic reform seeks to bring alignment to education systems in multiple ways, including consistency across instructional philosophies, alignment across grade levels, and vertical coherence from district to schools to classrooms. This project builds on the Consortium for Policy Research in Education's (CPRE) ongoing, NSF-funded experimental study of the impacts of the Ongoing Assessment Project's (OGAP) math assessment intervention on elementary teachers and students in Philadelphia-area schools. The project will combine the OGAP math intervention with research-based understandings of systemic reform. OGAP is based upon established theory and research demonstrating the impact of teachers' use of ongoing short- and medium-cycle formative assessment on student learning. It combines these understandings with recent research on learning trajectories within mathematics content domains. By bringing to bear the strengths of all three of these areas of research - formative assessment, learning trajectories, and systemic reform - the project promises a significant contribution to the knowledge base about the application of math learning research to classroom instruction on a large scale. This project will produce concrete tools, routines, and practices that can be applied to strengthen programs' implementation by ensuring the strategic support of school and district leaders. This project is funded by the Discovery Research PreK-12 (DRK-12) and EHR Core Research (ECR) Programs. The DRK-12 program supports research and development on STEM education innovations and approaches to teaching, learning, and assessment. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field.

CPRE and the School District of Philadelphia (SDP) will establish a research-practice partnership focused on developing, implementing, refining, and testing a systemic support model to strengthen implementation of the OGAP math intervention in elementary schools. CPRE's current experimental study of OGAP's impacts reveals, preliminarily, statistically significant positive effects on teacher knowledge and student learning. As a result, SDP has decided to expand OGAP into an additional 60 schools in 2016-17. However, the current OGAP study has also revealed weak implementation stemming from a lack of consistent leadership support for the intervention. The project will address this implementation challenge by developing, refining, supporting, and documenting a systemic support component that will accompany OGAP's classroom-level implementation. The systemic supports will be developed by a research-practice partnership between CPRE; SDP; OGAP; the Graduate School of Education at the University of Pennsylvania (PennGSE); and the Philadelphia Education Research Consortium (PERC). The team will use principles of design-based implementation research to iteratively refine and improve the systemic support model. Along with the design and development of the systemic support model, the project will conduct a mixed-methods study of its impacts and roll-out. A three-armed quasi-experimental study will examine the differential impacts of OGAP, with and without systemic supports, and business-as-usual math programming on teacher and student outcomes. A mixed-methods study will examine teacher and administrator experiences in both treatment groups, and will provide feedback to inform the iterative development of the systemic support model.

InquirySpace 2: Broadening Access to Integrated Science Practices

This project will create technology-enhanced classroom activities and resources that increase student learning of science practices in high school biology, chemistry, and physics. InquirySpace will incorporate several innovative technological and pedagogical features that will enable students to undertake scientific experimentation that closely mirrors current science research and learn what it means to be a scientist.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1621301
Funding Period: 
Thu, 09/15/2016 to Tue, 08/31/2021
Full Description: 

This project will create technology-enhanced classroom activities and resources that increase student learning of science practices in high school biology, chemistry, and physics courses. The project addresses the urgent national priority to improve science education as envisioned in the Next Generation Science Standards (NGSS) by focusing less on learning facts and equations and instead providing students with the time, skills, and resources to experience the conduct of science and what it means to be a scientist. This project builds on prior work that created a sequence of physics activities that significantly improved students' abilities to undertake data-based experiments and led to productive independent investigations. The goal of the InquirySpace project is to improve this physics sequence, extend the approach to biology and chemistry, and adapt the materials to the needs of diverse students by integrating tailored formative feedback in real time. The result will be student and teacher materials that any school can use to allow students to experience the excitement and essence of scientific investigations as an integral part of science instruction. The project plans to create and iteratively revise learning materials and technologies, and will be tested in 48 diverse classroom settings. The educational impact of the project's approach will be compared with that of business-as-usual approaches used by teachers to investigate to what extent it empowers students to undertake self-directed experiments. To facilitate the widest possible use of the project, a complete set of materials, software, teacher professional development resources, and curriculum design documents will be available online at the project website, an online teacher professional development course, and teacher community sites. The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

InquirySpace will incorporate several innovative technological and pedagogical features that will enable students to undertake scientific experimentation that closely mirrors current science research. These features will include (1) educational games to teach data analysis and interpretation skills needed in the approach, (2) reduced dependence on reading and writing through the use of screencast instructions and reports, (3) increased reliance on graphical analysis that can make equations unnecessary, and (4) extensive use of formative feedback generated from student logs. The project uses an overarching framework called Parameter Space Reasoning (PSR) to scaffold students through a type of experimentation applicable to a very large class of experiments. PSR involves an integrated set of science practices related to a question that can be answered with a series of data collection runs for different values of independent variables. Data can be collected from sensors attached to the computer, analysis of videos, scientific databases, or computational models. A variety of visual analytic tools will be provided to reveal patterns in the graphs. Research will be conducted in three phases: design and development of technology-enhanced learning materials through design-based research, estimation of educational impact using a quasi-experimental design, and feasibility testing across diverse classroom settings. The project will use two analytical algorithms to diagnose students' learning of data analysis and interpretation practices so that teachers and students can modify their actions based on formative feedback in real time. These algorithms use computationally optimized calculations to model the growth of student thinking and investigation patterns and provide actionable information to teachers and students almost instantly. Because formative feedback can improve instruction in any field, this is a major development that has wide potential.

Sensing Science through Modeling: Developing Kindergarten Students' Understanding of Matter and Its Changes

This project will develop a technology-supported, physical science curriculum that will facilitate kindergarten students' conceptual understanding of matter and how matter changes. The results of this investigation will contribute important data on the evolving structure and content of children's physical science models as well as demonstrate children's understanding of matter and its changes.

Lead Organization(s): 
Award Number: 
1621299
Funding Period: 
Sat, 10/01/2016 to Wed, 09/30/2020
Full Description: 

Despite recent research demonstrating the capacity of young children to engage deeply with science concepts and practices, challenging science curriculum is often lacking in the early grades. This project addresses this gap by developing a technology-supported, physical science curriculum that will facilitate kindergarten students' conceptual understanding of matter and how matter changes. To accomplish these goals, the curriculum will include opportunities for students to participate in model-based inquiry in conjunction with the use of digital probeware and simulations that enable students to observe dynamic visualizations and make sense of the phenomena. To support the capacity of kindergarten teachers, a continuous model of teacher development will be implemented.

Throughout development, the project team will collaborate with kindergarten teachers and more than 300 demographically diverse students across eight classrooms in Massachusetts and Indiana. A design based research approach will be used to iteratively design and revise learning activities, technological tools, and assessments that meet the needs and abilities of kindergarten students and teachers. The project team will: 1) work with kindergarten teachers to modify an existing Grade 2 curricular unit for use with their students; 2) design a parallel curricular unit incorporating technology; 3) evaluate both units for feasibility and maturation effects; and 4) iteratively revise and pilot an integrated unit and assess kindergarten student conceptual understanding of matter and its changes. The results of this investigation will contribute important data on the evolving structure and content of children's physical science models as well as demonstrate children's understanding of matter and its changes.

Modeling Assessment to Enhance Teaching and Learning (Collaborative Research: Wilson)

This project will modify an existing assessment system (BEAR Assessment System) to provide ongoing, instructionally productive evidence to teachers about student learning and to link student work products and formative assessments with summative assessments in models that generate useful estimates of student growth.

Award Number: 
1621265
Funding Period: 
Thu, 12/01/2016 to Mon, 08/31/2020
Full Description: 

Although many in education advocate for evidence-based teaching, implementation of such practices is often quite difficult for teachers. Assessment and accountability data are commonly used for system-level reforms, but are seldom designed to help teachers guide day-to-day instructional decision-making. More useful assessment systems should deliver actionable information to assist in guiding instructional decisions, communicate the development of student knowledge, and integrate various forms of data to assist teachers. Such systems must be logistically feasible to implement, provide suitable grounds for interpreting information about achievement, and exist in a teacher community interested in ongoing feedback about student learning.

This project will modify an existing assessment system (BEAR Assessment System) to provide ongoing, instructionally productive evidence to teachers about student learning and to link student work products and formative assessments with summative assessments in models that generate useful estimates of student growth. To design and test the assessment system, researchers will study teacher integration of assessment tools with instruction via classroom observations, video records, and interviews. Feedback from teachers and observations of their assessment practices will inform revisions to the assessment system. Multiple iterations will focus on how best to represent and display assessment results for tracking individual and group learning. Researchers will investigate new psychometric models that link information from student classroom work, responses to formative assessments, and summative evaluations to provide more reliable estimates of student learning.

Building a Next Generation Diagnostic Assessment and Reporting System within a Learning Trajectory-Based Mathematics Learning Map for Grades 6-8

This project will build on prior funding to design a next generation diagnostic assessment using learning progressions and other learning sciences research to support middle grades mathematics teaching and learning. The project will contribute to the nationally supported move to create, use, and apply research based open educational resources at scale.

Award Number: 
1621254
Funding Period: 
Thu, 09/15/2016 to Sat, 08/31/2019
Full Description: 

This project seeks to design a next generation diagnostic assessment using learning progressions and other research (in the learning sciences) to support middle grades mathematics teaching and learning. It will focus on nine large content ideas, and associated Common Core State Standards for Mathematics. The PIs will track students over time, and work within school districts to ensure feasibility and use of the assessment system.

The research will build on prior funding by multiple funding agencies and address four major goals. The partnership seeks to address these goals: 1) revising and strengthening the diagnostic assessments in mathematics by adding new item types and dynamic tools for data gathering 2) studying alternative ways to use measurement models to assess student mathematical progress over time using the concept of learning trajectories, 3) investigating how to assist students and teachers to effectively interpret reports on math progress, both at the individual and the class level, and 4) engineering and studying instructional strategies based on student results and interpretations, as they are implemented within competency-based and personalized learning classrooms. The learning map, assessment system, and analytics are open source and can be used by other research and implementation teams. The project will exhibit broad impact due to the number of states, school districts and varied kinds of schools seeking this kind of resource as a means to improve instruction. Finally, the research project contributes to the nationally supported move to create, use, and apply research based open educational resources at scale.

Improving the Implementation of Rigorous Instructional Materials in Middle-Grades Mathematics: Developing a System of Practical Measures and Routines (Collaborative Research: Smith)

The goal of this project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale through a system of practical measures and routines for collecting and using data that both assesses and supports implementation.

Award Number: 
1621238
Funding Period: 
Sat, 10/01/2016 to Thu, 09/30/2021
Full Description: 

The goal of this 5-year research project is to improve the implementation of rigorous instructional materials in middle-grades mathematics at scale. Many projects seek to improve mathematics instruction, but are not able to easily track their efforts at improvement. The primary product of this project will be a system of practical measures and routines for collecting and using data that both assesses and supports the implementation of rigorous instructional materials in middle-grades mathematics. In contrast to research and accountability measures, practical measures are assessments that require little time to administer and can thus be used frequently. The data can be analyzed rapidly so that teachers can receive prompt feedback on their progress, and instructional leaders can use the data to decide where to target resources to support improvement in the quality of instruction and student learning. The system of practical measures and routines will include 1) measures of high-leverage aspects of teachers' instructional practices that have been linked to student learning (e.g., rigor of tasks, quality of students' discourse) and attend to equitable student participation; and 2) measures of high-leverage aspects of key supports for improving the quality of teachers' practice (e.g., quality of professional development; coaching); and 3) a set of routines regarding how to use the resulting data to engage in rapid, improvement efforts. A key principle of the proposed project is that the system of measures and routines can be adapted to a wide range of school and district contexts. This project is supported by the Discovery Research preK-12 (DRK-12) program. The DRK-12 program supports research and development of STEM education innovations and approaches in assessment, learning, and teaching.

The project will establish three research-practice partnerships with five districts, in three different states, that are currently implementing rigorous instructional materials in middle-grades mathematics. Year 1 will focus on the development of a set of practical measures of classroom instruction. Year 2 will focus on testing the use of the classroom measures in the context of supports for teachers' learning, and the development of practical measures of key supports for teachers' learning. Years 3-4 will focus on how the project can "learn our way to scale" (Bryk et al., 2015), which requires strategically implementing measures and routines in increasingly diverse conditions. The project will engage in rapid improvement cycles in which researchers will work alongside district leaders and professional development (PD) facilitators to analyze the data from the measures of both classroom instruction and the quality of support for teacher learning to test the effectiveness of improvements in intended supports for teacher learning and to adjust the design of the support based on data. Across Years 1-4, the project will use recent developments in technology and information visualization to test and improve 1) the collection of practical measures in situ and 2) the design of data representations (or visualizations) that support teachers and leaders to make instructional improvement decisions. In Year 5, the project will conduct formal analyses of the relations between supports for teachers' learning; teachers' knowledge and classroom practices; and student learning.

Development and Empirical Recovery for a Learning Progression-Based Assessment of the Function Concept

The project will design an assessment based on learning progressions for the concept of function - a critical concept for algebra learning and understanding. The goal of the assessment and learning progression design is to specifically incorporate findings about the learning of students traditionally under-served and under-performing in algebra courses.

Lead Organization(s): 
Award Number: 
1621117
Funding Period: 
Thu, 09/15/2016 to Mon, 08/31/2020
Full Description: 

The project will design an assessment based on learning progressions for the concept of function. A learning progression describes how students develop understanding of a topic over time. Function is a critical concept for algebra learning and understanding. The goal of the assessment and learning progression design in this project is to specifically incorporate findings about the learning of students traditionally under-served and under-performing in algebra courses. The project will include accounting for the social and cultural experiences of the middle and high school students when creating assessment tasks. The resources developed should impact mathematics instruction (especially for algebra courses) by creating a learning progression which captures the range of student performance and appropriately places them at distinct levels of performance. The important contribution of the work is the development of a learning progression and related assessment tasks that account for the experiences of students often under-served in mathematics. The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

The learning progression development will begin by comparing and integrating existing learning progressions and current research on function learning. This project will develop an assessment of student knowledge of function based on learning progressions via empirical recovery (looking for the reconstruction of theoretical levels of the learning theory). Empirical recovery is the process through which data will be collected that reconstruct the various levels, stages, or sequences of said learning progression. The development of tasks and task models will include testing computer-delivered, interactive tasks and rubrics that can be used for human and automated scoring (depending on the task). Item response theory methods will be used to evaluate the assessment tasks' incorporation of the learning progression.


Project Videos

2019 STEM for All Video Showcase

Title: Concept of Function Learning Progression

Presenter(s): Edith Graf, Frank Davis, Chad Milner, Maisha Moses, & Sarah Ohls


Modeling Assessment to Enhance Teaching and Learning (Collaborative Research: Lehrer)

This project will modify an existing assessment system (BEAR Assessment System) to provide ongoing, instructionally productive evidence to teachers about student learning and to link student work products and formative assessments with summative assessments in models that generate useful estimates of student growth.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1621088
Funding Period: 
Thu, 12/01/2016 to Mon, 08/31/2020
Full Description: 

Although many in education advocate for evidence-based teaching, implementation of such practices is often quite difficult for teachers. Assessment and accountability data are commonly used for system-level reforms, but are seldom designed to help teachers guide day-to-day instructional decision-making. More useful assessment systems should deliver actionable information to assist in guiding instructional decisions, communicate the development of student knowledge, and integrate various forms of data to assist teachers. Such systems must be logistically feasible to implement, provide suitable grounds for interpreting information about achievement, and exist in a teacher community interested in ongoing feedback about student learning.

This project will modify an existing assessment system (BEAR Assessment System) to provide ongoing, instructionally productive evidence to teachers about student learning and to link student work products and formative assessments with summative assessments in models that generate useful estimates of student growth. To design and test the assessment system, researchers will study teacher integration of assessment tools with instruction via classroom observations, video records, and interviews. Feedback from teachers and observations of their assessment practices will inform revisions to the assessment system. Multiple iterations will focus on how best to represent and display assessment results for tracking individual and group learning. Researchers will investigate new psychometric models that link information from student classroom work, responses to formative assessments, and summative evaluations to provide more reliable estimates of student learning.

Developing Formative Assessment Tools and Routines for Additive Reasoning

This design and development project is an expansion of the Ongoing Assessment Project (OGAP), an established model for research-based formative assessment in grades 3-8, to the early elementary grades. The project will translate findings from research on student learning of early number, addition, and subtraction into tools and routines that teachers can use to formatively assess their students' understanding on a regular basis and develop targeted instructional responses.

Lead Organization(s): 
Award Number: 
1620888
Funding Period: 
Thu, 09/01/2016 to Thu, 02/28/2019
Full Description: 

This design and development project is an expansion of the Ongoing Assessment Project (OGAP), an established model for research-based formative assessment in grades 3-8, to the early elementary grades. OGAP brings together two powerful ideas in mathematics education - formative assessment and research based learning trajectories - to enhance teacher knowledge, instructional practices, and student learning. Building on a proven track record of success with this model, the current project will translate findings from research on student learning of early number, addition, and subtraction into tools and routines that teachers can use to formatively assess their students' understanding on a regular basis and develop targeted instructional responses. The project involves a development component focused on producing and field testing new resources (including frameworks, item banks, pre-assessments and professional development materials) and a research component designed to improve the implementation of these resources in school settings. The materials that are developed from this project will help teachers be able to more precisely assess student understanding in the major mathematical work of grades K-2 in order to better meet the needs of diverse learners. With the addition of these new early elementary materials, OGAP formative assessment resources will be available for use from kindergarten through grade 8.

Although much attention has been paid to the improvement of early literacy, building strong mathematical foundations and early computational fluency is equally critical for later success in school and preparation for STEM careers. This project will develop and field test tools, resources, and routines that teachers can employ to help young students develop deeper conceptual understandings and more powerful and efficient strategies in the early grades. The project emerged from the needs of school-based practitioners looking for instructional support in the primary grades and uses design-based research methodology. The new materials will be developed, tested, and revised through multiple iterations of implementation in schools. Research-based learning trajectories will be consolidated into simplified frameworks that illustrate the overall progression of major levels of student thinking in the domains of counting, addition, and subtraction. A bank of formative assessment items will be developed, field tested, and refined through a three-phase validation process. Professional development modules will be designed and field tested to support teacher knowledge and effective use of the formative assessment tools and routines. Data collected on key activities in the formative assessment process (including teacher selection of items, analysis of student work, instructional implications, and enacted instructional response) will be used to continually inform development as well as illuminate the conditions under which formative assessment leads to productive changes in instruction and student learning in the classroom. The project will yield a set of field tested tools and resources ready for both broader dissemination and further research on the promise of the intervention, as well as an understanding of how to support effective implementation.

Pages

Subscribe to Assessment