Evaluation

Embodied STEM Learning Across Technology-Based Learning Environments

This project conducts interdisciplinary research to advance understanding of embodied learning as it applies to STEM topics across a range of current technology-based learning environments (e.g., desktop simulations, interactive whiteboards, and 3D interactive environments). The project has two central research questions: How are student knowledge gains impacted by the degree of embodied learning and to what extent do the affordances of different technology-based learning environments constrain or support embodied learning for STEM topics?

Lead Organization(s): 
Award Number: 
1020367
Funding Period: 
Sun, 08/15/2010 to Sun, 07/31/2011
Project Evaluator: 
Susan Haag
Full Description: 

This project conducts interdisciplinary research to advance understanding of embodied learning as it applies to STEM topics across a range of current technology-based learning environments (e.g., desktop simulations, interactive whiteboards, and 3D interactive environments). The project builds on extensive research, including prior work of the PIs, regarding both embodied learning and statistical learning. The PIs describe embodied learning as engaging the neuromuscular systems of learners as they interact with the world around them visually, aurally, and kinesthetically in order to construct new knowledge structures. Statistical learning is described as the ability to learn, often without intent, which sequences of stimuli are consistent with a set of rules. An example of statistical learning is pattern recognition, which is central to mastery of complex topics in many STEM disciplines including physics and mathematics.

The project has two central research questions: How are student knowledge gains impacted by the degree of embodied learning and to what extent do the affordances of different technology-based learning environments constrain or support embodied learning for STEM topics? To investigate these questions, the PIs are conducting three series of experiments in five phases using two physics topics. The first four phases are developmental and the final phase implements and assesses the two modules in schools (20 plus teachers, 700 plus K-12 students) in Arizona and New York (15 total sites, 10 plus public schools, minimum one Title I school).

The aim of this project is to meld these two research trajectories to yield two key outcomes: 1) basic research regarding embodiment and statistical learning that can be applied to create powerful STEM learning experiences, and 2) the realization of exemplary models and principles to aid curriculum and technology designers in creating learning scenarios that take into account the level of embodiment that a given learning environment affords.

Efficacy Study of Project-Based Inquiry Science

This research and development project examines the impact of the Project-Based Inquiry Science (PBIS) middle school science curriculum. The research questions explored will look into efficacy, implementation, and teacher practice. A unique feature of the study’s design is an analytic focus on the conditions needed to implement the curriculum in ways that improve student learning in light of the Framework for K-12 Science Education.

Award Number: 
1020407
Funding Period: 
Sun, 08/15/2010 to Fri, 07/31/2015
Full Description: 

This research and development project studies the impact of Project-Based Inquiry Science (PBIS) on 6th grade students in a large urban school district. PBIS is a comprehensive, 3-year middle school science curriculum that focuses on standards-based science content and that uses project-based inquiry science units to help students learn. NSF funded the development of PBIS over the past two decades, with major investments made in the design of materials and with associated teacher professional development designed to help teachers understand the content of the units and how to teach them. Prior small-scale studies of PBIS have shown positive impact on student achievement and motivation, and on teacher use of reform-based instruction. The research questions explored are:

1. Efficacy. What is the impact of PBIS on student learning? To what extent do students in PBIS perform better than non-PBIS students on measures of learning?
2. Enactment and teacher practice. What is the impact of the curriculum on teaching quality? What is the fidelity of classroom implementation? How does the depth and level of implementation relate to student outcomes?

The study involves both quantitative and qualitative methods; the use of an experimental design allows estimates of causal impacts when combining professional development with the curriculum materials. This is a randomized control trial to test the efficacy of PBIS in 42 middle schools and with ˜120 teachers (21 schools and ˜60 teachers per condition), and affecting approximately 8,500 6th grade students. The dependent variables for students include results on state-level achievement tests and measures of their ability to develop and use models and construct explanations in the context of the curriculum units. Mediational analysis measures the association between contextual factors such as fidelity of implementation and quality of the professional development experience and student learning, allowing a deeper understanding of results.

This work is critical to the ongoing effort to support standards-based curriculum reform in science. PBIS has enjoyed some success in urban settings with diverse groups of students, including those from historically underrepresented groups in science, and now moves to larger scale. This curriculum is among a small number of science curriculum initiatives that are at a stage in the research and development cycle where implementation efforts are focused on scaling to a broader range of schools and districts. The curriculum units are based on design principles drawn from theory and research on how students learn and are aligned with learning goals found in state and national standards. Moreover, its design reflects where the science education field is headed – teaching a few big ideas and integrating scientific practices. Project outcomes will provide evidence about the effects of a published and available inquiry-based science curriculum.

Data Games: Tools and Materials for Learning Data Modeling (Collaborative Research: Finzer)

The Data Games project has developed software and curriculum materials in which data generated by students playing computer games form the raw material for mathematics classroom activities. Students play a short video game, analyze the game data, develop improved strategies, and test their strategies in another round of the game.

Project Email: 
Lead Organization(s): 
Award Number: 
0918735
Funding Period: 
Tue, 09/01/2009 to Fri, 08/31/2012
Project Evaluator: 
James Hammerman
Full Description: 

Students playing computer games generate large quantities of rich, interesting, highly variable data that mostly evaporate when the game ends. What if in a classroom setting, data from games students played remained accessible to them for analysis? In software and curriculum materials developed by the Data Games project at UMass Amherst and KCP Technologies, data generated by students playing computer games form the raw material for mathematics classroom activities. Students play a short video game, analyze the game data, develop improved strategies, and try their strategies in another round of the game.

 

The video games are embedded in an online data analysis learning environment that is based on desktop software tools Fathom® Dynamic Data and Tinkerplots® Dynamic Data Exploration, widely used in grades 5–8 and 8–14 respectively. The game data appear in graphs and tables in real time, allowing several cycles of strategy improvement in a short time. The games are designed so that these cycles improve understanding of specific data modeling and/or mathematics concepts.

 

The research strand of the Data Games project focuses on students’ creation of data representations that model a real-world context. Findings from this research have been incorporated into the design of the data structures in the software.

Communication in Science Inquiry Project (CISIP)

CISIP is a professional development program that enables English and science teachers to help students to learn content and communicate scientifically. The CISIP program: Translates How Students Learn Science in the Classroom and Common Core State Standards for student success; targets learning within a classroom discourse community that focuses on argumentation; and takes a team of science and English teachers at schools from middle level through university who collaborate.

Project Email: 
Partner Organization(s): 
Award Number: 
0353469
Funding Period: 
Thu, 07/01/2004 to Fri, 12/31/2010
Project Evaluator: 
M. J. Young

Mathematics Instruction Using Decision Science and Engineering Tools

A collaboration among educators, engineers, and mathematicians in three universities, this project is creating, implementing, and evaluating a one-year curriculum for teaching a non-calculus, fourth-year high school mathematics course and accompanied assessment instruments. The curriculum will draw on decision-making tools that include but go well beyond linear programming, to enhance student mathematical competence (particularly solving multi-step problems), improve students' attitudes toward mathematics, and promote states' adoption of the curriculum (initially NC and MI).

Project Email: 
Award Number: 
0733137
Funding Period: 
Sat, 09/15/2007 to Tue, 08/31/2010
Project Evaluator: 
Dr. Shlomo S. Sawilowsky
Full Description: 

Mathematics INstruction using Decision Science and Engineering Tools (MINDSET) is a collaboration among educators, engineers, and mathematicians at three universities to create, implement, and evaluate a new curriculum and textbook to teach standard mathematics concepts using math-based decision-making tools for a non-calculus fourth-year mathematics curriculum that several states now require and others may require in the near future. MINDSET has three goals: (1) enhancement of students’ mathematical ability, especially their ability to formulate and solve multi-step problems and interpret results; (2) improvement in students’ attitude toward mathematics, especially those from underrepresented groups, thereby motivating them to study mathematics; and (3) adoption of the curriculum initially in North Carolina and Michigan, then in other states.

Using decision-making tools from Operations Research and Industrial Engineering, we will develop a fourthyear high school curriculum in mathematics and support materials to teach standard content. Through a multi-state, multi-school district assessment, we will determine if a statistically significant improvement in students’ mathematical ability—particularly in multi-step problem solving and interpretation of results—and in motivation and attitude toward mathematics has occurred. Participating teachers will receive professional training, help to create a knowledge-based online community for support, and in-person and online technical assistance. Through extensive data collection and analysis, we will determine if this infrastructure is sustainable and sufficiently flexible to be reproduced and used by others.

A Longitudinal Randomized Trial Study of Middle School Science for English Language Learners (Project MSSELL) (Collaborative Research: Irby)

Project MSSELL will conduct a two-year randomized trial longitudinal evaluation of an enhanced standards-based science curriculum model. In Year 1, the project will refine and pilot the model based on learnings from its previous developmental phase and implementation with K-3 grade students. In Years 2 and 3, the enhanced model will be implemented and studied with fifth- and sixth-grade students.

Partner Organization(s): 
Award Number: 
0822153
Funding Period: 
Mon, 09/01/2008 to Fri, 08/31/2012
Project Evaluator: 
Dr. David Frances

Building an Understanding of Science

Understanding Science provides an accurate portrayal of the nature of science and tools for teaching associated concepts. This project has at its heart a public re-engagement with science that begins with teacher preparation. To this end, its immediate goals are (1) improve teacher understanding of the nature of the scientific enterprise and (2) provide resources and strategies that encourage and enable K-16 teachers to incorporate and reinforce the nature of science throughout their science teaching.

Award Number: 
0624436
Funding Period: 
Mon, 03/12/2007 to Wed, 05/11/2011
Project Evaluator: 
BSCS

Design and Use of Illustrations in Test Items as a Form of Accommodation for English Language Learners in Science and Mathematics Assessment

This project investigates how vignette illustrations minimize the impact of limited English proficiency on student performance in science tests. Different analyses will determine whether and how ELL and non-ELL students differ significantly on the ways they use vignettes to make sense of items; whether the use of vignettes reduces test-score differences due to language factors between ELL and non-ELL students; and whether the level of distance of the items moderates the effectiveness of vignette-illustrated items.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0822362
Funding Period: 
Mon, 09/01/2008 to Sun, 08/12/2012
Full Description: 

This exploratory project within the Contextual Strand (Challenge a) addresses validity in the assessment of science and mathematics for English language learners (ELLs), and the urgent need for effective testing accommodations for ELLs. Motivation for this investigation originated from a previous, NSF-funded project on the testing of ELLs. We observed that items which were accompanied by illustrations tended to be responded correctly by a higher percentage of students than items without illustrations. We will investigate the factors that are relevant to designing and using a new form of accommodation in the assessment of science and mathematics for ELLs—vignette illustrations.

This three-year project will be guided by four research questions: What principles underlie the effective design of science and mathematics test items with illustrations in ways that minimize limited English proficiency as a factor that prevents ELLs from understanding the items? Is the presence of an illustration a moderator in students’ understanding test items? If so, Is the effect due to the simple presence of a graphical component or due to characteristics of the illustrations that are created based on principled design? Does the presence of an illustration have a different effect on the performance of ELLs and the performance of non-ELL students?

We expect to be able to: 1) identify the role of illustrations in the cognitive activities elicited by vignette-illustrated items; 2) determine whether any differences between performance on vignette-illustrated items and other kinds of items are due to the this form of accommodation’s capacity to address language as a construct-irrelevant factor; 3) identify the set of practical and methodological issues that are critical to properly developing and using vignette-illustrated items; and 4) propose a set of documents and procedures for the systematic and cost-effective design and development of vignette-illustrated items. 

We will test ELL and non-ELL students with items of three types (vignette-illustrated items whose illustrations are designed systematically, vignette-illustrated items whose illustrations are created arbitrarily, and items without illustrations) at two levels of distance to the enacted curriculum (close and distal). Diverse forms of analysis will allow us to determine whether and how ELL and non-ELL students differ on the ways in which they use vignettes to make sense of items, whether the use of vignettes reduces test score differences due to language factors between ELL and non-ELL students, and whether the level of distance of the items moderates the effectiveness of vignette-illustrated items.

Intellectual merit. This project will provide information that will help to advance our understanding in two assessment arenas: effective accommodations for ELLs, and item development practices. While illustrations are frequently used in test items, there is not guidance in the assessment development literature on how to approach illustrations. Furthermore, the value of illustrations as a resource for ensuring that ELL students understand what a given item is about and what the item asks them to do has not been systematically investigated. Semiotics, cognitive psychology, and linguistics and socio-cultural theory are brought together to develop systematic procedures for developing illustrations as visual supports in tests. Understanding the role that images play in test taking is relevant to devising more effective ways of testing students. While this project aims to improve testing accommodations practices for ELLs, knowledge gained from it will inform test development practices relevant to all student populations.

Broader impact. We expect outcomes of this project to contribute to enhanced practice in both classroom and large-scale assessment. The push for including ELLs in large-scale testing programs with accountability purposes is not corresponded by effective testing accommodation practices. Many testing accommodations used by national and state assessment programs are not defensible, are not effective, or are improperly implemented. Vignette illustrations have the potential to become a low-cost, easy-to-implement form of testing accommodation for ELLs. Results form this investigation will allow us to identify a set of principles for the proper design and use of vignette illustrations as a form of testing accommodation for ELLs. The project is important not only because it explores the potential of an innovative form of accommodation but because it uses a systematic procedure for designing that form of accommodation.

National Conference on Benchmarking Student Evaluation Practices

This conference uses Student Evaluation Standards, published by the Joint Committee on Standards for Educational Evaluation, to engage a broad array of educational organizations in improving student achievement in STEM education through better evaluation practices that assess for learning. Participants learn more about the Student Evaluation Standards and use them together with a benchmarking process - distributed to them in the form of a toolkit - to enhance student evaluation processes.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0736491
Funding Period: 
Sat, 03/01/2008 to Mon, 02/28/2011

Tool Systems to Support Progress Toward Expert-like Teaching by Early Career Science Educators

The goal of this project is to accelerate the progress of early-career and pre-service science teachers from novice to expert-like pedagogical reasoning and practice by developing and studying a system of discourse tools. The tools are aimed at developing teachers' capabilities in shaping instruction around the most fundamental science ideas; scaffolding student thinking; and adapting instruction to diverse student populations by collecting and analyzing student data on their thinking levels.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0822016
Funding Period: 
Mon, 09/15/2008 to Fri, 08/31/2012
Project Evaluator: 
Jim Minstrell

Pages

Subscribe to Evaluation