Instrument

Identifying and Evaluating Adaptive Expertise in Teachers

This project examines the nature of adaptive expertise in mathematics education, exploring relationships between this concept from cognitive psychology and effective middle school mathematics instruction. One goal of the project is to operationalize adaptive expertise in mathematics classroom using three dimensions: cognitive models of professional competence, instructional practices, and professional learning. Then, researchers seek to determine whether teachers who are more effective at raising student achievement are more or less adaptive.

Lead Organization(s): 
Award Number: 
0732074
Funding Period: 
Sat, 09/01/2007 to Tue, 08/31/2010

Connected Chemistry as Formative Assessment

This project is developing, validating, and evaluating computer modeling-based formative assessments to improve student learning in chemistry. Activities include developing a series of computer models related to key topics in high school chemistry, developing questions to probe student understanding of matter and energy, identifying teaching and learning resources appropriate for different levels of student conceptual understanding, and developing professional development resources on integrating formative assessments into high school chemistry courses.

Project Email: 
Partner Organization(s): 
Award Number: 
0918295
Funding Period: 
Tue, 09/01/2009 to Fri, 08/31/2012
Project Evaluator: 
William Boone

SAVE Science: Situated Assessment Using Virtual Environments for Science Content and Inquiry

The SAVE Science project is creating an innovative system using immersive virtual environments for evaluating learning in science, consistent with research- and policy-based recommendations for science learning focused around the big ideas of science content and inquiry for middle school years. Motivation for this comes not only from best practices as outlined in the National Science Education Standards and AAAS' Project 2061, but also from the declining interest and confidence of today's student in science.

Project Email: 
Award Number: 
1157534
Funding Period: 
Mon, 09/01/2008 to Sat, 08/31/2013
Project Evaluator: 
Anthony Lutkus

Dynamic Geometry in Classrooms

This project is conducting repeated randomized control trials of an approach to high school geometry that utilizes Dynamic Geometry (DG) software and supporting instructional materials to supplement ordinary instructional practices. It compares effects of that intervention with standard instruction that does not make use of computer drawing tools.

Project Email: 
Lead Organization(s): 
Award Number: 
0918744
Funding Period: 
Tue, 09/01/2009 to Sat, 08/31/2013
Project Evaluator: 
Ed Dickey
Full Description: 

The project is conducting repeated randomized control trials of an approach to high school geometry that utilizes dynamic geometry (DG) software and supporting instructional materials to supplement ordinary instructional practices.  It compares effects of that intervention with standard instruction that does not make use of computer drawing/exploraction tools. The basic hypothesis of the study is that use of DG software to engage students in constructing mathematical ideas through experimentation, observation, data recording, conjecturing, conjecture testing, and proof results in better geometry learning for most students. The study tests that hypothesis by assessing student learning in 76 classrooms randomly assigned to treatment and control groups. Student learning is assessed by a geometry standardized test, a conjecturing-proving test, and a measure of student beliefs about the nature of geometry and mathematics in general. Teachers in both treatment and control groups receive relevant professional development, and they are provided with supplementary resource materials for teaching geometry. Fidelity of implementation for the experimental treatment is monitored carefully. Data for answering the several research questions of the study are analyzed by appropriate HLM methods. Results will provide evidence about the effectiveness of DG approach in high school teaching, evidence that can inform school decisions about innovation in that core high school mathematics course. The main research question of the project is: Is the dynamic geometry approach better than the business-as-usual approach in facilitating the geometric learning of our students (and more specifically our economically disadvantaged students) over the course of a full school year?

The main resources/products include geometry teachers’ professional development training materials, suggested dynamic geometry instructional activities to supplement current high school geometry curriculum, instruments such as Conjecturing-Proving Test, Geometry Belief Instrument, Classroom Observation Protocols, DG Implementation Questionnaire and Student Interview Protocols. 

The general plan for the four-year project is as follows:

Year 1: Preparation (All research instruments, professional development training and resource materials, recruitment and training of participants, etc.); 

Year 2: The first implementation of the dynamic geometry treatment, and related data collection and initial data analysis; 

Year 3: The second implementation of the DG treatment, and related data collection and data analysis; 

Year 4: Careful and detailed data analysis and reporting.

We are now in project year 3. Data are collected for the second implementation of the DG treatment. For data collected during project year 2, some initial analysis (the analysis on the geometry pretest and posttest data and the psychometric analysis on the project developed instruments) has been conducted. More thorough analysis of the collected data is still on going. The analysis on the geometry test shows that the experimental group significantly outperformed the control group on geometry performance.

The evaluation will be implemented throughout the project’s four-year duration, with an evolving balance of formative and summative evaluation activities.  In the project’s first three years, the evaluation will emphasize formative functions, designed to inform the project research team of the relative strengths and weaknesses of the research design and execution, and target corrections and improvements of the research components. Summative evaluation activities will also take place in these years with the collection of data on student achievement and teacher change. Evaluation activities for year 4 will focus on the summative evaluation of the project’s accomplishment and especially its impact on participating teachers and students. Evaluation reports will be issued annually with a final summative report presented at the end of year 4.

The research results will be disseminated via the following efforts: 1) Creating and constantly updating the project web site; 2) Publishing the related research articles in research journals such as Journal for Research in Mathematics Education; 3) Presenting at state, regional, national, and international research and professional meetings; 4) Meeting with state and local education agencies, schools, and mathematics teacher educators at other universities for presenting the research findings and using the DG approach in more schools and more mathematics teacher education programs; and 5) Contacting more school districts, with a view to developing relationships and ties that would smooth the way to disseminate the research results.

Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model

This project focuses on critical needs in the preparation and long-term development of pre-service, undergraduate, K-6 teachers of science. The project investigates the impact on these students of undergraduate, standards-based, reform entry level science courses developed by faculty based on their participation in the NASA Opportunities for Visionary Academics processional development program to identify: short-term impacts on undergraduate students and long-term effects on graduated teachers; characteristics of reform courses and characteristics of effective development efforts.

Project Email: 
Lead Organization(s): 
Award Number: 
0554594
Funding Period: 
Tue, 08/01/2006 to Sun, 07/31/2011
Full Description: 

The Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model project is informally known as the National Study of Education in Undergraduate Science (NSEUS). This 5-year project focuses on critical needs in the preparation and long-term development of pre-service, undergraduate, K-6 teachers of science. The goal is to investigate the impact on these students of undergraduate, standards-based, reform entry-level science courses developed by faculty in the NASA Opportunities for Visionary Academics (NOVA) professional development model. Twenty reform and 20 comparison undergraduate science courses from a national population of 101 diverse institutions participating in NOVA, stratified by institutional type, were be selected and compared in a professional development impact design model. Data is being collected in extended on-site visits using multiple quantitative and qualitative instruments and analyzed using comparative and relational studies at multiple points in the impact design model. Criteria for success of the project will be determined by conclusions drawn from the research questions; including evidence and effect sizes of short-term impacts on undergraduate students and long-term effects on graduated in-service teachers in their own classroom science teaching; identification of characteristics of undergraduate reformed courses that produce significant impacts; identification of characteristics of effective faculty, and effective dissemination.

Project Publications and Presentations:

Lardy, Corrine; Mason, Cheryl; Mojgan, Matloob-Haghanikar; Sunal, Cynthia Szymanski; Sunal, Dennis Wayne; Sundberg, Cheryl & Zollman, Dean (2009). How Are We Reforming Teaching in Undergraduate Science Courses? Journal of College Science Teaching, v. 39 (2), 12-14.  

Making Sciences: Data Modeling and Argumentation in Elementary Science

This project develops ecosystems-focused instructional materials that use sensor data and technology to help second and third graders become more proficient at data modeling and scientific argumentation. The goals are to provide elementary teachers with a research-based curriculum that engages students in exploring and visualizing environmental data and using the data to construct scientific arguments, and to contribute to the cognitive development literature on children's ideas about and abilities for scientific argumentation.

Award Number: 
0733233
Funding Period: 
Wed, 08/15/2007 to Sat, 07/31/2010

Visualizing to Integrate Science Understanding for All Learners (VISUAL)

This project is exploring how curricula and assessment using dynamic, interactive scientific visualizations of complex phenomena can ensure that all students learn significant science content. Dynamic visualizations provide an alternative pathway for students to understand science concepts, which can be exploited to increase the accessibility of a range of important science concepts. Computer technologies offer unprecedented opportunities to design curricula and assessments using visual technologies and to explore them in research, teaching, and learning.

Award Number: 
0918743
Funding Period: 
Tue, 09/01/2009 to Fri, 08/31/2012
Project Evaluator: 
Paul Holland

Cumulative Learning using Embedded Assessment Results (CLEAR)

This project focuses on the challenge of using assessment of relevant STEM content to improve K-12 teaching and learning. CLEAR takes advantage of new technologies and research findings to investigate ways that science assessments can both capture and contribute to cumulative, integrated learning of standards-based concepts in middle school courses. The project will research new forms of assessment that document students' accumulation of knowledge and also serve as learning events.

Award Number: 
0822388
Funding Period: 
Mon, 09/15/2008 to Fri, 08/31/2012
Project Evaluator: 
Paul Holland
Full Description: 

The CLEAR project takes advantage of new technologies and research findings to investigate ways that science assessments can both capture and contribute to cumulative, integrated learning of standards-based concepts in middle school courses.

Our research investigates how instructional activities can help middle school students develop a cumulative, integrated understanding of energy. Energy is a unifying scientific concept that has been shown to be difficult to learn due to its complexity and abstract nature.

Design and Use of Illustrations in Test Items as a Form of Accommodation for English Language Learners in Science and Mathematics Assessment

This project investigates how vignette illustrations minimize the impact of limited English proficiency on student performance in science tests. Different analyses will determine whether and how ELL and non-ELL students differ significantly on the ways they use vignettes to make sense of items; whether the use of vignettes reduces test-score differences due to language factors between ELL and non-ELL students; and whether the level of distance of the items moderates the effectiveness of vignette-illustrated items.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0822362
Funding Period: 
Mon, 09/01/2008 to Sun, 08/12/2012
Full Description: 

This exploratory project within the Contextual Strand (Challenge a) addresses validity in the assessment of science and mathematics for English language learners (ELLs), and the urgent need for effective testing accommodations for ELLs. Motivation for this investigation originated from a previous, NSF-funded project on the testing of ELLs. We observed that items which were accompanied by illustrations tended to be responded correctly by a higher percentage of students than items without illustrations. We will investigate the factors that are relevant to designing and using a new form of accommodation in the assessment of science and mathematics for ELLs—vignette illustrations.

This three-year project will be guided by four research questions: What principles underlie the effective design of science and mathematics test items with illustrations in ways that minimize limited English proficiency as a factor that prevents ELLs from understanding the items? Is the presence of an illustration a moderator in students’ understanding test items? If so, Is the effect due to the simple presence of a graphical component or due to characteristics of the illustrations that are created based on principled design? Does the presence of an illustration have a different effect on the performance of ELLs and the performance of non-ELL students?

We expect to be able to: 1) identify the role of illustrations in the cognitive activities elicited by vignette-illustrated items; 2) determine whether any differences between performance on vignette-illustrated items and other kinds of items are due to the this form of accommodation’s capacity to address language as a construct-irrelevant factor; 3) identify the set of practical and methodological issues that are critical to properly developing and using vignette-illustrated items; and 4) propose a set of documents and procedures for the systematic and cost-effective design and development of vignette-illustrated items. 

We will test ELL and non-ELL students with items of three types (vignette-illustrated items whose illustrations are designed systematically, vignette-illustrated items whose illustrations are created arbitrarily, and items without illustrations) at two levels of distance to the enacted curriculum (close and distal). Diverse forms of analysis will allow us to determine whether and how ELL and non-ELL students differ on the ways in which they use vignettes to make sense of items, whether the use of vignettes reduces test score differences due to language factors between ELL and non-ELL students, and whether the level of distance of the items moderates the effectiveness of vignette-illustrated items.

Intellectual merit. This project will provide information that will help to advance our understanding in two assessment arenas: effective accommodations for ELLs, and item development practices. While illustrations are frequently used in test items, there is not guidance in the assessment development literature on how to approach illustrations. Furthermore, the value of illustrations as a resource for ensuring that ELL students understand what a given item is about and what the item asks them to do has not been systematically investigated. Semiotics, cognitive psychology, and linguistics and socio-cultural theory are brought together to develop systematic procedures for developing illustrations as visual supports in tests. Understanding the role that images play in test taking is relevant to devising more effective ways of testing students. While this project aims to improve testing accommodations practices for ELLs, knowledge gained from it will inform test development practices relevant to all student populations.

Broader impact. We expect outcomes of this project to contribute to enhanced practice in both classroom and large-scale assessment. The push for including ELLs in large-scale testing programs with accountability purposes is not corresponded by effective testing accommodation practices. Many testing accommodations used by national and state assessment programs are not defensible, are not effective, or are improperly implemented. Vignette illustrations have the potential to become a low-cost, easy-to-implement form of testing accommodation for ELLs. Results form this investigation will allow us to identify a set of principles for the proper design and use of vignette illustrations as a form of testing accommodation for ELLs. The project is important not only because it explores the potential of an innovative form of accommodation but because it uses a systematic procedure for designing that form of accommodation.

Applying Research on Science Materials Implementation: Bringing Measurement of Fidelity of Implementation (FOI) to Scale

This project is creating a suite of instruments for measuring fidelity of implementation of several science and mathematics instructional materials programs and a User's Guide for customizing those instruments to other programs. The instruments are grounded in a shared conceptual framework that organizes "critical components" that the programs share. The suite was piloted and field tested in over 50 schools in Chicago.
Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0628052
Funding Period: 
Mon, 01/01/2007 to Wed, 06/30/2010

Pages

Subscribe to Instrument