Instrument

Engineering for All (EfA)

This project creates, tests and revises two-six week prototypical modules for middle school technology education classes, using the unifying themes and important social contexts of food and water. The modules employ engineering design as the core pedagogy and integrate content and practices from the standards for college and career readiness.

Lead Organization(s): 
Award Number: 
1316601
Funding Period: 
Sun, 09/15/2013 to Wed, 08/31/2016
Full Description: 

The Engineering for All project creates, tests and revises two-six week prototypical modules for middle school technology education classes, using the unifying themes and important social contexts of food and water. The modules employ engineering design as the core pedagogy and integrate content and practices from the standards for college and career readiness. Embedded assessments are developed and tested to make student learning visible to both teachers and students. Professional development for a limited group of teachers is used to increase their knowledge of engineering design and to test instruments being developed to measure (a) student and teacher capacity to employ informed design practices and (b) teacher design pedagogical content knowledge.

The project leadership is experienced at creating materials for engineering and technology and in providing professional development for teachers. The assessments and instruments are created by educational researchers. The advisory board includes engineers, science and engineering educators, and educational researchers to guide the development of the modules, the assessments and the instruments. An external evaluator reviews the protocols and their implementation.

This project has the potential to provide exemplary materials and assessments for engineering/technology education that address standards, change teacher practice, and increase the capacity of the engineering/technology education community to do research.

Common Online Data Analysis Platform (CODAP)

This project aims to engage students in meaningful scientific data collection, analysis, visualization, modeling, and interpretation. It targets grades 9-12 science instruction. The proposed research poses the question "How do learners conceive of and interact with empirical data, particularly when it has a hierarchical structure in which parameters and results are at one level and raw data at another?"

Lead Organization(s): 
Award Number: 
1435470
Funding Period: 
Tue, 10/01/2013 to Fri, 09/30/2016
Full Description: 

This project aims to engage students in meaningful scientific data collection, analysis, visualization, modeling, and interpretation. It targets grades 9-12 science instruction. The proposed research poses the question "How do learners conceive of and interact with empirical data, particularly when it has a hierarchical structure in which parameters and results are at one level and raw data at another?" As working with data becomes an integral part of students' learning across STEM curricula, understanding how students conceive of data grows ever more important. This is particularly timely as science becomes more and more data driven.

The team will develop and test a Common Online Data Analysis Platform (CODAP). STEM curriculum development has moved online, but development of tools for students to engage in data analysis has yet to follow suit. As a result, online curriculum development projects are often forced to develop their own data analysis tools, settle for desktop tools, or do without. In a collaboration with NSF-funded projects at the Concord Consortium, Educational Development Center, and University of Minnesota, the project team is developing an online, open source data analysis platform that can be used not only by these three projects, but subsequently by others.

The proposed research breaks new ground both in questions to be investigated and in methodology. The investigations build on prior research on students' understanding of data representation, measures of center and spread, and data modeling to look more closely at students' understanding of data structures especially as they appear in real scientific situations. Collaborative design based on three disparate STEM projects will yield a flexible data analysis environment that can be adopted by additional projects in subsequent years. Such a design process increases the likelihood that CODAP will be more than a stand-alone tool, and can be meaningfully integrated into online curricula. CODAP's overarching goal is to improve the preparation of students to fully participate in an increasingly data-driven society. It proposes to do so by improving a critical piece of infrastructure: namely, access to classroom-friendly data analysis tools by curriculum developers who wish to integrate student engagement with data into content learning.

This project is asociated with award number 1316728 with the same title.

Cross-National Comparison of School and District Supports for High-Quality Mathematics Instruction in the US and China

This RAPID project is a cross-national comparative study of U.S. and Chinese instructional support systems, building from earlier data about mathematics teaching and learning in large urban school districts of both the United States and the People's Republic of China. The study uses quantitative methods to compare and contrast the effectiveness of supports (e.g., professional development, teacher networks, school leadership) in improving teachers' instructional practices and student achievement using comparable instrumentation.

Lead Organization(s): 
Award Number: 
1321828
Funding Period: 
Sun, 09/15/2013 to Sun, 08/31/2014
Full Description: 

Since the publication of the result that students from Shanghai, China, outperformed students from all other participating countries on the 2009 Programme for International Student Assessment (PISA) in mathematics, researchers have sought to understand why Chinese mathematics education appears to be both more successful at boosting student learning and more equitably distributed. This RAPID project is a cross-national comparative study of U.S. and Chinese instructional support systems, building from earlier data about mathematics teaching and learning in large urban school districts of both the United States and the People's Republic of China. The work is being conducted by researchers from Vanderbilt University, Virginia Polytechnic Institute and State University and Beijing Normal University. The study uses quantitative methods to compare and contrast the effectiveness of supports (e.g., professional development, teacher networks, school leadership) in improving teachers' instructional practices and student achievement using comparable instrumentation.

The study contributes to research and policy in several ways. First, it is helping to identify supports that have been particularly effective in improving mathematics teaching and learning in China. This should inform current theories about how to best support mathematics education in the United States. Second, the cross-nationally validated instruments used to collect the data can be used by other researchers investigating curricular reform implementation cross-nationally. The findings of this study are especially relevant to district leaders as they develop support and accountability systems to effectively implement the content and practice standards of the Common Core State Standards for Mathematics.

This award is co-funded by NSF's International Science and Engineering Section, Office of International and Integrative Activities.

Misconceptions Oriented Standards-Based Assessment Resource for Teachers of High School Life Science (MOSART HSLS)

This project is developing and validating an assessment instrument that addresses the life sciences for students and teachers in grades 9 through 12 based on the Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART).

Lead Organization(s): 
Award Number: 
1316645
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

Researchers in the Harvard Smithsonian Center for Astrophysics are developing and validating an assessment instrument that addresses the life sciences for students and teachers in grades 9 through 12 based on the Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART). The project is developing 400 new test items that are based on core content domains for life science and are aligning these items with the previous National Science Education Standards to provide a connection to the earlier MOSART assessments. The project is also developing and validating two test instruments that address the cross cutting concepts of energy and matter for grades K-12, with a specific focus on flows, cycles and conservation. The new assessments will be made available to other researchers and practitioner through the project website and their on-line assessment system.

The assessment development is based on the process used in prior work that has produced the other MOSART instruments, including design efforts of assessment specialists, teachers, and learning scientists. Pilot items are tested through crowd-sourcing with online adult test takers. Classic test theory techniques, item response theory and Bayesian techniques model the student responses. Outcomes consist of item parameters, test and sub-test characteristics, and predictive linkages among items. A stratified, nationally representative sample of 250 high school biology teachers field test the items with classrooms of students. Descriptive statistics are generated to establish the state of student knowledge, pre-and post-test performance by item and by standard, and teacher knowledge, including the fraction of items for which teachers have correctly identified the most popular wrong answer. Descriptive analyses are followed by hierarchical linear modeling (HLM) of students within classrooms to examine the relationships between student and teacher knowledge. The dependent variables in HLM are student gain scores. Independent variables include teachers' knowledge, and student performance on grade K-8 assessments.

The MOSART instruments have been a strong line of assessment tools that are based on a model of cognition with a strong research base in misconceptions in science education. That research base is only slowly being augmented with a more coherent framework on learning progressions in STEM education, and the MOSART instruments will have the potential for extensive use for the foreseeable future. The grades 9-12 life science instrument based on coupling core ideas with science and engineering practices addresses the gaps in the current MOSART system of assessments. Given the rich literature on misconceptions in life science and the ubiquity of life science as a course at the high school level, the instrument promises to be as useful as the one for K-8 developed with MSP RETA funding. The new instruments on cross-cutting concepts provides a much needed set of assessments for researchers and practitioners, particularly teacher professional development providers. The transition to coupling core content and sciences practices with both the life sciences and the cross-cutting concepts is an opportunity to expand and update the suite of instruments.

Improving Competency in Elementary Science Teaching

This project provides elementary teachers, grades 3-5 with a pedagogical framework and related resources for distinguishing quality science teaching. The study focuses on developing and testing a framework, the Quality Science Teaching Continuum (QSTC), to determine its capacity to serve as a potent formative and collaborative tool with which teachers can reflect on their science teaching practices and recognize student behaviors that are indicators of engagement and science learning.

Lead Organization(s): 
Award Number: 
1317068
Funding Period: 
Mon, 07/01/2013 to Tue, 06/30/2015
Full Description: 

This Stanford University project provides elementary teachers, grades 3-5 with a pedagogical framework and related resources for distinguishing quality science teaching. The study focuses on developing and testing a framework, the Quality Science Teaching Continuum (QSTC), to determine its capacity to serve as a potent formative and collaborative tool with which teachers can reflect on their science teaching practices and recognize student behaviors that are indicators of engagement and science learning. The project includes an intensive professional development (PD) that will accompany the instrument designed to develop teachers' understanding of (1) pedagogy, (2) science process and content, (3) community building, and (4) use of QSTC to improve classroom instruction and student engagement.

Teachers will be videotaped during classroom science instruction at various points in the two-year process, and the resulting digital library of teaching videos provides an ongoing reference resource for teachers and others when reflecting on their practice. The project provides a proof of concept and examines the use of a specific, formative, integrative instrument, the QSTC, within an immersive teacher professional development program.

Language-Rich Inquiry Science with English Language Learners Through Biotechnology (LISELL-B)

This is a large-scale, cross-sectional, and longitudinal study aimed at understanding and supporting the teaching of science and engineering practices and academic language development of middle and high school students (grades 7-10) with a special emphasis on English language learners (ELLs) and a focus on biotechnology.

Award Number: 
1316398
Funding Period: 
Thu, 08/01/2013 to Tue, 07/31/2018
Full Description: 

This is a large-scale (4,000 students, 32 teachers, 5 classes per teacher per year); cross-sectional (four grade levels); and longitudinal (three years) study aimed at understanding and supporting the teaching of science and engineering practices and academic language development of middle and high school students (grades 7-10) with a special emphasis on English language learners (ELLs) and a focus on biotechnology. It builds on and extends the pedagogical model, professional development framework, and assessment instruments developed in a prior NSF-funded exploratory project with middle school teachers. The model is based on the research-supported notion that science and engineering practices and academic language practices are synergistic and should be taught simultaneously. It is framed around four key learning contexts: (a) a teacher professional learning institute; (b) rounds of classroom observations; (c) steps-to-college workshops for teachers, students, and families; and (d) teacher scoring sessions to analyze students' responses to assessment instruments.

The setting of this project consists of four purposefully selected middle schools and four high schools (six treatment and two control schools) in two Georgia school districts. The study employs a mixed-methods approach to answer three research questions: (1) Does increased teacher participation with the model and professional development over multiple years enhance the teachers' effectiveness in promoting growth in their students' understanding of scientific practices and use of academic language?; (2) Does increased student participation with the model over multiple years enhance their understanding of science practices and academic language?; and (3) Is science instruction informed by the pedagogical model more effective than regular instruction in promoting ELLs' understanding of science practices and academic language at all grade levels? Data gathering strategies include: (a) student-constructed response assessment of science and engineering practices; (b) student-constructed response assessment of academic language use; (c) teacher focus group interview protocol; (d) student-parent family interview protocol; (e) classroom observation protocol; (f) teacher pedagogical content knowledge assessment; and (g) teacher log of engagement with the pedagogical model. Quantitative data analysis to answer the first research question includes targeted sampling and longitudinal analysis of pretest and posttest scores. Longitudinal analysis is used to answer the second research question as well; whereas the third research question is addressed employing cross-sectional analysis. Qualitative data analysis includes coding of transcripts, thematic analysis, and pattern definition.

Outcomes are: (a) a research-based and field-tested prototype of a pedagogical model and professional learning framework to support the teaching of science and engineering practices to ELLs; (b) curriculum materials for middle and high school science teachers, students, and parents; (c) a teacher professional development handbook; and (d) a set of valid and reliable assessment instruments usable in similar learning environments.

Secondary Science Teaching with English Language and Literacy Acquisition (SSTELLA)

This is a four-year project to develop, implement, and study an experimental model of secondary science pre-service teacher education designed to prepare novice school teachers to provide effective science instruction to English language learners (ELLs). The project incorporates the principles underlying the Next Generation Science Standards with a focus on promoting students' scientific sense-making, comprehension and communication of scientific discourse, and productive use of language.

Award Number: 
1316834
Funding Period: 
Thu, 08/01/2013 to Tue, 07/31/2018
Full Description: 

This is a four-year Discovery Research K-12 project to develop, implement, and study an experimental model of secondary science pre-service teacher education designed to prepare novice school teachers to provide effective science instruction to English language learners (ELLs). The project incorporates the principles underlying the Next Generation Science Standards with a focus on promoting students' scientific sense-making, comprehension and communication of scientific discourse, and productive use of language. It articulates theory and practice related to the teaching of science content and the development of English language and literacy, and provides teachers with models of integrated practice in video cases and curriculum units. To test the efficacy of the study, a longitudinal, mixed-methods, quasi-experimental study is conducted at four institutions: the University of California-Santa Cruz, Arizona State University, the University of Arizona, and the University of Texas at San Antonio.

The three research questions are: (1) What is the impact of the project's pre-service teacher education program on novice secondary science teachers' knowledge, beliefs, and practice from the pre-service program into the second year of teaching?; (2) What is the relationship between science method instructors' fidelity of implementation of the project's practices and novice teachers' outcomes (knowledge, beliefs, and practice)?; and (3) What is the relationship between novice teachers' implementation of project-promoted practices and their students' learning? To answer these questions, the project collects and analyzes quantitative and qualitative data on novice teachers (85 treatment group and 85 control group) over three years utilizing surveys, interviews, observations, and student assessment instruments. Teachers' beliefs and knowledge about teaching science to ELLs are measured using the project-developed Science Teaching Survey, which provides quantitative scores based on a Likert-type scale, and the science teacher interview protocol to provide qualitative data, including the contextual factors affecting implementation of project-promoted practices. Classroom observations are captured through qualitative field notes and the Classroom Observation Rubric--a systematic project-developed observation instrument that measures implementation of the practices. Student learning outcomes are measured using (a) the Woodcock-Muñoz Language Survey (students' proficiency at applying listening, reading, writing, and comprehension abilities); (b) the Literacy in Science Assessment (students' productive use of language in authentic science literacy tasks); (c) the Scientific Sense-Making Assessment (how students make sense of core science ideas through scientific and engineering practices); and (d) appropriate state standardized assessments. In addition, the Opportunity to Learn Survey gauges students' perceptions of implementation of literacy integration, motivation in class, and identity as readers.

Project outcomes are: (a) a research-based and field-tested model for pre-service secondary science teacher education, including resources for science methods courses instructors and pre-service teachers; and (b) valid and reliable instrumentation usable in similar research and development environments.


Project Videos

2019 STEM for All Video Showcase

Title: Preparing Science Teachers to Support English Learners

Presenter(s): Edward Lyon


Piloting Graph Literacy Activities in Maine

The goal of this project is to develop and pilot test a limited number of free computer-based instructional activities that improve student graph comprehension, aimed especially at science students in grades 7 and 8. Because of growing interest in use of online resources for teaching and learning, this work is potentially transformative for a wide range of audiences, including teachers, students, researchers, and the developers and publishers of instructional materials across vSTEM areas and grades.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1256490
Funding Period: 
Mon, 10/01/2012 to Tue, 09/30/2014
Full Description: 

The goal of this project is to develop and pilot test a limited number of free computer-based instructional activities that improve student graph comprehension, aimed especially at science students in grades 7 and 8. In addition, the project is developing a pilot assessment instrument focusing on students' comprehension of graphs ("graph literacy"). The activities and the assessment instrument are being pilot tested in Maine, a rural state where family income is below the national average and students are underrepresented in studying STEM topics after high school. The state has identified this topic as an important one to focus on in the coming year.

Graph literacy is the ability to identify the important features of a wide variety of graphs and relate those features to the context of the graphs. This increases the students' understanding not only of how to interpret graphs, but also of the science content. This definition of graph literacy, while based in the math and science standards, goes beyond skills tested by many assessments of graph knowledge because they focus primarily on reading points off a graph, typically a type of graph that students have studied and are familiar with. While broadening the usual definition for graph skills, the project focuses on scatter and line graphs of the type encountered in many mathematics and science courses in grades 7-12, as well as in newspapers and magazines.

Graphs are central to STEM learning in many subjects and at almost all education levels. In spite of the vital role of graphs, students at all ages demonstrate difficulties using and interpreting graphs. The computer-based Graph Literacy activities being developed are based on extensive prior research about students' use and understanding of graphs, as well as continuing advances in delivering education activities through dynamic, interactive Web pages that do not require schools to install any software. Based on the research literature, there is a consensus that students need to be taught graph literacy in three steps: identifying and encoding the important superficial features of a graph they want to understand, such as the titles, units, and axis labels; linking visual features of that graph to mathematical relationships, based on recurring patterns (e.g., linear increase or decrease); and, integrating all of these features with the context of the graph. The activities we are developing are based on this approach, as are the validated assessments being developed to measure students' graph literacy.

The project is conducting a small, randomized experimental trial of the graph literacy activities in year 2 of the project. The goal of is to determine the effectiveness of the graph literacy activities in improving students' understanding of graphs. The open source software and approaches developed under the prior grant contribute directly to the likely success of this project. Because of growing interest in use of online resources for teaching and learning, this work is potentially transformative for a wide range of audiences, including teachers, students, researchers, and the developers and publishers of instructional materials across all STEM areas and grades. The underlying software technology for Graph Literacy is being made available as open source computer code, and any activities that use the code are released under a creative commons license. As a result, the graph literacy activities, and the pilot assessment instrument, can be widely adopted at no cost.

FUN: A Finland US Network for Engagement and STEM Learning in Games

As part of a SAVI, researchers from the U.S. and from Finland will collaborate on investigating the relationships between engagement and learning in STEM transmedia games. The project involves two intensive, 5 day workshops to identify new measurement instruments to be integrated into each other's research and development work. The major research question is to what degree learners in the two cultures respond similarly or differently to the STEM learning games.

Lead Organization(s): 
Award Number: 
1252709
Funding Period: 
Mon, 10/01/2012 to Tue, 09/30/2014
Full Description: 

As part of a SAVI, researchers from the U.S. and from Finland will collaborate on investigating the relationships between engagement and learning in STEM transmedia games. The members of U.S. Team for this project come from TERC, WGBH and Northern Illinois University. The project involves two intensive, 5 day workshops to identify new measurement instruments to be integrated into each other's research and development work. The major research question is to what degree learners in the two cultures respond similarly or differently to the STEM learning games.

Assessing, Validating, and Developing Content Knowledge for Teaching Energy (Collaborative Research: Gitomer)

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. The project uses a framework that includes tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222777
Funding Period: 
Sat, 09/01/2012 to Thu, 08/31/2017
Full Description: 

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. There is significant research that indicates that teacher content knowledge differs from what people in other professions need to know about particular domains such as mathematics, and the development of a Content Knowledge of Teaching Energy in mechanics is an extension of those research and development efforts. The project embeds the development of the instrument in a program of measuring effective teaching of physics in the classroom and develops a strong validity argument for the resulting assessment based on its use as a measure in a professional development project that intends to improve teachers' understanding of energy in physics. The research team consists of experts in physics, assessment and classroom teaching of physics. The collaborative project includes researchers at Rutgers, University of Maine, Seattle Pacific University, Facets Innovation, and the Educational Testing Service.

The project uses a framework for effective teaching developed in the Measures of Effective Teaching project funded by the Gates Foundation to construct a theoretical framework for the teaching of mechanical energy. That framework includes items and tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning. A strong framework of validation based on multiple lines of evidence of the relationship between the items developed for the study and observations, analysis of video, and artifacts from the classroom is one element of the study. Another element of the study examines multiple psychometric lines of evidence to determine the reliability of the instruments and the validity of the inferences drawn from them. The resulting instruments will be used in the measurement of changes of teacher content knowledge for teaching in professional development programs as another source of validation.

The improvement of teachers' content knowledge for teaching is an important intermediary goal of professional development of teachers. Without adequate understanding of the gaps in teacher knowledge and precise evidence of the improvement through professional development, the efficacy of different professional development projects is not possible. This project develops a model of teacher assessment instrument development that addresses a cross-cutting theme in the Next Generation Science Standards and contributes an important tool to the research and evaluation processes that are needed to make those standards a reality in the classroom. Findings from the use of the instruments across multiple projects inform policy decisions on local, state and federal levels.

Pages

Subscribe to Instrument