Assessment

Piloting Graph Literacy Activities in Maine

The goal of this project is to develop and pilot test a limited number of free computer-based instructional activities that improve student graph comprehension, aimed especially at science students in grades 7 and 8. Because of growing interest in use of online resources for teaching and learning, this work is potentially transformative for a wide range of audiences, including teachers, students, researchers, and the developers and publishers of instructional materials across vSTEM areas and grades.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1256490
Funding Period: 
Mon, 10/01/2012 to Tue, 09/30/2014
Full Description: 

The goal of this project is to develop and pilot test a limited number of free computer-based instructional activities that improve student graph comprehension, aimed especially at science students in grades 7 and 8. In addition, the project is developing a pilot assessment instrument focusing on students' comprehension of graphs ("graph literacy"). The activities and the assessment instrument are being pilot tested in Maine, a rural state where family income is below the national average and students are underrepresented in studying STEM topics after high school. The state has identified this topic as an important one to focus on in the coming year.

Graph literacy is the ability to identify the important features of a wide variety of graphs and relate those features to the context of the graphs. This increases the students' understanding not only of how to interpret graphs, but also of the science content. This definition of graph literacy, while based in the math and science standards, goes beyond skills tested by many assessments of graph knowledge because they focus primarily on reading points off a graph, typically a type of graph that students have studied and are familiar with. While broadening the usual definition for graph skills, the project focuses on scatter and line graphs of the type encountered in many mathematics and science courses in grades 7-12, as well as in newspapers and magazines.

Graphs are central to STEM learning in many subjects and at almost all education levels. In spite of the vital role of graphs, students at all ages demonstrate difficulties using and interpreting graphs. The computer-based Graph Literacy activities being developed are based on extensive prior research about students' use and understanding of graphs, as well as continuing advances in delivering education activities through dynamic, interactive Web pages that do not require schools to install any software. Based on the research literature, there is a consensus that students need to be taught graph literacy in three steps: identifying and encoding the important superficial features of a graph they want to understand, such as the titles, units, and axis labels; linking visual features of that graph to mathematical relationships, based on recurring patterns (e.g., linear increase or decrease); and, integrating all of these features with the context of the graph. The activities we are developing are based on this approach, as are the validated assessments being developed to measure students' graph literacy.

The project is conducting a small, randomized experimental trial of the graph literacy activities in year 2 of the project. The goal of is to determine the effectiveness of the graph literacy activities in improving students' understanding of graphs. The open source software and approaches developed under the prior grant contribute directly to the likely success of this project. Because of growing interest in use of online resources for teaching and learning, this work is potentially transformative for a wide range of audiences, including teachers, students, researchers, and the developers and publishers of instructional materials across all STEM areas and grades. The underlying software technology for Graph Literacy is being made available as open source computer code, and any activities that use the code are released under a creative commons license. As a result, the graph literacy activities, and the pilot assessment instrument, can be widely adopted at no cost.

Sensing Science: Temperature and Heat Readiness for Early Elementary Students

Concord Consortium is exploring K-2 students' understanding of heat and temperature in two Massachusetts school districts using sensors that display temperatures as colors. Exploration activities are being created, and students are being videotaped carrying out the activities. Students complete a short assessment for each activity. The exploration activities, assessments, and project data are available via open source through a website at Concord Consortium and are being presented to multiple professional audiences.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222892
Funding Period: 
Mon, 10/01/2012 to Tue, 09/30/2014
Full Description: 

Concord Consortium is exploring K-2 students' understanding of heat and temperature in two Massachusetts school districts using sensors that display temperatures as colors. The project is investigating the following research questions:

- How can visualizations, data collection linked to everyday experiences, and student reflection integrated into creative exploration address student preconceptions and promote K-2 student understanding of temperature and heat?

- How can the use of replay of video and reflection aid in addressing K-2 student preconceptions and improve student understanding of heat and temperature?

- Can the use of visualizations and data collection via digital sensing technology advance K-2 students past the goals of the K-2 science frameworks?

The project is being implemented in 10 classrooms for about 250 children representing diverse populations. Exploration activities are being created, and students are being videotaped carrying out the activities. Students complete a short assessment for each activity. Project staff will revisit the videos with the students to explore student concepts at a deeper level. David Reider of Education Design Inc. is conducting the evaluation which will focus on 1) program efficacy and design and 2) alignment with research design. It is formative in design with annual summative reports. From their data, the project is constructing a progressive hierarchy of student theories of heat and temperature. The project is also producing a protocol that teachers can use to have better dialogues with children that support children's reconstruction of their initial conceptions. The exploration activities, assessments, and project data are available via open source through a website at Concord Consortium and are being presented to multiple professional audiences.

Evaluation of the Sustainability and Effectiveness of Inquiry-Based Advanced Placement Science Courses: Evidence From an In-Depth Formative Evaluation and Randomized Controlled Study

This study examines the impact of the newly revised Advanced Placement (AP) Biology and Chemistry courses on students' understanding of and ability to utilize scientific inquiry, on students' confidence in engaging in college-level material, and on students’ enrollment and persistence in college STEM majors. The project provides estimates of the impact of students' AP-course taking on their progress into postsecondary educational experiences and their intent to continue to prepare to be future engineers and scientists.

Award Number: 
1220092
Funding Period: 
Sat, 09/15/2012 to Wed, 08/31/2016
Full Description: 

This study examines the impact of the newly revised Advanced Placement (AP) Biology and Chemistry courses on students' understanding of and ability to apply scientific inquiry, on students' confidence in successfully engaging in college-level material, and on students enrollment and persistence in college STEM majors. AP Biology and Chemistry courses represent an important educational program that operates at a large scale across the country. The extent to which the AP curricula vary in implementation across the schools in the study is also examined to determine the range of students' opportunity to learn the disciplinary content and the knowledge and skills necessary to engage in inquiry in science. Schools that are newly implementing AP courses are participants in this research and the challenges and successes that they experience are also a component of the research plan. Researchers at the University of Washington, George Washington University and SRI International are conducting the study.

The research design for this study includes both formative components and a randomized control experiment. Formative elements include observations, interviews and surveys of teachers and students in the AP courses studied. The experimental design includes the random assignment of students to the AP offered and follows the performances of the treatment and control students in two cohorts into their matriculation into postsecondary educational experiences. Surveys measure students' experiences in the AP courses, their motivations to study AP science, the level of stress they experience in their high school coursework and their scientific inquiry skills and depth of disciplinary knowledge. The study examines the majors chosen by those students who enter into colleges and universities to ascertain the extent to which they continue in science and engineering.

This project informs educators about the challenges and successes schools encounter when they expand access to AP courses. The experiences of the teachers who will be teaching students with variable preparation inform future needs for professional development and support. The project provides estimates of the impact of students' AP-course taking on their progress into postsecondary educational experiences and their intent to continue to prepare to be future engineers and scientists. It informs policy efforts to improve the access to more rigorous advanced courses in STEM and provides strong experimental evidence of the impact of AP course taking. The project has the potential to demonstrate to educational researchers how to study an educational program that operates at scale.

Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Sword)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Award Number: 
1222426
Funding Period: 
Wed, 08/15/2012 to Sun, 07/31/2016
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Matsuura)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222340
Funding Period: 
Wed, 08/15/2012 to Tue, 07/31/2018
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Supporting the Emergence of a Professional Teaching Community Through Collective Knowledge-Building in Assessment and Feedback of Mathematical Thinking (Collaborative Research: Brandt)

This collaborative project is developing an online, professional teaching community that addresses issues of assessment in mathematics classes. The developers are building on the success of the NSF-supported Math Forum's Problem of the Week program to create a community that works to increase students' mathematics learning by helping teachers stimulate student thinking, assess that thinking, and provide useful feedback to students.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1221351
Funding Period: 
Sat, 09/01/2012 to Fri, 08/31/2018
Full Description: 

This collaborative work involves Drexel University and Temple University where they are developing an online, professional teaching community that is addressing issues of assessment in mathematics classes. The developers are building on the success of the NSF-supported Math Forum's Problem of the Week program to create a community that is working to increase students' mathematics learning by helping teachers stimulate student thinking, assess that thinking, and provide useful feedback to students. The teachers are working together to create rubrics for assessing the progress of students as they solve challenging mathematics problems. The program is structured so that the teachers are learning mathematics and assessment strategies in addition to establishing a research-based model for online, professional communities.

Researchers are studying how specific activities (e.g., discourse, active participation, use of rubrics, feedback, and reflection) and an online community support teachers' engagement in authentic and generative assessment. Researchers are using ethnographic methods to understand the development of the community, and conducting focus groups and individual interviews to determine the impact of participation in the community on mathematics teachers. In addition, they are collecting data through discourse analysis, student work analysis, and rubric analysis to determine the optimal design of the products. The intentional structure of the online community builds on research findings on creating professional communities and research on assessing mathematics learning.

Online professional teaching communities offer new venues for communication, professional development, and shared work among mathematics teachers. The Math Forum provides an optimal, online context for expanding the popular Problem of the Week into a productive discussion of assessment of problem solving, the building of specific rubrics, and the related reflection on how to encourage student thinking. This collaborative work will offer rubrics for assessing mathematical problem solving, a new model for online professional development, and extensive information on building an online mathematics community.

Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Stevens)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Partner Organization(s): 
Award Number: 
1222496
Funding Period: 
Wed, 08/15/2012 to Sun, 07/31/2016
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Supporting the Emergence of a Professional Teaching Community Through Collective Knowledge-Building in Assessment and Feedback of Mathematical Thinking (Collaborative Research: Silverman)

This collaborative project is developing an online, professional teaching community that addresses issues of assessment in mathematics classes. The developers are building on the success of the NSF-supported Math Forum's Problem of the Week program to create a community that works to increase students' mathematics learning by helping teachers stimulate student thinking, assess that thinking, and provide useful feedback to students.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222355
Funding Period: 
Sat, 09/01/2012 to Fri, 08/31/2018
Full Description: 

This collaborative work involves Drexel University and Temple University where they are developing an online, professional teaching community that is addressing issues of assessment in mathematics classes. The developers are building on the success of the NSF-supported Math Forum's Problem of the Week program to create a community that is working to increase students' mathematics learning by helping teachers stimulate student thinking, assess that thinking, and provide useful feedback to students. The teachers are working together to create rubrics for assessing the progress of students as they solve challenging mathematics problems. The program is structured so that the teachers are learning mathematics and assessment strategies in addition to establishing a research-based model for online, professional communities.

Researchers are studying how specific activities (e.g., discourse, active participation, use of rubrics, feedback, and reflection) and an online community support teachers' engagement in authentic and generative assessment. Researchers are using ethnographic methods to understand the development of the community, and conducting focus groups and individual interviews to determine the impact of participation in the community on mathematics teachers. In addition, they are collecting data through discourse analysis, student work analysis, and rubric analysis to determine the optimal design of the products. The intentional structure of the online community builds on research findings on creating professional communities and research on assessing mathematics learning.

Online professional teaching communities offer new venues for communication, professional development, and shared work among mathematics teachers. The Math Forum provides an optimal, online context for expanding the popular Problem of the Week into a productive discussion of assessment of problem solving, the building of specific rubrics, and the related reflection on how to encourage student thinking. This collaborative work will offer rubrics for assessing mathematical problem solving, a new model for online professional development, and extensive information on building an online mathematics community.

SimScientists Assessments: Physical Science Links

The goal of this project is to develop and validate a middle school physical science assessment strand composed of four suites of simulation-based assessments for integrating into balanced (use of multiple measures), large-scale accountability science testing systems. It builds on the design templates, technical infrastructure, and evidence of the technical quality, feasibility, and instructional utility of the NSF-funded Calipers II project. The evaluation plan addresses both formative and summative aspects.

Lead Organization(s): 
Award Number: 
1221614
Funding Period: 
Mon, 10/01/2012 to Fri, 09/30/2016
Full Description: 

The goal of this project is to develop and validate a middle school physical science assessment strand composed of four suites of simulation-based assessments for integrating into balanced (use of multiple measures), large-scale accountability science testing systems. It builds on the design templates, technical infrastructure, and evidence of the technical quality, feasibility, and instructional utility of the NSF-funded Calipers II project. The assessment strand consists of multilevel (increased thinking levels) assessment designs grounded on evidence-centered principles that target practices and key disciplinary conceptual schemes, such as matter, motion, energy, and waves identified in the National Research Council report "A Framework for K-12 Science Education: Practices, Crosscutting Knowledge, and Core Ideas". The assessment model vertically links simulations (interactive with feedback to students, coaching, and reflection); curriculum-embedded assessments for formative use; unit benchmark assessment for interim summative purposes; and a set of "signature tasks" (short-term simulations on recurring problem types). Members of the Advisory Board and an Assessment Review Panel actively participate in the development and implementation of this effort. Heller Research Associates is the external evaluator. The evaluation plan addresses both formative and summative aspects.

The project's theory of action is based on model-based learning and evidence-centered design reflective of the notion that the construct of science is multidimensional, requiring (a) understanding how the components of a science conceptual system interact to produce behaviors of the system; and (b) the use of inquiry practices to investigate the dynamic behaviors and underlying components' interactions of the system. A total of eight research and development questions guide the scope of work. The questions focus on: (a) validity (substantive and technical quality) of the individual simulation assessments; and (b) classroom implementation (feasibility, fidelity, utility). The methodology for test construction and revision follows the testing standards of major professional organizations (i.e., American Educational Research Association, American Psychological Association, and National Council of Measurement in Education) through three development phases. Phase I (Assessment Development) focuses on the alignment, quality, and prototype testing, including leverage and modification of prior work, and design of new assessment suites and signature tasks. Phase II (Pilot and Validation Studies) deals with the testing of all assessments, research instruments, and study methods. Phase III (Cross-Validation Studies) substantiates the multilevel integration assessment model, cross-validates the assessments piloted in Phase II, and establishes a reliable argument that the assessments measure the intended content and inquiry practices suitable for use in district and state-level assessment systems.

Expected outcomes are: (1) a research-informed and field-tested physical science simulations-based assessment model with high potential for extended use in middle school grades; and (2) a policy brief that provides recommendations for integrating assessments into districts and state large-scale, multi-level, balanced science assessments.

Cluster Randomized Trial of the Efficacy of Early Childhood Science Education for Low-Income Children

The research goal of this project is to evaluate whether an early childhood science education program, implemented in low-income preschool settings produces measurable impacts for children, teachers, and parents. The study is determining the efficacy of the program on Science curriculum in two models, one in which teachers participate in professional development activities (the intervention), and another in which teachers receive the curriculum and teachers' guide but no professional development (the control).

Project Email: 
Award Number: 
1119327
Funding Period: 
Mon, 08/15/2011 to Mon, 07/31/2017
Project Evaluator: 
Brian Dates, Southwest Counseling Services
Full Description: 

The research goal of this project is to evaluate whether an early childhood science education program, Head Start on Science, implemented in low-income preschool settings (Head Start) produces measurable impacts for children, teachers, and parents. The study is being conducted in eight Head Start programs in Michigan, involving 72 classrooms, 144 teachers, and 576 students and their parents. Partners include Michigan State University, Grand Valley State University, and the 8 Head Start programs. Southwest Counseling Solutions is the external evaluator.

The study is determining the efficacy of the Head Start on Science curriculum in two models, one in which 72 teachers participate in professional development activities (the intervention), and another in which 72 teachers receive the curriculum and teachers' guide but no professional development (the control). The teacher study is a multi-site cluster randomized trial (MSCRT) with the classroom being the unit of randomization. Four time points over two years permit analysis through multilevel latent growth curve models. For teachers, measurement instruments include Attitudes Toward Science (ATS survey), the Head Start on Science Observation Protocol, the Preschool Classroom Science Materials/Equipment Checklist, the Preschool Science Classroom Activities Checklist, and the Classroom Assessment Scoring System (CLASS). For students, measures include the "mouse house problem," Knowledge of Biological Properties, the physics of falling objects, the Peabody Picture Vocabulary Test-Fourth Edition, the Expressive Vocabulary Test-2, the Test of Early Mathematics Ability-3, Social Skills Improvement System-Rating Scales, and the Emotion Regulation Checklist. Measures for parents include the Attitudes Toward Science survey, and the Community and Home Activities Related to Science and Technology for Preschool Children (CHARTS/PS). There are Spanish versions of many of these instruments which can be used as needed. The external evaluation is monitoring the project progress toward its objectives and the processes of the research study.

This project meets a critical need for early childhood science education. Research has shown that very young children can achieve significant learning in science. The curriculum Head Start on Science has been carefully designed for 3-5 year old children and is one of only a few science programs for this audience with a national reach. This study intends to provide a sound basis for early childhood science education by demonstrating the efficacy of this important curriculum in the context of a professional development model for teachers.

Pages

Subscribe to Assessment