Instrument

Assessing, Validating, and Developing Content Knowledge for Teaching Energy (Collaborative Research: Vokos)

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. The project uses a framework that includes tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222732
Funding Period: 
Sat, 09/01/2012 to Wed, 08/31/2016
Full Description: 

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. There is significant research that indicates that teacher content knowledge differs from what people in other professions need to know about particular domains such as mathematics, and the development of a Content Knowledge of Teaching Energy in mechanics is an extension of those research and development efforts. The project embeds the development of the instrument in a program of measuring effective teaching of physics in the classroom and develops a strong validity argument for the resulting assessment based on its use as a measure in a professional development project that intends to improve teachers' understanding of energy in physics. The research team consists of experts in physics, assessment and classroom teaching of physics. The collaborative project includes researchers at Rutgers, University of Maine, Seattle Pacific University, Facets Innovation, and the Educational Testing Service.

The project uses a framework for effective teaching developed in the Measures of Effective Teaching project funded by the Gates Foundation to construct a theoretical framework for the teaching of mechanical energy. That framework includes items and tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning. A strong framework of validation based on multiple lines of evidence of the relationship between the items developed for the study and observations, analysis of video, and artifacts from the classroom is one element of the study. Another element of the study examines multiple psychometric lines of evidence to determine the reliability of the instruments and the validity of the inferences drawn from them. The resulting instruments will be used in the measurement of changes of teacher content knowledge for teaching in professional development programs as another source of validation.

The improvement of teachers' content knowledge for teaching is an important intermediary goal of professional development of teachers. Without adequate understanding of the gaps in teacher knowledge and precise evidence of the improvement through professional development, the efficacy of different professional development projects is not possible. This project develops a model of teacher assessment instrument development that addresses a cross-cutting theme in the Next Generation Science Standards and contributes an important tool to the research and evaluation processes that are needed to make those standards a reality in the classroom. Findings from the use of the instruments across multiple projects inform policy decisions on local, state and federal levels.

Assessing, Validating, and Developing Content Knowledge for Teaching Energy (Collaborative Research: Minstrell)

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. The project uses a framework that includes tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning.

Lead Organization(s): 
Award Number: 
1222598
Funding Period: 
Sat, 09/01/2012 to Thu, 08/31/2017
Full Description: 

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. There is significant research that indicates that teacher content knowledge differs from what people in other professions need to know about particular domains such as mathematics, and the development of a Content Knowledge of Teaching Energy in mechanics is an extension of those research and development efforts. The project embeds the development of the instrument in a program of measuring effective teaching of physics in the classroom and develops a strong validity argument for the resulting assessment based on its use as a measure in a professional development project that intends to improve teachers' understanding of energy in physics. The research team consists of experts in physics, assessment and classroom teaching of physics. The collaborative project includes researchers at Rutgers, University of Maine, Seattle Pacific University, Facets Innovation, and the Educational Testing Service.

The project uses a framework for effective teaching developed in the Measures of Effective Teaching project funded by the Gates Foundation to construct a theoretical framework for the teaching of mechanical energy. That framework includes items and tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning. A strong framework of validation based on multiple lines of evidence of the relationship between the items developed for the study and observations, analysis of video, and artifacts from the classroom is one element of the study. Another element of the study examines multiple psychometric lines of evidence to determine the reliability of the instruments and the validity of the inferences drawn from them. The resulting instruments will be used in the measurement of changes of teacher content knowledge for teaching in professional development programs as another source of validation.

The improvement of teachers' content knowledge for teaching is an important intermediary goal of professional development of teachers. Without adequate understanding of the gaps in teacher knowledge and precise evidence of the improvement through professional development, the efficacy of different professional development projects is not possible. This project develops a model of teacher assessment instrument development that addresses a cross-cutting theme in the Next Generation Science Standards and contributes an important tool to the research and evaluation processes that are needed to make those standards a reality in the classroom. Findings from the use of the instruments across multiple projects inform policy decisions on local, state and federal levels.

Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Sword)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Award Number: 
1222426
Funding Period: 
Wed, 08/15/2012 to Sun, 07/31/2016
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Matsuura)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222340
Funding Period: 
Wed, 08/15/2012 to Tue, 07/31/2018
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Assessing, Validating, and Developing Content Knowledge for Teaching Energy (Collaborative Research: Wittmann)

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. The project uses a framework that includes tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1222580
Funding Period: 
Sat, 09/01/2012 to Wed, 08/31/2016
Full Description: 

This project develops an instrument to measure the content knowledge that teachers need to teach about energy in high school classroom instruction that focuses on mechanical energy. There is significant research that indicates that teacher content knowledge differs from what people in other professions need to know about particular domains such as mathematics, and the development of a Content Knowledge of Teaching Energy in mechanics is an extension of those research and development efforts. The project embeds the development of the instrument in a program of measuring effective teaching of physics in the classroom and develops a strong validity argument for the resulting assessment based on its use as a measure in a professional development project that intends to improve teachers' understanding of energy in physics. The research team consists of experts in physics, assessment and classroom teaching of physics. The collaborative project includes researchers at Rutgers, University of Maine, Seattle Pacific University, Facets Innovation, and the Educational Testing Service.

The project uses a framework for effective teaching developed in the Measures of Effective Teaching project funded by the Gates Foundation to construct a theoretical framework for the teaching of mechanical energy. That framework includes items and tasks based on instructional practices in the classroom that can identify the extent to which the teacher understands both the disciplinary knowledge and the appropriate teaching processes that support student learning. A strong framework of validation based on multiple lines of evidence of the relationship between the items developed for the study and observations, analysis of video, and artifacts from the classroom is one element of the study. Another element of the study examines multiple psychometric lines of evidence to determine the reliability of the instruments and the validity of the inferences drawn from them. The resulting instruments will be used in the measurement of changes of teacher content knowledge for teaching in professional development programs as another source of validation.

The improvement of teachers' content knowledge for teaching is an important intermediary goal of professional development of teachers. Without adequate understanding of the gaps in teacher knowledge and precise evidence of the improvement through professional development, the efficacy of different professional development projects is not possible. This project develops a model of teacher assessment instrument development that addresses a cross-cutting theme in the Next Generation Science Standards and contributes an important tool to the research and evaluation processes that are needed to make those standards a reality in the classroom. Findings from the use of the instruments across multiple projects inform policy decisions on local, state and federal levels.

Assessing Secondary Teachers' Algebraic Habits of Mind (Collaborative Research: Stevens)

This collaborative project is developing instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement.

Partner Organization(s): 
Award Number: 
1222496
Funding Period: 
Wed, 08/15/2012 to Sun, 07/31/2016
Full Description: 

Boston University, Education Development Center, Inc., and St. Olaf College are collaborating on Assessing Secondary Teachers' Algebraic Habits of Mind (ASTAHM) to develop instruments to assess secondary teachers' Mathematical Habits of Mind (MHoM). These habits bring parsimony, focus, and coherence to teachers' mathematical thinking and, in turn, to their work with students. MHoM is a critical component of mathematical knowledge for teaching at the secondary level. Recognizing the need for a scientific approach to investigate the ways in which MHoM is an indicator of teacher effectiveness, the partnership is researching the following questions:

1. How do teachers who engage MHoM when doing mathematics for themselves also bring MHoM to their teaching practice?

2. How are teachers' engagement with MHoM and their use of these habits in teaching related to student understanding and achievement?

To investigate these questions, ASTAHM is developing two instruments: a paper and pencil (P&P) assessment and an observation protocol that measure teachers' knowledge and classroom use, respectively, of MHoM.

The work is being conducted in two phases: (1) an instrument-refinement and learning phase, and (2) an instrument-testing and research phase. Objectives of Phase 1 are to gather data to refine the project's existing instruments and to learn about the bridge factors that impact the relationship between teachers' knowledge and classroom use of MHoM. Specific research activities include: administering the pilot P&P assessment to 40 teachers, videotaping Algebra instructions of 8 teachers, performing initial testing and refinement of the instruments, and using the data to analyze the bridge factors. Phase 2 is a large-scale study involving field-testing the P&P assessment with 200 teachers, videotaping 20 teachers and studying them using the observation protocol, collecting achievement data from 3000 students, and checking P&P content validity with 200 mathematicians. With these validated instruments in hand, the project will then conduct an investigation into the above research questions. Lesley University's Program Evaluation and Research Group (PERG) is the external evaluator. PERG is assessing ASTAHM's overall success in developing valid and reliable instruments to investigate the extent to which a relationship exists between teachers' MHoM and their classroom practice, as well as student achievement. Evaluators are also investigating whether users' coding guides for both instruments enable field-testers to effectively use and adequately score them.

This work fits into a larger research agenda with the ultimate goal of understanding the connections between secondary teachers' mathematical knowledge for teaching and secondary students' mathematical understanding and achievement. The MHoM construct is closely aligned with the Common Core State Standards-Mathematics (CCSS-M); especially its Standards for Mathematical Practice. For example, both place importance on seeking and using mathematical structure. Thus the instruments this project produces can act as pre- and post-measures of the effectiveness of professional development programs in preparing teachers to implement the CCSS-M. Mathematics teacher knowledge at the secondary level is an understudied field. Through analyses of the practices and habits of mind that teachers bring to their work, ASTAHM is developing instruments that can be used to shed light on effective secondary teaching.


Project Videos

2019 STEM for All Video Showcase

Title: Studying Teachers' Mathematical Habits of Mind

Presenter(s): Sarah Sword, Eden Badertscher, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2017 STEM for All Video Showcase
Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Sarah Sword, Courtney Arthur, Al Cuoco, Miriam Gates, Ryota Matsuura, & Glenn Stevens

2016 STEM for All Video Showcase

Title: Assessing Secondary Teachers' Algebraic Habits of Mind

Presenter(s): Ryota Matsuura, Al Cuoco, Glenn Stevens, & Sarah Sword


Cluster Randomized Trial of the Efficacy of Early Childhood Science Education for Low-Income Children

The research goal of this project is to evaluate whether an early childhood science education program, implemented in low-income preschool settings produces measurable impacts for children, teachers, and parents. The study is determining the efficacy of the program on Science curriculum in two models, one in which teachers participate in professional development activities (the intervention), and another in which teachers receive the curriculum and teachers' guide but no professional development (the control).

Project Email: 
Award Number: 
1119327
Funding Period: 
Mon, 08/15/2011 to Mon, 07/31/2017
Project Evaluator: 
Brian Dates, Southwest Counseling Services
Full Description: 

The research goal of this project is to evaluate whether an early childhood science education program, Head Start on Science, implemented in low-income preschool settings (Head Start) produces measurable impacts for children, teachers, and parents. The study is being conducted in eight Head Start programs in Michigan, involving 72 classrooms, 144 teachers, and 576 students and their parents. Partners include Michigan State University, Grand Valley State University, and the 8 Head Start programs. Southwest Counseling Solutions is the external evaluator.

The study is determining the efficacy of the Head Start on Science curriculum in two models, one in which 72 teachers participate in professional development activities (the intervention), and another in which 72 teachers receive the curriculum and teachers' guide but no professional development (the control). The teacher study is a multi-site cluster randomized trial (MSCRT) with the classroom being the unit of randomization. Four time points over two years permit analysis through multilevel latent growth curve models. For teachers, measurement instruments include Attitudes Toward Science (ATS survey), the Head Start on Science Observation Protocol, the Preschool Classroom Science Materials/Equipment Checklist, the Preschool Science Classroom Activities Checklist, and the Classroom Assessment Scoring System (CLASS). For students, measures include the "mouse house problem," Knowledge of Biological Properties, the physics of falling objects, the Peabody Picture Vocabulary Test-Fourth Edition, the Expressive Vocabulary Test-2, the Test of Early Mathematics Ability-3, Social Skills Improvement System-Rating Scales, and the Emotion Regulation Checklist. Measures for parents include the Attitudes Toward Science survey, and the Community and Home Activities Related to Science and Technology for Preschool Children (CHARTS/PS). There are Spanish versions of many of these instruments which can be used as needed. The external evaluation is monitoring the project progress toward its objectives and the processes of the research study.

This project meets a critical need for early childhood science education. Research has shown that very young children can achieve significant learning in science. The curriculum Head Start on Science has been carefully designed for 3-5 year old children and is one of only a few science programs for this audience with a national reach. This study intends to provide a sound basis for early childhood science education by demonstrating the efficacy of this important curriculum in the context of a professional development model for teachers.

Levels of Conceptual Understanding in Statistics (LOCUS)

LOCUS (Levels of Conceptual Understanding in Statistics) is an NSF Funded DRK12 project (NSF#118618) focused on developing assessments of statistical understanding. These assessments will measure students’ understanding across levels of development as identified in the Guidelines for Assessment and Instruction in Statistics Education (GAISE). The intent of these assessments is to provide teachers and researchers with a valid and reliable assessment of conceptual understanding in statistics consistent with the Common Core State Standards (CCSS).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1118168
Funding Period: 
Thu, 09/01/2011 to Fri, 08/31/2012
Project Evaluator: 
TERC, Jim Hammerman
Full Description: 

The goal of this project is to develop two tests (instruments) to assess conceptual understanding of statistics. The instruments are based on the levels A/B and on level C of statistical understanding development as described in the American Statistical Association Guidelines for Assessment and Instruction of Statistics Education (GAISE) framework. These instruments will be used to assess knowledge of statistics by grades 6-12 students. The instruments will have multiple-choice and constructed response (CR) items. The CR items will have scoring rubrics. The assessments will be pilot tested in school districts in six states. The instruments will be used by teachers to analyze students' growth in understanding of statistics and will be useable for both formative and summative purposes. An assessment blueprint will be developed based on the GAISE framework for selecting and constructing both fixed-choice and open-ended items. An evidenced-based designed process will be used to develop the assessments. The blueprint will be used by the test development committee to develop items. These items will be reviewed by the advisory board considering the main statistics topics to be included on the assessments. Through a layering process, the assessments will be piloted, revised, and field tested with students in grades 6-12 in six states. A three-parameter IRT model will be used in analyzing the items. The work will be done by researchers at the University of Florida with the support of those at the University of Minnesota, the Educational Testing Service, and Kenyon College. Researchers from TERC will conduct a process evaluation with several feedback and redesign cycles.

The assessments will be aligned with the Common Core State Standards for mathematics (CCSSM) and made available as open-source to teachers through a website. The research team will interact with the state consortia developing assessments to measure students' attainment of the CCSSM. As such, the assessments have the potential of being used by a large proportion of students in the country. The more conceptually-based items will provide teachers with concrete examples of what statistics students in grades 6-12 should know.

Opportunities and Challenges for Developing and Evaluating Diagnostic Assessments in STEM Education

Day: 
Fri

Four DR K-12 projects will discuss opportunities and challenges that they have encountered when trying to harness psychometric models for diagnosis in science and mathematics education.

Date/Time: 
10:00 am to 12:00 pm
Session Type: 
Panel

What are the opportunities and challenges that arise when trying to harness psychometric models for diagnosis in STEM education?        

Efficacy Study of Metropolitan Denver's Urban Advantage Program: A Project to Improve Scientific Literacy Among Urban Middle School Students

This is an efficacy study to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The study aims to answer the following questions: How does participation in the program affect students' science knowledge, skills, and attitudes toward science; teachers' science knowledge, skills, and abilities; and families engagement in and support for their children's science learning and aspirations?

Award Number: 
1020386
Funding Period: 
Wed, 09/15/2010 to Wed, 08/31/2011
Project Evaluator: 
Maggie Miller
Full Description: 

This is an efficacy study through which the Denver Museum of Nature and Science, the Denver Zoo, the Denver Botanic Gardens, and three of Denver's urban school districts join efforts to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The Metropolitan Denver Urban Advantage (UA Denver) program is used for this purpose. This program consists of three design elements: (a) student-driven investigations, (b) STEM-related content, and (c) alignment of schools and informal science education institutions; and six major components: (a) professional development for teachers, (b) classroom materials and resources, (c) access to science-rich organizations, (d) outreach to families, (e) capacity building and sustainability, and (e) program assessment and student learning. Three research questions guide the study: (1) How does the participation in the program affect students' science knowledge, skills, and attitudes toward science relative to comparison groups of students? (2) How does the participation in the program affect teachers' science knowledge, skills, and abilities relative to comparison groups of teachers? and (3) How do families' participation in the program affect their engagement in and support for their children's science learning and aspirations relative to comparison families?

The study's guiding hypothesis is that the UA Denver program should improve science literacy in urban middle school students measured by (a) students' increased understanding of science, as reflected in their science investigations or "exit projects"; (b) teachers' increased understanding of science and their ability to support students in their exit projects, as documented by classroom observations, observations of professional development activities, and surveys; and (c) school groups' and families' increased visits to participating science-based institutions, through surveys. The study employs an experimental research design. Schools are randomly assigned to either intervention or comparison groups and classrooms will be the units of analysis. Power analysis recommended a sample of 18 intervention and 18 comparison middle schools, with approximately 72 seventh grade science teachers, over 5,000 students, and 12,000 individual parents in order to detect differences among intervention and comparison groups. To answer the three research questions, data gathering strategies include: (a) students' standardized test scores from the Colorado Student Assessment Program, (b) students' pre-post science learning assessment using the Northwest Evaluation Association's Measures for Academic Progress (science), (c) students' pre-post science aspirations and goals using the Modified Attitude Toward Science Inventory, (d) teachers' fidelity of implementation using the Teaching Science as Inquiry instrument, and (e) classroom interactions using the Science Teacher Inquiry Rubric, and the Reformed Teaching Observation protocol. To interpret the main three levels of data (students, nested in teachers, nested within schools), hierarchical linear modeling (HLM), including HLM6 application, are utilized. An advisory board, including experts in research methodologies, science, informal science education, assessment, and measurement oversees the progress of the study and provides guidance to the research team. An external evaluator assesses both formative and summative aspects of the evaluation component of the scope of work.

The key outcome of the study is a research-informed and field-tested intervention implemented under specific conditions for enhancing middle school science learning and teaching, and supported by partnerships between formal and informal organizations.

Pages

Subscribe to Instrument