Achievement/Growth

Exploring the Efficacy of Engineering is Elementary (E4)

This project is developing evidence about the efficacy of the Engineering is Elementary curriculum under ideal conditions by studying the student and teacher-level effects of implementation. The project seeks to determine the core elements of the curriculum that support successful use. The findings from this study have broad implications for how engineering design curricular can be developed and implemented at the elementary level.

Lead Organization(s): 
Award Number: 
1220305
Funding Period: 
Sat, 09/15/2012 to Fri, 08/31/2018
Full Description: 

This project is developing evidence about the efficacy of the Engineering is Elementary curriculum under ideal conditions by studying the student and teacher-level effects of implementation. The rigorous level of evidence that is developed in this study has significant utility as a support for the kinds of elementary engineering curricula that are needed as the Next Generation Science Standards come online and emphasize engineering design. The study is a randomized control trial where the assignment of teachers will be to the EiE curricular materials or to a counterfactual condition, the use of more standard design engineering curricular materials. The project studies the impact of the use of the curriculum on student learning and on teachers' use of the curriculum in a fidelity of implementation study to determine the core elements of the curriculum that support successful use. The study examines the implementation of the curricular materials in a number of contexts to more fully understand the conditions under which they work best and to explicate what aspects of such project-based inquiry materials most support student learning.

This study uses a randomized cluster trial to examine the efficacy of the EiE curriculum across 75 schools in the treatment and 75 schools in the control group samples. Two teachers per school are included in one treatment/control condition per school. Outcome measures for students include performances on project-specific measures that have been examined for technical quality of validity and reliability. A set of additional research-based survey instruments validated for use in the EiE context are also used to collect data about students' attitudes, perceptions, interest and motivation toward science and engineering. A robust fidelity of implementation research plan is being implemented that will include teachers surveys, pre and post assessments, teacher logs, as well as student engineering journals and student work from classroom implementation. The fidelity of implementation is further studied with forty treatment and ten control teachers through classroom observations and interviews.

The findings from this study have broad implications for how engineering design curricular can be developed and implemented at the elementary level. Engineering design has not been emphasized in the elementary classroom, lagging behind instruction in science with which teachers are more familiar. The results of this study inform practitioners and policy makers about what works, for whom and under what conditions. Information about the different contexts in which the curriculum has been implemented supports the dissemination of evidence-based research and development practices to strengthen STEM learning for all students.

Project ATOMS: Accomplished Elementary Teachers of Mathematics and Science

The project is studying the impact of the mathematics and science intensive pre-service preparation program for elementary school teachers.  The project includes assessments of pre-service teachers' math and science content, teacher performance, self-report surveys, and teacher interviews. Each of the study dimensions (Knowledge Dimension, Teaching Performance, and Perspectives on the Program) will be assessed at three time points across this longitudinal study, providing a model for elementary teacher development of STEM teaching.

Partner Organization(s): 
Award Number: 
1118894
Funding Period: 
Thu, 09/01/2011 to Sat, 08/31/2019
Full Description: 

The project is studying the impact of the mathematics and science intensive pre-service preparation program for elementary school teachers at North Carolina State University called the Accomplished Elementary Teachers of Mathematics and Science (ATOMS). Faculty in NCSU's Department of Elementary Education, researchers at the Duke University Sanford School of Public Policy's Education Research Data Center and the NC State College Professional Education Office are involved in conducting this project.

The project includes assessments of pre-service teachers' math and science content, teacher performance, self-report surveys, and teacher interviews. Researchers are also tracking participants' perspectives on the program and comparing knowledge dimensions and teaching performance of a sub-sample of ATOMS teachers to a similar group of non-ATOMS teachers. Each of the study dimensions (Knowledge Dimension, Teaching Performance, and Perspectives on the Program) will be assessed at three time points across this longitudinal study, providing a model for elementary teacher development of STEM teaching.

The study has potential to advance current understanding regarding teacher preparation, especially in terms of supporting elementary teachers' instruction in science and math. The project is also innovative and potentially transformative by asking interesting and pertinent questions of how teachers can affect the learning of their students. Besides generating new knowledge, this project also has the potential to impact STEM education research. The ATOMS pre-service teacher preparation program may serve as a model for effective pre-service teacher education across the nation if the researchers can clearly demonstrate the effect of participating in the program in changing teachers' knowledge, attitudes, and skills, as well as their students' achievement. Investigators propose the dissemination of findings to both K-12 audiences and institutions of higher education. Additionally, key findings will be bulleted for policy makers in brief reports or brochures sent to deans of Colleges of Education nationwide, highlighting recommendations based on the findings.

An Innovative Approach to Earth Science Teacher Preparation: Uniting Science, Informal Science Education, and Schools to Raise Student Achievement

The American Museum of Natural History in New York City, in partnership with New York University, and in collaboration with five high-needs schools, is developing, implementing, and researching a five-year pilot Master of Arts in Teaching (MAT) program in Earth Science. The program is delivered by the Museum's scientific and education teams and its evaluation covers aspects of the program from recruitment to first year of teaching.

Project Email: 
Award Number: 
1119444
Funding Period: 
Thu, 09/01/2011 to Thu, 08/31/2017
Project Evaluator: 
David Silvernail, Center for Education and Policy, University of Southern Maine
Full Description: 

The American Museum of Natural History (AMNH), in collaboration with New York University's Institute for Education and Social Policy and the University of Southern Maine Center for Evaluation and Policy, will develop and evaluate a new teacher education program model to prepare science teachers through a partnership between a world class science museum and high need schools in metropolitan New York City (NYC). This innovative pilot residency model was approved by the New York State (NYS) Board of Regents as part of the state’s Race To The Top award. The program will prepare a total of 50 candidates in two cohorts (2012 and 2013) to earn a Board of Regents-awarded Masters of Arts in Teaching (MAT) degree with a specialization in Earth Science for grades 7-12. The program focuses on Earth Science both because it is one of the greatest areas of science teacher shortages in urban areas and because AMNH has the ability to leverage the required scientific and educational resources in Earth Science and allied disciplines, including paleontology and astrophysics.

The proposed 15-month, 36-credit residency program is followed by two additional years of mentoring for new teachers. In addition to a full academic year of residency in high-needs public schools, teacher candidates will undertake two AMNH-based clinical summer residencies; a Museum Teaching Residency prior to entering their host schools, and a Museum Science Residency prior to entering the teaching profession. All courses will be taught by teams of doctoral-level educators and scientists.

The project’s research and evaluation components will examine the factors and outcomes of a program offered through a science museum working with the formal teacher preparation system in high need schools. Formative and summative evaluations will document all aspects of the program. In light of the NYS requirement that the pilot program be implemented in high-need, low-performing schools, this project has the potential to engage, motivate and improve the Earth Science achievement and interest in STEM careers of thousands of students from traditionally underrepresented populations including English language learners, special education students, and racial minority groups. In addition, this project will gather meaningful data on the role science museums can play in preparing well-qualified Earth Science teachers. The research component will examine the impact of this new teacher preparation model on student achievement in metropolitan NYC schools. More specifically, this project asks, "How do Earth Science students taught by first year AMNH MAT Earth Science teachers perform academically in comparison with students taught by first year Earth Science teachers not prepared in the AMNH program?.”

Energy: A Multidisciplinary Approach for Teachers (EMAT) Designing and Studying a Multidisciplinary, Online Course for High School Teachers

This project will iteratively design, develop, field test, refine, and rigorously study a six-unit, facilitated, online professional development (PD) course focusing on energy-related concepts in the context of alternative energy. The primary audience is high school science teachers teaching out of their field of endorsement and serving students underrepresented in the sciences. The project will investigate whether the PD will precipitate changes in teacher knowledge and practice that result in higher student achievement.

Award Number: 
1118643
Funding Period: 
Thu, 09/01/2011 to Thu, 08/31/2017
Project Evaluator: 
RMC Research Corporation
Full Description: 

The Energy: A Multidisciplinary Approach for Teachers (EMAT) project will iteratively design, develop, field test, refine, and rigorously study a seven-unit, facilitated, online professional development (PD) course focusing on energy-related concepts in the context of alternative energy. The primary audience is high school science teachers teaching out of their field of endorsement and serving students underrepresented in the sciences. The project will investigate whether the PD will precipitate changes in teacher knowledge and practice that result in higher student achievement. As a result, EMAT will improve the science achievement of underrepresented students and enhance their future participation in science. Biological Sciences Curriculum Study and partners Oregon Public Broadcasting, the National Teacher Enhancement Network, the National Renewable Energy Laboratory, the Great Lakes Bioenergy Research Center, and RMC Research Corporation bring significant resources and are highly qualified to develop and research EMAT.

The EMAT project advances knowledge in the field of teacher professional development by merging two facets of PD that have hitherto been studied separately and testing hypotheses about the degree to which this pairing enhances learning and practice. These facets are structured constructivist experiences and experiences grounded in situated cognition learning theory. Teachers reflect on research-based teaching practices in the lesson analysis process through Science Content Storyline and Student Thinking lenses. EMAT tests longitudinal impacts on teachers' content knowledge, pedagogical content knowledge, and teaching practices and students' content knowledge, contributing much needed data for future PD projects. EMAT also studies which aspects of online environments are most effective for teachers. Data collected will inform full revisions of the course and will help address significant gaps in our understanding of online PD.

EMAT advances the field's understanding of which elements of online PD are effective and the extent to which high-quality online PD translates to improved student learning. Simultaneously, the project develops and tests a scalable, flexible resource to enhance teacher learning and practice. As a result, EMAT will have a broad impact by promoting research-based teaching and learning while advancing discovery and understanding. Furthermore, by targeting the recruitment of teacher participants from large urban districts with high numbers of teachers teaching out of field, EMAT impacts students traditionally underrepresented in the sciences. EMAT will not only contribute to the research on PD, but also will be available for use in diverse settings. A facilitation guide allows the course to be freely used by school districts and teacher education and certification programs across the country. In addition, the facilitated course will be offered for graduate credit through the National Teacher Enhancement Network and will be freely available to individuals for independent study. Results of all research and evaluation will be published in science education journals and practitioner journals for teachers, and presented to PD groups at conferences. EMAT will benefit society by impacting teacher and student understanding of energy-related concepts, thereby increasing the capacity of U.S. citizens to creatively address energy challenges from a foundation of scientifically sound knowledge.

Promoting Science Among English Language Learners (P-SELL) Scale-Up

This effectiveness study focuses on the scale-up of a model of curricular and teacher professional development intervention aimed at improving science achievement of all students, especially English language learners (ELLs). The model consists of three basic components: (a) inquiry-oriented science curriculum, (b) teacher professional development for science instruction with these students, and (c) school resources for science instruction.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1209309
Funding Period: 
Mon, 08/15/2011 to Fri, 07/31/2015
Project Evaluator: 
Lauren Scher
Full Description: 

This four-year effectiveness study focuses on the scale-up of a model of curricular and teacher professional development intervention aimed at improving science achievement of all students, especially English language learners (ELLs). The model consists of three basic components: (a) inquiry-oriented science curriculum, (b) teacher professional development for science instruction with these students, and (c) school resources for science instruction. The project's main goals are: (1) to evaluate the effect of the intervention on student achievement, (2) to determine the effect of the intervention on teacher knowledge, practices, and school resources, and (3) to assess how teacher knowledge, practices, and resources mediate student achievement. The project is conducted in the context of the Florida current science education policies and accountability system (e.g., adoption of the Next Generation Sunshine State Standards in Science, assessment of science at the fifth grade, a Race to the Top award state). The study draws on findings from research on a previous NSF-funded efficacy study (035331) in which the model to be scaled-up was tested in a single school district. The effectiveness study includes three (of 67) school districts as key partners, representative of racially, ethnically, linguistically, and socioeconomically diverse student populations; 64 elementary schools, 320 science teachers, and 24,000 fifth-grade students over a three-year period. Science learning is the primary subject matter, inclusive of life, physical, and earth/space sciences. Six research questions corresponding to three research areas guide the proposed scope of work. For the research area of Student Science Achievement, questions are: (1) What is the effect of the intervention on fifth-grade students' science achievement, compared to "business as usual"?, and (2) To what extent are the effects of the intervention moderated by students' English as a Second Language (ESOL) level, SES status, and racial/ethnic backgrounds? For Teacher Knowledge and Practices as a research area, questions are: (3) What is the effect of the intervention on teachers' science knowledge and teaching practices?, and (4) To what extent is students' science achievement predicted by school resources for science instruction? For School Resources for Science, questions are: (5) What is the effect of the intervention on school resources for science instruction?, and (6) To what extent is student achievement predicted by school resources for science instruction? To assess the effect of the intervention on students' and teachers' outcomes, a cluster-randomized-control trial is used, resulting in a total of 64 randomly selected schools (after stratifying them by school-level percent of ESOL and Free Reduced Lunch students). All science teachers and students from the 64 schools participate in the project: 32 in the treatment group (project curriculum for fifth grade, teacher professional development, and instructional resources), and 32 in the control group (district-adopted fifth-grade curriculum, no teacher professional development, and no instructional resources). To address the research area of Student Science Achievement, formative assessment items are used at the end of each curriculum unit, along with two equated forms of a project-developed science test (to be used as pre-and posttests) with both treatment and control groups, in addition to the Florida's Comprehensive Assessment Tests-Science. Data interpretation for this research area employs a set of three-level HLMs (students, nested in classrooms, nested in schools). To address the research area of Teacher Knowledge and Practices and School Resources for Science, the project uses three measures: (a) two equated forms of a 35-items test of teacher science knowledge, (b) a classroom observation instrument measuring third-party ratings of teacher knowledge and teaching practices, and (c) a questionnaire measuring teachers' self-reports of science knowledge and teaching practices. All measures are administered to both treatment and control groups. Data interpretation strategies include a series of HLMs with emphasis on the relevant teacher outcomes as a function of time, and of school-level mediating variables. External project evaluation is conducted by Concentric Research and Evaluation using quantitative and qualitative methods and addressing both formative and summative components. Project research findings contribute to the refinement of a model reflective of the new science standards in the State and the emerging national science standards. The value added of this effort consists of its potential to inform effective implementation of science curricula and teacher professional development in other learning settings, including ELLs and traditionally marginalized student populations at the elementary school level. It constitutes practically the only research study focused on the issue of scale-up and sustainability of effective science education practices with this student subpopulation, which has become prominent due to the dramatic growth of a racially, ethnically, and linguistically diverse school-aged population, low levels of U.S. student science achievement, and the role of science and mathematics in current accountability systems nationwide.

School Structure and Science Success: Organization and Leadership Influences on Student Achievement (Collaborative Research: Butler)

This project will document factors explaining variations in science achievement across schools enrolling ethnically and linguistically diverse students. The research question is: what leadership and organizational features at the school level are associated with mitigating science achievement gaps? Researchers, in collaboration with school districts, will identify school leadership practices that can be connected with reductions in achievement gaps related to student ethnicity, English fluency, and social status.

Partner Organization(s): 
Award Number: 
1338512
Funding Period: 
Fri, 02/15/2013 to Sat, 06/30/2018
Full Description: 

The School Organization and Science Achievement (SOSA) Project will document factors explaining variations in science achievement across schools enrolling ethnically and linguistically diverse students. The research question is: what leadership and organizational features at the school level are associated with mitigating science achievement gaps? Previous school effectiveness studies demonstrate school leadership and social capital influencing student achievement; the SOSA project is unique with its focus on science achievement. Researchers at the University of Connecticut and the University of South Florida St. Petersburg, in collaboration with school districts in their respective states, will identify school leadership practices that can be connected with reductions in achievement gaps related to student ethnicity, English fluency, and social status. At the conclusion of the five-year project, the findings will take the form of recommendations about leadership practices and school organization that can be implemented in other school settings.

The project uses a mixed methods design by combining statistical modeling and qualitative data. Multiple regression analyses highlight those schools populated by fifth graders that have greater or lesser achievement gaps in science. Using social capital theory (i.e., school norms, communication channels, and trustworthiness) comparisons of positive and negative outlier schools will be made via interviews of building principals, classroom teachers and community representatives. The expectation is that schools providing more equitable science experiences to all students will exhibit stronger social capital compared to buildings with disparities in science test scores across demographic categories. These insights will be supplemented by multilevel structural equation modeling to determine the strength of association between various school climate measures (e.g., teacher-to-principal trust, correspondence between teacher and principal perceptions of leadership, and school/community ties) and science achievement as measured by statewide fifth grade science tests. In addition, growth analyses will be used to detect shifts over time and provide insights about the links between policy changes or leadership adjustments, inasmuch as science achievement gaps are affected.

By working with 150 schools in two states, this collaborative research project is designed to generate findings applicable in other school systems. Particularly in settings where science achievement gaps are large, and especially when such gaps vary between schools even when the student populations are similar, the findings from this study will have practical leadership implications. Expertise in this project includes science education, educational leadership, and statistical modeling. This complementary combination increases the depth of the project's efforts along with expanding its potential impacts. Key questions addressed by this project include: to what extent is leadership in science similar to or different from leadership in other subject areas? how do variations in leadership design (e.g., top-down versus distributed leadership) contribute to reductions in science achievement gaps? to what degree can effective leadership mitigate other factors that exacerbate the challenges of providing high quality science learning experiences for every child? Findings will be disseminated via the SOSA Project website, along with leadership development strategies. Deliverables include templates to replicate the study, case studies for professional development, and strategies for supporting the development of science teacher-leaders.

This project was previously funded under award # 1119359.

Further Development and Testing of the Target Inquiry Model for Middle and High School Science Teacher Professional Development (Collaborative Research: Yezierski)

This project scales and further tests the Target Inquiry professional development model. The model involves teachers in three core experiences: 1) a research experience for teachers, 2) materials adaptation, and 3) an action research project. The original program was implemented with high school chemistry teachers, and was shown to result in significant increases, with large effect sizes, in teachers' understanding of science inquiry and quality of instruction, and in science achievement of those teachers' students.

Partner Organization(s): 
Award Number: 
1118749
Funding Period: 
Mon, 08/15/2011 to Wed, 07/31/2013
Full Description: 

This project scales and further tests the Target Inquiry (TI) professional development model. The TI model involves teachers in three core experiences: 1) a research experience for teachers, 2) materials adaptation, and 3) an action research project. The original program was implemented with high school chemistry teachers at Grand Valley State University (GVSU), and was shown to result in significant increases, with large effect sizes, in teachers' understanding of science inquiry and quality of instruction, and in science achievement of those teachers' students. The scale-up and further testing would involve adding physics, biology and geology at Grand Valley State University, and implementing the program at Miami University (MU) with chemistry teachers. Three research questions will be studied:

1) How do the three TI core experiences influence in-service high school science teachers' (i) understanding of the nature of science; (ii) attitudes and beliefs about inquiry instruction; and (iii) classroom instructional methods in the derivatives of the TI model?

2) How does teacher participation in TI affect students' process skills (scientific reasoning and metacognition) and conceptual understanding of science in the derivatives of the TI model?

3) What are the challenges and solutions related to implementing TI in science disciplines beyond chemistry and in other regions?

The research design is quasi-experimental and longitudinal, incorporating implementation with research, and using quantitative and qualitative methods blended in a design research framework. A total of 54 middle and high school science teachers are being recruited for the study. The TI group is completing the TI program (N = 27; 15 at GVSU; 12 at MU) while the comparison group (same sizes and locations) is not. The comparison group is matched according to individual characteristics and school demographics. All teachers are being studied, along with their students, for 4 years (pre-program, post-RET, post-MA, post-AR/post-program). TI teachers are taking 15 credits of graduate level science courses over three years, including summers. Courses include a graduate seminar focused on preparing for the research experience, the research experience in a faculty member's science lab during the summer, application of research to teaching, action research project development, adaptation and evaluation of inquiry-focused curricula, and interpretation and analysis of classroom data from action research. Consistent feedback from professional development, teachers, and evaluation, including the previous implementation, contributes to a design-based approach. Teacher factors being studied include nature of science, inquiry teaching knowledge and beliefs, and quality of inquiry instruction. Student factors being studied include scientific reasoning; metacognition, self-efficacy, and learning processes in science; and content knowledge and conceptual understanding. Only established quantitative and qualitative instruments are being used. Quantitative analysis includes between-group comparisons by year on post-tests, with pre-tests as covariates, and multi-level models with students nested with teachers, and teachers within sites, with the teacher level as the primary unit of change. Trends over time between the treatment and comparison groups are being examined. The evaluation is using a combination of pre/post causal comparative quantitative measures and relevant qualitative data from project leaders and participants, as well as from the comparison group, to provide formative and summative evaluation input.

Outcomes of the project include documentation and understanding of the impacts on science teachers' instruction and student outcomes of research experiences for teachers when they are supported by materials adaptation and action research, and an understanding of what it takes to scale the model to different science disciplines and a different site. The project is also producing a website of instructional materials for middle and secondary science.

Leveling Up: Supporting and Measuring High School STEM Knowledge Building in Social Digital Games

This project designs, develops and tests a digital gaming environment for high school students that fosters and measures science learning within alternate reality games about saving Earth's ecosystems. Players work together to solve scientific challenges using a broad range of tools including a centralized web-based gaming site and social networking tools, along with handheld smart-phones, and an avatar-based massively multiplayer online environment. The game requires players to contribute to a scientific knowledge building community.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1119144
Funding Period: 
Fri, 07/01/2011 to Mon, 06/30/2014
Project Evaluator: 
New Knowledge Organization
Full Description: 

This project designs, develops and tests a digital gaming environment for high school students that fosters and measures science learning within alternate reality games about saving Earth's ecosystems. Players work together to solve scientific challenges using a broad range of tools including a centralized web-based gaming site and social networking tools, along with handheld smart-phones, and an avatar-based massively multiplayer online environment (MMO). EdGE at TERC joins with GameGurus, high school teachers and assessment specialists to develop Leveling Up. The game requires players to contribute to a scientific knowledge building community; and players rate each other's contributions for their value to the communities' learning and decision-making in solving the challenge. Designers also work with high-school teachers to develop bridge activities that leverage science learning in games for use in formal education. Overall, the project goal is to understand the potential of the gaming environment as a direct intervention and as a catalyst to transform and measure high school STEM learning.

The research on Leveling Up compares the science learning measured within social digital games to class-based assessments of similar content and skills and explains the results using data from design documents, participant observations, surveys, interviews and student work. Formative research and iterative design with a cohort of with 15 testbed classes (grades 10-12) result in a set of assessments that have been validated in terms of scientific constructs and a set of common equivalent curriculum and assessments for implementation studies. In the third year of the project, researchers study 12 treatment classes and 3 control classes to compare students' advancement in the game to their gains on classroom assessments. In addition, half of the testbed classes use the classroom bridge activities and half do not, yielding samples of 180 students for each treatment and 90 students for the control sample. Researchers use multilevel models to examine the impact of the Leveling Up game play and bridge activities on high-school students' science knowledge. Independent evaluators (ILI) validate the interpretation of findings from the formative and implementation research.

Leveling Up is a fundamental first step for the STEM education field to understand how the pervasive social media emerging in today's society, including the phenomena of social digital gaming, can be leveraged to create exciting and productive STEM learning environments for the future. These technologies and knowledge building processes are critical for building a workforce of tomorrow that is scientifically, technologically, and data literate and also embody the inquiry and collaboration skills to contribute to productive and informed decisions about Earth's ecosystems and other important scientific and societal issues of our times. The project, Leveling Up, results in an ongoing STEM gaming environment for the public as well as a model for high school STEM assessment that may be used in other social digital games. Finally, Leveling Up also contributes a model for activities that bridge scientific inquiry occurring in social digital games with skills and content taught in high school STEM classes.

Levels of Conceptual Understanding in Statistics (LOCUS)

LOCUS (Levels of Conceptual Understanding in Statistics) is an NSF Funded DRK12 project (NSF#118618) focused on developing assessments of statistical understanding. These assessments will measure students’ understanding across levels of development as identified in the Guidelines for Assessment and Instruction in Statistics Education (GAISE). The intent of these assessments is to provide teachers and researchers with a valid and reliable assessment of conceptual understanding in statistics consistent with the Common Core State Standards (CCSS).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1118168
Funding Period: 
Thu, 09/01/2011 to Fri, 08/31/2012
Project Evaluator: 
TERC, Jim Hammerman
Full Description: 

The goal of this project is to develop two tests (instruments) to assess conceptual understanding of statistics. The instruments are based on the levels A/B and on level C of statistical understanding development as described in the American Statistical Association Guidelines for Assessment and Instruction of Statistics Education (GAISE) framework. These instruments will be used to assess knowledge of statistics by grades 6-12 students. The instruments will have multiple-choice and constructed response (CR) items. The CR items will have scoring rubrics. The assessments will be pilot tested in school districts in six states. The instruments will be used by teachers to analyze students' growth in understanding of statistics and will be useable for both formative and summative purposes. An assessment blueprint will be developed based on the GAISE framework for selecting and constructing both fixed-choice and open-ended items. An evidenced-based designed process will be used to develop the assessments. The blueprint will be used by the test development committee to develop items. These items will be reviewed by the advisory board considering the main statistics topics to be included on the assessments. Through a layering process, the assessments will be piloted, revised, and field tested with students in grades 6-12 in six states. A three-parameter IRT model will be used in analyzing the items. The work will be done by researchers at the University of Florida with the support of those at the University of Minnesota, the Educational Testing Service, and Kenyon College. Researchers from TERC will conduct a process evaluation with several feedback and redesign cycles.

The assessments will be aligned with the Common Core State Standards for mathematics (CCSSM) and made available as open-source to teachers through a website. The research team will interact with the state consortia developing assessments to measure students' attainment of the CCSSM. As such, the assessments have the potential of being used by a large proportion of students in the country. The more conceptually-based items will provide teachers with concrete examples of what statistics students in grades 6-12 should know.

Continuous Learning and Automated Scoring in Science (CLASS)

This five-year project investigates how to provide continuous assessment and feedback to guide students' understanding during science inquiry-learning experiences, as well as detailed guidance to teachers and administrators through a technology-enhanced system. The assessment system integrates validated automated scorings for students' written responses to open-ended assessment items into the "Web-based Inquiry Science Environment" (WISE) program.

Award Number: 
1119670
Funding Period: 
Thu, 09/01/2011 to Mon, 08/31/2015
Full Description: 

This five-year project investigates how to provide continuous assessment and feedback to guide students' understanding during science inquiry-learning experiences, as well as detailed guidance to teachers and administrators through a technology-enhanced system. The assessment system integrates validated automated scorings for students' written responses to open-ended assessment items (i.e., short essays, science narratives, concept mapping, graphing problems, and virtual experiments) into the "Web-based Inquiry Science Environment" (WISE) program. WISE is an online science-inquiry curricula that supports deep understanding through visualization of processes not directly observable, virtual experiments, graphing results, collaboration, and response to prompts for explanations. In partnership with Educational Testing Services (ETS), project goals are: (1) to develop five automated inquiry assessment activities that capture students' abilities to integrate their ideas and form coherent scientific arguments; (2) to customize WISE by incorporating automated scores; (3) to investigate how students' systematic feedback based on these scores improve their learning outcomes; and (4) to design professional development resources to help teachers use scores to improve classroom instruction, and administrators to make better informed decisions about teacher professional development and inquiry instruction. The project targets general science (life, physical, and earth) in three northern California school districts, five middle schools serving over 4,000 6th-8th grade students with diverse cultural and linguistic backgrounds, and 29 science teachers. It contributes to increase opportunities for students to improve their science achievement, and for teachers and administrators to make efficient, evidence-based decisions about high-quality teaching and learning.

A key research question guides this effort: How automated scoring of inquiry assessments can increase success for diverse students, improve teachers' instructional practices, and inform administrators' decisions about professional development, inquiry instruction, and assessment? To develop science inquiry assessment activities, scoring written responses include semantic, syntax, and structure of meaning analyses, as well as calibration of human-scored items with a computer-scoring system through the c-rater--an ETS-developed cyber learning technology. Validity studies are conducted to compare automated scores with human-scored items, teacher, district, and state scores, including sensitivity to the diverse student population. To customize the WISE curriculum, the project modifies 12 existing units and develops nine new modules. To design adaptive feedback to students, comparative studies explore options for adaptive guidance and test alternatives based on automated scores employing linear models to compare student performance across randomly assigned guidance conditions; controlling for covariates, such as prior science scores, gender, and language; and grouping comparison studies. To design teacher professional development, synthesis reports on auto-scored data are created to enable them to use evidence to guide curricular decisions, and comments' analysis to improve feedback quality. Workshops, classroom observations, and interviews are conducted to measure longitudinal teachers' change over time. To empower administrators' decision making, special data reports, using-evidence activities, individual interviews, and observation of administrators' meetings are conducted. An advisory board charged with project evaluation addresses both formative and summative aspects.

A research-informed model to improve science teaching and learning at the middle school level through cyber-enabled assessment is the main outcome of this effort. A total of 21 new, one- to three-week duration standards-based science units, each with four or more automatically scored items, serve as prototypes to improve students' performance, teachers' instructional approaches, and administrators' school policies and practices.

Pages

Subscribe to Achievement/Growth