Evaluation

Options for DR K-12 Project Evaluations

Day: 
Wed

CADRE invites feedback on a draft paper that describes how evaluation can add value to an R&D project, with special attention to the challenges of making a distinction between evaluation and the project’s research activities.

Date/Time: 
3:00 pm to 4:00 pm
Session Type: 
Product Feedback Session
Presenters: 

CADRE is developing a concise "options paper" on project evaluation and invites your feedback in this session. The draft paper, to be distributed to participants, describes challenges in evaluating an R&D project (e.g., how to distinguish between evaluation and the project's research activities). It identifies options for current project leaders, evaluators, and prospective grant applicants to consider. Your feedback in the session will help improve the paper before CADRE revises and distributes it.

Levels of Conceptual Understanding in Statistics (LOCUS)

LOCUS (Levels of Conceptual Understanding in Statistics) is an NSF Funded DRK12 project (NSF#118618) focused on developing assessments of statistical understanding. These assessments will measure students’ understanding across levels of development as identified in the Guidelines for Assessment and Instruction in Statistics Education (GAISE). The intent of these assessments is to provide teachers and researchers with a valid and reliable assessment of conceptual understanding in statistics consistent with the Common Core State Standards (CCSS).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1118168
Funding Period: 
Thu, 09/01/2011 to Fri, 08/31/2012
Project Evaluator: 
TERC, Jim Hammerman
Full Description: 

The goal of this project is to develop two tests (instruments) to assess conceptual understanding of statistics. The instruments are based on the levels A/B and on level C of statistical understanding development as described in the American Statistical Association Guidelines for Assessment and Instruction of Statistics Education (GAISE) framework. These instruments will be used to assess knowledge of statistics by grades 6-12 students. The instruments will have multiple-choice and constructed response (CR) items. The CR items will have scoring rubrics. The assessments will be pilot tested in school districts in six states. The instruments will be used by teachers to analyze students' growth in understanding of statistics and will be useable for both formative and summative purposes. An assessment blueprint will be developed based on the GAISE framework for selecting and constructing both fixed-choice and open-ended items. An evidenced-based designed process will be used to develop the assessments. The blueprint will be used by the test development committee to develop items. These items will be reviewed by the advisory board considering the main statistics topics to be included on the assessments. Through a layering process, the assessments will be piloted, revised, and field tested with students in grades 6-12 in six states. A three-parameter IRT model will be used in analyzing the items. The work will be done by researchers at the University of Florida with the support of those at the University of Minnesota, the Educational Testing Service, and Kenyon College. Researchers from TERC will conduct a process evaluation with several feedback and redesign cycles.

The assessments will be aligned with the Common Core State Standards for mathematics (CCSSM) and made available as open-source to teachers through a website. The research team will interact with the state consortia developing assessments to measure students' attainment of the CCSSM. As such, the assessments have the potential of being used by a large proportion of students in the country. The more conceptually-based items will provide teachers with concrete examples of what statistics students in grades 6-12 should know.

Urban Ecology Course Materials Created with a Universal Design for Learning Framework

The Lynch School of Education and the Urban Ecology Institute at Boston College are partnering with the Center for Applied Special Technology (CAST) to develop, test, evaluate and disseminate a year-long set of urban ecology course materials for use in high school-level capstone science courses. The standards-based materials emphasize locally-relevant field studies and incorporate principles of Universal Design for Learning and Educative Curriculum.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1110524
Funding Period: 
Fri, 10/01/2010 to Fri, 09/30/2011
Full Description: 

The Lynch School of Education and the Urban Ecology Institute at Boston College are partnering with the Center for Applied Special Technology (CAST) to develop, test, evaluate and disseminate a year-long set of urban ecology course materials for use in high school-level capstone science courses. The standards-based materials emphasize locally-relevant field studies and incorporate principles of Universal Design for Learning and Educative Curriculum. Other features include (1) an on-line resource center that links to professional development resources, (2) a student-written urban environment newspaper called "Green Times," and (3) a writing support toolkit. Evaluation and research studies focus on measuring effectiveness of the materials in promoting content understanding, self-efficacy in science and inquiry abilities of students in urban high schools, particularly those from underrepresented groups.

(Note: This project was originally awarded to the Lead Organization, Boston University under the Award #0628143 and for the Funding Period: Sun, 10/01/2006 - Thu, 09/30/2010. Due to a change in institution by the PI of the project, a new award was issued: Award # 1110524)

A Framework for Assessing Environmental Literacy

This workshop developed a new, comprehensive, research-based framework for assessing environmental literacy. By bringing together, for the first time, experts in research, assessment, and evaluation from the fields of science education, environmental education, and related social science fields, this project accessed and built its work on the literature and the insights of many disciplines.

Award Number: 
1033934
Funding Period: 
Mon, 11/15/2010 to Wed, 10/31/2012
Project Evaluator: 
Joe Heimlich, OSU
Full Description: 
This workshop developed a new, comprehensive, research-based framework for assessing environmental literacy. By bringing together, for the first time, experts in research, assessment, and evaluation from the fields of science education, environmental education, and related social science fields, this project accessed and built its work on the literature and the insights of many disciplines. The North American Association for Environmental Education (NAAEE) worked with the leaders of the only two large-scale assessments of environmental literacy used in the U.S. to date (Programme for International Student Assessment [PISA] and the National Environmental Literacy Assessment [NELA]) to conduct the workshop. The project leaders analyzed PISA and NELA and used a multi-disciplinary search and review of the literature to prepare a draft framework. At the workshop and thereafter, a diverse array of invited experts critiqued that draft and provided suggestions for revision. Then, the leaders/organizers produced a final Environmental Literacy Framework and disseminated it both electronically and at a nationally advertised event to a wide audience of assessment specialists, funding and policy-making agencies, and organizations working to develop assessments and achieve environmental literacy. Many institutions and agencies have noted the need to create an environmentally literate population, and government and private entities are investing hundreds of millions of dollars in projects aimed at enhancing environmental literacy. Given the scope and scale of these investments and the interest in this arena on the part of federal agencies, professional organizations, and corporations, assessments for gauging our progress in transforming our preK-12 education system to achieve that end are needed. The new Framework for assessing environmental literacy provides a foundation for measuring the extent to which we are enabling all learners to acquire the knowledge, skills, dispositions, and behaviors vital for competently making decisions about local, regional, national and global issues.
Alternative video text
Alternative video text: 
A video of the National Press Club dissemination event is posted at www.NAAEE.net/Framework

Research and Evaluation

Day: 
Fri

Researchers and evaluators will consider the purposes and functions of evaluation in DR K-12 research and development projects, and test different approaches in a working session.

Date/Time: 
10:00 am to 12:00 pm
Session Type: 
Panel

The session will focus on:  1) considerations about the relationship of evaluation and research in terms of purposes/functions of evaluation for a DR K-12 development and/or research project; and 2) balancing internal/external/advisory board roles within those purposes/functions of evaluation. Panelists will comment on how projects they are involved in—as investigators or evaluators—have negotiated these considerations.

Efficacy Study of Metropolitan Denver's Urban Advantage Program: A Project to Improve Scientific Literacy Among Urban Middle School Students

This is an efficacy study to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The study aims to answer the following questions: How does participation in the program affect students' science knowledge, skills, and attitudes toward science; teachers' science knowledge, skills, and abilities; and families engagement in and support for their children's science learning and aspirations?

Award Number: 
1020386
Funding Period: 
Wed, 09/15/2010 to Wed, 08/31/2011
Project Evaluator: 
Maggie Miller
Full Description: 

This is an efficacy study through which the Denver Museum of Nature and Science, the Denver Zoo, the Denver Botanic Gardens, and three of Denver's urban school districts join efforts to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The Metropolitan Denver Urban Advantage (UA Denver) program is used for this purpose. This program consists of three design elements: (a) student-driven investigations, (b) STEM-related content, and (c) alignment of schools and informal science education institutions; and six major components: (a) professional development for teachers, (b) classroom materials and resources, (c) access to science-rich organizations, (d) outreach to families, (e) capacity building and sustainability, and (e) program assessment and student learning. Three research questions guide the study: (1) How does the participation in the program affect students' science knowledge, skills, and attitudes toward science relative to comparison groups of students? (2) How does the participation in the program affect teachers' science knowledge, skills, and abilities relative to comparison groups of teachers? and (3) How do families' participation in the program affect their engagement in and support for their children's science learning and aspirations relative to comparison families?

The study's guiding hypothesis is that the UA Denver program should improve science literacy in urban middle school students measured by (a) students' increased understanding of science, as reflected in their science investigations or "exit projects"; (b) teachers' increased understanding of science and their ability to support students in their exit projects, as documented by classroom observations, observations of professional development activities, and surveys; and (c) school groups' and families' increased visits to participating science-based institutions, through surveys. The study employs an experimental research design. Schools are randomly assigned to either intervention or comparison groups and classrooms will be the units of analysis. Power analysis recommended a sample of 18 intervention and 18 comparison middle schools, with approximately 72 seventh grade science teachers, over 5,000 students, and 12,000 individual parents in order to detect differences among intervention and comparison groups. To answer the three research questions, data gathering strategies include: (a) students' standardized test scores from the Colorado Student Assessment Program, (b) students' pre-post science learning assessment using the Northwest Evaluation Association's Measures for Academic Progress (science), (c) students' pre-post science aspirations and goals using the Modified Attitude Toward Science Inventory, (d) teachers' fidelity of implementation using the Teaching Science as Inquiry instrument, and (e) classroom interactions using the Science Teacher Inquiry Rubric, and the Reformed Teaching Observation protocol. To interpret the main three levels of data (students, nested in teachers, nested within schools), hierarchical linear modeling (HLM), including HLM6 application, are utilized. An advisory board, including experts in research methodologies, science, informal science education, assessment, and measurement oversees the progress of the study and provides guidance to the research team. An external evaluator assesses both formative and summative aspects of the evaluation component of the scope of work.

The key outcome of the study is a research-informed and field-tested intervention implemented under specific conditions for enhancing middle school science learning and teaching, and supported by partnerships between formal and informal organizations.

Interactive Science and Technology Instruction for English Learners (RAPID)

This project examines the first-year implementation of a program that will provide low-cost netbook computers and specialized software to fifth and sixth grade students in four schools in Southern California. The PIs collect baseline and early implementation data to determine effects of the intervention on students' academic achievement in science, academic writing in science, and interest in further STEM study.

Project Email: 
Award Number: 
1053767
Funding Period: 
Fri, 10/01/2010 to Fri, 09/30/2011
Full Description: 

This is a RAPID award to investigators at the University of California, Irvine, to examine the first-year implementation of a program that will provide low-cost netbook computers and specialized software to fifth and sixth grade students in four schools in Southern California. The PIs collect baseline and early implementation data to determine effects if the intervention on students' academic achievement in science, academic writing in science, and interest in further STEM study. They also examine the extent to which participation in the program improves student access to, use of, and self-perceived proficiency with technology and how these attributes are mediated by socioeconomic status, ethnicity, and English learner status. Additionally, they examine the effect of the program on teachers' knowledge of and use of technology for instruction.

Four schools from the same school district with similar demographics serve as comparison schools in the study. Additionally, all fifth and sixth grade teachers participate in the study with four program teachers (two at fifth grade and two at sixth grade) participating more extensively as focus teachers. Both qualitative and quantitative methods are used to examine the effects of the program. 

The products include analysis of extensive data on implementation, learning and attitudes. A total of 531 students are involved in the study as well as their teachers. The findings are likely to guide subsequent implementation and research on full implementation within the targeted schools.

Studying Topography, Orographic Rainfall, and Ecosystems (STORE) with Geospatial Information Technology

This project is using innovative Geospatial Information Technology-based learning in high school environmental science studies with a focus on the meteorological and ecological impacts of climate change. The resources developed are using ArcGIS Explorer Desktop and Google Earth software applications to increase students' learning and interest in science and careers and will be adaptable for teachers to improve classroom implementation.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1019645
Funding Period: 
Wed, 09/01/2010 to Sat, 08/31/2013
Project Evaluator: 
Haynie Research and Evaluation
Full Description: 

STORE is developing and piloting classroom uses of GIS-based interactive data files displaying climatological, topographical, and biological data about an especially ecologically and topographically diverse section of mid-California and a section of western New York State, plus projected climate change outcomes in 2050 and 2099 from an IPCC climate change model. Both areas contain weather stations. The participating students and teachers live in those areas, hence the place-based focus of the project.

To help teachers make curricular decisions about how to use these data with their students, the project has, with input from six design partner teachers, produced a curriculum module exemplar consisting of six lessons. The lessons start with basic meteorological concepts about the relationship between weather systems and topography, then focus on recent climatological and land cover data. The last two lessons focus on IPCC-sanctioned climate change projections in relation to possible fates of different regional species. Technology light versions of these lessons send students directly to map layers displaying the data for scientific analysis. Technology-heavy versions address the additional goal of building students' capacities to manipulate features of geographic information systems (GIS). Hence, the technology-heavy versions require use of the ARC GIS Explorer Desktop software, whereas the technology light versions are available in both the ARC software and in Google Earth. Google Earth makes possible some student interactivity such as drawing transects and studying elevation profiles, but does not support more advanced use of geographic information system technology such as queries of data-containing shape files or customization of basemaps and data representational symbology.

Answer keys are provided for each lesson. Teachers have in addition access to geospatial data files that display some storm systems that moved over California in the winter of 2010-2001 so that students can study relationships between actual data about storm behavior and relationship to topography and the climatological data which displays those relationships in a summary manner. This provides the student the opportunity to explore differences between weather and climate.

To increase the likelihood of successful classroom implementation and impact on student learning, the professional development process provides the conditions for teachers to make good adaptability decisions for successful follow-through. Teachers can implement the six lessons or adapt them or design their own from scratch. The project requires that they choose from these options, explain on content representation forms their rationales for those decisions, and provide assessment information about student learning outcomes from their implementations. The project provides the teachers with assessment items that are aligned to each of the six lessons, plus some items that test how well the students can interpret the STORE GIS data layers.

All of this work is driven by the hypothesis that science teachers are more likely to use geospatial information technology in their classrooms when provided with the types of resources that they are provided in this project. In summary, these resources include:

1.     tutorials about how to use the two GIS applications

2.     sufficiently adaptive geospatial data available in free easily transportable software applications

3.     lessons that they can implement as is, adapt, or discard if they want to make up their own (as long as they use the data)

4.     supportive resources to build their content knowledge (such as overview documents about their states' climates and information about the characteristics of each data layer and each data set available to them).

 

The growth and evolution of the teachers' technological pedagogical content knowledge is being tracked through interviews, face-to-face group meetings, and classroom observations. Also being tracked is the extent to which the teachers and students can master the technology applications quickly and on their own without workshops, and how well teachers provide feedback to the students and assess their learning outcomes when implementing STORE lessons. As the project moves into its third and final year, we will be studying outcomes from the first classroom implementation year (i.e. year two of the project) and determining to what extent the professional development strategies need to be revised in relation to how the teachers are responding to the project resources and forms of professional support. In the end, the project will contribute to the knowledge base about what professional development strategies are appropriate for getting teachers to use these types of resources, what decisions teachers make about how to use the resources for different courses and student groups they teach, and what are the outcomes of those uses in terms of curricular material, instructional strategies, and student learning.

Evaluating the Developing Mathematical Ideas Professional Development Program: Researching its Impact on Teaching and Student Learning

This is a 3.5-year efficacy study of the Developing Mathematical Ideas (DMI) elementary math teacher professional development (PD) program. DMI is a well-known, commercially available PD program with substantial prior evidence showing its impact on elementary teachers' mathematical and pedagogical knowledge. However, no studies have yet linked DMI directly with changes in teachers' classroom practice, or with improved student outcomes in math. This study aims to remedy this gap.

Project Email: 
Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1019769
Funding Period: 
Wed, 09/01/2010 to Fri, 08/31/2012
Project Evaluator: 
Bill Nave
Full Description: 

This is a 3.5-year efficacy study of the Developing Mathematical Ideas (DMI) elementary math teacher professional development (PD) program. DMI was developed by staff from Education Development Center (EDC), SummerMath for Teachers, and TERC, the STEM research and development institution responsible for this research. DMI is a well-known, commercially available PD program with substantial prior evidence showing its impact on elementary teachers' mathematical and pedagogical knowledge. However, no studies have yet linked DMI directly with changes in teachers' classroom practice, or with improved student outcomes in math. This study aims to remedy this gap.

The research questions for the study are:

1) Does participation in the Developing Mathematical Ideas (DMI) professional development program lead to increases in reform-oriented teaching?

2) Does participation in DMI lead to increases in students' mathematics learning and achievement, especially in their ability to explain their thinking and justify their answers?

3) What is the process by which a reform-oriented professional development program can influence teaching practice and, thus, student learning? Through what mechanisms does DMI have impact, and with what kinds of support do we see the desired changes on our outcome measures when the larger professional development context is examined?

The dependent variables for this study include a) teachers' pedagogical and mathematics knowledge for teaching; b) the nature of their classroom practice; and c) student learning/ achievement in mathematics.

The study uses experimental and quasi-experimental methods, working with about 195 elementary grades teachers and their students in Boston, Springfield, Leominster, Fitchburg, and other Massachusetts public schools. Volunteer teachers are randomly assigned either to PD with DMI in the first year of the efficacy study, or to a control group that will wait until the second year of the study to receive DMI PD. Both groups of teachers will be followed through two academic years. Analyses use OLS regression, hierarchical modeling, and structural equation modeling, as appropriate, to compare the two groups and to track changes over time. In this way, the project explores several aspects of a conceptual framework hypothesizing relationships among PD, teacher mathematical and pedagogical knowledge, classroom teaching practice, and student outcomes. There are multiple measures of each construct, including video-analysis of teacher practice, and a new video-based measure of teacher knowledge.

The study tests the impact of DMI in a range of districts (large urban, small urban, suburban) serving an ethnically and economically diverse mix of students. It provides much needed, rigorous evidence testing the efficacy of this reform-oriented professional development program. It also directly explores the commonplace theory that teachers' understanding of content and student thinking and their encouragement of rich mathematical discourse for student sense-making lead to improvement on measures of mathematics achievement. Findings from the study are disseminated to both research and practitioner communities. The project provides professional development in mathematics to about 195 teachers to improve their ability to teach important concepts. If the evidence for efficacy is positive, then even larger-scale use of this PD program is likely.

Pages

Subscribe to Evaluation