Assessment

Levels of Conceptual Understanding in Statistics (LOCUS)

LOCUS (Levels of Conceptual Understanding in Statistics) is an NSF Funded DRK12 project (NSF#118618) focused on developing assessments of statistical understanding. These assessments will measure students’ understanding across levels of development as identified in the Guidelines for Assessment and Instruction in Statistics Education (GAISE). The intent of these assessments is to provide teachers and researchers with a valid and reliable assessment of conceptual understanding in statistics consistent with the Common Core State Standards (CCSS).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1118168
Funding Period: 
Thu, 09/01/2011 to Fri, 08/31/2012
Project Evaluator: 
TERC, Jim Hammerman
Full Description: 

The goal of this project is to develop two tests (instruments) to assess conceptual understanding of statistics. The instruments are based on the levels A/B and on level C of statistical understanding development as described in the American Statistical Association Guidelines for Assessment and Instruction of Statistics Education (GAISE) framework. These instruments will be used to assess knowledge of statistics by grades 6-12 students. The instruments will have multiple-choice and constructed response (CR) items. The CR items will have scoring rubrics. The assessments will be pilot tested in school districts in six states. The instruments will be used by teachers to analyze students' growth in understanding of statistics and will be useable for both formative and summative purposes. An assessment blueprint will be developed based on the GAISE framework for selecting and constructing both fixed-choice and open-ended items. An evidenced-based designed process will be used to develop the assessments. The blueprint will be used by the test development committee to develop items. These items will be reviewed by the advisory board considering the main statistics topics to be included on the assessments. Through a layering process, the assessments will be piloted, revised, and field tested with students in grades 6-12 in six states. A three-parameter IRT model will be used in analyzing the items. The work will be done by researchers at the University of Florida with the support of those at the University of Minnesota, the Educational Testing Service, and Kenyon College. Researchers from TERC will conduct a process evaluation with several feedback and redesign cycles.

The assessments will be aligned with the Common Core State Standards for mathematics (CCSSM) and made available as open-source to teachers through a website. The research team will interact with the state consortia developing assessments to measure students' attainment of the CCSSM. As such, the assessments have the potential of being used by a large proportion of students in the country. The more conceptually-based items will provide teachers with concrete examples of what statistics students in grades 6-12 should know.

Continuous Learning and Automated Scoring in Science (CLASS)

This five-year project investigates how to provide continuous assessment and feedback to guide students' understanding during science inquiry-learning experiences, as well as detailed guidance to teachers and administrators through a technology-enhanced system. The assessment system integrates validated automated scorings for students' written responses to open-ended assessment items into the "Web-based Inquiry Science Environment" (WISE) program.

Award Number: 
1119670
Funding Period: 
Thu, 09/01/2011 to Mon, 08/31/2015
Full Description: 

This five-year project investigates how to provide continuous assessment and feedback to guide students' understanding during science inquiry-learning experiences, as well as detailed guidance to teachers and administrators through a technology-enhanced system. The assessment system integrates validated automated scorings for students' written responses to open-ended assessment items (i.e., short essays, science narratives, concept mapping, graphing problems, and virtual experiments) into the "Web-based Inquiry Science Environment" (WISE) program. WISE is an online science-inquiry curricula that supports deep understanding through visualization of processes not directly observable, virtual experiments, graphing results, collaboration, and response to prompts for explanations. In partnership with Educational Testing Services (ETS), project goals are: (1) to develop five automated inquiry assessment activities that capture students' abilities to integrate their ideas and form coherent scientific arguments; (2) to customize WISE by incorporating automated scores; (3) to investigate how students' systematic feedback based on these scores improve their learning outcomes; and (4) to design professional development resources to help teachers use scores to improve classroom instruction, and administrators to make better informed decisions about teacher professional development and inquiry instruction. The project targets general science (life, physical, and earth) in three northern California school districts, five middle schools serving over 4,000 6th-8th grade students with diverse cultural and linguistic backgrounds, and 29 science teachers. It contributes to increase opportunities for students to improve their science achievement, and for teachers and administrators to make efficient, evidence-based decisions about high-quality teaching and learning.

A key research question guides this effort: How automated scoring of inquiry assessments can increase success for diverse students, improve teachers' instructional practices, and inform administrators' decisions about professional development, inquiry instruction, and assessment? To develop science inquiry assessment activities, scoring written responses include semantic, syntax, and structure of meaning analyses, as well as calibration of human-scored items with a computer-scoring system through the c-rater--an ETS-developed cyber learning technology. Validity studies are conducted to compare automated scores with human-scored items, teacher, district, and state scores, including sensitivity to the diverse student population. To customize the WISE curriculum, the project modifies 12 existing units and develops nine new modules. To design adaptive feedback to students, comparative studies explore options for adaptive guidance and test alternatives based on automated scores employing linear models to compare student performance across randomly assigned guidance conditions; controlling for covariates, such as prior science scores, gender, and language; and grouping comparison studies. To design teacher professional development, synthesis reports on auto-scored data are created to enable them to use evidence to guide curricular decisions, and comments' analysis to improve feedback quality. Workshops, classroom observations, and interviews are conducted to measure longitudinal teachers' change over time. To empower administrators' decision making, special data reports, using-evidence activities, individual interviews, and observation of administrators' meetings are conducted. An advisory board charged with project evaluation addresses both formative and summative aspects.

A research-informed model to improve science teaching and learning at the middle school level through cyber-enabled assessment is the main outcome of this effort. A total of 21 new, one- to three-week duration standards-based science units, each with four or more automatically scored items, serve as prototypes to improve students' performance, teachers' instructional approaches, and administrators' school policies and practices.

Development of a Cognition-Guided, Formative-Assessment-Intensive, Individualized Computer-Based Dynamic Geometry Learning System for Grades 3-8

This project is focused on creating, testing, refining, and studying a computer-based, individualized, interactive learning system for intermediate/middle school students or by teachers in classrooms. This learning system is called Individualized Dynamic Geometry Instruction and will contain four instructional modules in geometry and measurement that reflect the recommendations of the Common Core State Standards.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1119034
Funding Period: 
Thu, 09/01/2011 to Mon, 08/31/2015
Project Evaluator: 
Jeff Shih
Full Description: 

Developers and researchers at Ohio State University and KCP Technologies are creating, testing, refining, and studying a computer-based, individualized, interactive learning system for intermediate/middle school students that can be used by them independently (online or offline) or by teachers in classrooms. This learning system is called Individualized Dynamic Geometry Instruction (iDGi) and will contain four instructional modules in geometry and measurement that reflect the recommendations of the Common Core State Standards (CCSS). iDGi courseware fully integrates research-based Learning Progressions (LPs) for guiding students' reasoning; formative-assessment linked to LPs; instructional sequencing that interactively adapts to students' locations in LPs; built-in student monitoring, feedback, and guidance; and research-based principles of educational media into the modules. The software platform for iDGi development is an extended version of the dynamic geometry computer environment, The Geometer's Sketchpad.

The development process follows recommendations in Douglas Clements' Curriculum Research Framework and includes sequences of development, trials with students, data collection, and revision. The research and evaluation are based on random assignment of approximately 350 students to treatment and control groups. Achievement data are collected using developer-constructed instruments with items that reflect the mathematics topics in the CCSS. Researchers explore the variability at the student, teacher, and school levels using the appropriate level of hierarchical linear models.

Commercial publishers have expressed strong interest in publishing online and offline computer versions of iDGi, an iPad version of iDGi, an online management system for iDGi, and support materials for users and teachers.

Curriculum and Assessment Partnership Conference

This conference's primary goal is to discuss issues and recommend actions regarding assessment design and production that arise in an environment of Common Core State Standards (CCSS). Participants will include representatives from the governing states as well as from the leadership teams of the assessment consortia along with experts in curriculum development, standards implementation and assessment.

Award Number: 
1108723
Funding Period: 
Tue, 03/01/2011 to Wed, 02/29/2012
Full Description: 

The Consortium for Mathematics and Its Applications is engaged in an effort to bring together members of the community of mathematics curriculum developers and experts in assessment, including representatives and stakeholders from the two assessment consortia recently funded by the US Department of Education. The primary goal of the meeting is to discuss issues and recommend actions regarding assessment design and production that arise in an environment of Common Core State Standards (CCSS). The format will allow information exchange and resource sharing in the service of states and districts as they work to effectively implement the CCSS. Participants will include representatives from the governing states as well as from the leadership teams of the assessment consortia along with experts in curriculum development, standards implementation and assessment.

This work is being carried out with the support and cooperation of the National Council of Teachers of Mathematics, the National Council of Supervisors of Mathematics, the Association of Mathematics Teacher Educators, and the Association of State Supervisors of Mathematics.

A Framework for Assessing Environmental Literacy

This workshop developed a new, comprehensive, research-based framework for assessing environmental literacy. By bringing together, for the first time, experts in research, assessment, and evaluation from the fields of science education, environmental education, and related social science fields, this project accessed and built its work on the literature and the insights of many disciplines.

Award Number: 
1033934
Funding Period: 
Mon, 11/15/2010 to Wed, 10/31/2012
Project Evaluator: 
Joe Heimlich, OSU
Full Description: 
This workshop developed a new, comprehensive, research-based framework for assessing environmental literacy. By bringing together, for the first time, experts in research, assessment, and evaluation from the fields of science education, environmental education, and related social science fields, this project accessed and built its work on the literature and the insights of many disciplines. The North American Association for Environmental Education (NAAEE) worked with the leaders of the only two large-scale assessments of environmental literacy used in the U.S. to date (Programme for International Student Assessment [PISA] and the National Environmental Literacy Assessment [NELA]) to conduct the workshop. The project leaders analyzed PISA and NELA and used a multi-disciplinary search and review of the literature to prepare a draft framework. At the workshop and thereafter, a diverse array of invited experts critiqued that draft and provided suggestions for revision. Then, the leaders/organizers produced a final Environmental Literacy Framework and disseminated it both electronically and at a nationally advertised event to a wide audience of assessment specialists, funding and policy-making agencies, and organizations working to develop assessments and achieve environmental literacy. Many institutions and agencies have noted the need to create an environmentally literate population, and government and private entities are investing hundreds of millions of dollars in projects aimed at enhancing environmental literacy. Given the scope and scale of these investments and the interest in this arena on the part of federal agencies, professional organizations, and corporations, assessments for gauging our progress in transforming our preK-12 education system to achieve that end are needed. The new Framework for assessing environmental literacy provides a foundation for measuring the extent to which we are enabling all learners to acquire the knowledge, skills, dispositions, and behaviors vital for competently making decisions about local, regional, national and global issues.
Alternative video text
Alternative video text: 
A video of the National Press Club dissemination event is posted at www.NAAEE.net/Framework

Establishing a Roadmap for Large-Scale Improvement of K-12 Education in the Geographical Sciences

This project will engage in a community-wide effort to synthesize the literature from a broad range of fields and to use the findings to create frameworks that will guide the planning, implementation, and scale-up of efforts to improve geographic education over the next decade. This will result in a set of publicly reviewed, consensus reports that will guide collaborative efforts and broaden awareness of the acute need for geographic literacy and geographic science education.

Award Number: 
1049437
Funding Period: 
Wed, 09/15/2010 to Fri, 08/31/2012
Project Evaluator: 
Education and Training Institute
Full Description: 

Having a geographically literate population will be critical to the economic stability, physical security, and environmental sustainability of the United States in the 21st century. Yet the U.S. still lags far behind the other developed nations in education in the geographical sciences. Recognizing the risk that geographic illiteracy poses for our country, the National Geographic Society (NGS), in collaboration with the Association of American Geographers, American Geographical Society, and National Council for Geographic Education, proposes to engage in a set of research synthesis and dissemination activities that will provide road maps for the design of assessment, professional development, instructional materials, public information, and educational research for the next decade. The work will be done by a broad range of experts from K-12 institutions as well as the geographical science and educational research communities

Building on a 25 year collaboration, NGS and its partners propose to engage in a community-wide effort to synthesize the literature from a broad range of fields and to use the findings to create frameworks that will guide the planning, implementation, and scale-up of efforts to improve geographic education over the next decade. The result of this effort will be a set of publicly reviewed, consensus reports that will guide the collaborative efforts of the project partners and the larger geographic education community, as well as broaden awareness of the increasingly significant and acute need for geographic literacy and education in the geographical sciences in our country.

This project will create three in-depth "roadmap" reports targeted at practitioners, takeholders, and policymakers. Developed by expert committees, these three reports will be on:

- Assessment frameworks for systematic monitoring and continuous improvement of geographic education programs.

- Professional development for teachers and instructional materials to support large-scale educational improvement across diverse contexts.

- Educational research agenda to set priorities and identify appropriate methodologies for research that will improve geographic education into the future.

These three reports will be summarized in an executive summary written for a broad audience of educators, policymakers, and concerned citizens.

In addition to these consensus reports, the project will also conduct research on public understanding of the nature and importance of geographic literacy, with particular attention to the key audiences of educators, policymakers, and citizens. In addition to shaping the project's reports, this research will inform the broader communications and dissemination efforts of this project and its partners.

A Learning Progression-Based System for Promoting Understanding of Carbon-Transforming Processes

This project builds on prior efforts with learning progressions, and is focused on key carbon-transforming processes in socio-ecological systems at multiple scales, including cellular and organismal metabolism, ecosystem energetics and carbon cycling, carbon sequestration, and combustion of fossil fuels. The primary project outcomes will be coordinated instructional tools that are useful to professionals at all levels in the science education system--classroom teachers, professional developers, and developers of curricula, standards and assessments.

Lead Organization(s): 
Award Number: 
1020187
Funding Period: 
Wed, 09/15/2010 to Mon, 08/31/2015
Project Evaluator: 
Rose Shaw
Full Description: 

This project--led by science educators at Michigan State University, the National Geographic Society, the Natural Resource Ecology Laboratory (NREL) at Colorado State University, the Berkeley Evaluation and Assessment Research (BEAR) Center, and AAAS Project 2061, and including schools in California, Colorado, Maryland, Michigan, and Washington--builds on prior efforts with learning progressions, and is focused on key carbon-transforming processes in socio-ecological systems at multiple scales, including cellular and organismal metabolism, ecosystem energetics and carbon cycling, carbon sequestration, and combustion of fossil fuels.

The project uses an iterative design research process to develop and refine a suite of tools for reasoning and test efficacy of those tools in geographically and culturally diverse schools. The project team is:

1. Refining and validating a detailed learning progression framework covering the middle and high school years; ultimately, the framework will describe the development of students' capacity to use fundamental principles such as conservation of matter and energy to reason about carbon-transforming processes at multiple scales.

2. Refining 'Tools for Reasoning' that make hidden scientific principles - matter, energy, and scale - visible to students; the power of these tools lies in their flexible use for different processes, systems, scales, and curricular contexts.

3. Developing and refining flexible teaching strategies that engage students in cognitive apprenticeship in the practices of environmental science literacy: a) inquiry and argumentation, b) explanations and predictions, and c) decision-making about environmental issues.

4. Using and refining existing summative assessments, and developing and testing formative assessment tools; these assessment tools will provide teachers and researchers with immediate information about their students' intellectual resources and will be linked to the learning progression framework.

5. Developing, field testing, and assessing the effectiveness of six middle school and six high school units that use project tools and enact project principles; the units introduce students to fundamental principles, engage them in reasoning about carbon-transforming processes at organismal scale, and at landscape and global scales. Each unit includes a) an online formative assessment and b) activity sequences that use tools for reasoning and teaching strategies.

6. Developing, field testing, and assessing professional development materials in both face-to-face and facilitated online forms; the materials introduce teachers to learning progressions in environmental science literacy, assessment tools, tools for reasoning, teaching strategies, and teaching materials and activities, and also address difficulties that teachers encounter in using learning progressions and enacting teaching strategies.

The primary project outcomes will be coordinated instructional tools that are useful to professionals at all levels in the science education system--classroom teachers, professional developers, and developers of curricula, standards and assessments.

Efficacy Study of Metropolitan Denver's Urban Advantage Program: A Project to Improve Scientific Literacy Among Urban Middle School Students

This is an efficacy study to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The study aims to answer the following questions: How does participation in the program affect students' science knowledge, skills, and attitudes toward science; teachers' science knowledge, skills, and abilities; and families engagement in and support for their children's science learning and aspirations?

Award Number: 
1020386
Funding Period: 
Wed, 09/15/2010 to Wed, 08/31/2011
Project Evaluator: 
Maggie Miller
Full Description: 

This is an efficacy study through which the Denver Museum of Nature and Science, the Denver Zoo, the Denver Botanic Gardens, and three of Denver's urban school districts join efforts to determine if partnerships among formal and informal organizations demonstrate an appropriate infrastructure for improving science literacy among urban middle school science students. The Metropolitan Denver Urban Advantage (UA Denver) program is used for this purpose. This program consists of three design elements: (a) student-driven investigations, (b) STEM-related content, and (c) alignment of schools and informal science education institutions; and six major components: (a) professional development for teachers, (b) classroom materials and resources, (c) access to science-rich organizations, (d) outreach to families, (e) capacity building and sustainability, and (e) program assessment and student learning. Three research questions guide the study: (1) How does the participation in the program affect students' science knowledge, skills, and attitudes toward science relative to comparison groups of students? (2) How does the participation in the program affect teachers' science knowledge, skills, and abilities relative to comparison groups of teachers? and (3) How do families' participation in the program affect their engagement in and support for their children's science learning and aspirations relative to comparison families?

The study's guiding hypothesis is that the UA Denver program should improve science literacy in urban middle school students measured by (a) students' increased understanding of science, as reflected in their science investigations or "exit projects"; (b) teachers' increased understanding of science and their ability to support students in their exit projects, as documented by classroom observations, observations of professional development activities, and surveys; and (c) school groups' and families' increased visits to participating science-based institutions, through surveys. The study employs an experimental research design. Schools are randomly assigned to either intervention or comparison groups and classrooms will be the units of analysis. Power analysis recommended a sample of 18 intervention and 18 comparison middle schools, with approximately 72 seventh grade science teachers, over 5,000 students, and 12,000 individual parents in order to detect differences among intervention and comparison groups. To answer the three research questions, data gathering strategies include: (a) students' standardized test scores from the Colorado Student Assessment Program, (b) students' pre-post science learning assessment using the Northwest Evaluation Association's Measures for Academic Progress (science), (c) students' pre-post science aspirations and goals using the Modified Attitude Toward Science Inventory, (d) teachers' fidelity of implementation using the Teaching Science as Inquiry instrument, and (e) classroom interactions using the Science Teacher Inquiry Rubric, and the Reformed Teaching Observation protocol. To interpret the main three levels of data (students, nested in teachers, nested within schools), hierarchical linear modeling (HLM), including HLM6 application, are utilized. An advisory board, including experts in research methodologies, science, informal science education, assessment, and measurement oversees the progress of the study and provides guidance to the research team. An external evaluator assesses both formative and summative aspects of the evaluation component of the scope of work.

The key outcome of the study is a research-informed and field-tested intervention implemented under specific conditions for enhancing middle school science learning and teaching, and supported by partnerships between formal and informal organizations.

Supporting Scientific Practices in Elementary and Middle School Classrooms

This project will develop a learning progression that characterizes how learners integrate and interrelate scientific argumentation, explanation and scientific modeling, building ever more sophisticated versions of practice over time using the three common elements of sense-making, persuading peers and developing consensus.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1020316
Funding Period: 
Wed, 09/01/2010 to Fri, 08/31/2012
Full Description: 

Research on student learning has developed separate progressions for scientific argumentation, explanation and scientific modeling. Engaging Learners in Scientific Practices develops a learning progression that characterizes how learners integrate and interrelate scientific argumentation, explanation and scientific modeling, building ever more sophisticated versions of practice over time using the three common elements of sense-making, persuading peers and developing consensus. The learning progression is constructed through improvements in students' performance and understanding of scientific practice as measured by their attention to generality of explanation, attention to clarity of communication and audience understanding, attention to evidentiary support, and attention to mechanistic versus descriptive accounts. The project is led by researchers at Northwestern University, the University of Texas, Wright State University, Michigan State University, and the BEAR assessment group. Two cohorts of 180 students each are followed for two years from 4th to 5th grade in Illinois and two cohorts of 180 students each are followed for two years from 5th to 6th grade in Michigan The elementary school students will work with FOSS curriculum units modified to embed supports for scientific practices. Two cohorts of 500 middle school students are followed for three years from 6th to 8th grade as they work with coordinated IQWST units over three years. The outcome measures include analyses of classroom discourse, pre- and pos-test assessments of student learning, and reflective interviews grounded in students' own experiences with practices in the classroom to assess their growth across the dimensions. The BEAR team is responsible for validation and calibration of the frameworks and instruments, and design of the scheme for analysis of the data. Horizon Research performs the formative and summative evaluation. The project will produce an empirically-tested learning progression for scientific practices for grades 4-8 along with tested curriculum materials and validated assessment items that support and measure students' ability in the scientific practices of explanation, argumentation and modeling. In the process of development, an understanding is gained about how to design and test this learning progression. The framework is articulated on a website for use by other researchers and developers. The project also builds capacity by educating several graduate students.


Project Videos

2019 STEM for All Video Showcase

Title: Science Storylines

Presenter(s): Brian Reiser, Kelsey Edwards, Barbara Hug, Tara McGill, Jamie Noll, Michael Novak, Bill Penuel, Trey Smith, & Aliza Zivic


Cultivating Hispanics and African Americans Reading, Math, Science (CHARMS) in Elementary Schools for Girls Conference

This project is analyzing and sharing baseline data on the achievement of African American and Hispanic girls on national and state assessments. The objectives of the project are to: (1) conduct a critical analysis of achievement data for African American and Hispanic female students; (2) organize a conference featuring presentation of the data analysis and a national speaker; (3) provide STEM career information and materials; and (4) share results of the achievement data analysis.

Lead Organization(s): 
Award Number: 
1048544
Funding Period: 
Wed, 09/01/2010 to Wed, 08/31/2011
Full Description: 

Led by STEM educators at Texas A&M University, this project is analyzing and sharing baseline data on the achievement of African American and Hispanic girls on national and state assessments. The objectives of the project are: (1) To conduct a critical analysis of National Assessment of Educational Progress (NAEP) and Texas Assessment of Knowledge and Skills (TAKS) achievement data for African American and Hispanic female students in grades 3-6 with a focus on sub-test objectives for science, mathematics, and reading over the years 2000-2010; (2) To organize a one-day conference for 100 teachers, administrators and parents from urban, rural and suburban school districts featuring presentation of the data analysis and a national speaker who will share information and lead discussion on why African American and Hispanic girls at the elementary level should begin to think about seeking STEM careers and the required expected academic preparations; (3) To provide conference participants with STEM career information and materials; and (4) To share results of the achievement data analysis at international/national conferences (National Council for Teachers of Mathematics, National Science Teachers Association, American Educational Research Association) and submit papers for publication in scholarly journals.

Quantitative and qualitative methodology will be used to respond to three research questions: (1) What are the differences in the academic achievement of African American and Hispanic girls in grades 3-6 on the National Assessment of Educational Progress (NAEP) and Texas Assessment of Knowledge and Skills (TAKS) during the years 2000-2010? (2) What are the voices of African American and Hispanic 6th grade girls about their TAKS test from third grade to sixth grade? (3) What is the impact of a one-day conference on raising the awareness level of educators and parents about academic achievement among African American and Hispanic girls on national and state assessment in grades 3-6 in reading, mathematics, and science? To address question number one, the study will determine if statistically significant differences exist among the variables of race, class, and gender by grades and subject on student performance on the NAEP and TAKS tests and sub-tests in the areas of reading, mathematics and science. To address question number two, a qualitative analysis will be conducted. Students will be interviewed and data will be transcribed, sorted, and categorized into themes. Member checks and triangulation of data will be used to establish validity and reliability of the findings. To address question number three, descriptive statistics will be used to analyze a Likert-type survey instrument that will be developed by the project PI and CoPIs to assess conference objectives. In addition, a purposive sample of participants (teachers and parents) will be interviewed about their participation in the conference and their responses analyzed using qualitative analysis.

With a focus on African American and Hispanic girls' academic achievement, the project will provide educators, parents and students through a conference venue and other outlets with valuable information to understand their competency in subjects that can impact their decisions to seek STEM careers.

Pages

Subscribe to Assessment