Application of Evidence-centered Design to States Large-scale Science Assessment

This project aims to (1) determine ways in which Evidence-Centered Design enhances the quality of large-scale, technology-based science assessments for middle school grades and high school equivalency; (2) implement resulting procedures in operational test development; (3) evaluate the efficiency, effectiveness and generalizability of these procedures, and (4) disseminate findings to the assessment community.

Award Number: 
0733172
Funding Period: 
Saturday, September 1, 2007 to Friday, August 31, 2012
Project Evaluator: 
Haynie Research and Evaluation--Doreen Finkelstein, Kathleen Haynie
Full Description: 

The project began as a collaborative research effort among six organizations—a non-profit research company (SRI International), a university (University of Maryland), a commercial test publishing company (Pearson), Minnesota’s (MN) state department of education, a software engineering firm (Codeguild, Inc.), and an educational evaluation firm (Haynie Research and Evaluation). Due to changes in the affiliation of key personnel, the project transitioned to a collaboration among five organizations--SRI International, ETS, University of Maryland, Pearson and Haynie Research and Evaluation. Together these groups designed and implemented several studies to document the influence of evidence-centered design when applied to Pearson's science assessment design and development processes.

The goals of the project are: (1) to determine leverage points by which ECD can enhance the quality of large-scale technology-based assessments and the efficiency of their design, (2) to implement resulting procedures in operational test development cycles, (3) to evaluate efficiency, effectiveness and generalizability of these procedures, (4) to develop two software wizards to support design of task-based scenarios and assessment items, and (5) to disseminate findings to the assessment community.

This project will develop an exemplar set of design patterns based on the critical benchmarks identified in the Minnesota Academic Standards for science and on the GED science practice indicators and content targets. It is of particular interest in this project that elements of ECD will be applied to an existing large-scale accountability and credentialing assessments, in the context of existing test development and delivery processes.  The project is constrained to maintain adherence to existing test specifications, “look and feel” of tasks, timelines, and delivery and scoring procedures.  Rather than designing new assessment systems or re-engineering existing ones, the present project seeks to identify and implement ideas from ECD in existing large-scale, high-stakes testing programs. Principles of ECD have been implemented in several training workshops for assessment designers and item writers to support the development of scenario-based science tasks. The project's technical report series is available at http://ecd.sri.com.