Login

Validity

Examining Formative Assessment Practices for English Language Learners in Science Classrooms (Collaborative Research: Li)

This is an exploratory study to identify critical aspects of effective science formative assessment (FA) practices for English Language Learners (ELLs), and the contextual factors influencing such practices. FA, in the context of the study, is viewed as a process contributing to the science learning of ELLs, as opposed to the administration of discrete sets of instruments to collect data from students. The study targets Spanish-speaking, elementary and middle school students.

Lead Organization(s): 
Award Number: 
1118951
Funding Period: 
Thu, 09/01/2011 - Sat, 08/31/2013
Project Evaluator: 
Advisory board members
Full Description: 

This is a two-year exploratory study to identify critical aspects of effective science formative assessment (FA) practices for English Language Learners (ELLs), and the contextual factors influencing such practices. Three institutions join efforts for this purpose: University of Colorado at Boulder, University of Colorado at Denver, and University of Washington. FA, in the context of the study, is viewed as a process contributing to the science learning of ELLs, as opposed to the administration of discrete sets of instruments to collect data from students. The study targets Spanish-speaking, elementary and middle school students. Findings from this study contribute to advance knowledge and understanding of FA as an inherent component of the science learning process in linguistically diverse classrooms, and to define a research agenda aimed at enhancing science teachers' ability to enact equitable and effective assessment practices for this student subpopulation.

Three research questions guide the work: (1) What FA practices are occurring in science classrooms that serve predominantly mainstream students and in those serving predominantly ELLs?; (2) How are teachers' FA practices for mainstream students different from or similar to those used with ELLs?; and (3) How do contextual factors and teachers' cultural and linguistic competencies influence FA practices? To address these questions, two conceptual frameworks are used--one for characterizing FA events; the other for examining FA events as a communication process. The study employs a mixed-methods research approach with emphasis on case studies. The sample size consists of three school districts in Colorado and Washington, 16 classrooms (8 elementary, 8 middle school), 16 teachers, and 96 ELLs. Classrooms are selected to represent a particular combination of four factors: (a) teacher ethnicity, (b) teacher formal academic preparation in teaching ELLs, (c) type of linguistic student background, and (d) grade level. Students are selected through a stratified random sample, identified by achievement level (i.e., low, medium, high), and linguistic background (i.e., mainstream, ELL). Data collection strategies to document the implementation of FA at the beginning, during, and at the end of a science unit include: (a) classroom observation protocols, (b) classroom video-recording, (c) video/artifact simulated recall, (d) assessment artifacts, (e) student interviews, (f) teacher questionnaires, (g) teacher interviews, (h) school principal interviews, and (i) school observations. Reliability and validity of most of the data-gathering instruments is determined through pilot studies. Data interpretation strategies include: (a) coding based on the two conceptual frameworks, (b) scoring rubrics to identify levels of effectiveness, and (c) narratives and profiles to describe FA patterns. Publications and the development of a website constitute the main dissemination strategies. A technical advisory board is responsible for formative and summative evaluation. Key evaluation questions are: (1) To what extent does the project enhance research on ELL FA practices through case studies?, and (2) How effectively do the project dissemination activities facilitate understanding of FA practices?

Major project outcomes include: (1) a description of the patterns of formal and informal FA practices for ELLs; (2) a comparison of the FA practices observed in classrooms that vary on the dimensions of teacher characteristics and linguistic diversity; and (3) an empirically and theoretically informed set of findings and strategies for supporting teachers to enact and enhance FA practices sensitive to cultural and linguistic diversity. Three main products are developed: (1) a monograph describing the FA practices observed across the different classrooms with concrete examples; (2) a description of possible professional development strategies to improve in-service FA practices for linguistically diverse students; and (3) a research-informed approach for analyzing FA practices. Besides filling the existing research gap on FA with ELLs, outcomes and products serve as a foundation for a future research agenda and a comprehensive project aimed at ensuring equitable science learning for all students, including ELLs.

Examining Formative Assessment Practices for English Language Learners in Science Classrooms (Collaborative Research: Li)

Levels of Conceptual Understanding in Statistics (LOCUS)

LOCUS (Levels of Conceptual Understanding in Statistics) is an NSF Funded DRK12 project (NSF#118618) focused on developing assessments of statistical understanding. These assessments will measure students’ understanding across levels of development as identified in the Guidelines for Assessment and Instruction in Statistics Education (GAISE). The intent of these assessments is to provide teachers and researchers with a valid and reliable assessment of conceptual understanding in statistics consistent with the Common Core State Standards (CCSS).

Lead Organization(s): 
Award Number: 
1118168
Funding Period: 
Thu, 09/01/2011 - Fri, 08/31/2012
Project Evaluator: 
TERC, Jim Hammerman
Full Description: 

The goal of this project is to develop two tests (instruments) to assess conceptual understanding of statistics.

The instruments are based on the levels A/B and on level C of statistical understanding development as described in the American Statistical Association Guidelines for Assessment and Instruction of Statistics Education (GAISE) framework. These instruments will be used to assess knowledge of statistics by grades 6-12 students. The instruments will have multiple-choice and constructed response (CR) items. The CR items will have scoring rubrics. The assessments will be pilot tested in school districts in six states. The instruments will be used by teachers to analyze students' growth in understanding of statistics and will be useable for both formative and summative purposes. An assessment blueprint will be developed based on the GAISE framework for selecting and constructing both fixed-choice and open-ended items. An evidenced-based designed process will be used to develop the assessments. The blueprint will be used by the test development committee to develop items. These items will be reviewed by the advisory board considering the main statistics topics to be included on the assessments. Through a layering process, the assessments will be piloted, revised, and field tested with students in grades 6-12 in six states. A three-parameter IRT model will be used in analyzing the items. The work will be done by researchers at the University of Florida with the support of those at the University of Minnesota, the Educational Testing Service, and Kenyon College. Researchers from TERC will conduct a process evaluation with several feedback and redesign cycles.

The assessments will be aligned with the Common Core State Standards for mathematics (CCSSM) and made available as open-source to teachers through a website. The research team will interact with the state consortia developing assessments to measure students? attainment of the CCSSM. As such, the assessments have the potential of being used by a large proportion of students in the country. The more conceptually-based items will provide teachers with concrete examples of what statistics students in grades 6-12 should know.

Levels of Conceptual Understanding in Statistics (LOCUS)

Using Rule Space and Poset-Based Adaptive Testing Methodologies to Identify Ability Patterns in Early Mathematics and Create a Comprehensive Mathematics Ability Test

This project will develop a new assessment for children ages 3-7 to provide teachers with diagnostic information on a child's development of mathematics facility on ten domains such as counting, sequencing, adding/subtracting, and measurement. The Comprehensive Research-based Mathematics Ability (CREMAT) is being developed using innovative psychometric models to reveal information about children on specific attributes for each of the 10 domains.

Project Email: 
Douglas.Clements@du.edu
Lead Organization(s): 
Award Number: 
1313695
Funding Period: 
Sat, 09/01/2012 - Sun, 08/31/2014
Full Description: 

A new assessment for children ages 3-7 is being developed to provide teachers with diagnostic information on a child's development of mathematics facility on ten domains such as counting, sequencing, adding/subtracting, and measurement. The Comprehensive Research-based Mathematics Assessment (CREMA) is being developed using innovative psychometric models to reveal information about children on specific attributes for each of the 10 domains. The CREMA will produce information based on carefully developed learning trajectories in a relative short period of time by using computer adaptive testing. The project is guided by two goals: 1) to produce a cognitively diagnostic adaptive assessment that will yield more useful and detailed information about students' knowledge of mathematics than previously possible, and 2) subject the developmental progressions to close cognitive diagnosis using cutting-edge psychometric approaches. An item pool of about 350 items is being developed that can be used to identify the level of understanding children ages 3-7 have on the 10 domains that have been identified as foundational to further learning in mathematics. A research team headed by Dr. Douglas Clements at the University of Buffalo is conducting the development work while being assisted by Dr. Curtis Tatsuoka, a statistician at Case Western Reserve University.

The CREMA is being developed using leading-edge psychometric models based on Q-Matrix theory, rule-state models, and posets. The initial item pool includes items from the REMA, a previously developed instrument based on unidemensional IRT models. New items are being piloted with at least 200 students from a group of a total of 800 students evenly distributed among pre-K to grade 2. The successful items then are used to create the new CREMA. The new assessment is being field tested with 300 children, pre-K to grade 2. A random sample of 50 students (at least 10 from each grade) is being video taped as they work the items. Specific criteria of convergence are being used for feedback on how specific items are performing to meet the required specifications. An external evaluator is auditing the process and is doing spot checks of item codings and other analyses performed.

The main product will be the CREMA that will be made widely available. This instrument using computer adaptive testing will provide teachers with ready information on young children's understanding of critical mathematical ideas. The new psychometric models that will be used and developed to process multiple attributes from individual items will make large strives to move forward the field of mathematics assessment of young children. A publisher has expressed interest to make the assessment widely available that increases the likelihood the assessment will have large impact on early childhood mathematics learning.

This project was previously funded under award # 1019925.

Using Rule Space and Poset-Based Adaptive Testing Methodologies to Identify Ability Patterns in Early Mathematics and Create a Comprehensive Mathematics Ability Test

Iterative Model Building (IMB): A Program for Training Quality Teachers and Measuring Teacher Quality

This project aims to improve professional development programs for pre-service teachers (PSTs) as a way to improve student learning in mathematics and science. PSTs engage in a series of teaching cycles, and then engage in lesson study groups to develop, teach, and analyze a whole-class lesson. The cycle is completed by reexamining students' knowledge in teaching experiments with pairs of students. These teaching cycles are called Iterative Model Building (IMB).

Lead Organization(s): 
Award Number: 
0732143
Funding Period: 
Wed, 08/15/2007 - Tue, 07/31/2012
Project Evaluator: 
Center for Evaluation and Education Policy
Iterative Model Building (IMB): A Program for Training Quality Teachers and Measuring Teacher Quality

Diagnostic E-learning Trajectories Approach (DELTA) Applied to Rational Number Reasoning for Grades 3-8

This project aims to develop a software diagnostic tool for integrating diagnostic interviews, group administered assessments, and student data in real-time so that teachers can enter and view student status information. This project would concentrate on rational number learning in grades 3-8. The design is based on a model of learning trajectories developed from existing research studies.

Project Email: 
gismo.fi@gmail.com
Lead Organization(s): 
Partner Organization(s): 
Award Number: 
0733272
Funding Period: 
Sat, 09/01/2007 - Tue, 08/31/2010
Project Evaluator: 
William Penuel (SRI)
Full Description: 

This project aims to develop a software diagnostic tool for integrating diagnostic interviews, group administered assessments, and student data in real-time so that teachers can enter and view student status information. This project would concentrate on rational number learning in grades 3-8. The design is based on a model of learning trajectories developed from existing research studies.

The diagnostic system to be developed for teachers would be used in assessing their students' knowledge and would identify difficulties in understanding five key clusters of concepts and skills in rational number reasoning. It would also investigate the diagnostic system's effects on student and teacher learning in relation to state standards, assessments, and curricular programs. The five areas include understanding: (1) multiplicative and division space; (2) fractions, ratio, proportion and rates; (3) rectangular area and volume; (4) decimals and percents; and (5) similarity and scaling.

The diagnostic measures will include diagnostic interviews collecting data using a handheld computer, two types of group-administered assessments of student progress, one set along learning trajectories for each of the five sub-constructs and one composite measurement per grade. The diagnostic system will produce computer-based progress maps, summarizing individual student and class performance and linking to state assessments.

Diagnostic E-learning Trajectories Approach (DELTA) Applied to Rational Number Reasoning for Grades 3-8

Mathematics Instruction Using Decision Science and Engineering Tools

A collaboration among educators, engineers, and mathematicians in three universities, this project is creating, implementing, and evaluating a one-year curriculum for teaching a non-calculus, fourth-year high school mathematics course and accompanied assessment instruments. The curriculum will draw on decision-making tools that include but go well beyond linear programming, to enhance student mathematical competence (particularly solving multi-step problems), improve students' attitudes toward mathematics, and promote states' adoption of the curriculum (initially NC and MI).

Project Email: 
mindset@ncsu.edu
Lead Organization(s): 
Award Number: 
0733137
Funding Period: 
Sat, 09/15/2007 - Tue, 08/31/2010
Project Evaluator: 
Dr. Shlomo S. Sawilowsky
Full Description: 

Mathematics INstruction using Decision Science and Engineering Tools (MINDSET) is a collaboration among educators, engineers, and mathematicians at three universities to create, implement, and evaluate a new curriculum and textbook to teach standard mathematics concepts using math-based decision-making tools for a non-calculus fourth-year mathematics curriculum that several states now require and others may require in the near future. MINDSET has three goals: (1) enhancement of students’ mathematical ability, especially their ability to formulate and solve multi-step problems and interpret results; (2) improvement in students’ attitude toward mathematics, especially those from underrepresented groups, thereby motivating them to study mathematics; and (3) adoption of the curriculum initially in North Carolina and Michigan, then in other states.

Using decision-making tools from Operations Research and Industrial Engineering, we will develop a fourthyear high school curriculum in mathematics and support materials to teach standard content. Through a multi-state, multi-school district assessment, we will determine if a statistically significant improvement in students’ mathematical ability—particularly in multi-step problem solving and interpretation of results—and in motivation and attitude toward mathematics has occurred. Participating teachers will receive professional training, help to create a knowledge-based online community for support, and in-person and online technical assistance. Through extensive data collection and analysis, we will determine if this infrastructure is sustainable and sufficiently flexible to be reproduced and used by others.

Mathematics Instruction Using Decision Science and Engineering Tools

Chemistry Facets: Formative Assessment to Improve Student Understanding in Chemistry

This project implemented a facets-of-thinking perspective to design tools and practices to improve high school chemistry teachers' formative assessment practices. Goals are to identify and develop clusters of facets related to key chemistry concepts; develop assessment items; enhance the assessment system for administering items, reporting results, and providing teacher resource materials; develop teacher professional development and resource materials; and examine whether student learning in chemistry improves in classes that incorporate a facet-based assessment system.

Award Number: 
0733169
Funding Period: 
Sat, 09/15/2007 - Wed, 08/31/2011
Project Evaluator: 
Heller Research Associates
Full Description: 

Supported by research on students' preconceptions, particularly in chemistry, and the need to build on the knowledge and skills that students bring to the classroom, this project implements a facets-of-thinking perspective for the improvement of formative assessment, learning, and instruction in high school chemistry. Its goals are: to identify and develop clusters of facets (students' ideas and understandings) related to key high school chemistry concepts; to develop assessment items that diagnose facets within each cluster; to enhance the existing web-based Diagnoser assessment system for administering items, reporting results, and providing teacher resource materials for interpreting and using the assessment data; to develop teacher professional development and resource materials to support their use of facet-based approaches in chemistry; and to examine whether student learning in chemistry improves in classes that incorporate a facet-based assessment system.

The proposed work builds on two previously NSF-funded projects focused on designing Diagnoser (ESI-0435727) in the area of physics and on assessment development to support the transition to complex science learning (REC-0129406). The work plan is organized in three strands: (1) Assessment Development, consisting of the development and validation of facet clusters related to the Atomic Structure of Matter and Changes in Matter and the development and validation of question sets related to each facet cluster, including their administration to chemistry classes; (2) Professional Development, through which materials will be produced for a teacher workshop focused on the assessment-for-learning cycle; and (3) Technology Development, to upgrade the Diagnoser authoring system and to include chemistry facets and assessments.

Anticipated products include: (1) 8-10 validated facet clusters related to the Atomic Structure of Matter and Changes in Matter; (2) 12-20 items per facet cluster that provide diagnostic information about student understanding in relation to the facet clusters; (3) additional instructional materials related to each facet cluster, including 1-3 questions to elicit inital student ideas, a developmental lesson to encourage students' exploration of new concepts, and 3-5 prescriptive lessons to address persistent problematic ideas; and (4) a publically-available web-based Diagnoser for chemistry (www.Diagnoser.com), including student assessments and instructional materials.

Chemistry Facets: Formative Assessment to Improve Student Understanding in Chemistry

Application of Evidence-centered Design to States Large-scale Science Assessment

This project aims to (1) determine ways in which Evidence-Centered Design enhances the quality of large-scale, technology-based science assessments for middle school grades and high school equivalency; (2) implement resulting procedures in operational test development; (3) evaluate the efficiency, effectiveness and generalizability of these procedures, and (4) disseminate findings to the assessment community.

Award Number: 
0733172
Funding Period: 
Sat, 09/01/2007 - Fri, 08/31/2012
Project Evaluator: 
Haynie Research and Evaluation--Doreen Finkelstein, Kathleen Haynie
Full Description: 

The project began as a collaborative research effort among six organizations—a non-profit research company (SRI International), a university (University of Maryland), a commercial test publishing company (Pearson), Minnesota’s (MN) state department of education, a software engineering firm (Codeguild, Inc.), and an educational evaluation firm (Haynie Research and Evaluation). Due to changes in the affiliation of key personnel, the project transitioned to a collaboration among five organizations--SRI International, ETS, University of Maryland, Pearson and Haynie Research and Evaluation. Together these groups designed and implemented several studies to document the influence of evidence-centered design when applied to Pearson's science assessment design and development processes.

The goals of the project are: (1) to determine leverage points by which ECD can enhance the quality of large-scale technology-based assessments and the efficiency of their design, (2) to implement resulting procedures in operational test development cycles, (3) to evaluate efficiency, effectiveness and generalizability of these procedures, (4) to develop two software wizards to support design of task-based scenarios and assessment items, and (5) to disseminate findings to the assessment community.

This project will develop an exemplar set of design patterns based on the critical benchmarks identified in the Minnesota Academic Standards for science and on the GED science practice indicators and content targets. It is of particular interest in this project that elements of ECD will be applied to an existing large-scale accountability and credentialing assessments, in the context of existing test development and delivery processes.  The project is constrained to maintain adherence to existing test specifications, “look and feel” of tasks, timelines, and delivery and scoring procedures.  Rather than designing new assessment systems or re-engineering existing ones, the present project seeks to identify and implement ideas from ECD in existing large-scale, high-stakes testing programs. Principles of ECD have been implemented in several training workshops for assessment designers and item writers to support the development of scenario-based science tasks. The project's technical report series is available at http://ecd.sri.com.

Application of Evidence-centered Design to States Large-scale Science Assessment

SAVE Science: Situated Assessment Using Virtual Environments for Science Content and Inquiry

The SAVE Science project is creating an innovative system using immersive virtual environments for evaluating learning in science, consistent with research- and policy-based recommendations for science learning focused around the big ideas of science content and inquiry for middle school years. Motivation for this comes not only from best practices as outlined in the National Science Education Standards and AAAS' Project 2061, but also from the declining interest and confidence of today's student in science.

Project Email: 
savescience@gmail.com
Lead Organization(s): 
Award Number: 
1157534
Funding Period: 
Mon, 09/01/2008 - Sat, 08/31/2013
Project Evaluator: 
Anthony Lutkus
SAVE Science: Situated Assessment Using Virtual Environments for Science Content and Inquiry

Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model

This project focuses on critical needs in the preparation and long-term development of pre-service, undergraduate, K-6 teachers of science. The project investigates the impact on these students of undergraduate, standards-based, reform entry level science courses developed by faculty based on their participation in the NASA Opportunities for Visionary Academics processional development program to identify: short-term impacts on undergraduate students and long-term effects on graduated teachers; characteristics of reform courses and characteristics of effective development efforts.

Project Email: 
dwsunal@bama.ua.edu
Lead Organization(s): 
Award Number: 
0554594
Funding Period: 
Tue, 08/01/2006 - Sun, 07/31/2011
Full Description: 

The Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model project is informally known as the National Study of Education in Undergraduate Science (NSEUS). This 5-year project focuses on critical needs in the preparation and long-term development of pre-service, undergraduate, K-6 teachers of science. The goal is to investigate the impact on these students of undergraduate, standards-based, reform entry-level science courses developed by faculty in the NASA Opportunities for Visionary Academics (NOVA) professional development model. Twenty reform and 20 comparison undergraduate science courses from a national population of 101 diverse institutions participating in NOVA, stratified by institutional type, were be selected and compared in a professional development impact design model. Data is being collected in extended on-site visits using multiple quantitative and qualitative instruments and analyzed using comparative and relational studies at multiple points in the impact design model. Criteria for success of the project will be determined by conclusions drawn from the research questions; including evidence and effect sizes of short-term impacts on undergraduate students and long-term effects on graduated in-service teachers in their own classroom science teaching; identification of characteristics of undergraduate reformed courses that produce significant impacts; identification of characteristics of effective faculty, and effective dissemination.

Project Publications and Presentations:

Lardy, Corrine; Mason, Cheryl; Mojgan, Matloob-Haghanikar; Sunal, Cynthia Szymanski; Sunal, Dennis Wayne; Sundberg, Cheryl & Zollman, Dean (2009). How Are We Reforming Teaching in Undergraduate Science Courses? Journal of College Science Teaching, v. 39 (2), 12-14.  

Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model
Syndicate content