Assessment

Human Variance and Assessment for Learning Implications for Diverse Learners of STEM: A National Conference

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests.

Lead Organization(s): 
Award Number: 
1939192
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The conference purpose is to stimulate a national conversation concerning the relationships between assessment, teaching and learning that include scholarly research and development of tests; members of city and state boards of education; officials from states and major school systems; policymakers; and representatives of teachers' associations and parents' associations. This conference aims to attract these important professionals has important co-sponsors like the Urban Institute. This national conference flows from the work of the Gordon Commission on the Future of Assessment for Education that addressed the advancement of achievement in STEM disciplines (PreK-12) for students who are underrepresented among high achieving students. This issue of advancement of underrepresented high achieving students has received little concentrated effort and a conference would help in providing greater understanding of this special concern, which includes a student in poverty in complexed family structures.

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests. The conference will stimulate national conversation and ultimately a market that demands educational assessments that inform and improve teaching and learning transactions. The conference will be organized around four conceptual and theoretical papers that focus on the knowledge base upon which six concurrent workshops will be based. The four papers are: (1) Human Diversity and Assessment; (2) The Limits of Test Bias and Its Corrections; (3) Towards an Assessment Science Capable of Informing and Improving Learning; and  (4) Assessment in the Service of Learning. The workshops will focus on models of pedagogical practice that show promise for informing and improving teaching and learning processes and their outcomes. These issues will be discussed by 11-15 expert presenters who understand student learning and the types of information gleaned from different types of assessments. The attention to URMs and their needs and contexts are prioritized in discussions surrounding measurement science and the integration of assessment. Several important issues that address understanding of student learning, and the relationship between the varieties of information concerning students that can be accessed through assessments are: (1) The importance of the broader and more productive use of educational testing to improve the learning of STEM subject matter and values; (2) Curriculum embedded assessment and the reduction in disparities in achievement by STEM learners from diverse social divisions; (3) Innovative procedures and programs for the use of data concerning learners and teaching and learning transactions in the teaching and learning of STEM with learners who are underrepresented among high achieving STEM learners.

Advancing Coherent and Equitable Systems of Science Education

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education.

Lead Organization(s): 
Award Number: 
1920249
Funding Period: 
Thu, 08/01/2019 to Mon, 07/31/2023
Full Description: 

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education. This is an important study because in most states, the student population is becoming more diverse, and states need help in finding ways to better serve schools and districts within their jurisdictions. Through this effort, state science leaders will participate in a networked improvement community model organized to develop and test state-level strategies. Specifically, the focus will be on the adaptation of instructional materials and formative assessment as linked policy strategies for aligning curriculum, instruction, and assessment and for relating instruction to the interests and histories of local communities. State science leaders and researchers will investigate how and under what conditions certain strategies support the emergence of coherent and equitable state systems of science education in which all students have opportunities to meet challenging new science standards. The project will build knowledge and theory about the conditions under which a network of state teams can promote coherent guidance for culturally-based instruction in local districts and schools. Together the partners will collaborate to diagnose current challenges to promoting coherence and equity and then develop knowledge and resources about conditions that promote coherence and equity by testing and studying strategies for cultivating it.

An iterative design-based research approach will be used to build foundational knowledge for the equitable implementation of the vision of science and engineering learning that integrates disciplinary core ideas, science and engineering practices, and crosscutting concepts working from a cultural perspective on learning. A multiple-case study will be used to collect data about the impact of the networked improvement community model on leadership development to effectively improve state efforts. Surveys and interviews will be used to gather information on co-designing efforts, use and adaptation of resources, and knowledge gained by state science leaders. Data will also be collected on political conditions and infrastructures of teamwork as potential facilitators and barriers to the development of strategic knowledge leadership. Analyses of data will identify patterns or configurations of conditions associated with growth in science leaders' strategic knowledge leadership related to equity. This technique will generate evidence-based claims for how and when supports and barriers matter for growth in strategic knowledge leadership for equity.

Developing and Validating Early Assessments of College Readiness: Differential Effects for Underrepresented Groups, Optimal Timing of Assessments, and STEM-Specific Indicators

This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

Project Email: 
Award Number: 
1908630
Funding Period: 
Mon, 07/15/2019 to Wed, 06/30/2021
Project Evaluator: 
Full Description: 

One third of all college freshmen are academically unprepared for entry-level college coursework and require remedial course. That figure is much higher at many colleges. The problem is more acute in STEM disciplines, particularly among students from underrepresented ethnic groups and low socioeconomic status families. This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

This project will use two recent and complementary large-scale, nationally representative federal databases: the High School Longitudinal Study of 2009 and the Education Longitudinal Study of 2002. Factor analysis will be used to develop composite variables of college readiness and multilevel regression will be used to develop predictive models on a range of college outcomes to test the predictive validity of composite and individual predictors. The models will be extended to conduct multiple group analyses to test for differential prediction for students from underrepresented groups. The project intends to promote 1) the use of a wider range of assessments of academic preparedness, 2) the use of measures that are more sensitive for assessing college readiness from underrepresented groups and among STEM majors, 3) earlier assessment using indicators and models with predictive validity and 4) progress monitoring of college readiness by providing a detailed example of how that can be developed and implemented. Findings will also raise student, parental, teacher, and other school personnel awareness of the range of factors relevant for preparing students for college.

Alternative video text
Alternative video text: 

Case Studies of a Suite of Next Generation Science Instructional, Assessment, and Professional Development Materials in Diverse Middle School Settings

This project addresses a gap between vision and implementation of state science standards by designing a coordinated suite of instructional, assessment and teacher professional learning materials that attempt to enact the vision behind the Next Generation Science Standards. The study focuses on using state-of-the-art technology to create an 8-week long, immersive, life science field experience organized around three investigations.

Lead Organization(s): 
Award Number: 
1907944
Funding Period: 
Mon, 07/01/2019 to Fri, 06/30/2023
Full Description: 

New state science standards are ambitious and require important changes to instructional practices, accompanied by a coordinated system of curriculum, assessment, and professional development materials. This project addresses a gap between vision and implementation of such standards by designing a coordinated suite of instructional, assessment and teacher professional learning materials that attempt to enact the vision behind the Next Generation Science Standards. The study focuses on the design of such materials using state-of-the-art technology to create an 8-week long, immersive, life science field experience organized around three investigations. Classes of urban students in two states will collect data on local insect species with the goal of understanding, sharing, and critiquing environmental management solutions. An integrated learning technology system, the Learning Navigator, draws on big data to organize student-gathered data, dialogue, lessons, an assessment information. The Learning Navigator will also amplify the teacher's role in guiding and fostering next generation science learning. This project advances the field through an in-depth exploration of the goals for the standards documents. The study begins to address questions about what works when, where, and for whom in the context of the Next Generation Science Standards.

The project uses a series of case studies to create, test, evaluate and refine the system of instructional, assessment and professional development materials as they are enacted in two distinct urban school settings. It is designed with 330 students and 22 teachers in culturally, racially and linguistically diverse, under-resourced schools in Pennsylvania and California. These schools are located in neighborhoods that are economically challenged and have students who demonstrate patterns of underperformance on state standardized tests. It will document the process of team co-construction of Next Generation Science-fostering instructional materials; develop assessment tasks for an instructional unit that are valid and reliable; and, track the patterns of use of the instructional and assessment materials by teachers. The study will also record if new misconceptions are revealed as students develop Next Generation Science knowledge,  comparing findings across two diverse school locations in two states. Data collection will include: (a) multiple types of data to establish validity and reliability of educational assessments, (b) the design, evaluation and use of a classroom observation protocol to gather information on both frequency and categorical degree of classroom practices that support the vision, and (c) consecutive years of ten individual classroom enactments through case studies analyzed through cross-case analyses. This should lead to stronger and better developed understandings about what constitutes strong Next Generation Science learning and the classroom conditions, instructional materials, assessments and teacher development that foster it.

Building a Teacher Knowledge Base for the Implementation of High-Quality Instructional Resources through the Collaborative Investigation of Video Cases (Collaborative Research: Murray)

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation.

Lead Organization(s): 
Award Number: 
1908319
Funding Period: 
Mon, 07/01/2019 to Fri, 06/30/2023
Full Description: 

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation. The backbone of AIM-TRU is a growing, open repository of video cases available to teachers and teacher educators across the U.S. who use or are interested in using the lessons. The repository will include tools such as a facilitator's guide to support teachers and teacher educators to engage in the model and collaboratively investigate the video cases. Consequently, the work will have the potential to engage teachers and teacher educators in improving mathematics education at scale. Because the video cases will capture implementation and ideas for improving instruction in schools serving populations who are underrepresented in mathematics, AIM-TRU will serve to improve mathematics education equitably.

Research questions focus on what teachers learn about high-quality mathematics instruction and instructional materials within a community of practice, and how that learning influences their teaching. In AIM-TRU, teachers engage in the collaborative investigation of video cases utilizing a shared repertoire that includes questioning protocols adapted from the Teaching for Robust Understanding (TRU) framework. This framework articulates five dimensions of classroom instruction that are necessary and sufficient to support students in becoming powerful mathematical thinkers. This affords teachers opportunities to use the TRU dimensions as lenses to diagnose common problems of practice that arise in implementation, and propose innovations and theories for improving instruction that can be tested in real classrooms and documented in new video cases. Analytic tools will be used from frame analysis to produce empirical evidence of what teachers are learning about instruction and instructional materials along the five dimensions of TRU. These data will be mapped to a random sample of video recordings of participating teachers' instruction, scored using the TRU Math Rubric, in order to link learning outcomes from the professional development to changes in instruction. Addressing these research questions will provide a deeper understanding and empirical evidence of learning within teacher collectives, the pressing national need to develop mechanisms to produce collective professional knowledge for teaching, and further efforts to understand the types of knowledge required for effective teaching.

Building a Teacher Knowledge Base for the Implementation of High-Quality Instructional Resources through the Collaborative Investigation of Video Cases (Collaborative Research: Jabon)

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation.

Lead Organization(s): 
Award Number: 
1908311
Funding Period: 
Mon, 07/01/2019 to Fri, 06/30/2023
Full Description: 

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation. The backbone of AIM-TRU is a growing, open repository of video cases available to teachers and teacher educators across the U.S. who use or are interested in using the lessons. The repository will include tools such as a facilitator's guide to support teachers and teacher educators to engage in the model and collaboratively investigate the video cases. Consequently, the work will have the potential to engage teachers and teacher educators in improving mathematics education at scale. Because the video cases will capture implementation and ideas for improving instruction in schools serving populations who are underrepresented in mathematics, AIM-TRU will serve to improve mathematics education equitably.

Research questions focus on what teachers learn about high-quality mathematics instruction and instructional materials within a community of practice, and how that learning influences their teaching. In AIM-TRU, teachers engage in the collaborative investigation of video cases utilizing a shared repertoire that includes questioning protocols adapted from the Teaching for Robust Understanding (TRU) framework. This framework articulates five dimensions of classroom instruction that are necessary and sufficient to support students in becoming powerful mathematical thinkers. This affords teachers opportunities to use the TRU dimensions as lenses to diagnose common problems of practice that arise in implementation, and propose innovations and theories for improving instruction that can be tested in real classrooms and documented in new video cases. Analytic tools will be used from frame analysis to produce empirical evidence of what teachers are learning about instruction and instructional materials along the five dimensions of TRU. These data will be mapped to a random sample of video recordings of participating teachers' instruction, scored using the TRU Math Rubric, in order to link learning outcomes from the professional development to changes in instruction. Addressing these research questions will provide a deeper understanding and empirical evidence of learning within teacher collectives, the pressing national need to develop mechanisms to produce collective professional knowledge for teaching, and further efforts to understand the types of knowledge required for effective teaching.

Building a Teacher Knowledge Base for the Implementation of High-Quality Instructional Resources through the Collaborative Investigation of Video Cases (Collaborative Research: Wilson)

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation.

Lead Organization(s): 
Award Number: 
1908185
Funding Period: 
Mon, 07/01/2019 to Fri, 06/30/2023
Full Description: 

This project will address the pressing national need to generate shared, practice-based knowledge about how to implement freely available, high-quality instructional resources (mathematics formative assessment lessons) that have been shown to produce significant gains in student learning outcomes. It will expand a professional development model (Analyzing Instruction in Mathematics using the Teaching for Robust Understanding Framework (AIM-TRU)) that supports teacher learning about effective lesson implementation. The backbone of AIM-TRU is a growing, open repository of video cases available to teachers and teacher educators across the U.S. who use or are interested in using the lessons. The repository will include tools such as a facilitator's guide to support teachers and teacher educators to engage in the model and collaboratively investigate the video cases. Consequently, the work will have the potential to engage teachers and teacher educators in improving mathematics education at scale. Because the video cases will capture implementation and ideas for improving instruction in schools serving populations who are underrepresented in mathematics, AIM-TRU will serve to improve mathematics education equitably.

Research questions focus on what teachers learn about high-quality mathematics instruction and instructional materials within a community of practice, and how that learning influences their teaching. In AIM-TRU, teachers engage in the collaborative investigation of video cases utilizing a shared repertoire that includes questioning protocols adapted from the Teaching for Robust Understanding (TRU) framework. This framework articulates five dimensions of classroom instruction that are necessary and sufficient to support students in becoming powerful mathematical thinkers. This affords teachers opportunities to use the TRU dimensions as lenses to diagnose common problems of practice that arise in implementation, and propose innovations and theories for improving instruction that can be tested in real classrooms and documented in new video cases. Analytic tools will be used from frame analysis to produce empirical evidence of what teachers are learning about instruction and instructional materials along the five dimensions of TRU. These data will be mapped to a random sample of video recordings of participating teachers' instruction, scored using the TRU Math Rubric, in order to link learning outcomes from the professional development to changes in instruction. Addressing these research questions will provide a deeper understanding and empirical evidence of learning within teacher collectives, the pressing national need to develop mechanisms to produce collective professional knowledge for teaching, and further efforts to understand the types of knowledge required for effective teaching.

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Harris)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Lead Organization(s): 
Award Number: 
1903103
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Rutstein)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Lead Organization(s): 
Award Number: 
1853923
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Billman)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Partner Organization(s): 
Award Number: 
1853951
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Pages

Subscribe to Assessment