Assessment

Looking Back and Looking Forward: Increasing the Impact of Educational Research on Practice

The focus of this conference is to carefully examine past and current research with an eye toward improving its impact on practice and to create concrete steps that could shape the nature and impact of mathematics education research.

Lead Organization(s): 
Award Number: 
1941494
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The focus of the proposed conference is to carefully examine past and current research with an eye toward improving its impact on practice. This conference is designed to create concrete steps that could shape the nature and impact of mathematics education research for years to come. A diverse group of 50 participants will be invited to participate. Participants include 10 experienced K-12 educators whose perspectives will be used to anchor the conference in problems of practice. Other participants represent senior through more junior scholars who have demonstrated a commitment to addressing the disconnect between research and practice, along with technology experts to advise participants on capabilities and innovative uses of modern technologies for instruction, assessment and data management.

The overarching goal for the conference is to help the field of mathematics education think deeply about the most productive ways to answer the following questions: [1] Why hasn't past research had a more direct impact on practice? What can be learned from this historical analysis for future research? [2] What is a possible vision for research that would have a more direct impact on practice? What questions should be asked? What methods should be used? What concrete steps can be taken to launch the new research programs? [3] What are the implications of adopting new kinds of research programs? If they gain traction, how will such changes affect the broader education community and infrastructure, including preservice teacher education, teacher professional development, and the training of future researchers? How should the roles of researchers and teachers change? What incentive structures might motivate these changes? How will new programs of research interact with existing programs?

Human Variance and Assessment for Learning Implications for Diverse Learners of STEM: A National Conference

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests.

Lead Organization(s): 
Award Number: 
1939192
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The conference purpose is to stimulate a national conversation concerning the relationships between assessment, teaching and learning that include scholarly research and development of tests; members of city and state boards of education; officials from states and major school systems; policymakers; and representatives of teachers' associations and parents' associations. This conference aims to attract these important professionals has important co-sponsors like the Urban Institute. This national conference flows from the work of the Gordon Commission on the Future of Assessment for Education that addressed the advancement of achievement in STEM disciplines (PreK-12) for students who are underrepresented among high achieving students. This issue of advancement of underrepresented high achieving students has received little concentrated effort and a conference would help in providing greater understanding of this special concern, which includes a student in poverty in complexed family structures.

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests. The conference will stimulate national conversation and ultimately a market that demands educational assessments that inform and improve teaching and learning transactions. The conference will be organized around four conceptual and theoretical papers that focus on the knowledge base upon which six concurrent workshops will be based. The four papers are: (1) Human Diversity and Assessment; (2) The Limits of Test Bias and Its Corrections; (3) Towards an Assessment Science Capable of Informing and Improving Learning; and  (4) Assessment in the Service of Learning. The workshops will focus on models of pedagogical practice that show promise for informing and improving teaching and learning processes and their outcomes. These issues will be discussed by 11-15 expert presenters who understand student learning and the types of information gleaned from different types of assessments. The attention to URMs and their needs and contexts are prioritized in discussions surrounding measurement science and the integration of assessment. Several important issues that address understanding of student learning, and the relationship between the varieties of information concerning students that can be accessed through assessments are: (1) The importance of the broader and more productive use of educational testing to improve the learning of STEM subject matter and values; (2) Curriculum embedded assessment and the reduction in disparities in achievement by STEM learners from diverse social divisions; (3) Innovative procedures and programs for the use of data concerning learners and teaching and learning transactions in the teaching and learning of STEM with learners who are underrepresented among high achieving STEM learners.

Advancing Coherent and Equitable Systems of Science Education

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education.

Lead Organization(s): 
Award Number: 
1920249
Funding Period: 
Thu, 08/01/2019 to Mon, 07/31/2023
Full Description: 

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education. This is an important study because in most states, the student population is becoming more diverse, and states need help in finding ways to better serve schools and districts within their jurisdictions. Through this effort, state science leaders will participate in a networked improvement community model organized to develop and test state-level strategies. Specifically, the focus will be on the adaptation of instructional materials and formative assessment as linked policy strategies for aligning curriculum, instruction, and assessment and for relating instruction to the interests and histories of local communities. State science leaders and researchers will investigate how and under what conditions certain strategies support the emergence of coherent and equitable state systems of science education in which all students have opportunities to meet challenging new science standards. The project will build knowledge and theory about the conditions under which a network of state teams can promote coherent guidance for culturally-based instruction in local districts and schools. Together the partners will collaborate to diagnose current challenges to promoting coherence and equity and then develop knowledge and resources about conditions that promote coherence and equity by testing and studying strategies for cultivating it.

An iterative design-based research approach will be used to build foundational knowledge for the equitable implementation of the vision of science and engineering learning that integrates disciplinary core ideas, science and engineering practices, and crosscutting concepts working from a cultural perspective on learning. A multiple-case study will be used to collect data about the impact of the networked improvement community model on leadership development to effectively improve state efforts. Surveys and interviews will be used to gather information on co-designing efforts, use and adaptation of resources, and knowledge gained by state science leaders. Data will also be collected on political conditions and infrastructures of teamwork as potential facilitators and barriers to the development of strategic knowledge leadership. Analyses of data will identify patterns or configurations of conditions associated with growth in science leaders' strategic knowledge leadership related to equity. This technique will generate evidence-based claims for how and when supports and barriers matter for growth in strategic knowledge leadership for equity.

Validation of the Equity and Access Rubrics for Mathematics Instruction (VEAR-MI)

The main goal of this project is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. The project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

Award Number: 
1908481
Funding Period: 
Mon, 07/15/2019 to Fri, 06/30/2023
Full Description: 

High-quality mathematics instruction remains uncommon and opportunities for students to develop the mathematical understanding are not distributed equally. This is particularly true for students of color and students for whom English is not their first language. While educational research has made progress in identifying practices that are considered high-quality, little attention has been given to specific instructional practices that support historically marginalized groups of students particularly as they participate in more rigorous mathematics. The main goal is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. In addition, the project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

This project will make use of two existing large-scale datasets focusing on mathematics teachers to develop rubrics on mathematics instructional quality. The datasets include nearly 3,000 video-recorded mathematics lessons and student achievement records from students in Grades 3 through 8. The four phases of this research and development project include training material development, an observation and rubric generalizability study, a coder reliability study, and structural analysis. Data analysis plans involve case studies, exploratory and confirmatory factor analyses, and cognitive interviews. 

Validity Evidence for Measurement in Mathematics Education (V-M2ED) (Collaborative Research: Bostic)

The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1920621
Funding Period: 
Thu, 08/01/2019 to Wed, 07/31/2024
Full Description: 

As education has shifted more towards data-driven policy and research initiatives in the last several decades, data for policy-related aspects are often expected to be more quantitative in nature.  This has led to the increase in use of more quantitative measures in STEM education, including mathematics education. Unfortunately, evidence regarding the validity and reliability of mathematics education measures is lacking. Furthermore, the evidence for validity for quantitative tools and measures is not conceptualized or defined consistently by researchers in the field. The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education. Drawing on the results of the synthesis study, the researchers will design, curate, and disseminate a repository of quantitative assessments used in mathematics education teaching and research. The researchers will also create materials and online training for a variety of scholars and practitioners to use the repository.

The team will address two main research questions: 1) How might validity evidence related to quantitative assessments used in mathematics education research be categorized and described? and 2) What validity evidence exists for quantitative instruments used in mathematics education scholarship since 2000? Researchers will use a cross-comparative methodology which involves conducting a literature search and then analyzing and categorizing features of instruments. The research team will examine cases (i.e., assessments described in manuscripts) in which quantitative instruments have been used, alongside specific features such as the construct measured, evidence related to sources of validity, and study sample. The team will then design, develop, and deploy a free online digital repository for the categorization of instruments and describe their associated validity evidence.

Validity Evidence for Measurement in Mathematics Education (V-M2ED) (Collaborative Research: Krupa)

The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education.

Partner Organization(s): 
Award Number: 
1920619
Funding Period: 
Thu, 08/01/2019 to Wed, 07/31/2024
Full Description: 

As education has shifted more towards data-driven policy and research initiatives in the last several decades, data for policy-related aspects are often expected to be more quantitative in nature.  This has led to the increase in use of more quantitative measures in STEM education, including mathematics education. Unfortunately, evidence regarding the validity and reliability of mathematics education measures is lacking. Furthermore, the evidence for validity for quantitative tools and measures is not conceptualized or defined consistently by researchers in the field. The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education. Drawing on the results of the synthesis study, the researchers will design, curate, and disseminate a repository of quantitative assessments used in mathematics education teaching and research. The researchers will also create materials and online training for a variety of scholars and practitioners to use the repository.

The team will address two main research questions: 1) How might validity evidence related to quantitative assessments used in mathematics education research be categorized and described? and 2) What validity evidence exists for quantitative instruments used in mathematics education scholarship since 2000? Researchers will use a cross-comparative methodology which involves conducting a literature search and then analyzing and categorizing features of instruments. The research team will examine cases (i.e., assessments described in manuscripts) in which quantitative instruments have been used, alongside specific features such as the construct measured, evidence related to sources of validity, and study sample. The team will then design, develop, and deploy a free online digital repository for the categorization of instruments and describe their associated validity evidence.

Developing and Validating Early Assessments of College Readiness: Differential Effects for Underrepresented Groups, Optimal Timing of Assessments, and STEM-Specific Indicators

This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

Project Email: 
Award Number: 
1908630
Funding Period: 
Mon, 07/15/2019 to Wed, 06/30/2021
Project Evaluator: 
Full Description: 

One third of all college freshmen are academically unprepared for entry-level college coursework and require remedial course. That figure is much higher at many colleges. The problem is more acute in STEM disciplines, particularly among students from underrepresented ethnic groups and low socioeconomic status families. This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

This project will use two recent and complementary large-scale, nationally representative federal databases: the High School Longitudinal Study of 2009 and the Education Longitudinal Study of 2002. Factor analysis will be used to develop composite variables of college readiness and multilevel regression will be used to develop predictive models on a range of college outcomes to test the predictive validity of composite and individual predictors. The models will be extended to conduct multiple group analyses to test for differential prediction for students from underrepresented groups. The project intends to promote 1) the use of a wider range of assessments of academic preparedness, 2) the use of measures that are more sensitive for assessing college readiness from underrepresented groups and among STEM majors, 3) earlier assessment using indicators and models with predictive validity and 4) progress monitoring of college readiness by providing a detailed example of how that can be developed and implemented. Findings will also raise student, parental, teacher, and other school personnel awareness of the range of factors relevant for preparing students for college.

Alternative video text
Alternative video text: 

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Harris)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Lead Organization(s): 
Award Number: 
1903103
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Rutstein)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Lead Organization(s): 
Award Number: 
1853923
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Articulating a Transformative Approach for Designing Tasks that Measure Young Learners' Developing Proficiencies in Integrated Science and Literacy (Collaborative Research: Billman)

The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades.

Partner Organization(s): 
Award Number: 
1853951
Funding Period: 
Sat, 12/15/2018 to Sat, 11/30/2019
Full Description: 

SRI International, University of California-Berkeley (Lawrence Hall of Science), and WestEd will join efforts to articulate a potentially transformative approach for designing new kinds of classroom-based, three-dimensional assessment tasks that measure first graders' proficiencies in integrated science and literacy learning. The main goal of this study will be to conduct exploratory-design work to produce both the design approach and the early-stage tasks that are critical inputs for creating a program of research and development to more fully develop a suite of innovative assessment tasks for the early grades. Specific goals of the effort will be: (1) to iteratively develop and refine a design approach that enables assessment designers to develop Next Generation Science Standards (NGSS)-aligned tasks and rubrics that include a literacy component for the early grades; (2) to use this design approach to create two exemplar assessment tasks that are feasible for classroom use; and (3) to collect initial evidence that informs the promise of the design approach.

The work's research question will be: How can we extend current methodology to create assessments that integrate the three dimensions of the NGSS and literacy for early learners? The study will select first grade as the learning environment and two of the NGSS first grade performance expectations as the assessment targets. First grade students are typically at a critical point in developing their language and literacy proficiencies, which will allow the team to take on the challenges of variation in language and literacy skills. Correspondingly, the study will select two NGSS first grade life sciences performance expectations, because they include direct ties to literacy practices in science: (1) From Molecules to Organisms: Structures and Processes (Read texts and use media to determine patterns in behavior of parents and offspring that help offspring survive); and (2) Heredity: Inheritance and Variation of Traits (Make observations to construct an evidence-based account that young plants and animals are like, but not exactly like their parents). The design phase of the activity will consist of an assessment of the learning context and targets of the study, and the development of an assessment framework following the National Research Center's report, "Designing Assessments for the Next Generation Science Standards" (2014), including the principled assessment evidence-centered-design methodology. Data gathering, and interpretation strategies will include Experts' Review of the design approach, a focus group of teachers (n=8), and one-on-one cognitive interviews with students (n=20), conducted by researchers, which will be recorded to determine the quality and usability of the assessments using qualitative methods. The ultimate outcome of the proposed work will be a design approach for creating assessment tasks in a principled way across science disciplines for early elementary grade students. An advisory board will provide formative assessment feedback to the research team.

Pages

Subscribe to Assessment