Assessment

Learning Progressions in Science: Analyzing and Deconstructing the Multiple Dimensions in Assessment

Through this project, researchers will develop internet-based assessments designed to capture learning outcomes that (a) measure the higher order cognitive skills that are essential to current reform efforts, and (b) that report results in ways that are readily accessible and interpretable.

Award Number: 
2010322
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

Assessments are a crucial tool to enable the success of teaching and learning in science classrooms. Hence, to realize the vision of current reform efforts assessments must be developed that (a) measure the higher order cognitive skills that are essential to those reforms, and (b) that report results in ways that are readily accessible and interpretable. Through this project, researchers will develop internet-based assessments that capture such learning outcomes. These assessments can influence policy and practice by providing tangible products that exemplify the kind of learning outcomes and performances expected in today's science classrooms. The measures will also ensure that all students are assessed fairly. This project has the potential to enable students to express their knowledge and skills in a variety of ways which are less demanding and more creative than typical in traditional assessments.

Researchers will develop assessment materials aligned to the Next Generation Science Standards practice of scientific argumentation, the cross-cutting concept of patterns, and content in each of the following foundational middle school science domains - physical, life, and Earth sciences. Researchers will work closely with administrators and educators to ensure the relevance and alignment of materials to teachers' needs. Data will be gathered from middle and high school students from ethnically, culturally, and linguistically diverse school districts in California and Arizona. Key measures to be developed include embedding questions about patterns into the nine existing scenarios (tasks) and in new scenarios about natural resources. The team will develop assessments that can be used in both open-ended and selected response formats, to enhance their usefulness to teachers for diagnostic understanding, and their efficiency for summative classroom use. Data will also be gathered from teachers, to help develop interpretational materials. Teachers will have access, in real time, to their own students' responses, and estimates of students' performance on learning progressions within each of the science practices, crosscutting concepts and domains mentioned. The team will use the BEAR Assessment System to develop and refine assessment materials. This system is an integrated approach to developing assessments that seeks to provide meaningful interpretations of student work relative to cognitive and developmental goals. The researchers will gather empirical evidence to develop and improve the assessment materials, and then gather reliability and validity evidence to support their use. In total, item response data will be collected from several thousand students across the two districts. Student response data will be analyzed using multidimensional item response theory models.

Preparing Teachers to Design Tasks to Support, Engage, and Assess Science Learning in Rural Schools

This study focuses on working with teachers to develop assessment practices that focuses on the three NGSS dimensions of science ideas, practices and cross-cutting concepts, and adds two more dimensions; teachers will develop assessment tasks interesting to students, and promote the development of their science identities. To advance equitable opportunities for all students to learn science, this project will design and provide an online course to support rural teachers who teach science in grades 6-12.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2010086
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

Nationally, a third of US students attend rural or remote schools, yet rural teachers receive fewer opportunities to work together and engage in professional learning than their suburban and urban counterparts. This, in turn, can reflect on the opportunities for rural students to learn the high quality, up-to-date science ideas, practices, and concepts that are required by state standards, especially those aligned with the Next Generation Science Standards (NGSS). To advance equitable opportunities for all students to learn science, the project team will design and provide an online course to support rural teachers who teach science in grades 6-12. The course will focus on improving classroom science assessment practices and instruction to meet the unique needs of rural educators and their students. Too often, science concepts are removed from the lives of rural students, although their homes, communities and natural environments are filled with ideas and experiences that can make science come alive. When teachers link assessment and instruction to students' everyday lives, students have enhanced interest in and identification with science. This, in turn, can lead more students to pursue science and applied science fields beyond high school, to broaden the STEM pipeline. In addition, students are better prepared to participate in science in their communities as empowered citizens. This study focuses on working with teachers to develop assessment practices that not only focus on the three NGSS dimensions of science ideas, practices and cross-cutting concepts, it also adds two more dimensions; teachers will develop assessment tasks interesting to students, and promote the development of their science identities. The researchers refer to this as 5D assessment.

Researchers at Colorado University Boulder and BSCS Science Learning will use design-based implementation research to collaboratively design an online course sequence that targets 5D assessment in science. The study will proceed in three phases: a rapid ethnographic study to assess the needs of teachers serving a variety of rural communities, a study of teachers' use of an online platform for their professional learning, and lastly an experimental study to research the effects of the online course on teacher and student outcomes. Researchers will recruit 10 teachers to take the on-line course for the professional development and collect data on participating teachers' implementation of the course ideas through classroom videotaping and surveys designed to capture their changing practices. In the third year of the project, researchers will conduct an impact study with 70 secondary science teachers taking the re-designed on-line course, and compare their outcomes with a "business-as-usual" condition. The impact of course will be measured by questionnaires that address their vision for teaching the NGSS and self-reported instructional practices; classroom observations; and, teacher-constructed student assessments. Student outcomes will be measured using science interest and identity surveys, and an examination of student work products that demonstrate students' ability to use the science and engineering practice of modeling, a practice likely to be encountered in all NGSS science classrooms. The project will identify conditions under which learning about 5D assessment task design can support instructional improvement, increase student interest in science and engineering, and enhance students' opportunities to learn. The researchers hypothesize that the online program will have a positive impact on teachers' vision, classroom practices, and their use of high-quality 5D tasks. They also hypothesize that teacher participation will result in significant increases in student interest in and identification with science, and that these effects will be mediated by teacher outcomes. Finally, the researchers hypothesize that effects will be equitable across demographic variables in rural communities.

Leveraging Simulations in Preservice Preparation to Improve Mathematics Teaching for Students with Disabilities (Collaborative Research: Jones)

This project aims to support the mathematics learning of students with disabilities through the development and use of mixed reality simulations for elementary mathematics teacher preparation. These simulations represent low-stakes opportunities for preservice teachers to practice research-based instructional strategies to support mathematics learning, and to receive feedback on their practices.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2010298
Funding Period: 
Fri, 05/01/2020 to Tue, 04/30/2024
Full Description: 

The preparation of general education teachers to support the mathematics learning of students with disabilities is critical, as students with disabilities are overrepresented in the lower ranks of mathematics achievement. This project aims to address this need in the context of elementary mathematics teacher preparation through the development and use of mixed reality simulations. These simulations represent low-stakes opportunities for preservice teachers to practice research-based instructional strategies to support mathematics learning, and to receive feedback on their practices. Learning units that use the simulations will focus on two high leverage practices: teacher modeling of self-monitoring and reflection strategies during problem solving and using strategy instruction to teach students to support problem solving. These high-leverage teaching practices will support teachers engaging all students, including students with disabilities, in conceptually sophisticated mathematics in which students are treated as sense-makers and empowered to do mathematics in culturally meaningful ways.

The project work encompasses three primary aims. The first aim is to develop a consensus around shared definitions of high-leverage practices across the mathematics education and special education communities. To accomplish this goal, the project will convene a series of consensus-building panels with mathematics education and special education experts to develop shared definitions of the two targeted high leverage practices. This work will include engaging with current research, group discussion, and production of documents with specifications for the practices. The second aim is to develop learning units for elementary mathematics methods courses grounded in mixed reality simulation. These simulations will allow teacher candidates to enact the high leverage practices with simulated students and to receive coaching on their practice from the research team. The impact of this work will be assessed through the analysis of interviews with teacher educators implementing the units and observations and artifacts from the implementations. The third aim will be to assess the effectiveness of the simulations on teacher candidates? practices and beliefs through small-scaled randomized control trials. Teacher candidates will be randomly assigned to conditions that address the practices and make use of simulations, and a business as usual condition focused on lesson planning, student assessment, and small group discussions of the high leverage practices. The impact of the work will be assessed through the analysis of baseline and exit simulations, measures of teacher self-efficacy for teaching students with disabilities, and observations of classroom teaching in their clinical placement settings.

Looking Back and Looking Forward: Increasing the Impact of Educational Research on Practice

The focus of this conference is to carefully examine past and current research with an eye toward improving its impact on practice and to create concrete steps that could shape the nature and impact of mathematics education research.

Lead Organization(s): 
Award Number: 
1941494
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The focus of the proposed conference is to carefully examine past and current research with an eye toward improving its impact on practice. This conference is designed to create concrete steps that could shape the nature and impact of mathematics education research for years to come. A diverse group of 50 participants will be invited to participate. Participants include 10 experienced K-12 educators whose perspectives will be used to anchor the conference in problems of practice. Other participants represent senior through more junior scholars who have demonstrated a commitment to addressing the disconnect between research and practice, along with technology experts to advise participants on capabilities and innovative uses of modern technologies for instruction, assessment and data management.

The overarching goal for the conference is to help the field of mathematics education think deeply about the most productive ways to answer the following questions: [1] Why hasn't past research had a more direct impact on practice? What can be learned from this historical analysis for future research? [2] What is a possible vision for research that would have a more direct impact on practice? What questions should be asked? What methods should be used? What concrete steps can be taken to launch the new research programs? [3] What are the implications of adopting new kinds of research programs? If they gain traction, how will such changes affect the broader education community and infrastructure, including preservice teacher education, teacher professional development, and the training of future researchers? How should the roles of researchers and teachers change? What incentive structures might motivate these changes? How will new programs of research interact with existing programs?

Human Variance and Assessment for Learning Implications for Diverse Learners of STEM: A National Conference

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests.

Lead Organization(s): 
Award Number: 
1939192
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The conference purpose is to stimulate a national conversation concerning the relationships between assessment, teaching and learning that include scholarly research and development of tests; members of city and state boards of education; officials from states and major school systems; policymakers; and representatives of teachers' associations and parents' associations. This conference aims to attract these important professionals has important co-sponsors like the Urban Institute. This national conference flows from the work of the Gordon Commission on the Future of Assessment for Education that addressed the advancement of achievement in STEM disciplines (PreK-12) for students who are underrepresented among high achieving students. This issue of advancement of underrepresented high achieving students has received little concentrated effort and a conference would help in providing greater understanding of this special concern, which includes a student in poverty in complexed family structures.

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests. The conference will stimulate national conversation and ultimately a market that demands educational assessments that inform and improve teaching and learning transactions. The conference will be organized around four conceptual and theoretical papers that focus on the knowledge base upon which six concurrent workshops will be based. The four papers are: (1) Human Diversity and Assessment; (2) The Limits of Test Bias and Its Corrections; (3) Towards an Assessment Science Capable of Informing and Improving Learning; and  (4) Assessment in the Service of Learning. The workshops will focus on models of pedagogical practice that show promise for informing and improving teaching and learning processes and their outcomes. These issues will be discussed by 11-15 expert presenters who understand student learning and the types of information gleaned from different types of assessments. The attention to URMs and their needs and contexts are prioritized in discussions surrounding measurement science and the integration of assessment. Several important issues that address understanding of student learning, and the relationship between the varieties of information concerning students that can be accessed through assessments are: (1) The importance of the broader and more productive use of educational testing to improve the learning of STEM subject matter and values; (2) Curriculum embedded assessment and the reduction in disparities in achievement by STEM learners from diverse social divisions; (3) Innovative procedures and programs for the use of data concerning learners and teaching and learning transactions in the teaching and learning of STEM with learners who are underrepresented among high achieving STEM learners.

Advancing Coherent and Equitable Systems of Science Education

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education.

Lead Organization(s): 
Award Number: 
1920249
Funding Period: 
Thu, 08/01/2019 to Mon, 07/31/2023
Full Description: 

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education. This is an important study because in most states, the student population is becoming more diverse, and states need help in finding ways to better serve schools and districts within their jurisdictions. Through this effort, state science leaders will participate in a networked improvement community model organized to develop and test state-level strategies. Specifically, the focus will be on the adaptation of instructional materials and formative assessment as linked policy strategies for aligning curriculum, instruction, and assessment and for relating instruction to the interests and histories of local communities. State science leaders and researchers will investigate how and under what conditions certain strategies support the emergence of coherent and equitable state systems of science education in which all students have opportunities to meet challenging new science standards. The project will build knowledge and theory about the conditions under which a network of state teams can promote coherent guidance for culturally-based instruction in local districts and schools. Together the partners will collaborate to diagnose current challenges to promoting coherence and equity and then develop knowledge and resources about conditions that promote coherence and equity by testing and studying strategies for cultivating it.

An iterative design-based research approach will be used to build foundational knowledge for the equitable implementation of the vision of science and engineering learning that integrates disciplinary core ideas, science and engineering practices, and crosscutting concepts working from a cultural perspective on learning. A multiple-case study will be used to collect data about the impact of the networked improvement community model on leadership development to effectively improve state efforts. Surveys and interviews will be used to gather information on co-designing efforts, use and adaptation of resources, and knowledge gained by state science leaders. Data will also be collected on political conditions and infrastructures of teamwork as potential facilitators and barriers to the development of strategic knowledge leadership. Analyses of data will identify patterns or configurations of conditions associated with growth in science leaders' strategic knowledge leadership related to equity. This technique will generate evidence-based claims for how and when supports and barriers matter for growth in strategic knowledge leadership for equity.

Validation of the Equity and Access Rubrics for Mathematics Instruction (VEAR-MI)

The main goal of this project is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. The project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

Award Number: 
1908481
Funding Period: 
Mon, 07/15/2019 to Fri, 06/30/2023
Full Description: 

High-quality mathematics instruction remains uncommon and opportunities for students to develop the mathematical understanding are not distributed equally. This is particularly true for students of color and students for whom English is not their first language. While educational research has made progress in identifying practices that are considered high-quality, little attention has been given to specific instructional practices that support historically marginalized groups of students particularly as they participate in more rigorous mathematics. The main goal is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. In addition, the project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

This project will make use of two existing large-scale datasets focusing on mathematics teachers to develop rubrics on mathematics instructional quality. The datasets include nearly 3,000 video-recorded mathematics lessons and student achievement records from students in Grades 3 through 8. The four phases of this research and development project include training material development, an observation and rubric generalizability study, a coder reliability study, and structural analysis. Data analysis plans involve case studies, exploratory and confirmatory factor analyses, and cognitive interviews. 

Validity Evidence for Measurement in Mathematics Education (V-M2ED) (Collaborative Research: Bostic)

The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1920621
Funding Period: 
Thu, 08/01/2019 to Wed, 07/31/2024
Full Description: 

As education has shifted more towards data-driven policy and research initiatives in the last several decades, data for policy-related aspects are often expected to be more quantitative in nature.  This has led to the increase in use of more quantitative measures in STEM education, including mathematics education. Unfortunately, evidence regarding the validity and reliability of mathematics education measures is lacking. Furthermore, the evidence for validity for quantitative tools and measures is not conceptualized or defined consistently by researchers in the field. The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education. Drawing on the results of the synthesis study, the researchers will design, curate, and disseminate a repository of quantitative assessments used in mathematics education teaching and research. The researchers will also create materials and online training for a variety of scholars and practitioners to use the repository.

The team will address two main research questions: 1) How might validity evidence related to quantitative assessments used in mathematics education research be categorized and described? and 2) What validity evidence exists for quantitative instruments used in mathematics education scholarship since 2000? Researchers will use a cross-comparative methodology which involves conducting a literature search and then analyzing and categorizing features of instruments. The research team will examine cases (i.e., assessments described in manuscripts) in which quantitative instruments have been used, alongside specific features such as the construct measured, evidence related to sources of validity, and study sample. The team will then design, develop, and deploy a free online digital repository for the categorization of instruments and describe their associated validity evidence.

Validity Evidence for Measurement in Mathematics Education (V-M2ED) (Collaborative Research: Krupa)

The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education.

Partner Organization(s): 
Award Number: 
1920619
Funding Period: 
Thu, 08/01/2019 to Wed, 07/31/2024
Full Description: 

As education has shifted more towards data-driven policy and research initiatives in the last several decades, data for policy-related aspects are often expected to be more quantitative in nature.  This has led to the increase in use of more quantitative measures in STEM education, including mathematics education. Unfortunately, evidence regarding the validity and reliability of mathematics education measures is lacking. Furthermore, the evidence for validity for quantitative tools and measures is not conceptualized or defined consistently by researchers in the field. The purpose of this project is to fully explore the mathematics education literature to synthesize what validity evidence is available for quantitative assessments in mathematics education. Drawing on the results of the synthesis study, the researchers will design, curate, and disseminate a repository of quantitative assessments used in mathematics education teaching and research. The researchers will also create materials and online training for a variety of scholars and practitioners to use the repository.

The team will address two main research questions: 1) How might validity evidence related to quantitative assessments used in mathematics education research be categorized and described? and 2) What validity evidence exists for quantitative instruments used in mathematics education scholarship since 2000? Researchers will use a cross-comparative methodology which involves conducting a literature search and then analyzing and categorizing features of instruments. The research team will examine cases (i.e., assessments described in manuscripts) in which quantitative instruments have been used, alongside specific features such as the construct measured, evidence related to sources of validity, and study sample. The team will then design, develop, and deploy a free online digital repository for the categorization of instruments and describe their associated validity evidence.

Developing and Validating Early Assessments of College Readiness: Differential Effects for Underrepresented Groups, Optimal Timing of Assessments, and STEM-Specific Indicators

This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

Project Email: 
Award Number: 
1908630
Funding Period: 
Mon, 07/15/2019 to Wed, 06/30/2021
Project Evaluator: 
Full Description: 

One third of all college freshmen are academically unprepared for entry-level college coursework and require remedial course. That figure is much higher at many colleges. The problem is more acute in STEM disciplines, particularly among students from underrepresented ethnic groups and low socioeconomic status families. This purpose of this project is to develop and validate a range of assessments with a focus on academic preparedness for higher education. The team will explore relevant qualities of assessments such as their differential predictive validity to ensure they are appropriate for underrepresented groups, the optimal grade level to begin assessing readiness, and measures that are most appropriate for predicting STEM-specific readiness.

This project will use two recent and complementary large-scale, nationally representative federal databases: the High School Longitudinal Study of 2009 and the Education Longitudinal Study of 2002. Factor analysis will be used to develop composite variables of college readiness and multilevel regression will be used to develop predictive models on a range of college outcomes to test the predictive validity of composite and individual predictors. The models will be extended to conduct multiple group analyses to test for differential prediction for students from underrepresented groups. The project intends to promote 1) the use of a wider range of assessments of academic preparedness, 2) the use of measures that are more sensitive for assessing college readiness from underrepresented groups and among STEM majors, 3) earlier assessment using indicators and models with predictive validity and 4) progress monitoring of college readiness by providing a detailed example of how that can be developed and implemented. Findings will also raise student, parental, teacher, and other school personnel awareness of the range of factors relevant for preparing students for college.

Alternative video text
Alternative video text: 

Pages

Subscribe to Assessment