Assessment

Assessing College-Ready Computational Thinking (Collaborative Research: Wilson)

The goal of this project is to develop learning progressions and assessment items targeting computational thinking. The items will be used for a test of college-ready critical reasoning skills and will be integrated into an existing online assessment system, the Berkeley Assessment System Software.

Award Number: 
2010314
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

Because of the growing need for students to be college and career ready, high-quality assessments of college readiness skills are in high demand. To realize the goal of preparing students for college and careers, assessments must measure important competencies and provide rapid feedback to teachers. It is necessary to go beyond the limits of multiple-choice testing and foster the skills and thinking that lie at the core of college and career ready skills, such as computational thinking. Computational thinking is a set of valuable skills that can be used to solve problems, design systems, and understand human behavior, and is thus essential to developing a more STEM-literate public. Computational thinking is increasingly seen as a fundamental analytical skill that everyone, not just computer scientists, can use. The goal of this project is to develop learning progressions and assessment items targeting computational thinking. The items will be used for a test of college-ready critical reasoning skills and will be integrated into an existing online assessment system, the Berkeley Assessment System Software.

The project will address a set of research questions focused on 1) clarifying computational thinking constructs, 2) usability, reliability of validity of assessment items and the information they provide, 3) teachers' use of assessments, and 4) relationships to student performance. The study sample of 2,700 used for the pilot and field tests will include all levels of students in 10th through 12th grade and first year college students (both community college and university level). The target population is students in schools which are implementing the College Readiness Program (CRP) of the National Mathematics and Science Institute. In the 2020-21 academic year 54 high schools across 11 states (CA, GA, FL, ID, LA, NC, NM, OH, TX, VA, and WA) will participate. This will include high school students in Advanced Placement classes as well as non-Advanced Placement classes.  The team will use the BEAR Assessment System to develop and refine assessment materials. This system is an integrated approach to developing assessments that seeks to provide meaningful interpretations of student work relative to cognitive and developmental goals. The researchers will gather empirical evidence to develop and improve the assessment materials, and then gather reliability and validity evidence to support their use. In total, item response data will be collected from several thousand students. Student response data will be analyzed using multidimensional item response theory models.

Supporting Students' Language, Knowledge, and Culture through Science

This project will test and refine a teaching model that brings together current research about the role of language in science learning, the role of cultural connections in students' science engagement, and how students' science knowledge builds over time. The outcome of this project will be to provide an integrated framework that can guide current and future science teachers in preparing all students with the conceptual and linguistic practices they will need to succeed in school and in the workplace.

Lead Organization(s): 
Award Number: 
2010633
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

The Language, Culture, and Knowledge-building through Science project seeks to explore and positively influence the work of science teachers at the intersection of three significant and ongoing challenges affecting U.S. STEM education. First, U.S. student demographics are rapidly changing, with an increasing number of students learning STEM subjects in their second language. This change means that all teachers need new skills for meeting students where they currently are, linguistically, culturally, and in terms of prior science knowledge. Second, the needs and opportunities of the national STEM workforce are changing rapidly within a shifting employment landscape. This shift means that teachers need to better understand future job opportunities and the knowledge and skills that will be necessary in those careers. Third, academic expectations in schools have changed, driven by changes in education standards. These new expectations mean that teachers need new skills to support all students to master a range of practices that are both conceptual and linguistic. To address these challenges, teachers require new models that bring together current research about the role of language in science learning, the role of cultural connections in students' science engagement, and how students' science knowledge builds over time. This project begins with such an initial model, developed collaboratively with science teachers in a prior project. The model will be rigorously tested and refined in a new geographic and demographic context. The outcome will be to provide an integrated framework that can guide current and future science teachers in preparing all students with the conceptual and linguistic practices they will need to succeed in school and in the workplace.

This project model starts with three theoretical constructs that have been integrated into an innovative framework of nine practices. These practices guide teachers in how to simultaneously support students' language development, cultural sustenance, and knowledge building through science with a focus on supporting and challenging multilingual learners. The project uses a functional view of language development, which highlights the need to support students in understanding both how and why to make shifts in language use. For example, students' attention will be drawn to differences in language use when they shift from language that is suited to peer negotiation in a lab group to written explanations suitable for a lab report. Moving beyond a funds of knowledge approach to culture, the team view of integrating students' cultural knowledge includes strengthening the role of home knowledge in school, but also guiding students to apply school knowledge to their out-of-school interests and passions. Finally, the project team's view of cumulative knowledge building, informed by work in the sociology of knowledge, highlights the need for teachers and students to understand the norms for meaning making within a given discipline. In the case of science, the three-dimensional learning model in the Next Generation Science Standards makes these disciplinary norms visible and serves as a launching point for the project's work. Teachers will be supported to structure learning opportunities that highlight what is unique about meaning making through science. Using a range of data collection and analysis methods, the project team will study changes in teachers' practices and beliefs related to language, culture and knowledge building, as teachers work with all students, and particularly with multilingual learners. The project work will take place in both classrooms and out of class science learning settings. By working closely over several years with a group of fifty science teachers spread across the state of Oregon, the project team will develop a typology of teachers (design personas) to increase the field's understanding of how to support different teachers, given their own backgrounds, in preparing all students for the broad range of academic and occupational pathways they will encounter.

Learning Progressions in Science: Analyzing and Deconstructing the Multiple Dimensions in Assessment

Through this project, researchers will develop internet-based assessments designed to capture learning outcomes that (a) measure the higher order cognitive skills that are essential to current reform efforts, and (b) that report results in ways that are readily accessible and interpretable.

Award Number: 
2010322
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

Assessments are a crucial tool to enable the success of teaching and learning in science classrooms. Hence, to realize the vision of current reform efforts assessments must be developed that (a) measure the higher order cognitive skills that are essential to those reforms, and (b) that report results in ways that are readily accessible and interpretable. Through this project, researchers will develop internet-based assessments that capture such learning outcomes. These assessments can influence policy and practice by providing tangible products that exemplify the kind of learning outcomes and performances expected in today's science classrooms. The measures will also ensure that all students are assessed fairly. This project has the potential to enable students to express their knowledge and skills in a variety of ways which are less demanding and more creative than typical in traditional assessments.

Researchers will develop assessment materials aligned to the Next Generation Science Standards practice of scientific argumentation, the cross-cutting concept of patterns, and content in each of the following foundational middle school science domains - physical, life, and Earth sciences. Researchers will work closely with administrators and educators to ensure the relevance and alignment of materials to teachers' needs. Data will be gathered from middle and high school students from ethnically, culturally, and linguistically diverse school districts in California and Arizona. Key measures to be developed include embedding questions about patterns into the nine existing scenarios (tasks) and in new scenarios about natural resources. The team will develop assessments that can be used in both open-ended and selected response formats, to enhance their usefulness to teachers for diagnostic understanding, and their efficiency for summative classroom use. Data will also be gathered from teachers, to help develop interpretational materials. Teachers will have access, in real time, to their own students' responses, and estimates of students' performance on learning progressions within each of the science practices, crosscutting concepts and domains mentioned. The team will use the BEAR Assessment System to develop and refine assessment materials. This system is an integrated approach to developing assessments that seeks to provide meaningful interpretations of student work relative to cognitive and developmental goals. The researchers will gather empirical evidence to develop and improve the assessment materials, and then gather reliability and validity evidence to support their use. In total, item response data will be collected from several thousand students across the two districts. Student response data will be analyzed using multidimensional item response theory models.

Learning to Teach During COVID-19: Leveraging Simulated Classrooms as Practice-based Spaces for Preservice Elementary Teachers within Online Teacher Education Courses

The COVID-19 pandemic has significantly disrupted the ability of teacher education programs to place their teacher candidates in typical K-12 teaching settings as a part of learning to teach. This project examines how simulated classroom field experiences for preservice teachers can be implemented in online and emergency remote teacher education courses.

Lead Organization(s): 
Award Number: 
2032179
Funding Period: 
Mon, 06/15/2020 to Mon, 05/31/2021
Full Description: 

School-based field experiences are a critical part of preservice teacher education. The COVID-19 pandemic has significantly disrupted the ability of teacher education programs to place their teacher candidates in typical K-12 teaching settings as a part of learning to teach. This project examines how simulated classroom field experiences for preservice teachers can be implemented in online and emergency remote teacher education courses. Elementary mathematics and science teacher educators are provided with opportunities to engage their preservice teachers in practice-based spaces using mixed-reality simulated classroom environments. These simulations are real-time lessons with animated student avatars that are voiced by an interactor who is responding to the teacher's lesson in real time in ways that represent authentic student thinking. This project aims to develop support materials for integrating simulated field experiences into elementary mathematics and science teacher education courses. The research will seek to understand what preservice teachers learn about teaching from these experiences, how teacher educators integrate the simulated field experiences into coursework, and how such simulated experiences can be integrated in remote, online courses in ways that support preservice teacher learning.

This project advances knowledge through the development and deployment of simulation-based tools that develop preservice elementary teachers' abilities to teach mathematics and science. Preservice teachers use performance tasks to deliver instruction in the simulated classroom. The project develops support materials for teacher educators to integrate this work into online and/or emergency remote teacher education courses (in response to COVID-19) in ways that support engagement in ambitious teaching practice. The project assesses impact on preservice teachers' ambitious teaching practice through artifacts of the simulated classroom practice, including observations and recordings of the simulated interactions and preservice teacher surveys and assessments of their use of ambitious teaching practices. The project evaluates the ways in which teacher educators integrate the simulated field experience into their emergency remote teacher education courses through surveys and interviews. The research addresses the immediate COVID-19 pandemic challenges in providing field experiences for students and provides long-term support for the ongoing challenge of finding field experience settings that are conducive to preparing highly-qualified elementary mathematics and science teachers.

Preparing Teachers to Design Tasks to Support, Engage, and Assess Science Learning in Rural Schools

This study focuses on working with teachers to develop assessment practices that focuses on the three NGSS dimensions of science ideas, practices and cross-cutting concepts, and adds two more dimensions; teachers will develop assessment tasks interesting to students, and promote the development of their science identities. To advance equitable opportunities for all students to learn science, this project will design and provide an online course to support rural teachers who teach science in grades 6-12.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2010086
Funding Period: 
Tue, 09/01/2020 to Sat, 08/31/2024
Full Description: 

Nationally, a third of US students attend rural or remote schools, yet rural teachers receive fewer opportunities to work together and engage in professional learning than their suburban and urban counterparts. This, in turn, can reflect on the opportunities for rural students to learn the high quality, up-to-date science ideas, practices, and concepts that are required by state standards, especially those aligned with the Next Generation Science Standards (NGSS). To advance equitable opportunities for all students to learn science, the project team will design and provide an online course to support rural teachers who teach science in grades 6-12. The course will focus on improving classroom science assessment practices and instruction to meet the unique needs of rural educators and their students. Too often, science concepts are removed from the lives of rural students, although their homes, communities and natural environments are filled with ideas and experiences that can make science come alive. When teachers link assessment and instruction to students' everyday lives, students have enhanced interest in and identification with science. This, in turn, can lead more students to pursue science and applied science fields beyond high school, to broaden the STEM pipeline. In addition, students are better prepared to participate in science in their communities as empowered citizens. This study focuses on working with teachers to develop assessment practices that not only focus on the three NGSS dimensions of science ideas, practices and cross-cutting concepts, it also adds two more dimensions; teachers will develop assessment tasks interesting to students, and promote the development of their science identities. The researchers refer to this as 5D assessment.

Researchers at Colorado University Boulder and BSCS Science Learning will use design-based implementation research to collaboratively design an online course sequence that targets 5D assessment in science. The study will proceed in three phases: a rapid ethnographic study to assess the needs of teachers serving a variety of rural communities, a study of teachers' use of an online platform for their professional learning, and lastly an experimental study to research the effects of the online course on teacher and student outcomes. Researchers will recruit 10 teachers to take the on-line course for the professional development and collect data on participating teachers' implementation of the course ideas through classroom videotaping and surveys designed to capture their changing practices. In the third year of the project, researchers will conduct an impact study with 70 secondary science teachers taking the re-designed on-line course, and compare their outcomes with a "business-as-usual" condition. The impact of course will be measured by questionnaires that address their vision for teaching the NGSS and self-reported instructional practices; classroom observations; and, teacher-constructed student assessments. Student outcomes will be measured using science interest and identity surveys, and an examination of student work products that demonstrate students' ability to use the science and engineering practice of modeling, a practice likely to be encountered in all NGSS science classrooms. The project will identify conditions under which learning about 5D assessment task design can support instructional improvement, increase student interest in science and engineering, and enhance students' opportunities to learn. The researchers hypothesize that the online program will have a positive impact on teachers' vision, classroom practices, and their use of high-quality 5D tasks. They also hypothesize that teacher participation will result in significant increases in student interest in and identification with science, and that these effects will be mediated by teacher outcomes. Finally, the researchers hypothesize that effects will be equitable across demographic variables in rural communities.

Leveraging Simulations in Preservice Preparation to Improve Mathematics Teaching for Students with Disabilities (Collaborative Research: Jones)

This project aims to support the mathematics learning of students with disabilities through the development and use of mixed reality simulations for elementary mathematics teacher preparation. These simulations represent low-stakes opportunities for preservice teachers to practice research-based instructional strategies to support mathematics learning, and to receive feedback on their practices.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2010298
Funding Period: 
Fri, 05/01/2020 to Tue, 04/30/2024
Full Description: 

The preparation of general education teachers to support the mathematics learning of students with disabilities is critical, as students with disabilities are overrepresented in the lower ranks of mathematics achievement. This project aims to address this need in the context of elementary mathematics teacher preparation through the development and use of mixed reality simulations. These simulations represent low-stakes opportunities for preservice teachers to practice research-based instructional strategies to support mathematics learning, and to receive feedback on their practices. Learning units that use the simulations will focus on two high leverage practices: teacher modeling of self-monitoring and reflection strategies during problem solving and using strategy instruction to teach students to support problem solving. These high-leverage teaching practices will support teachers engaging all students, including students with disabilities, in conceptually sophisticated mathematics in which students are treated as sense-makers and empowered to do mathematics in culturally meaningful ways.

The project work encompasses three primary aims. The first aim is to develop a consensus around shared definitions of high-leverage practices across the mathematics education and special education communities. To accomplish this goal, the project will convene a series of consensus-building panels with mathematics education and special education experts to develop shared definitions of the two targeted high leverage practices. This work will include engaging with current research, group discussion, and production of documents with specifications for the practices. The second aim is to develop learning units for elementary mathematics methods courses grounded in mixed reality simulation. These simulations will allow teacher candidates to enact the high leverage practices with simulated students and to receive coaching on their practice from the research team. The impact of this work will be assessed through the analysis of interviews with teacher educators implementing the units and observations and artifacts from the implementations. The third aim will be to assess the effectiveness of the simulations on teacher candidates? practices and beliefs through small-scaled randomized control trials. Teacher candidates will be randomly assigned to conditions that address the practices and make use of simulations, and a business as usual condition focused on lesson planning, student assessment, and small group discussions of the high leverage practices. The impact of the work will be assessed through the analysis of baseline and exit simulations, measures of teacher self-efficacy for teaching students with disabilities, and observations of classroom teaching in their clinical placement settings.

Looking Back and Looking Forward: Increasing the Impact of Educational Research on Practice

The focus of this conference is to carefully examine past and current research with an eye toward improving its impact on practice and to create concrete steps that could shape the nature and impact of mathematics education research.

Lead Organization(s): 
Award Number: 
1941494
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The focus of the proposed conference is to carefully examine past and current research with an eye toward improving its impact on practice. This conference is designed to create concrete steps that could shape the nature and impact of mathematics education research for years to come. A diverse group of 50 participants will be invited to participate. Participants include 10 experienced K-12 educators whose perspectives will be used to anchor the conference in problems of practice. Other participants represent senior through more junior scholars who have demonstrated a commitment to addressing the disconnect between research and practice, along with technology experts to advise participants on capabilities and innovative uses of modern technologies for instruction, assessment and data management.

The overarching goal for the conference is to help the field of mathematics education think deeply about the most productive ways to answer the following questions: [1] Why hasn't past research had a more direct impact on practice? What can be learned from this historical analysis for future research? [2] What is a possible vision for research that would have a more direct impact on practice? What questions should be asked? What methods should be used? What concrete steps can be taken to launch the new research programs? [3] What are the implications of adopting new kinds of research programs? If they gain traction, how will such changes affect the broader education community and infrastructure, including preservice teacher education, teacher professional development, and the training of future researchers? How should the roles of researchers and teachers change? What incentive structures might motivate these changes? How will new programs of research interact with existing programs?

Human Variance and Assessment for Learning Implications for Diverse Learners of STEM: A National Conference

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests.

Lead Organization(s): 
Award Number: 
1939192
Funding Period: 
Sun, 09/01/2019 to Mon, 08/31/2020
Full Description: 

The conference purpose is to stimulate a national conversation concerning the relationships between assessment, teaching and learning that include scholarly research and development of tests; members of city and state boards of education; officials from states and major school systems; policymakers; and representatives of teachers' associations and parents' associations. This conference aims to attract these important professionals has important co-sponsors like the Urban Institute. This national conference flows from the work of the Gordon Commission on the Future of Assessment for Education that addressed the advancement of achievement in STEM disciplines (PreK-12) for students who are underrepresented among high achieving students. This issue of advancement of underrepresented high achieving students has received little concentrated effort and a conference would help in providing greater understanding of this special concern, which includes a student in poverty in complexed family structures.

The conference will attract thought leaders, policy makers, supervisors of practice and scholars of measurement science to be informed of emerging thought and developments and to discuss selected models for the implementation of new ways of generating and utilizing data from education tests. The conference will stimulate national conversation and ultimately a market that demands educational assessments that inform and improve teaching and learning transactions. The conference will be organized around four conceptual and theoretical papers that focus on the knowledge base upon which six concurrent workshops will be based. The four papers are: (1) Human Diversity and Assessment; (2) The Limits of Test Bias and Its Corrections; (3) Towards an Assessment Science Capable of Informing and Improving Learning; and  (4) Assessment in the Service of Learning. The workshops will focus on models of pedagogical practice that show promise for informing and improving teaching and learning processes and their outcomes. These issues will be discussed by 11-15 expert presenters who understand student learning and the types of information gleaned from different types of assessments. The attention to URMs and their needs and contexts are prioritized in discussions surrounding measurement science and the integration of assessment. Several important issues that address understanding of student learning, and the relationship between the varieties of information concerning students that can be accessed through assessments are: (1) The importance of the broader and more productive use of educational testing to improve the learning of STEM subject matter and values; (2) Curriculum embedded assessment and the reduction in disparities in achievement by STEM learners from diverse social divisions; (3) Innovative procedures and programs for the use of data concerning learners and teaching and learning transactions in the teaching and learning of STEM with learners who are underrepresented among high achieving STEM learners.

Advancing Coherent and Equitable Systems of Science Education

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education.

Lead Organization(s): 
Award Number: 
1920249
Funding Period: 
Thu, 08/01/2019 to Mon, 07/31/2023
Full Description: 

This project will examine how partnerships among state science leaders, education researchers and education practitioners cultivate vertical coherence and equity in state science education. This is an important study because in most states, the student population is becoming more diverse, and states need help in finding ways to better serve schools and districts within their jurisdictions. Through this effort, state science leaders will participate in a networked improvement community model organized to develop and test state-level strategies. Specifically, the focus will be on the adaptation of instructional materials and formative assessment as linked policy strategies for aligning curriculum, instruction, and assessment and for relating instruction to the interests and histories of local communities. State science leaders and researchers will investigate how and under what conditions certain strategies support the emergence of coherent and equitable state systems of science education in which all students have opportunities to meet challenging new science standards. The project will build knowledge and theory about the conditions under which a network of state teams can promote coherent guidance for culturally-based instruction in local districts and schools. Together the partners will collaborate to diagnose current challenges to promoting coherence and equity and then develop knowledge and resources about conditions that promote coherence and equity by testing and studying strategies for cultivating it.

An iterative design-based research approach will be used to build foundational knowledge for the equitable implementation of the vision of science and engineering learning that integrates disciplinary core ideas, science and engineering practices, and crosscutting concepts working from a cultural perspective on learning. A multiple-case study will be used to collect data about the impact of the networked improvement community model on leadership development to effectively improve state efforts. Surveys and interviews will be used to gather information on co-designing efforts, use and adaptation of resources, and knowledge gained by state science leaders. Data will also be collected on political conditions and infrastructures of teamwork as potential facilitators and barriers to the development of strategic knowledge leadership. Analyses of data will identify patterns or configurations of conditions associated with growth in science leaders' strategic knowledge leadership related to equity. This technique will generate evidence-based claims for how and when supports and barriers matter for growth in strategic knowledge leadership for equity.

Validation of the Equity and Access Rubrics for Mathematics Instruction (VEAR-MI)

The main goal of this project is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. The project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

Award Number: 
1908481
Funding Period: 
Mon, 07/15/2019 to Fri, 06/30/2023
Full Description: 

High-quality mathematics instruction remains uncommon and opportunities for students to develop the mathematical understanding are not distributed equally. This is particularly true for students of color and students for whom English is not their first language. While educational research has made progress in identifying practices that are considered high-quality, little attention has been given to specific instructional practices that support historically marginalized groups of students particularly as they participate in more rigorous mathematics. The main goal is to validate a set of rubrics that attend to the existence and the quality of instructional practices that support equity and access in mathematics classes. In addition, the project team will clarify the relationships between the practices outlined in the rubrics and aspects of teachers' perspectives and knowledge as well as student learning outcomes.

This project will make use of two existing large-scale datasets focusing on mathematics teachers to develop rubrics on mathematics instructional quality. The datasets include nearly 3,000 video-recorded mathematics lessons and student achievement records from students in Grades 3 through 8. The four phases of this research and development project include training material development, an observation and rubric generalizability study, a coder reliability study, and structural analysis. Data analysis plans involve case studies, exploratory and confirmatory factor analyses, and cognitive interviews. 

Pages

Subscribe to Assessment