Assessment

AI-based Assessment in STEM Education Conference

The Framework for K-12 Science Education has set forth an ambitious vision for science learning by integrating disciplinary science ideas, scientific and engineering practices, and crosscutting concepts, so that students could develop competence to meet the STEM challenges of the 21st century. Achieving this vision requires transformation of assessment practices from relying on multiple-choice items to performance-based knowledge-in-use tasks.

Lead Organization(s): 
Award Number: 
2138854
Funding Period: 
Sun, 08/01/2021 to Sun, 07/31/2022
Full Description: 

The Framework for K-12 Science Education has set forth an ambitious vision for science learning by integrating disciplinary science ideas, scientific and engineering practices, and crosscutting concepts, so that students could develop competence to meet the STEM challenges of the 21st century. Achieving this vision requires transformation of assessment practices from relying on multiple-choice items to performance-based knowledge-in-use tasks. Such novel assessment tasks serve the purpose of both engaging students in using knowledge to solve problems and tracking students’ learning progression so that teachers could adjust instruction to meet students’ needs. However, these performance-based constructed-response items often prohibit timely feedback, which, in turn, has hindered science teachers from using these assessments. Artificial Intelligence (AI) has demonstrated great potential to meet this assessment challenge. To tackle this challenge, experts in assessment, AI, and science education will gather for a two-day conference at University of Georgia to generate knowledge of integrating AI in science assessment.

The conference is organized around four themes: (a) AI and Domain Specific Learning Theory; (b) AI and validity theory and assessment design principles; (c) AI and technology integration theory; and (d) AI and pedagogical theory focusing on assessment practices. It allows participants to share theoretical perspectives, empirical findings, as well as research experiences. It can also help identify challenges and future research directions to increase the broad use of AI-based assessments in science education. The conference will be open to other researchers, postdocs, and students via Zoom. It is expected that conference participants establish a network in this emergent area of science assessment. Another outcome of the conference, Applying AI in STEM Assessment, will be published as an edited volume by Harvard Education Press.

Facilitating Formative Feedback: Using Simulations to Impact the Capability of Novice Mathematics Teachers

This project explores the ways in which thoughtfully designed simulations can provide preservice teachers with formative assessment opportunities that serve as a complement to, or alternative to as needed, feedback derived from field placement contexts. A set of simulations will be designed with a focus on eliciting and interpreting student thinking. These simulations will be used with preservice teachers in three elementary teacher preparation programs of varying size and demographics.

Lead Organization(s): 
Award Number: 
2101343
Funding Period: 
Wed, 09/01/2021 to Sun, 08/31/2025
Full Description: 

During their initial teacher preparation experiences, preservice teachers need meaningful formative assessment that can support them in developing their skills and practices as new teachers of mathematics. While field placements offer some such opportunities, too often preservice teachers are not able to see, experience, and enact a full range of research-based effective mathematics teaching practices. This level II four-year design and development study in the assessment strand explores the ways in which thoughtfully designed simulations can provide preservice teachers with formative assessment opportunities that serve as a complement to, or alternative to as needed, feedback derived from field placement contexts. A set of simulations will be designed with a focus on eliciting and interpreting student thinking. These simulations will be used with preservice teachers in three elementary teacher preparation programs of varying size and demographics. Data will be collected to understand the ways in which the feedback from engaging in the simulations serves to strengthen preservice teachers' abilities to elicit and interpret student thinking through an analysis of performance in the simulations, interviews with preservice teachers, and feedback from teacher educators. An associated study will establish the reliability and validity of the simulations as assessment tools.

Simulations will be developed and tested in three cycles, with iterative improvements made between each cycle. The first cycle will involve 10 preservice teachers in a pilot study separate from participation in a course, in which preservice teachers engage in a simulation, receive formative feedback, and engage in a second similar simulation. This cycle will evaluate the extent to which feedback appears to influence subsequent performance. In the second cycle, the project will work with three teacher educators in diverse contexts to enact the simulations with all preservice teachers in one section of their elementary mathematics methods courses. In the final cycle, the use of the simulations will shift from a research team actor playing the role of the student to a site-based actor recruited by the teacher educators at each of the three institutions. To validate the tools, researcher reliability and teacher educator reliability studies will be conducted to asses the extent to which the four different simulation assessments provide consistent feedback on the targeted teaching practices and the extent to which the scoring of the assessments are reliable. A G study (generalizability study) will be conducted to evaluate the extent to which the teacher participant is the primary source of variation as compared to variations from student actors or the rater administering the assessment. Results will be disseminated in a variety of mathematics education settings and the simulation materials will be made available to practitioners and adapted for additional use in  mixed-reality simulation platforms.

Learning about Viral Epidemics through Engagement with Different Types of Models

The COVID-19 pandemic has highlighted the need for supporting student learning about viral outbreaks and other complex societal issues. Given the complexity of issues like viral outbreaks, engaging learners with different types of models (e.g., mechanistic, computational and system models) is critical. However, there is little research available regarding how learners coordinate sense making across different models.

Award Number: 
2101083
Funding Period: 
Wed, 09/01/2021 to Sun, 08/31/2025
Full Description: 

The project will develop new curriculum and use it to research how high school students learn about viral epidemics while developing competencies for scientific modeling. The COVID-19 pandemic has highlighted the need for supporting student learning about viral outbreaks and other complex societal issues. Given the complexity of issues like viral outbreaks, engaging learners with different types of models (e.g., mechanistic, computational and system models) is critical. However, there is little research available regarding how learners coordinate sense making across different models. This project will address the gap by studying student learning with different types of models and will use these findings to develop and study new curriculum materials that incorporate multiple models for teaching about viral epidemics in high school biology classes. COVID-19 caused devasting impacts, and marginalized groups including the Latinx community suffered disproportionately negative outcomes. The project will directly recruit Latinx students to ensure that design products are culturally responsive and account for Latinx learner needs. The project will create new pathways for engaging Latinx students in innovative, model-based curriculum about critically important issues. Project research and resources will be widely shared via publications, conference presentations, and professional development opportunities for teachers.

The project will research three aspects of student learning: a) conceptual understandings about viral epidemics, b) epistemic understandings associated with modeling, and c) model-informed reasoning about viral epidemics and potential solutions. The research will be conducted in three phases. Phase 1 will explore how students make sense of viral epidemics through different types of models. This research will be conducted with small groups of students as they work through learning activities and discourse opportunities associated with viral epidemic models. Phase 2 will research how opportunities to engage in modeling across different types of models should be supported and sequenced for learning about viral epidemics. These findings will make it possible to revise the learning performance which will be used to develop a curricular module for high school biology classes. Phase 3 will study the extent to which students learn about viral epidemics through engagement in modeling practices across different models. For this final phase, teachers will participate in professional development about viral epidemics and modeling and then implement the viral epidemic module in their biology classes. A pre- and post-test research design will be used to explore student conceptual understandings, model-informed reasoning, and epistemic understandings.

Developing and Evaluating Assessments of Problem-Solving in Computer Adaptive Testing Environments (Collaborative Research: Bostic)

The Common Core State Standards for Mathematics (CCSSM) problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2100988
Funding Period: 
Sun, 08/01/2021 to Fri, 07/31/2026
Full Description: 

Problem solving has been a priority within K-12 mathematics education for over four decades and is reflected throughout the Common Core State Standards for Mathematics (CCSSM) initiative, which have been adopted in some form by 41 states. Broadly defined, problem solving involves the mathematical practices in which students engage as they solve intellectually-challenging mathematical tasks. In prior research, problem-solving measures aligned to CCSSM for grades 3-5 were developed and validated to supplement previously established problem-solving measures in grades 6-8. The problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests. Few measures designed to assess students’ mathematical problem-solving ability use this technology. Shorter tests require less in-class time for assessment than current paper-pencil problem-solving measures and increase classroom instruction time. The computer-adaptive problem-solving measures have sufficient reliability and strong validity evidence, and may limit test-taker fatigue. Finally, the project will benchmark current grades 6-8 instruments using an objective standard-setting method, which allows for improved score interpretations with content-related feedback. Immediate results of student- and class-level reports will be produced through the computer adaptive testing system allowing for teachers to modify instruction to improve students’ learning.

This five-year project aims to advance the use of computer adaptive testing and assessment development for use in mathematics instruction. The project applies an iterative and stakeholder-informed design science-based methodology as well as employs the use of Rasch modeling for the psychometric analysis during item development and validation. The project aims to: (a) benchmark the previously established grades 6-8 problem-solving measures; (b) develop, calibrate, and validate criterion-referenced computer adaptive testing for each measure; (c) construct student- and class-level score reports for integration into the computer adaptive testing system; and (d) investigate teachers’ capacity for implementing, interpreting, and using the assessments and results in STEM learning settings. The project addresses the following set of research questions: (RQ1) What benchmark performance standards define different proficiency levels on problem-solving measures for each grade level? (RQ2) What are the psychometric properties of new problem-solving measures items developed for the computer adaptive testing item bank? (RQ3) Is there significant item drift across student populations on the new problem-solving measure items? (RQ4) To what extent are problem-solving measures item calibrations stable within the computer adaptive testing system? (RQ5) What recommendations for improvements do teachers and students have for the new problem-solving measures items, computer adaptive testing platform and reporting system, if any? (RQ6) To what extent do teachers interact with, perceive, and make sense of the assessment information generated for use in practice? and (RQ7) Does an online learning module build teacher capacity for problem solving measures, computer adaptive testing implementation, interpretation, and use of student assessment outcomes in STEM learning settings? An experimental design will be utilized to investigate teachers’ capacity for implementing, interpreting, and using problem solving measures in a computer adaptive testing system. The project has the potential to impact the field by providing school districts and researchers a means to assess students’ mathematical problem-solving performance at one time or growth over time efficiently and effectively; address future online learning needs; and improve classroom teaching through more precise information about students’ strengths with less class time focused on assessment.

Developing and Evaluating Assessments of Problem-Solving in Computer Adaptive Testing Environments (Collaborative Research: Sondergeld)

The Common Core State Standards for Mathematics (CCSSM) problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
2101026
Funding Period: 
Sun, 08/01/2021 to Fri, 07/31/2026
Full Description: 

Problem solving has been a priority within K-12 mathematics education for over four decades and is reflected throughout the Common Core State Standards for Mathematics (CCSSM) initiative, which have been adopted in some form by 41 states. Broadly defined, problem solving involves the mathematical practices in which students engage as they solve intellectually-challenging mathematical tasks. In prior research, problem-solving measures aligned to CCSSM for grades 3-5 were developed and validated to supplement previously established problem-solving measures in grades 6-8. The problem-solving measures assess students’ problem-solving performance within the context of CCSSM math content and practices. This project expands the scope of the problem-solving measures use and score interpretation. The project work advances mathematical problem-solving assessments into computer adaptive testing. Computer adaptive testing allows for more precise and efficient targeting of student ability compared to static tests. Few measures designed to assess students’ mathematical problem-solving ability use this technology. Shorter tests require less in-class time for assessment than current paper-pencil problem-solving measures and increase classroom instruction time. The computer-adaptive problem-solving measures have sufficient reliability and strong validity evidence, and may limit test-taker fatigue. Finally, the project will benchmark current grades 6-8 instruments using an objective standard-setting method, which allows for improved score interpretations with content-related feedback. Immediate results of student- and class-level reports will be produced through the computer adaptive testing system allowing for teachers to modify instruction to improve students’ learning.

This five-year project aims to advance the use of computer adaptive testing and assessment development for use in mathematics instruction. The project applies an iterative and stakeholder-informed design science-based methodology as well as employs the use of Rasch modeling for the psychometric analysis during item development and validation. The project aims to: (a) benchmark the previously established grades 6-8 problem-solving measures; (b) develop, calibrate, and validate criterion-referenced computer adaptive testing for each measure; (c) construct student- and class-level score reports for integration into the computer adaptive testing system; and (d) investigate teachers’ capacity for implementing, interpreting, and using the assessments and results in STEM learning settings. The project addresses the following set of research questions: (RQ1) What benchmark performance standards define different proficiency levels on problem-solving measures for each grade level? (RQ2) What are the psychometric properties of new problem-solving measures items developed for the computer adaptive testing item bank? (RQ3) Is there significant item drift across student populations on the new problem-solving measure items? (RQ4) To what extent are problem-solving measures item calibrations stable within the computer adaptive testing system? (RQ5) What recommendations for improvements do teachers and students have for the new problem-solving measures items, computer adaptive testing platform and reporting system, if any? (RQ6) To what extent do teachers interact with, perceive, and make sense of the assessment information generated for use in practice? and (RQ7) Does an online learning module build teacher capacity for problem solving measures, computer adaptive testing implementation, interpretation, and use of student assessment outcomes in STEM learning settings? An experimental design will be utilized to investigate teachers’ capacity for implementing, interpreting, and using problem solving measures in a computer adaptive testing system. The project has the potential to impact the field by providing school districts and researchers a means to assess students’ mathematical problem-solving performance at one time or growth over time efficiently and effectively; address future online learning needs; and improve classroom teaching through more precise information about students’ strengths with less class time focused on assessment.

Dimensions of Success: Transforming Quality Assessment in Middle School Science and Engineering

This project represents a new approach to quality assessment of K-12 science and engineering learning experiences. By updating and expanding the Dimensions of Success (DoS) observation tool initially established for informal science learning settings to middle school science and engineering classrooms (DoS-MSSE), the project will create and implement a sustainable and scalable system of support for teachers who are learning how to implement the Next Generation Science Standards (NGSS) Framework for K-12 effectively and equitably.

Lead Organization(s): 
Award Number: 
2101554
Funding Period: 
Sat, 05/15/2021 to Wed, 04/30/2025
Full Description: 

This project represents a new approach to quality assessment of K-12 science and engineering learning experiences. By updating and expanding the Dimensions of Success (DoS) observation tool initially established for informal science learning settings to middle school science and engineering classrooms (DoS-MSSE), the project will create and implement a sustainable and scalable system of support for teachers who are learning how to implement the Next Generation Science Standards (NGSS) Framework for K-12 effectively and equitably. The project’s goals are as follows. (1) Develop and test the DoS-MSSE observation tool, including alignment to the NGSS and equity, to ensure scores are valid and reliable across all dimensions. (2) Establish a video scoring method to increase access to continuous improvement systems of support. (3) Create profiles of STEM quality for schools and districts to inform policy decisions, professional development opportunities, practice guidelines, and other efforts. (4) Develop and implement a training program to certify teachers and staff in the use of the tool and provide professional development. Extending DoS to middle schools will support and advancing educational practice by identifying schools’ strengths and challenges in science and engineering learning, making equity more concrete, and enhancing school and teacher capacity through established DoS systems of support. The tool will be developed to assess learning experiences for quality improvement, not used for accountability or comparison. DoS-MSSE will also support research and evaluation by providing a validated quality assessment tool, specific to middle school science and engineering, that is flexible and can be applied to a variety of proposals and projects, enhancing research infrastructure. Finally, it will improve the STEM education field by providing stakeholders with “quality profiles” to guide policy and approaches. By promoting equitable, high-quality science and engineering learning inside and outside of school, the expanded DoS framework will strengthen efforts across sectors and help communities provide the inspiration, knowledge, and skills youth need to thrive in the workforce and in life.

Over four years, this project will transform the DoS framework to address multiple educational priorities for formal middle school settings by: (1) creating a new domain that aligns to science and engineering practices, disciplinary core ideas, and cross-cutting concepts from the NGSS/Framework for K-12; (2) identifying observable and measurable diversity, equity, inclusion, and access practices (and related indicators, including social-emotional learning and development) in the context of science and engineering in classrooms, and further revising rubric criteria. The project will build a validity argument for DoS-MSSE using established methods for observation protocols, and analyze the psychometric properties of all dimensions and domains using item-response theory (IRT) methods (e.g., Many-Faceted Rasch Measurement, to model the data which will allow for estimation of teachers’ effectiveness, rater severity, and the difficulty of each dimension). The rubrics will be informed by content experts and pilot observations; scores will be analyzed across dimensions, observers, disciplinary domains, and time; and scores will be compared to other data, including student and teacher self-report and observation scores collected with other established measures. The project will culminate in a larger-scale validation study, collecting observational and survey data from three geographically different school districts (N=30 schools). This effort will characterize the strength of evidence for indicators of classroom quality and provide data to identify support needs across districts, schools, and teachers. Given the ongoing need for remote learning, and the general need to increase school capacity for non-punitive assessment, the tool will be expanded to support both live and recorded modes of data collection, establishing a video scoring method. A training and certification system will be field-tested in the fourth year of the project to assure accessibility, scalability, and sustainability.

Empowering Teachers to See and Support Student Use of Crosscutting Concepts in the Life Sciences

The project focuses on the development of formative assessment tools that highlight assets of students’ use of crosscutting concepts (CCCs) while engaged in science and engineering practices in grades 9-12 Life Sciences.

Lead Organization(s): 
Award Number: 
2100822
Funding Period: 
Sun, 08/01/2021 to Wed, 07/31/2024
Full Description: 

The project focuses on the development of formative assessment tools that highlight assets of students’ use of crosscutting concepts (CCCs) while engaged in science and engineering practices in grades 9-12 Life Sciences. In response to the calls set forth by the Framework for K-12 Science Education and Next Generation Science Standards (NGSS), the field has most successfully researched and developed assessment tools for disciplinary core ideas and the science and engineering practices. The CCCs, which serve as the connective links across science domains, however, remain more abstractly addressed. Presently, science educators have little guidance for what student use of CCCs looks like or how to assess and nurture such use. This project, with its explicit attention to the CCCs, advances true three-dimensional scientific understanding in both research and the classroom. Leveraging formative assessment as a vehicle for student and teacher development taps into proven successful instructional strategies (e.g., sharing visions of successful learning, descriptive feedback, self-assessment), while also advancing formative assessment, itself, by strengthening and illustrating how these strategies may focus on the CCCs. Further, a strengths-based approach will center culturally related differences in students’ use of CCCs to achieve more equitable opportunities to engage in classroom sensemaking practices. This work impacts the field of science education by 1) enabling a more thorough realization of NGSS ideals, 2) strengthening teachers’ abilities to identify diverse demonstrations of CCCs, and 3) showcasing the impact of novel classroom tools to sharpen teachers’ abilities to solicit, notice, and act upon evidence of emergent student scientific thinking within their instructional practices.

This design-based implementation research project will engage teachers in the iterative development and refinement of rubrics that support three-dimensional science understanding through formative assessment. The high school biology classrooms that compose the study site are engaged in ambitious science teaching-inspired instruction. An inductive, bottom-up approach (Brookhart, 2013) will allow researchers, teachers, and students to co-construct rubrics. Analysis of classroom observations, artifact collection, interviews with teachers and students, and expert-panel ratings will produce a rubric for each CCC that integrates relevant science and engineering practices and is applicable across a range of disciplinary core ideas. These rubrics will illustrate progressions of increasingly advanced use of each of the CCCs, to guide the construction, pursuit, and assessment of learning goals. There will be two design cycles that allow for the collection of validity evidence and produce rubrics with the potential for broad application by educators. Complementary lines of qualitative and quantitative (i.e., psychometric) analysis will contribute to development and validation of the rubrics and their formative uses. Project inquiry will focus on 1) how the rubrics can represent CCCs for key disciplinary practices, 2) the extent to which teachers’ and students’ understandings of the rubrics align, and 3) how implementation of the rubrics impacts teachers’ and students’ understandings of the CCCs.

Supporting Instructional Decision Making: The Potential of Automatically Scored Three-Dimensional Assessment System (Collaborative Research: Krajcik)

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems.

Lead Organization(s): 
Award Number: 
2100964
Funding Period: 
Wed, 09/01/2021 to Sun, 08/31/2025
Full Description: 
This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems. Led by collaborators from University of Georgia, Michigan State University, University of Illinois at Chicago, and WestEd, the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments.
 
The project will achieve the research goals using a mixed-methods design in three phases. Phase I: Develop AutoRs. Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so that teachers can use them with more efficiency and productivity. Phase II: Develop and test PCKSs. Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning with teachers on how to use the AutoRs and PCKSs. The project will research how teachers use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs. Phase III: Classroom implementation. In this phase a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, and the corresponding professional development programs, and publications to facilitate 3D instruction and learning.

Supporting Instructional Decision Making: The Potential of Automatically Scored Three-Dimensional Assessment System (Collaborative Research: Zhai)

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems.

Lead Organization(s): 
Award Number: 
2101104
Funding Period: 
Wed, 09/01/2021 to Sun, 08/31/2025
Full Description: 
This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems. Led by collaborators from University of Georgia, Michigan State University, University of Illinois at Chicago, and WestEd, the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments.
 
The project will achieve the research goals using a mixed-methods design in three phases. Phase I: Develop AutoRs. Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so that teachers can use them with more efficiency and productivity. Phase II: Develop and test PCKSs. Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning with teachers on how to use the AutoRs and PCKSs. The project will research how teachers use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs. Phase III: Classroom implementation. In this phase a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, and the corresponding professional development programs, and publications to facilitate 3D instruction and learning.

Supporting Instructional Decision Making: The Potential of Automatically Scored Three-Dimensional Assessment System (Collaborative Research: Weiser)

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems.

Lead Organization(s): 
Award Number: 
2101112
Funding Period: 
Wed, 09/01/2021 to Sun, 08/31/2025
Full Description: 
This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems. Led by collaborators from University of Georgia, Michigan State University, University of Illinois at Chicago, and WestEd, the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments.
 
The project will achieve the research goals using a mixed-methods design in three phases. Phase I: Develop AutoRs. Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so that teachers can use them with more efficiency and productivity. Phase II: Develop and test PCKSs. Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning with teachers on how to use the AutoRs and PCKSs. The project will research how teachers use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs. Phase III: Classroom implementation. In this phase a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, and the corresponding professional development programs, and publications to facilitate 3D instruction and learning.

Pages

Subscribe to Assessment