Four DR K-12 projects will discuss opportunities and challenges that they have encountered when trying to harness psychometric models for diagnosis in science and mathematics education.
What are the opportunities and challenges that arise when trying to harness psychometric models for diagnosis in STEM education?
A growing number of projects, many funded by the National Science Foundation, have been examining how psychometric models may be harnessed to pursue STEM education research. Several of these projects are addressing the need for STEM assessments that go beyond large-scale standardized testing practice to provide beneficial information directly to professional developers, teachers, and students. These projects seek to open up new classroom assessment capabilities, new use models for classroom assessments, and new directions for STEM education research that in recent decades has relied largely on case studies and other qualitative methods to study the teaching and learning of STEM content.
The proposed panel will bring together four currently funded DR K–12 projects that are examining how diagnostic modeling may be applied to further research in STEM education. Interest in diagnostic testing reflects recent developments in psychometric theory and modeling that are creating new possibilities for assessing STEM concepts and facets of understanding that go beyond summary or composite measures of teachers’ knowledge or students’ achievement. In particular, psychometric theory has moved from models based on test scores (e.g., classical test theory) to models based on answers to individual test items (e.g., item response theory) to models based on skills or components of reasoning required to answer particular test items (e.g., diagnostic classification). The four projects demonstrate the diversity of work in applications of diagnostic testing to STEM education research. Some of the projects are developing and implementing diagnostic tests for use in STEM classrooms or professional development of STEM teachers. Other projects are conducting retrospective diagnostic modeling of existing classroom assessments. Some projects are examining science content, others mathematics content; some are assessing teachers, others students; some are focusing on paper-and-pencil instruments, others simulation-based assessments.
The proposed session will highlight the opportunities and challenges that arise when harnessing psychometric models for diagnosis of STEM content. A 10-minute introduction to diagnosis from a psychometric perspective will be followed by 15-minute presentations from each of the four projects. Each presentation will provide background for the given project and identify opportunities and challenges in applying diagnostic psychometric models. Example challenges include fostering interdisciplinary collaboration among content experts, cognitive scientists, and psychometricians; coordinating psychometric models with perspectives on learning and knowledge within STEM disciplines; understanding and balancing cognitive and conceptual structures of assessments with psychometric models; and determining relationships among the number of questions needed on a test form, the number of concepts that can be diagnosed on a test form, and the number of examinees required to satisfactorily estimate diagnostic models. The presentations will be followed by 15 minutes for questions to the panel. During the final portion of the session, the panelists will be available for individual discussion with attendees at project posters. The session format will afford opportunities for attendees to consider the range of issues across projects, to learn more about particular projects, and to network with one another.
Panel presentations will allow for a broader discussion of the opportunities and challenges that the four projects have encountered when harnessing psychometric models for diagnosis in science and mathematics education. Poster presentations will allow attendees to learn more details about the projects in which they are particularly interested and afford networking.