Assessment

DIMEs: Immersing Teachers and Students in Virtual Engineering Internships

This project will provide curricular and pedagogical support by developing and evaluating teacher-ready curricular Digital Internship Modules for Engineering (DIMEs). DIMES will be designed to support middle school science teachers in providing students with experiences that require students to use engineering design practices and science understanding to solve a real-world problem, thereby promoting a robust understanding of science and engineering, and motivating students to increased interest in science and engineering.

Award Number: 
1417939
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

The Next Generation Science Standards (NGSS) outline the science competencies students should demonstrate through their K-12 years and represent a commitment to integrate engineering design into the structure of science education. However, achieving this new ideal of teaching and learning will require new curricular and pedagogical supports for teachers as well as new and time-efficient assessment methods. This project will provide such curricular and pedagogical support by developing and evaluating teacher-ready curricular Digital Internship Modules for Engineering (DIMEs). DIMES will be designed to support middle school science teachers in providing students with experiences that require students to use engineering design practices and science understanding to solve a real-world problem, thereby promoting a robust understanding of science and engineering, and motivating students to increased interest in science and engineering. The modules will also assess students' ability to apply their science knowledge in solving the engineering problem, thereby providing teachers with actionable data about the depth of their students' science and engineering understanding. The DIMEs will be environments where students work as interns at a simulated engineering firm. 

The Digital Internship Modules for Engineering will provide immersive experiences that simultaneously serve as learning and assessment opportunities. DIMEs will assess not only whether students understand NGSS science and engineering concepts, but also whether they can use them in the context of real-world problem solving. Teachers will orchestrate DIMEs using a custom-designed teacher interface that will show student work, auto-generated assessments, and reports on each student's learning progress. This project will build on prior work on NSF-funded computer-based STEM learning environments called epistemic games. Epistemic games are computer role-playing games that have been successfully used in both undergraduate engineering courses and informal settings for K-12 populations to teach students to think like STEM professionals, thereby preparing them to solve 21st century problems. The project will create six ten-day activities, two each in Physical Science, Life Science and Earth Science units that are typically taught in middle school. An iterative research and design process is used to conduct pilot tests of the six DIMEs in local classrooms, field test a beta version of each DIME with 15 teachers and up to 1500 students in national classrooms, and then implement final versions of each DIME in research trials with 30 teachers and up to 3000 students in national classrooms. By bringing cutting-edge developments in learning science and undergraduate engineering education to middle school STEM education, the project aims to improve educational practice, and enhance assessment of learning outcomes in middle school science classroom settings.

Computer Science in Secondary Schools (CS3): Studying Context, Enactment, and Impact

This project will examine the relationships among the factors that influence the implementation of the Exploring Computer Science (ECS), a pre-Advanced Placement curriculum that prepares students for further study in computer science. This study elucidates how variation in curricular implementation influences student learning and determines not only what works, but also for whom and under what circumstances.

Lead Organization(s): 
Award Number: 
1418149
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

Computational thinking is an important set of 21st century knowledge and skills that has implications for the heavily technological world in which we live. Multiple industries indicate the under supply of those trained to be effective in the computer science workforce. In addition, there are increasing demands for broadening the participation in the computer science workforce by women and members of minority populations. SRI International will examine the relationships among the factors that influence the implementation of the Exploring Computer Science (ECS), a pre-Advanced Placement curriculum that prepares students for further study in computer science. SRI will work in partnership with the ECS curriculum developers, teachers, and the nonprofit Code.org who are involved in the scaling of ECS. This study elucidates how variation in curricular implementation influences student learning and determines not only what works, but also for whom and under what circumstances.

SRI will conduct a pilot study in which they develop, pilot, and refine measures as they recruit school districts for the implementation study. The subsequent implementation study will be a 2 year examination of curriculum enactment, teacher practice, and evidence of student learning. Because no comparable curriculum currently exists, the study will examine the conditions needed to implement the ECS curriculum in ways that improve student computational thinking outcomes rather than determine whether the ECS curriculum is more effective than other CS-related curricula. The study will conduct two kinds of analyses: 1) an analysis of the influence of ECS on student learning gains, and 2) an analysis of the relationship between classroom-level implementation and student learning gains. Because of the clustered nature of the data (students nested within classrooms nested within schools), the project will use hierarchical linear modeling to examine the influence of the curriculum.

CodeR4STATS - Code R for AP Statistics

This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts.

Lead Organization(s): 
Award Number: 
1418163
Funding Period: 
Mon, 09/01/2014 to Sat, 08/31/2019
Full Description: 

Increasingly, all STEM fields rely on being able to understand data and use statistics. This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts. Thus, this project attempts to help students learn coding, statistics, and STEM simultaneously in the context of AP Stats. In addition, researchers will examine the extent to which students learn statistical concepts, computational fluency, and critical reasoning skills better with the online tools.

The resources developed by the project aim to enhance statistics learning through an integrated application of strategies previously documented to be effective: a focus on data visualization and representation, engaging students in meaningful investigations with complex real-world data sets, utilizing computational tools and techniques to analyze data, and better preparing educators for the needs of a more complex and technologically-rich mathematical landscape. This project will unite these lines of work into one streamlined pedagogical environment called CodeR4STATS with three kinds of resources: computing resources, datasets, and assessment resources. Computing resources will include freely available access to an instance of the cloud-based R-studio with custom help pages. Data resources will include over 800 scientific datasets from Woods Hole Oceanographic Institute, Harvard University's Institute for Quantitative Social Science, Hubbard Brook Experimental Forest, Boston University, and Tufts University with several highlighted in case studies for students; these will be searchable within the online environment. Assessment and tutoring resources will be provided using the tutoring platform ASSISTments which uses example tracing to provide assessment, feedback, and tailored instruction. Teacher training and a teacher online discussion board will also be provided. Bringing these resources together will be programming lab activities, five real-world case studies, and sixteen statistics assignments linked to common core math standards. Researchers will use classroom observational case studies from three classrooms over two years, including cross-case comparison of lessons in the computational environment versus offline lessons; student and teacher interviews; and an analysis of learner data from the online system, especially the ASSISTments-based assessment data. This research will examine learning outcomes and help refine design principles for statistics learning environments.

Centers for Learning and Teaching: Research to Identify Changes in Mathematics Education Doctoral Preparation and the Production of New Doctorates

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs.

Lead Organization(s): 
Award Number: 
1434442
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

The quality of the mathematical education provided to teachers and ultimately to their students depends on the quality of teacher educators at the colleges and universities. For several decades, there has been a shortage of well-prepared mathematics teacher educators. Doctoral programs in mathematics education are the primary ways that these teacher educators learn the content and methods that they need to prepare teachers, but the quality of these programs varies and the number of qualified graduates has been insufficient to meet the demand.

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs. It will also gather data on the number of doctorates in mathematics education from the CLT institutions prior to the establishment of the CLT and after their CLT ended. A comparison group of Doctoral granting institutions will be studied over the same time frame to determine the number of doctoral students graduated during similar time frames as the CLTs. Follow-up data from graduates of the CLTs will be gathered to identify programmatic strengths and weaknesses as graduates will be asked to reflect on how their doctoral preparation aligned with their current career path. The research questions are: What were the effects of CLTs on the production of new doctorates in mathematics education? What changes were made to doctoral programs in mathematics education by the CLT institutions? How well prepared were the CLT graduates for various career paths?

Access, Agency, and Allies in Mathematical Systems (A3IMS)

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important math practices, followed by ongoing interactions for the math teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1417672
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

Given the role that mathematics plays as a gateway into STEM disciplines, addressing achievement gaps for underrepresented students is essential. By focusing on both equity and access, the project, funded in the Discovery Research K-12 program, seeks to improve students' opportunities to learn by focusing on helping middle grades teachers learn about and enact equitable mathematics instruction and encourage students to have agency in their own mathematics learning. The middle grades are a particularly important focus of the project, as this is a time in students' education when achievement gaps grow. The practice-based model of professional development focuses on creating systemic reforms through collaborative communities in which mathematics teacher educators, mathematics teachers, and students work together both to support the fair distribution of opportunities to learn and to empower each in their roles in supporting mathematics learning of all students. The project has promise for supporting students, teachers, and teacher educators who work with middle grades mathematics teachers with the potential to address mathematics achievement gaps of students via a focus on equitable mathematics teaching and learning.

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important mathematical practices like argumentation and justification, followed by ongoing interactions for the mathematics teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms. Field testing of the practice-based professional development in one urban district which will include research conducted on the nature of students', teachers', and teacher educators' opportunities to learn with respect to three features of an equitable mathematical system and from the perspective of three components of the system. The project studies the coherence and alignment of these components from the perspective of classroom mathematics teachers. The research addresses essential questions related to how to provide equitable opportunity to learn for students, teachers, and teacher educators. In particular it will generate models of PD, tools for assessing equity in mathematics teaching and learning, and a theory of equitable mathematics education systems that advances our understanding of the ways in which approaches to teaching, learning, and studying mathematics support equitable opportunities to learn.

 

A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in STEM Education will be established to address a set of issues posed as critical to the future of assessment for education and will undertake a series of activities culminating in the production of a report.

Lead Organization(s): 
Award Number: 
1433181
Funding Period: 
Mon, 09/01/2014 to Mon, 02/29/2016
Full Description: 

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education will be established to address a set of issues posed as critical to the future of assessment for education. Building on the work of the Gordon Commission on the Future of Assessment in Education, the Study Group will undertake a series of activities culminating in the production of a report to the field including conceptual analyses, knowledge syntheses findings, and recommended specifications for a program of research and development to advance STEM education through assessment for education.

The Study Group will consist of fifteen research scientists and scholars of pedagogical practice who are being organized to inquire into the potential of the measurement sciences to more effectively inform the teaching and learning of STEM subject matter for students whose development is challenged by the demands of diversity, equity and excellence in intellective competence. Group members are especially interested in balancing the capabilities of the measurement sciences in the assessment of developed abilities, with equal strength in the analysis, documentation and understanding of the learning and teaching processes by which intellective competence is developed. The Group will conduct a conceptual inquiry into the interrelatedness of diversity, equity, and achievement in STEM. Special attention will be given to variance in attributions, contexts and perspectives associated with differences in life conditions, cultural experience and cultural identity. The Study Group will utilize in-person and virtual deliberations; consultative conversations; and commissioned position and review papers to generate recommendations and suggested specifications for programs of research, development and praxis designed to better inform the teaching and, especially, the learning of STEM disciplines.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Krajcik)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1316908
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Pellegrino)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Partner Organization(s): 
Award Number: 
1316874
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

Publications & Presentations 

Alozie, N., Fujii, R., Leones, T., Cheng, B., Pennock, P. H. & Damery, K. (2017, May). An equity framework for the design and development of NGSS aligned formative assessment tasks. Paper in N. Alozie (Chair), Using NGSS to Inform and Provide Equitable Instruction, Learning, and Assessments to Diverse Students. Structured poster session at the annual meeting of the American Educational Research Association, San Antonio, TX.

Alozie, N., Madden, K., Zaidi, S., Haugabook Pennock, P., & Harris, C. J. (2018, April) Challenges in designing instructionally supportive science assessments using culturally relevant principles for diverse students.  Paper presented at the American Educational Research Association Annual Meeting, New York, NY.

Alozie, N., Pennock, P.H., Madden, K., Zaidi, S., Harris, C. J., & Krajcik, J. (2018, March). Designing and developing NGSS-aligned formative assessment tasks to promote equity. Paper presented at the annual conference of National Association for Research in Science Teaching, Atlanta, GA.

Dahsah, C., Lee, J., DeBarger, A., Damelin, D., & Krajcik, J. (2015, April). Involving teachers in developing assessments aligned with NGSS using a 7-step process. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Chicago, IL.

Damelin, D. (2014). Developing Assessments for the NGSS. @Concord, 18(2), 12-13.

Damelin, D. (2017). Using Technology to Enhance NGSS-Aligned Assessment Tasks for Classroom Formative Use. @Concord, 21(1), 8-9.

DeBarger, A. H., Harris, C. J., D’Angelo, C., Krajcik, J., Dahsah, C., Lee, J., & Beauvineau, Y. (2014). Constructing assessment items that blend core ideas and science practices. In J. L. Polman, E. A. Kyza, D. K. O'Neill, I. Tabak, W. R. Penuel, A. S. Jurow, K. O’Connor, T. Lee, & L. D’Amico (Eds.), Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) 2014, Vol. 3. Boulder, CO: International Society of the Learning Sciences.

Gane, B. D., McElhaney, K. W., Zaidi, S. Z., & Pellegrino, J. W. (2018, March). Analysis of student and item performance on three-dimensional constructed response assessment tasks. Paper presented at the 2018 NARST Annual International Conference, Atlanta, GA.

Gane, B. D., Zaidi, S. Z., & Pellegrino, J. W. (2018). Measuring what matters: Using technology to assess multidimensional learning. European Journal of Education, 53, 176–187. https://doi.org/10.1111/ejed.12269

Harris, C. J., Krajcik, J., Pellegrino, J. & DeBarger, A. H. (in press, 2019).  Designing knowledge-in-use assessments to promote deeper learning. Educational Measurement: Issues and Practice.

Harris, C. J., Krajcik, J. S., Pellegrino, J. W., & McElhaney, K. W. (2016). Constructing assessment tasks that blend disciplinary core ideas, crosscutting concepts, and science practices for classroom formative applications. Menlo Park, CA: SRI International.

Harris, C. J., Krajcik, J. S., Pellegrino, J. W., McElhaney, K. M., Pennock, P., H., & Gane, B. (2018, March). Designing classroom-based assessments for supporting three-dimensional teaching and learning. Paper presented at the NARST 2018 Annual International Conference, Atlanta, GA.

McElhaney, K., D’Angelo, C., Harris, C. J., Seeratan, K., Stanford, T., & DeBarger, A. (2015, April). Integrating crosscutting concepts into 3-dimensional scoring rubrics. Poster and paper presented at the annual meeting of the National Association for Research in Science Teaching, Chicago, IL.

McElhaney, K.W., Gane, B. D., DiBello, L.V., Fujii, R., Pennock, P.H, Vaishampayan, G., & Pellegrino, J.W. (2017, May). Designing scoring rubrics to support NGSS-aligned, classroom-based formative assessment. Paper and poster presented at the American Educational Research Association Annual Meeting, San Antonio, TX.

McElhaney, K., Gane, B. D., Harris, C. J., Pellegrino, J. W., DiBello, L. V., & Krajcik, J. S. (2016, April). Using learning performances to design three-dimensional assessments of science proficiency. Paper presented at the annual conference of National Association for Research in Science Teaching, Baltimore, MD.

McElhaney, K.W., Zaidi, S., Gane, B. D., Alozie, N., & Harris, C.J. (2018, March). Designing NGSS-aligned assessment tasks and rubrics to support classroom-based formative assessment. Paper presented at the NARST Annual International Conference, Atlanta, GA.

McElhaney, K., Vaishampayan, G., D’Angelo, C., Harris, C. J., Pellegrino, J. W., & Krajcik, J. (2016, June). Using learning performances to design science assessments that measure knowledge-in-use. In C. K. Looi, J. L. Polman, U. Cress, & P. Reiman (Eds.). Transforming learning, empowering learners: Proceedings of the 12th international conference of the learning sciences (ICLS) 2016, Vol. 2 (pp. 1211–1212).Singapore: International Society of the Learning Sciences.

Pellegrino, J. W. (2015, August). Measuring what matters: Challenges and opportunities in assessing science proficiency. In Proceedings of the Learning Assessments Research Conference (p. 54–58). Melbourne, Australia: Australian Council for Educational Research.

Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for defining and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51(1), 59-81.

Pellegrino, J. W., Gane, B. D., Zaidi, S. Z., Harris, C. J., McElhaney, K. W., Alozie, N., Haugabook Pennock, P., Severance, S., Neumann, K., Fortus, D., Krajcik, J., Nordine, J., Furtak, E. M., Briggs, D., Chattergoon, R, Penuel, B., Wingert, K. Van Horne, K. (2018). The challenge of assessing “knowledge in use”: Examples from three-dimensional science learning and instruction. In Kay, J. and Luckin, R. (Eds.). Rethinking Learning in the Digital Age: Making the Learning Sciences Count, Proceedings of the 13th International Conference of the Learning Sciences (ICLS) 2018, 2, 1211-1218. London, UK: International Society of the Learning Sciences.

Pellegrino, J. W., Harris, C. J., Krajcik, J., Gane, B. D., McElhaney, K. W., Pennock, P.H., Alozie, N., & Zaidi, S. Z. (2018). Design of next generation science assessments: Measuring what matters. In Kay, J. and Luckin, R. (Eds.). Rethinking Learning in the Digital Age: Making the Learning Sciences Count, Proceedings of the 13th International Conference of the Learning Sciences (ICLS) 2018, 2, 1212-1213. London, UK: International Society of the Learning Sciences.

Pellegrino, J. W., Krajcik, J., Harris, C., & Damelin, D. (2016). Constructing Science Assessment Tasks that Integrate Disciplinary Core Ideas, Science Practices and Crosscutting Concepts. Paper presented at the EARLI SIG on Assessment and Evaluation Conference. Munich, Germany.

Pennock, P. H., Alozie, N., & Morales, C. (March, 2018). Assessing 3-D learning with instructionally supportive tasks and rubrics. Paper presented at the 2018 Annual National Conference of the National Science Teachers Association (NSTA), Atlanta, GA.

Pennock, P. H., & Severance, S. (March, 2018). Comparative analysis of three-dimensional research-based and classroom based rubrics for formative assessment. Paper presented at the NARST Annual International Conference, Atlanta, GA.

Zaidi, S.Z., Ko, M., Gane, B.D., Madden, K., Gaur, D., & Pellegrino. J.W. (2018, March). Portraits of teachers using three-dimensional assessment tasks to inform instruction. Paper presented at the NARST Annual International Conference, Atlanta, GA.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Harris)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Lead Organization(s): 
Award Number: 
1903103
Funding Period: 
Sun, 09/01/2013 to Sun, 06/30/2019
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

This project was previously funded under award #1316903.

Assessing Student Engagement in Math and Science in Middle School: Classroom, Family, and Peer Effects on Engagement

The project will use a comprehensive mixed methods design to develop theoretically-grounded measures of student engagement in middle school math and science classes that reflect a multidimensional construct within an ethnically and socioeconomically diverse sample of urban youth. The project conceptualizes student engagement as a multidimensional construct including behavioral, emotional, and cognitive components. This multidimensional perspective of student engagement provides a rich characterization of how students act, feel, and think.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1315943
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

The proposed project uses a comprehensive mixed methods design to develop theoretically-grounded measures of student engagement in middle school math and science classes that reflect a multidimensional construct within an ethnically and socioeconomically diverse sample of urban youth. The project conceptualizes student engagement as a multidimensional construct including behavioral, emotional, and cognitive components. This multidimensional perspective of student engagement provides a rich characterization of how students act, feel, and think. The project has three aims which are to 1) develop reliable and valid measures of student engagement in middle school math and science classes for the use of teachers and researchers; 2) field test and validate these measures of student engagement in math and science classes; and 3) test a) whether classroom, peer, and family characteristics predict student engagement in math and science classes, which in turn, predicts their course enrollment patterns, academic achievement, and educational and career aspirations in math and science and b) whether these associations differ by gender, race, and socioeconomic status.

To meet these goals, the proposed project includes two studies. In study 1, twenty-five middle school students and 10 math and 10 science teachers participate in focus groups and individual interviews to inform the development of survey instruments in fulfillment of Aim 1. In study 2, 450 middle school students and their teachers and parents participate in a field study to test the psychometric quality of the newly developed instruments in fulfillment of Aims 2 and 3. The sample is recruited from four middle schools located in a socioeconomically and ethnically diverse community. Data to be collected includes information on math and science course enrollment, performance, educational and career aspirations, student engagement in math and science, and support from teachers, peers, and parents.

This project develops easily-administered and psychometrically sound instruments for teachers and researchers to assess student engagement in math and science classes, so they can identify groups of students who are at risk for disengagement and potentially turning away from STEM careers as a first step towards designing appropriate school interventions. It is anticipated that the project findings provide research-based solutions to some of the specific behaviors that influence youth motivation in math and science. Specifically, the project identifies family, peer, and classroom predictors and educational outcomes of student engagement in math and science classes that are amenable to future interventions, as well as examines differences in the relations between context, engagement, and educational outcomes by gender, race, and socioeconomic status. The study also yields information that can directly and immediately support teachers in the partner school districts to enhance the quality of math and science education. As findings are disseminated to math and science teachers, they are able to develop effective strategies to promote student engagement in math and science. This multidimensional approach will advance current scholarship and practice concerning middle school students' pursuit of math and science related fields.

Pages

Subscribe to Assessment