Assessment

CodeR4STATS - Code R for AP Statistics

This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts.

Lead Organization(s): 
Award Number: 
1418163
Funding Period: 
Mon, 09/01/2014 to Sat, 08/31/2019
Full Description: 

Increasingly, all STEM fields rely on being able to understand data and use statistics. This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts. Thus, this project attempts to help students learn coding, statistics, and STEM simultaneously in the context of AP Stats. In addition, researchers will examine the extent to which students learn statistical concepts, computational fluency, and critical reasoning skills better with the online tools.

The resources developed by the project aim to enhance statistics learning through an integrated application of strategies previously documented to be effective: a focus on data visualization and representation, engaging students in meaningful investigations with complex real-world data sets, utilizing computational tools and techniques to analyze data, and better preparing educators for the needs of a more complex and technologically-rich mathematical landscape. This project will unite these lines of work into one streamlined pedagogical environment called CodeR4STATS with three kinds of resources: computing resources, datasets, and assessment resources. Computing resources will include freely available access to an instance of the cloud-based R-studio with custom help pages. Data resources will include over 800 scientific datasets from Woods Hole Oceanographic Institute, Harvard University's Institute for Quantitative Social Science, Hubbard Brook Experimental Forest, Boston University, and Tufts University with several highlighted in case studies for students; these will be searchable within the online environment. Assessment and tutoring resources will be provided using the tutoring platform ASSISTments which uses example tracing to provide assessment, feedback, and tailored instruction. Teacher training and a teacher online discussion board will also be provided. Bringing these resources together will be programming lab activities, five real-world case studies, and sixteen statistics assignments linked to common core math standards. Researchers will use classroom observational case studies from three classrooms over two years, including cross-case comparison of lessons in the computational environment versus offline lessons; student and teacher interviews; and an analysis of learner data from the online system, especially the ASSISTments-based assessment data. This research will examine learning outcomes and help refine design principles for statistics learning environments.

Centers for Learning and Teaching: Research to Identify Changes in Mathematics Education Doctoral Preparation and the Production of New Doctorates

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs.

Lead Organization(s): 
Award Number: 
1434442
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

The quality of the mathematical education provided to teachers and ultimately to their students depends on the quality of teacher educators at the colleges and universities. For several decades, there has been a shortage of well-prepared mathematics teacher educators. Doctoral programs in mathematics education are the primary ways that these teacher educators learn the content and methods that they need to prepare teachers, but the quality of these programs varies and the number of qualified graduates has been insufficient to meet the demand.

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs. It will also gather data on the number of doctorates in mathematics education from the CLT institutions prior to the establishment of the CLT and after their CLT ended. A comparison group of Doctoral granting institutions will be studied over the same time frame to determine the number of doctoral students graduated during similar time frames as the CLTs. Follow-up data from graduates of the CLTs will be gathered to identify programmatic strengths and weaknesses as graduates will be asked to reflect on how their doctoral preparation aligned with their current career path. The research questions are: What were the effects of CLTs on the production of new doctorates in mathematics education? What changes were made to doctoral programs in mathematics education by the CLT institutions? How well prepared were the CLT graduates for various career paths?

Access, Agency, and Allies in Mathematical Systems (A3IMS)

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important math practices, followed by ongoing interactions for the math teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1417672
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

Given the role that mathematics plays as a gateway into STEM disciplines, addressing achievement gaps for underrepresented students is essential. By focusing on both equity and access, the project, funded in the Discovery Research K-12 program, seeks to improve students' opportunities to learn by focusing on helping middle grades teachers learn about and enact equitable mathematics instruction and encourage students to have agency in their own mathematics learning. The middle grades are a particularly important focus of the project, as this is a time in students' education when achievement gaps grow. The practice-based model of professional development focuses on creating systemic reforms through collaborative communities in which mathematics teacher educators, mathematics teachers, and students work together both to support the fair distribution of opportunities to learn and to empower each in their roles in supporting mathematics learning of all students. The project has promise for supporting students, teachers, and teacher educators who work with middle grades mathematics teachers with the potential to address mathematics achievement gaps of students via a focus on equitable mathematics teaching and learning.

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important mathematical practices like argumentation and justification, followed by ongoing interactions for the mathematics teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms. Field testing of the practice-based professional development in one urban district which will include research conducted on the nature of students', teachers', and teacher educators' opportunities to learn with respect to three features of an equitable mathematical system and from the perspective of three components of the system. The project studies the coherence and alignment of these components from the perspective of classroom mathematics teachers. The research addresses essential questions related to how to provide equitable opportunity to learn for students, teachers, and teacher educators. In particular it will generate models of PD, tools for assessing equity in mathematics teaching and learning, and a theory of equitable mathematics education systems that advances our understanding of the ways in which approaches to teaching, learning, and studying mathematics support equitable opportunities to learn.

 

A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in STEM Education will be established to address a set of issues posed as critical to the future of assessment for education and will undertake a series of activities culminating in the production of a report.

Lead Organization(s): 
Award Number: 
1433181
Funding Period: 
Mon, 09/01/2014 to Mon, 02/29/2016
Full Description: 

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education will be established to address a set of issues posed as critical to the future of assessment for education. Building on the work of the Gordon Commission on the Future of Assessment in Education, the Study Group will undertake a series of activities culminating in the production of a report to the field including conceptual analyses, knowledge syntheses findings, and recommended specifications for a program of research and development to advance STEM education through assessment for education.

The Study Group will consist of fifteen research scientists and scholars of pedagogical practice who are being organized to inquire into the potential of the measurement sciences to more effectively inform the teaching and learning of STEM subject matter for students whose development is challenged by the demands of diversity, equity and excellence in intellective competence. Group members are especially interested in balancing the capabilities of the measurement sciences in the assessment of developed abilities, with equal strength in the analysis, documentation and understanding of the learning and teaching processes by which intellective competence is developed. The Group will conduct a conceptual inquiry into the interrelatedness of diversity, equity, and achievement in STEM. Special attention will be given to variance in attributions, contexts and perspectives associated with differences in life conditions, cultural experience and cultural identity. The Study Group will utilize in-person and virtual deliberations; consultative conversations; and commissioned position and review papers to generate recommendations and suggested specifications for programs of research, development and praxis designed to better inform the teaching and, especially, the learning of STEM disciplines.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Krajcik)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1316908
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Pellegrino)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Partner Organization(s): 
Award Number: 
1316874
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

Publications & Presentations 

Alozie, N., Fujii, R., Leones, T., Cheng, B., Pennock, P. H. & Damery, K. (2017, May). An equity framework for the design and development of NGSS aligned formative assessment tasks. Paper in N. Alozie (Chair), Using NGSS to Inform and Provide Equitable Instruction, Learning, and Assessments to Diverse Students. Structured poster session at the annual meeting of the American Educational Research Association, San Antonio, TX.

Alozie, N., Madden, K., Zaidi, S., Haugabook Pennock, P., & Harris, C. J. (2018, April) Challenges in designing instructionally supportive science assessments using culturally relevant principles for diverse students.  Paper presented at the American Educational Research Association Annual Meeting, New York, NY.

Alozie, N., Pennock, P.H., Madden, K., Zaidi, S., Harris, C. J., & Krajcik, J. (2018, March). Designing and developing NGSS-aligned formative assessment tasks to promote equity. Paper presented at the annual conference of National Association for Research in Science Teaching, Atlanta, GA.

Dahsah, C., Lee, J., DeBarger, A., Damelin, D., & Krajcik, J. (2015, April). Involving teachers in developing assessments aligned with NGSS using a 7-step process. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Chicago, IL.

Damelin, D. (2014). Developing Assessments for the NGSS. @Concord, 18(2), 12-13.

Damelin, D. (2017). Using Technology to Enhance NGSS-Aligned Assessment Tasks for Classroom Formative Use. @Concord, 21(1), 8-9.

DeBarger, A. H., Harris, C. J., D’Angelo, C., Krajcik, J., Dahsah, C., Lee, J., & Beauvineau, Y. (2014). Constructing assessment items that blend core ideas and science practices. In J. L. Polman, E. A. Kyza, D. K. O'Neill, I. Tabak, W. R. Penuel, A. S. Jurow, K. O’Connor, T. Lee, & L. D’Amico (Eds.), Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) 2014, Vol. 3. Boulder, CO: International Society of the Learning Sciences.

Gane, B. D., McElhaney, K. W., Zaidi, S. Z., & Pellegrino, J. W. (2018, March). Analysis of student and item performance on three-dimensional constructed response assessment tasks. Paper presented at the 2018 NARST Annual International Conference, Atlanta, GA.

Gane, B. D., Zaidi, S. Z., & Pellegrino, J. W. (2018). Measuring what matters: Using technology to assess multidimensional learning. European Journal of Education, 53, 176–187. https://doi.org/10.1111/ejed.12269

Harris, C. J., Krajcik, J., Pellegrino, J. & DeBarger, A. H. (in press, 2019).  Designing knowledge-in-use assessments to promote deeper learning. Educational Measurement: Issues and Practice.

Harris, C. J., Krajcik, J. S., Pellegrino, J. W., & McElhaney, K. W. (2016). Constructing assessment tasks that blend disciplinary core ideas, crosscutting concepts, and science practices for classroom formative applications. Menlo Park, CA: SRI International.

Harris, C. J., Krajcik, J. S., Pellegrino, J. W., McElhaney, K. M., Pennock, P., H., & Gane, B. (2018, March). Designing classroom-based assessments for supporting three-dimensional teaching and learning. Paper presented at the NARST 2018 Annual International Conference, Atlanta, GA.

McElhaney, K., D’Angelo, C., Harris, C. J., Seeratan, K., Stanford, T., & DeBarger, A. (2015, April). Integrating crosscutting concepts into 3-dimensional scoring rubrics. Poster and paper presented at the annual meeting of the National Association for Research in Science Teaching, Chicago, IL.

McElhaney, K.W., Gane, B. D., DiBello, L.V., Fujii, R., Pennock, P.H, Vaishampayan, G., & Pellegrino, J.W. (2017, May). Designing scoring rubrics to support NGSS-aligned, classroom-based formative assessment. Paper and poster presented at the American Educational Research Association Annual Meeting, San Antonio, TX.

McElhaney, K., Gane, B. D., Harris, C. J., Pellegrino, J. W., DiBello, L. V., & Krajcik, J. S. (2016, April). Using learning performances to design three-dimensional assessments of science proficiency. Paper presented at the annual conference of National Association for Research in Science Teaching, Baltimore, MD.

McElhaney, K.W., Zaidi, S., Gane, B. D., Alozie, N., & Harris, C.J. (2018, March). Designing NGSS-aligned assessment tasks and rubrics to support classroom-based formative assessment. Paper presented at the NARST Annual International Conference, Atlanta, GA.

McElhaney, K., Vaishampayan, G., D’Angelo, C., Harris, C. J., Pellegrino, J. W., & Krajcik, J. (2016, June). Using learning performances to design science assessments that measure knowledge-in-use. In C. K. Looi, J. L. Polman, U. Cress, & P. Reiman (Eds.). Transforming learning, empowering learners: Proceedings of the 12th international conference of the learning sciences (ICLS) 2016, Vol. 2 (pp. 1211–1212).Singapore: International Society of the Learning Sciences.

Pellegrino, J. W. (2015, August). Measuring what matters: Challenges and opportunities in assessing science proficiency. In Proceedings of the Learning Assessments Research Conference (p. 54–58). Melbourne, Australia: Australian Council for Educational Research.

Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for defining and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51(1), 59-81.

Pellegrino, J. W., Gane, B. D., Zaidi, S. Z., Harris, C. J., McElhaney, K. W., Alozie, N., Haugabook Pennock, P., Severance, S., Neumann, K., Fortus, D., Krajcik, J., Nordine, J., Furtak, E. M., Briggs, D., Chattergoon, R, Penuel, B., Wingert, K. Van Horne, K. (2018). The challenge of assessing “knowledge in use”: Examples from three-dimensional science learning and instruction. In Kay, J. and Luckin, R. (Eds.). Rethinking Learning in the Digital Age: Making the Learning Sciences Count, Proceedings of the 13th International Conference of the Learning Sciences (ICLS) 2018, 2, 1211-1218. London, UK: International Society of the Learning Sciences.

Pellegrino, J. W., Harris, C. J., Krajcik, J., Gane, B. D., McElhaney, K. W., Pennock, P.H., Alozie, N., & Zaidi, S. Z. (2018). Design of next generation science assessments: Measuring what matters. In Kay, J. and Luckin, R. (Eds.). Rethinking Learning in the Digital Age: Making the Learning Sciences Count, Proceedings of the 13th International Conference of the Learning Sciences (ICLS) 2018, 2, 1212-1213. London, UK: International Society of the Learning Sciences.

Pellegrino, J. W., Krajcik, J., Harris, C., & Damelin, D. (2016). Constructing Science Assessment Tasks that Integrate Disciplinary Core Ideas, Science Practices and Crosscutting Concepts. Paper presented at the EARLI SIG on Assessment and Evaluation Conference. Munich, Germany.

Pennock, P. H., Alozie, N., & Morales, C. (March, 2018). Assessing 3-D learning with instructionally supportive tasks and rubrics. Paper presented at the 2018 Annual National Conference of the National Science Teachers Association (NSTA), Atlanta, GA.

Pennock, P. H., & Severance, S. (March, 2018). Comparative analysis of three-dimensional research-based and classroom based rubrics for formative assessment. Paper presented at the NARST Annual International Conference, Atlanta, GA.

Zaidi, S.Z., Ko, M., Gane, B.D., Madden, K., Gaur, D., & Pellegrino. J.W. (2018, March). Portraits of teachers using three-dimensional assessment tasks to inform instruction. Paper presented at the NARST Annual International Conference, Atlanta, GA.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Harris)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Lead Organization(s): 
Award Number: 
1903103
Funding Period: 
Sun, 09/01/2013 to Sun, 06/30/2019
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

This project was previously funded under award #1316903.

Assessing Student Engagement in Math and Science in Middle School: Classroom, Family, and Peer Effects on Engagement

The project will use a comprehensive mixed methods design to develop theoretically-grounded measures of student engagement in middle school math and science classes that reflect a multidimensional construct within an ethnically and socioeconomically diverse sample of urban youth. The project conceptualizes student engagement as a multidimensional construct including behavioral, emotional, and cognitive components. This multidimensional perspective of student engagement provides a rich characterization of how students act, feel, and think.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1315943
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

The proposed project uses a comprehensive mixed methods design to develop theoretically-grounded measures of student engagement in middle school math and science classes that reflect a multidimensional construct within an ethnically and socioeconomically diverse sample of urban youth. The project conceptualizes student engagement as a multidimensional construct including behavioral, emotional, and cognitive components. This multidimensional perspective of student engagement provides a rich characterization of how students act, feel, and think. The project has three aims which are to 1) develop reliable and valid measures of student engagement in middle school math and science classes for the use of teachers and researchers; 2) field test and validate these measures of student engagement in math and science classes; and 3) test a) whether classroom, peer, and family characteristics predict student engagement in math and science classes, which in turn, predicts their course enrollment patterns, academic achievement, and educational and career aspirations in math and science and b) whether these associations differ by gender, race, and socioeconomic status.

To meet these goals, the proposed project includes two studies. In study 1, twenty-five middle school students and 10 math and 10 science teachers participate in focus groups and individual interviews to inform the development of survey instruments in fulfillment of Aim 1. In study 2, 450 middle school students and their teachers and parents participate in a field study to test the psychometric quality of the newly developed instruments in fulfillment of Aims 2 and 3. The sample is recruited from four middle schools located in a socioeconomically and ethnically diverse community. Data to be collected includes information on math and science course enrollment, performance, educational and career aspirations, student engagement in math and science, and support from teachers, peers, and parents.

This project develops easily-administered and psychometrically sound instruments for teachers and researchers to assess student engagement in math and science classes, so they can identify groups of students who are at risk for disengagement and potentially turning away from STEM careers as a first step towards designing appropriate school interventions. It is anticipated that the project findings provide research-based solutions to some of the specific behaviors that influence youth motivation in math and science. Specifically, the project identifies family, peer, and classroom predictors and educational outcomes of student engagement in math and science classes that are amenable to future interventions, as well as examines differences in the relations between context, engagement, and educational outcomes by gender, race, and socioeconomic status. The study also yields information that can directly and immediately support teachers in the partner school districts to enhance the quality of math and science education. As findings are disseminated to math and science teachers, they are able to develop effective strategies to promote student engagement in math and science. This multidimensional approach will advance current scholarship and practice concerning middle school students' pursuit of math and science related fields.

Using Research-Based Formative Assessment to Improve Mathematics Teaching and Learning

This project provides professional development and support for teachers of mathematics in Grades 3-5 and assesses the impacts of the project through a rigorous cluster randomized control trial. The project supports teachers to provide instruction that helps all students reach ambitious academic goals in mathematics.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1316527
Funding Period: 
Wed, 01/01/2014 to Mon, 07/31/2017
Full Description: 

Using Research-Based Formative Assessment to Improve Mathematics Teaching and Learning builds on almost a decade of research and development by the Vermont Mathematics Partnership's Ongoing Assessment Project (OGAP). The project provides professional development and support for teachers of mathematics in Grades 3-5 and assesses the impacts of the project through a rigorous cluster randomized control trial. The project supports teachers to provide instruction that helps all students reach ambitious academic goals in mathematics by: 1) increasing teachers' knowledge of mathematics and of how students learn specific mathematics content, and 2) providing teachers with specific tools and routines for enacting formative assessment and adapting their instruction. The project has three integrated components: 1) professional development and ongoing support in 30 New York City public schools, 2) research on teachers' use of assessment evidence in instruction, and 3) research on student and teacher outcomes.

Helping students deeply understand mathematical concepts requires teachers to become skilled in formative assessment, particularly in the ongoing analysis of evidence in student work when making instructional decisions: moving beyond right and wrong answers into the more important questions of how students think and reason mathematically, where their misconceptions lie, and how they can be addressed instructionally. Yet research shows that teachers struggle with the analytic aspects of formative assessment, and little is known about how teachers use evidence from student work or thinking to improve their instruction. The project addresses both of these concerns by: 1) implementing a rigorous, research-driven approach to formative assessment in 30 schools; and, 2) studying the effects of the intervention in ways that clearly measure impact on teachers and students, including the link between how teachers interpret student work and how they respond instructionally. The creativity and originality of the project lie in its synthesis of a vast body of knowledge about mathematics teaching and learning into a clearly packaged and presented set of tools, routines, and strategies which are directly usable in practice and can dramatically improve the quality of mathematics instruction. The project is organized around the central goal of improving teachers' formative assessment practice, with the research design providing rigorous evidence of project impacts while simultaneously informing the field.

The project will be implemented in a highly diverse school district serving a large number of students from groups traditionally underrepresented in mathematics and the sciences. The formative assessment system developed through this project will ultimately be made available, through a website and multi-media booklets, to all teachers in New York City public schools and across the country. The OGAP formative assessment system will be tied to college and career readiness standards in mathematics rather than a particular curriculum-although it addresses the same content as the major mathematics curricula-which means the materials, knowledge, and strategies will be usable across settings.

The Validity of Technology-Enhanced Assessment in Geometry

This project contributes to the small research base by exploring the validity of Technology-Enhanced Items (TEIs) in the context of elementary geometry. The project addresses three research questions: 1) To what extent are TEIs a valid measurement of geometry standards in the elementary grades?; 2) To what extent do TEIs provide an improved measurement compared to SR items?; and 3) What are the general characteristics of mathematics standards that might be better measured through TEIs?

Lead Organization(s): 
Award Number: 
1316557
Funding Period: 
Thu, 08/01/2013 to Thu, 12/31/2015
Full Description: 

Assessment developers, state departments of education, and national consortia have focused extensive efforts on including Technology-Enhanced Items (TEIs) on summative and formative assessments. TEIs have a number of potential benefits over traditional, selected-response (SR) items, including the potential to measure higher-level constructs, the reduction of the effects of test-taking skills and guessing, the capture of rich diagnostic information, the reduction of cognitive load from non-relevant constructs, and the engaging nature of their design. The first three benefits are true of constructed-response (CR) items, but TEIs have the added benefit of being automatically scored by computer. Despite the potential benefits of TEIs, and the strong push to include these types of items in assessments, there is a death of research on the validity of inferences made by TEIs and on whether TEIs provide improved measurement over traditional item types. The Validity of Technology-Enhanced Assessment in Geometry (VTAG) project contributes to the small research base by exploring the validity of TEIs in the context of elementary geometry.

The project addresses three research questions:

RQ1: To what extent are TEIs a valid measurement of geometry standards in the elementary grades?

RQ2: To what extent do TEIs provide an improved measurement compared to SR items?

RQ3: What are the general characteristics of mathematics standards that might be better measured through TEIs?

To address these research questions, the researchers develop 20 items (ten SR items and 10 TEIs) for each of the seven Common Core State Standards in fourth and fifth grade geometry. The researchers collect validity evidence using expert review, cognitive labs, and classroom administration of the items. The first two research questions are addressed by evaluating the validity of the items based on a variety of sources, including test content, internal structure, the relationship to other variables, and student response processes. To address the third research question, informed by the results of the prior two, the researchers use qualitative analysis to identify common themes of the standards that were identified as being better measured through TEIs.

Pages

Subscribe to Assessment