Assessment

Empowering Teachers through VideoReview

This project  will develop a video recording and analysis system called VideoReView (VRV) that allows grade four science teachers to record, tag, and analyze video in their classroom in real time. The investigators will then study and enhance the system in the context of professional learning communities of teachers. 

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1415898
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

This project represents a collaboration between TERC and IntuVision to develop a video recording and analysis system called VideoReView (VRV) that allows grade four science teachers to record, tag, and analyze video in their classroom in real time. The system will contain a number of features---such as a sophisticated system of tagging and the automatic detection of important video segments---designed to speed and assist the teacher in its use. The investigators will then study and enhance the system in the context of professional learning communities of teachers. The system is expected to enable teachers to examine their own teaching, and that of others, in a much more dynamic and specific way and to integrate video into their ongoing structures of professional learning. To date, video analysis of teaching is out of the reach of ordinary teachers. If successful, this research could change the way teachers engage in their own profession and their understanding of, for example, student thinking and argumentation in science---something emphasized in the Next Generation Science Standards---but previously more difficult to do without being able to replay and refine teaching episodes.

The complete VRV System will be tested with 18 Grade four teachers and approximately 400 students from six schools in the Newton Public School System in a Boston suburb. The emphasis of the study will be on the ability of teachers to use the system with little outside assistance, means of enhancing its features and usability, and its integration into professional learning communities. A mixed methods research design will be used that includes surveys and interviews. The study outcomes will be disseminated through publications and conference presentations.

EarSketch: An Authentic, Studio-based STEAM Approach to High School Computing Education

This project will study the influence on positive student achievement and engagement (particularly among populations traditionally under-represented in computer science) of an intervention that integrates a computational music remixing tool -EarSketch- with the Computer Science Principles, a view of computing literacy that is emerging as a new standard for Advanced Placement and other high school computer science courses.

Award Number: 
1417835
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Project Evaluator: 
Mary Moriarity
Full Description: 

This project will study the influence on positive student achievement and engagement (particularly among populations traditionally under-represented in computer science) of an intervention that integrates a computational music remixing tool -EarSketch- with the Computer Science Principles, a view of computing literacy that is emerging as a new standard for Advanced Placement and other high school computer science courses. The project is grounded on the premise that EarSketch, a STEM + Art (STEAM) learning environment, embodies authenticity (i.e., its cultural and industry relevance in both arts and STEM domains), along with a context that facilitates communication and collaboration among students (i.e., through a studio-based learning approach). These elements are critical to achieving successful outcomes across diverse student populations. Using agent-based modeling, the research team will investigate what factors enhance or impede implementation of authentic STEAM tools in different school settings.

The researchers will be engaged in a multi-stage process to develop: a) an implementation-ready, web-based EarSketch learning environment that integrates programming, digital audio workstation, curriculum, audio loop library, and social sharing features, along with studio-based learning functionality to support student presentation, critique, discussion, and collaboration; and b) an online professional learning course for teachers adopting EarSketch in Computer Science Principles courses. Using these resources, the team will conduct a quasi-experimental study of EarSketch in Computer Science Principles high school courses across the state of Georgia; measure student learning and engagement across multiple demographic categories; and determine to what extent an EarSketch-based CS Principles course promotes student achievement and engagement across different student populations. The project will include measures of student performance, creativity, collaboration, and communication in student programming tasks to determine the extent to which studio-based learning in EarSketch promotes success in these important areas. An agent-based modeling framework in multiple school settings will be developed to determine what factors enhance or impede implementation of EarSketch under conditions of routine practice.

Teaching STEM with Robotics: Design, Development, and Testing of a Research-based Professional Development Program for Teachers

Using design-based research, with teachers as design partners, the project will create and refine project-based, hands-on robotics curricula such that science and math content inherent in robotics and related engineering design practices are learned. To provide teachers with effective models to capitalize on robotics for elucidating science and math concepts, a design-based Professional Development program will be built using principles of technological, pedagogical, and content knowledge (TPACK).

Lead Organization(s): 
Award Number: 
1417769
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

Offering meaningful and motivating engineering contexts, such as robotics, within science and math courses constitutes a compelling strategy to address the Next Generation Science Standards and the Common Core State Standards for Math while enhancing science and math learning for all students. Using design-based research, with teachers as design partners, the project will create and refine project-based, hands-on curricula such that science and math content inherent in robotics and related engineering design practices are learned. To provide teachers with effective models to capitalize on robotics for elucidating science and math concepts, a design-based Professional Development program will be built using principles of technological, pedagogical, and content knowledge (TPACK). To ensure that teachers are well prepared, research-based practices and features of effective Professional Development will be adopted. Experts in robotics, engineering, education, curriculum design, and assessment--with experience in K-12 education, training, and outreach--have formed an interdisciplinary team to make robotics central to and sustainable in middle school science and math classrooms.

The research questions addressed in this project are qualitative in nature as appropriate for design research questions. The methodologies include teacher needs assessment, teachers' perceptions of robotics, pre and post testing, classroom observations, and surveys. Examples of the research questions are:

What characteristics of robotics promote effective learning of middle school science and math?

What elements of Professional Development engender teachers' TPACK of robotics and link it with classroom science and math?

What are student prerequisites to effectively use robotics in science and math learning?

What are the gains in students' STEM engagement, interest, persistence, and career awareness?

The robotics curriculum will include physical science used in robot performance expectations and motion stability. Additionally the curriculum will include the engineering design process consisting of problem definition, solution development, and design improvement. Robotics provides opportunities to support science and engineering practices of the Next Generation Science Standards such as developing and using models, planning and conducting investigations, designing solutions, and analyzing and interpreting data. The project will be aimed at middle school students and will provide substantial teacher professional development to implement the new curriculum modules. The partner schools have student bodies drawn from a diverse student population in New York City.

DIMEs: Immersing Teachers and Students in Virtual Engineering Internships

This project will provide curricular and pedagogical support by developing and evaluating teacher-ready curricular Digital Internship Modules for Engineering (DIMEs). DIMES will be designed to support middle school science teachers in providing students with experiences that require students to use engineering design practices and science understanding to solve a real-world problem, thereby promoting a robust understanding of science and engineering, and motivating students to increased interest in science and engineering.

Award Number: 
1417939
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

The Next Generation Science Standards (NGSS) outline the science competencies students should demonstrate through their K-12 years and represent a commitment to integrate engineering design into the structure of science education. However, achieving this new ideal of teaching and learning will require new curricular and pedagogical supports for teachers as well as new and time-efficient assessment methods. This project will provide such curricular and pedagogical support by developing and evaluating teacher-ready curricular Digital Internship Modules for Engineering (DIMEs). DIMES will be designed to support middle school science teachers in providing students with experiences that require students to use engineering design practices and science understanding to solve a real-world problem, thereby promoting a robust understanding of science and engineering, and motivating students to increased interest in science and engineering. The modules will also assess students' ability to apply their science knowledge in solving the engineering problem, thereby providing teachers with actionable data about the depth of their students' science and engineering understanding. The DIMEs will be environments where students work as interns at a simulated engineering firm. 

The Digital Internship Modules for Engineering will provide immersive experiences that simultaneously serve as learning and assessment opportunities. DIMEs will assess not only whether students understand NGSS science and engineering concepts, but also whether they can use them in the context of real-world problem solving. Teachers will orchestrate DIMEs using a custom-designed teacher interface that will show student work, auto-generated assessments, and reports on each student's learning progress. This project will build on prior work on NSF-funded computer-based STEM learning environments called epistemic games. Epistemic games are computer role-playing games that have been successfully used in both undergraduate engineering courses and informal settings for K-12 populations to teach students to think like STEM professionals, thereby preparing them to solve 21st century problems. The project will create six ten-day activities, two each in Physical Science, Life Science and Earth Science units that are typically taught in middle school. An iterative research and design process is used to conduct pilot tests of the six DIMEs in local classrooms, field test a beta version of each DIME with 15 teachers and up to 1500 students in national classrooms, and then implement final versions of each DIME in research trials with 30 teachers and up to 3000 students in national classrooms. By bringing cutting-edge developments in learning science and undergraduate engineering education to middle school STEM education, the project aims to improve educational practice, and enhance assessment of learning outcomes in middle school science classroom settings.

Computer Science in Secondary Schools (CS3): Studying Context, Enactment, and Impact

This project will examine the relationships among the factors that influence the implementation of the Exploring Computer Science (ECS), a pre-Advanced Placement curriculum that prepares students for further study in computer science. This study elucidates how variation in curricular implementation influences student learning and determines not only what works, but also for whom and under what circumstances.

Lead Organization(s): 
Award Number: 
1418149
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

Computational thinking is an important set of 21st century knowledge and skills that has implications for the heavily technological world in which we live. Multiple industries indicate the under supply of those trained to be effective in the computer science workforce. In addition, there are increasing demands for broadening the participation in the computer science workforce by women and members of minority populations. SRI International will examine the relationships among the factors that influence the implementation of the Exploring Computer Science (ECS), a pre-Advanced Placement curriculum that prepares students for further study in computer science. SRI will work in partnership with the ECS curriculum developers, teachers, and the nonprofit Code.org who are involved in the scaling of ECS. This study elucidates how variation in curricular implementation influences student learning and determines not only what works, but also for whom and under what circumstances.

SRI will conduct a pilot study in which they develop, pilot, and refine measures as they recruit school districts for the implementation study. The subsequent implementation study will be a 2 year examination of curriculum enactment, teacher practice, and evidence of student learning. Because no comparable curriculum currently exists, the study will examine the conditions needed to implement the ECS curriculum in ways that improve student computational thinking outcomes rather than determine whether the ECS curriculum is more effective than other CS-related curricula. The study will conduct two kinds of analyses: 1) an analysis of the influence of ECS on student learning gains, and 2) an analysis of the relationship between classroom-level implementation and student learning gains. Because of the clustered nature of the data (students nested within classrooms nested within schools), the project will use hierarchical linear modeling to examine the influence of the curriculum.

CodeR4STATS - Code R for AP Statistics

This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts.

Lead Organization(s): 
Award Number: 
1418163
Funding Period: 
Mon, 09/01/2014 to Sat, 08/31/2019
Full Description: 

Increasingly, all STEM fields rely on being able to understand data and use statistics. This project builds on prior efforts to create teaching resources for high-school Advanced Placement Statistics teachers to use an open source statistics programming language called "R" in their classrooms. The project brings together datasets from a variety of STEM domains, and will develop exercises and assessments to teach students how to program in R and learn the underlying statistics concepts. Thus, this project attempts to help students learn coding, statistics, and STEM simultaneously in the context of AP Stats. In addition, researchers will examine the extent to which students learn statistical concepts, computational fluency, and critical reasoning skills better with the online tools.

The resources developed by the project aim to enhance statistics learning through an integrated application of strategies previously documented to be effective: a focus on data visualization and representation, engaging students in meaningful investigations with complex real-world data sets, utilizing computational tools and techniques to analyze data, and better preparing educators for the needs of a more complex and technologically-rich mathematical landscape. This project will unite these lines of work into one streamlined pedagogical environment called CodeR4STATS with three kinds of resources: computing resources, datasets, and assessment resources. Computing resources will include freely available access to an instance of the cloud-based R-studio with custom help pages. Data resources will include over 800 scientific datasets from Woods Hole Oceanographic Institute, Harvard University's Institute for Quantitative Social Science, Hubbard Brook Experimental Forest, Boston University, and Tufts University with several highlighted in case studies for students; these will be searchable within the online environment. Assessment and tutoring resources will be provided using the tutoring platform ASSISTments which uses example tracing to provide assessment, feedback, and tailored instruction. Teacher training and a teacher online discussion board will also be provided. Bringing these resources together will be programming lab activities, five real-world case studies, and sixteen statistics assignments linked to common core math standards. Researchers will use classroom observational case studies from three classrooms over two years, including cross-case comparison of lessons in the computational environment versus offline lessons; student and teacher interviews; and an analysis of learner data from the online system, especially the ASSISTments-based assessment data. This research will examine learning outcomes and help refine design principles for statistics learning environments.

Centers for Learning and Teaching: Research to Identify Changes in Mathematics Education Doctoral Preparation and the Production of New Doctorates

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs.

Lead Organization(s): 
Award Number: 
1434442
Funding Period: 
Fri, 08/01/2014 to Tue, 07/31/2018
Full Description: 

The quality of the mathematical education provided to teachers and ultimately to their students depends on the quality of teacher educators at the colleges and universities. For several decades, there has been a shortage of well-prepared mathematics teacher educators. Doctoral programs in mathematics education are the primary ways that these teacher educators learn the content and methods that they need to prepare teachers, but the quality of these programs varies and the number of qualified graduates has been insufficient to meet the demand.

This project will research the programmatic changes that resulted from the NSF investment in Centers for Learning and Teaching of Mathematics (CLT) at the 31 participating institutions. It will provide information on the core elements of doctoral preparation in mathematics education at the institutions and ways in which participation in the CLTs has changed their programs. It will also gather data on the number of doctorates in mathematics education from the CLT institutions prior to the establishment of the CLT and after their CLT ended. A comparison group of Doctoral granting institutions will be studied over the same time frame to determine the number of doctoral students graduated during similar time frames as the CLTs. Follow-up data from graduates of the CLTs will be gathered to identify programmatic strengths and weaknesses as graduates will be asked to reflect on how their doctoral preparation aligned with their current career path. The research questions are: What were the effects of CLTs on the production of new doctorates in mathematics education? What changes were made to doctoral programs in mathematics education by the CLT institutions? How well prepared were the CLT graduates for various career paths?

Access, Agency, and Allies in Mathematical Systems (A3IMS)

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important math practices, followed by ongoing interactions for the math teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms.

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1417672
Funding Period: 
Mon, 09/01/2014 to Fri, 08/31/2018
Full Description: 

Given the role that mathematics plays as a gateway into STEM disciplines, addressing achievement gaps for underrepresented students is essential. By focusing on both equity and access, the project, funded in the Discovery Research K-12 program, seeks to improve students' opportunities to learn by focusing on helping middle grades teachers learn about and enact equitable mathematics instruction and encourage students to have agency in their own mathematics learning. The middle grades are a particularly important focus of the project, as this is a time in students' education when achievement gaps grow. The practice-based model of professional development focuses on creating systemic reforms through collaborative communities in which mathematics teacher educators, mathematics teachers, and students work together both to support the fair distribution of opportunities to learn and to empower each in their roles in supporting mathematics learning of all students. The project has promise for supporting students, teachers, and teacher educators who work with middle grades mathematics teachers with the potential to address mathematics achievement gaps of students via a focus on equitable mathematics teaching and learning.

This project involves designing, facilitating, and studying professional development (PD) to support equitable mathematics education. The PD will involve grades 4-8 mathematics teachers across three sites to support the design of a two-week institute focused on enhancing access and agency in relationship to important mathematical practices like argumentation and justification, followed by ongoing interactions for the mathematics teachers to engage in systematic inquiry of their practice over time to facilitate equitable mathematics teaching and learning in their classrooms. Field testing of the practice-based professional development in one urban district which will include research conducted on the nature of students', teachers', and teacher educators' opportunities to learn with respect to three features of an equitable mathematical system and from the perspective of three components of the system. The project studies the coherence and alignment of these components from the perspective of classroom mathematics teachers. The research addresses essential questions related to how to provide equitable opportunity to learn for students, teachers, and teacher educators. In particular it will generate models of PD, tools for assessing equity in mathematics teaching and learning, and a theory of equitable mathematics education systems that advances our understanding of the ways in which approaches to teaching, learning, and studying mathematics support equitable opportunities to learn.

 

A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in STEM Education will be established to address a set of issues posed as critical to the future of assessment for education and will undertake a series of activities culminating in the production of a report.

Lead Organization(s): 
Award Number: 
1433181
Funding Period: 
Mon, 09/01/2014 to Mon, 02/29/2016
Full Description: 

This project seeks to find ways to make the measurement sciences more useful to the production of intellective competence in diverse students of the STEM disciplines. A Study Group on Diversity, Equity and Excellence in Achievement and Assessment in Science, Technology, Engineering and Mathematics Education will be established to address a set of issues posed as critical to the future of assessment for education. Building on the work of the Gordon Commission on the Future of Assessment in Education, the Study Group will undertake a series of activities culminating in the production of a report to the field including conceptual analyses, knowledge syntheses findings, and recommended specifications for a program of research and development to advance STEM education through assessment for education.

The Study Group will consist of fifteen research scientists and scholars of pedagogical practice who are being organized to inquire into the potential of the measurement sciences to more effectively inform the teaching and learning of STEM subject matter for students whose development is challenged by the demands of diversity, equity and excellence in intellective competence. Group members are especially interested in balancing the capabilities of the measurement sciences in the assessment of developed abilities, with equal strength in the analysis, documentation and understanding of the learning and teaching processes by which intellective competence is developed. The Group will conduct a conceptual inquiry into the interrelatedness of diversity, equity, and achievement in STEM. Special attention will be given to variance in attributions, contexts and perspectives associated with differences in life conditions, cultural experience and cultural identity. The Study Group will utilize in-person and virtual deliberations; consultative conversations; and commissioned position and review papers to generate recommendations and suggested specifications for programs of research, development and praxis designed to better inform the teaching and, especially, the learning of STEM disciplines.

Designing Assessments in Physical Science Across Three Dimensions (Collaborative Research: Krajcik)

This is a collaborative project to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012).

Lead Organization(s): 
Partner Organization(s): 
Award Number: 
1316908
Funding Period: 
Sun, 09/01/2013 to Thu, 08/31/2017
Full Description: 

This is a collaborative proposal among the University of Illinois at Chicago, Michigan State University, and SRI International to develop, test, and analyze sets of technology-supported diagnostic classroom assessments for middle school (grades 6-8) physical science. Assessments are aligned with the performance assessment and evidence-centered design methodologies suggested in the Framework for K-12 Science Education (NRC, 2012). The study focuses on the development of new measures of learning that take into account the interdependence of science content and practice. Two disciplinary core ideas--Matter and its Interactions, and Energy--and two scientific and engineering practices--Constructing Explanations and Designing Solutions, and Developing and Using Models--are used for this purpose.

The research questions are: (1) What are the characteristic features of science assessments based upon systematic application of the Evidence-Centered Design (ECD) assessment process?; (2) To what extent can assessment designs incorporate critical core idea, crosscutting concept and science/engineering practice dimensions in ways that both separate and integrate these dimensions as part of the design architecture?; (3) What is the evidence that the multiple dimensions of science learning (e.g., content, practices and crosscutting concepts) are separable and recoverable in the performance of students who respond to these assessments?; (4) How instructionally sensitive are these assessments? (i.e., Do they show differential and appropriate sensitivity to students' opportunity to learn science in ways consistent with the vision contained in the NRC Framework?); (5) What forms of evidence can be provided for the validity of these assessments using a multifaceted validity framework that takes into account both the interpretive and evidentiary components of a validity argument for these new assessments?; (6) What are the characteristics of assessments that best serve the needs of classroom teachers relative to a formative assessment process and in what ways do such assessments and scoring processes need to be designed to support effective teacher implementation?; and (7) What are the unique affordances and opportunities provided by technology in designing and implementing assessments focused on merging content & practices performance expectations?

Assessments are iteratively designed and administered in three school districts and a laboratory school in Florida and one school district in Wisconsin using the "Investigating and Questioning our World through Science and Technology" curriculum. The three school districts in Florida have classrooms that are using typical curriculum. The assessments will also be administered and tested with students in these classrooms. To address the research questions, the project conducts five major tasks: (1) development of assessment items using the ECD process to document and guide coherence of items; (2) an alignment study to review design patterns and task templates; (3) a cognitive analysis study to empirically investigate the extent to which the items elicit the intended guidelines; (4) three empirical studies, including (a) an early-stage testing with teachers (n=6) and students (n=180) in Year 1, (b) a pilot testing in Year 2 with teachers (n=12) and students (n=360), and (c) a main study in Year 3 with teachers (n=30) and students (n=900); and (5) a study to investigate the formative use of the assessment items using teacher focus groups' feedback and analysis of student performance data from previous studies.

Project outcomes are: (a) research-informed and field-tested assessment prototypes that measure students' thinking around the two physical science core ideas and the two scientific and engineering practices; (b) relevant data and procedures used in the studies; and (c) a framework for the formative use of the assessments, including guidelines, scoring rubrics, and criteria for assessment design decisions.

Pages

Subscribe to Assessment