Supporting Science Learning and Teaching in Middle School Classrooms through Automated Analysis of Students' Writing (Collaborative Research: Passonneau)

This project will develop a novel, automated technology to provide middle-school students and their teachers with real-time feedback about students' written explanations of physics phenomena. The use of evidence to build scientific explanations is a central practice by which scientific knowledge is generated and learned. Students often do not understand what a scientific explanation is and frequently write incomplete, non-causal accounts of scientific phenomenon. Teachers often have difficulties in helping students write explanations, as it is complex and time-consuming. Working in groups of three or four and experimenting with designing a roller coaster, students will learn about key principles in physics such as the conservation of energy and the laws concerning forces and motion. Each student will be provided with a digital journal. The prompts and information in the journal will structure the roller coaster activities (supported 6-8 weeks of instruction) and provide the students with a place to record their written ideas and explanations. At the close of several rounds of experimentation and analysis, students will write causal explanations for their current design. Through the use of the wise crowd automated assessment system, students will receive feedback on their writing. (The automated wise crowd model uses a content assessment of the explanations of experts as the foundation for analyzing and providing feedback to students.) Teachers will also use information from the system to facilitate full class discussions and individualized support. Project research and development activities will result in a fully developed and tested mechanism for providing feedback for students' science explanations. Through automated support of the content analysis of student writing across multiple assignments, the project innovations will allow teachers to more fully integrate writing into their assignments. Ultimately, the project can help students understand how scientific explanations are developed and justified and make them more critical consumers of scientific knowledge so they can make better informed decisions about scientific issues in everyday life.

The project will use a design-based research approach in developing the automated system using wise-crowd analysis and in assessing the impacts of the system on student learning and teacher classroom practices. Four research questions will guide the research: (1) How does feedback from the wise crowd system affect students' written explanation of scientific phenomenon?; (2) How do students with different levels of prior knowledge and reading comprehension benefit from automated feedback and teacher scaffolding?; (3) How do teachers use automated assessment and aggregated summaries of students' explanations during instruction?; and (4) In what ways does scaffolding from the wise crowd system and feedback from teachers support students' written explanations of learning? Through the four-year project, an iterative development process will include the design of the system and testing of two iterations of the system; research of student responses across the progression of roller coaster design and written assignments; and use of some validated and custom instruments to assess student understanding of key forces and assessment of student abilities to use data to evaluate claims. Classroom studies will use video data and researcher field notes to help understand how teachers facilitated the use of the wise-crowd system. Research will culminate in testing of the final version of the wise crowd system. Using a quasi-experimental design, classes will be randomly assigned to the treatment or comparison conditions. Findings will advance knowledge in the field about the best ways to integrate content assessment and feedback from the automated system with classroom and individual support from teachers to optimize learning for students. Materials and results generated from the project will be broadly disseminated, resulting in significant impacts for researchers and practitioners.