Assessing Teaching Practice

Image of a teacher and two students working together with a manipulative.This Spotlight features expertise, tools, and examples from DRK-12 projects related to assessing aspects of teaching practice in STEM education. In addition to CADRE resources, four projects share information about the new resources, processes, or tools they are developing to assess teaching practice; approaches and strategies that have shown promise in their work; advice for other grantees; and available findings and products.

In this Spotlight:

Resources to Support the Assessment of Teaching Practice

Instructional Observation PanelCADRE Learning Series on Instructional Observation | 2023 Webinar Recording

Many DRK-12 projects seek to document changes in classroom instruction. To that end, researchers often seek to use or create reliable instructional observation instruments. In this Learning Series webinar, moderated by Ilana Horne, researchers Pauline Chinn, Kara Jackson, and Gillian Roehrig discuss how they have used or designed these tools in their DRK-12 research. Participants learned about a range of tools, as well as their various uses and limitations, to inform their own own project designs.

Webinar Recording | Slides: Intro (Horn), Roehrig, Jackson, Chinn

Compendium Cover ImageCADRE Compendium of Research Instruments for STEM Education, Part 1: Teacher Practices, PCK, and Content Knowledge | 2013 Report

This compendium provides an overview of STEM instrumentation being used by the first six cohorts of NSF's DRK-12 program and along with resources that may be useful to research and evaluation professionals. The information contained within was heavily dependent on information available on existing websites. For each instrument we provide information on the constructs/variables that are measured, the target audience of the instrument, the subject domains assessed, information on obtaining the instrument and related documents about reliability and validity evidence when it could be located. It is part of a two part series, where Part 2 focuses on measuring students’ content knowledge, reasoning Skills, and psychological attributes.

Download the Report

Measuring Teacher Change Poster Hall | 2023 DRK-2 PI Meeting

At the 2023 DRK-12 PI Meeting, in a session facilitated by Jinfa Cai and Jamie Mikeska, researchers from seven projects shared how they were approaching and innovating measuring change in teachers' knowledge and practice. The collection of posters is available on the CADRE website.

View the Poster Hall

Featured Projects

Design and Development of a K-12 STEM Observation Protocol

PIs: Emily Dare, Elizabeth Ring-Whalen, Gillian Roehrig | Co-PIs: Joshua Ellis, Mark Rouleau
STEM Disciplines: Integrated STEM education
Grade Levels: K-12

Project Description: The main goal of our project - Collaborative Research: Design and Development of a K-12 STEM Observation Protocol - was to create a valid and reliable observation instrument for use in K-12 science and engineering classrooms where integrated STEM instruction is taking place. Prior to our work, no instrument to characterize integrated STEM education existed. Our work has resulted in a detailed conceptual framework for integrated STEM education and the STEM Observation Protocol (STEM-OP), which includes 10 observable items with four descriptive levels for each item (scored 0-3): 1) Relating content to students’ lives, 2) Contextualizing student learning, 3) Developing multiple solutions, 4) Cognitive engagement in STEM, 5) Integrating STEM content, 6) Student agency, 7) Student collaboration, 8) Evidence-based reasoning, 9) Technology practices in STEM, and 10) STEM career awareness. The STEM-OP can be used in a variety of educational contexts (curriculum development, coaching, PD) and research. To assist others in learning about the STEM-OP and how to use it, we created an online training course via Canvas that guides users through informative videos about each item and their corresponding scoring levels, including examples of classroom video. The online platform includes opportunities for users to test their knowledge and receive feedback through course quizzes. The STEM-OP and its associated materials are appropriate for a variety of stakeholders, including K-12 teachers, district administrators, teacher educators, and educational researchers.

Conceptual Framework for STEM Observation Protocol Project

Focal Practices: The STEM-OP focuses on observing integrated STEM practices as evidenced by the teachers’ actions and words. It measures the degree of integrated STEM (or “STEM-iness”) within instruction rather than assessing quality of instruction, which can be done using other instruments. The ultimate goal of developing the STEM-OP was to find a way to talk about integrated STEM practices in action with a clear set of guidelines.

Challenging Aspects of Assessing Teaching Practice & Advice for Researchers: Creating an observation instrument is hard when the phenomenon you’re aiming to observe is not well-defined, so that was an initial challenge for our team to overcome! This meant we first had to develop a conceptual framework for integrated STEM education to guide what would eventually become the items on the STEM-OP. When we first developed it, we had over 70 items, which clearly wasn’t going to work for an instrument to be used during classroom observations. It took approximately 18 months to have a finalized version of the STEM-OP with high interrater-reliability among our team members; patience is certainly a key element when developing something new and novel. We had a big team that included faculty and graduate students with different backgrounds and perspectives, which was a great asset. Our differences caused us to think deeply about the items on the protocol and carefully consider the wording of each item’s levels. While this wasn’t always a smooth process, it helped us to create a better product. Along the way we had to remind ourselves that we were doing something that hadn’t been done before. This was sometimes a frustration for us to overcome because we set a high bar for ourselves. Is the STEM-OP perfect? No - no instrument out there is. Recognizing that our goal was really to create a starting point was extremely helpful for us to reframe the work we were doing to have a product available for others to use.

Initial Findings: Access to the large video repository of integrated STEM classroom observations allowed us to use the instrument to gauge what components of integrated STEM as defined by the STEM-OP are currently being implemented. A comparison of science-focused vs. engineering-focused lessons found that eight of the STEM-OP items scored significantly higher during engineering lessons than science lessons (Roehrig et al., 2022). This suggests the benefits of the integration of engineering into science classrooms in terms of improving students’ engagement in science and engineering practices, as well as 21st century skills. Dare et al. (2022) found distinct patterns when considering different grade bands (elementary, middle, high) and science content (physical science, life science, and earth science). In particular, implementation of integrated STEM lessons in elementary grade bands tended to score higher than middle and high school implementations and physical science lessons consistently outscored earth and life science-focused lessons. We have also examined how the STEM-OP scores vary when considering a fully implemented unit, which reveals that achieving higher scores is quite rare on a daily basis but can happen periodically throughout a unit of instruction (Dare & Ring-Whalen, in press). This helps us to set up goals for professional learning moving forward. Research involving a sample of teachers who participated in the early professional development demonstrated that while teachers are generally positive about integrated STEM in their science teaching, they are faced with practical barriers, but look to the STEM-OP as a guide for their curriculum and instruction, as well as a resource to share with colleagues interested in integrated STEM education (Dare et al., 2021; Dare et al., in review).

Products: Our biggest resource for those wanting to learn more about the STEM-OP and how to use it can be found on our Free for Teachers Canvas course. Individuals can self-enroll in the course using this link: https://canvas.instructure.com/enroll/Y7WRRG. This course is fitting for anyone who wants to learn more about the STEM-OP, including K-12 educators. The following three publications may also be of interest to those who want to understand the STEM-OP:

  1. Dare, E. A., Hiwatig, B., Keratithamkul, K., Ellis, J. A., Roehrig, G. H., Ring-Whalen, E. A., Rouleau, M. D., Faruqi, F., Rice, C., Titu, P., Li, F., Wieselmann, J. R., & Crotty, E. A. (2021). Improving STEM education: The design and development of a K-12 classroom observation instrument (RTP). In Proceedings of the 2021 ASEE Annual Conference and Exposition. https://peer.asee.org/37307
  2. Roehrig, G., Dare, E. A., Ellis, J. A., & Ring-Whalen, E. (2021). Beyond the basics: A detailed conceptual framework of integrated STEM. Disciplinary and Interdisciplinary Science Education Research, 3(11). https://doi.org/10.1186/s43031-021-00041-y
  3. Roehrig, G. H., Rouleau, M. D., Dare, E. A., & Ring-Whalen, E. A. (2023). Uncovering core dimensions of K-12 integrated STEM. Research in Integrated STEM Education, 1(1), 5-29. https://doi.org/10.1163/27726673-00101004

In total, this project has resulted in one dissertation (with another in progress), two book chapters, eight journal publications (with another in review), 10 conference proceedings, over 30 conference presentations and embedded workshops, and 11 invited keynote presentations. The following lists our additional published works.

  • Dare, E. A., Ellis, J. A., Rouleau, M. D., Roehrig, G. H., & Ring-Whalen, E. A. (2022, June). Current practices in K-12 integrated STEM education: A comparison across science content areas and grade-levels (Fundamental). In Proceedings of the 2022 ASEE Annual Conference and Exposition.
  • Dare, E. A., Keratithamkul, K., Hiwatig, B. M., & Li, F. (2021). Beyond content: Exploring the role of STEM disciplines, real-world problem 21st century skills, and STEM careers within science teachers’ conceptions of STEM education. Education Sciences, 11(11).
  • Dare, E. A., & Ring-Whalen, E. A. (2021). Eliciting and refining conceptions of STEM education: A series of activities for professional development. Innovations in Science Teacher Education, 6(2).
  • Dare, E. A., & Ring-Whalen, E. A. (in press). Understanding variation in integrated STEM practice as measured by the STEM Observation Protocol (STEM-OP): Comparing daily practice and curriculum unit implementation. Research in Integrated STEM Education. 
  • Dare, E.A., Roehrig, G.H., Ellis, J.A., Rouleau, M.D., & Ring-Whalen, E.A. (2022, September). Understanding current practices of integrated STEM education in K-12 science classrooms. In Proceedings of Japan Society for Science Education.
  • Ellis, J. A., Wieselmann, J. R., Sivaraj, R., Roehrig, G. H., Dare, E. A., & Ring-Whalen, E. A. (2020). Toward a productive definition of technology in science and STEM education. Contemporary Issues in Technology and Teacher Education – Science, 20(3), 472-496.
  • Faruqi, F., Karatithamkul, K., Roehrig, G. H., Hiwatig, B. M., Forde, E., & Ozturk, N. (2022, June). Manifestation of Integration into practice: A single case study of an elementary science teacher in action (Research to Practice). In Proceedings of the 2022 ASEE Annual Conference and Exposition.
  • Forde, E. N., Robinson, L., Ellis, J., & Dare, E. A. (2023). Investigating the presence of mathematics and the levels of cognitively demanding mathematical tasks in integrated STEM units. Disciplinary and Interdisciplinary Science Education Research, 5(3), 1-18.
  • Hiwatig, B. M. R. (2022). Exploring the relationship between aspects of integrated STEM education and student attitudes towards STEM. [Doctoral dissertation, University of Minnesota]. ProQuest.
  • Hiwatig, B. M., Roehrig, G. H., Ellis, J. A., & Rouleau, M. D. (2022, June). Examining student cognitive engagement in integrated STEM (Fundamental). In Proceedings of the 2022 ASEE Annual Conference and Exposition.
  • Matsubara K.; Roehrig, G.; Dare, E.; Chi, U.; Miyauchi, T.; KobayashI, Y.; Tanimoto, K. (2022, September). Toward development of lesson analysis system with a STEM perspective for improving inquiry lessons. In Proceedings of Japan Society for Science Education.
  • Robinson, L., & Dare, E. A. (2022, June). How to use the STEM-OP levels to support the engineering design-based lesson plan template in the Framework for P-12 Engineering Learning (Resource Exchange). In Proceedings of the 2022 ASEE Annual Conference and Exposition.
  • Roehrig, G.H., Dare, E.A., Ellis, J.A., & Ring-Whalen, E.A. (2022, September). Development of a framework and observation protocol for Integrated STEM. In Proceedings of Japan Society for Science Education.
  • Roehrig, G. H., Dare, E. A., Wieselmann, J. R., & Ring-Whalen, E. A. (2023). The rise of STEM Education: STEM curriculum development and implementation. In R. Tierney, F. Rizvi, & K. Ercikan (Eds), International Encyclopedia of Education (pp. 153-163). Elsevier.
  • Roehrig, G.H., Hiwatig, B., & Keratithamkul, K. (2020). The intersections of integrated STEM and socio-scientific issues, In W. Powell (Ed.) Socioscientific issues-based instruction for scientific literacy development (pp. 256-278). IGI Global.
  • Roehrig, G. H., Ring-Whalen, E. A., Dare, E. A., Ellis, J. A., & Wieselmann, J. R. (2019). WIP: The Development of a K-12 Integrated STEM Observation Protocol. In Proceedings of the 2019 ASEE Annual Conference and Exposition. Tampa, FL: ASEE.
  • Sivaraj, R., Ellis, J., & Roehrig, G. (2019). Conceptualizing the T in STEM: A systematic review. In K. Graziano (Ed.), Proceedings of Society for Information Technology & Teacher Education International Conference 2019 (pp. 2273-2280). Chesapeake, VA: Association for the Advancement of Computing in Education.
  • Sivaraj, R., Ellis, J. A., Wieselmann, J. R., & Roehrig, G. H. (2020). Computational participation and the learner‐technology pairing in K‐12 STEM education. Human Behavior and Emerging Technologies, 2(4), 387-400.

Doing the Math Logo

Doing the Math with Paraeducators: Enhancing and Expanding and Sustaining a Professional Development Model in PreK to Grade 3 Math Classrooms

PI: Judy Storeygard | Co-PIs: Audrey Martinez-Gudapakkam, Karen Mutch-Jones, Brandon Sorge
STEM Disciplines: Mathematics
Grade Levels: Prek-3

Project Description: We work with paraeducators/instructional assistants in the Boston Public Schools and Metropolitan School District Washington Township (MSDWT). These are educators, many from under-represented populations, who do not have well-defined job requirements, and receive little professional development yet are expected to assist in instruction in all academic subjects. In addition, they are not paid for planning time. We have focused on building their confidence and pedagogical content knowledge through a 35 hour PD program during the summer and the following school year We have provided sessions for teachers to encourage collaboration with their paraeducator back in the classroom. As part of our project, we developed and collected data from instruments that measure efficacy and pedagogical content knowledge. We are currently analyzing our data and presenting findings. Our initial results show a significant change in paraeducator efficacy for both cohorts. The external evaluation reports indicated that participants gained skills in questioning students and eliciting student thinking, and familiarizing themselves with a variety of tools, representations, and manipulatives to help students think differently about problems. The report indicated that because of their increase in self-confidence, paras expresses feeling more capable when facilitating small group instruction in their math classes. The paraeducators/instructional assistants also appreciated the opportunity to collaborate with their colleagues.

Focal Practices: We are prioritizing listening to students mathematical thinking, asking them questions that promote understanding, being familiar with a variety of representations and tools so that students are exposed to strategies that might work for them.

Innovation in Assessing Teaching Practice: There are currently no instruments that measure teacher practice of paraeducators. We expect that the instruments we have developed will be useful to the field.

Challenging Aspects of Assessing Teaching Practice: The process is time-consuming, so it is important to consider this in planning the timeline. 

Advice for Awardees:

  • Conduct a literature search on what instruments are available, and before beginning be sure to learn about  your population: education level and experience, language preferences, facility with technology (should assessment be paper or electronic.
  • Consult with advisors and other colleagues who have had experience with the “type” of the educators in the study prior to designing any measures.
  • Conduct a small pilot test and revise the intervention and the measures in accordance with participant feedback. Participants should be from a population similar to the ones in the project. This process may have to be repeated several times.
  • Pilot all measures, particularly those which were designed for teachers and are being adapted for a different audience.

Initial Findings: The math efficacy survey results from the 2022-23 and 2023-24 cohorts in both Boston and Indiana combined are: mathematics teaching self-efficacy [t(65) = -4.240, p<.001] and outcomes expectancy [t(65) = -3.438, p=.001]. Both showed statistically significant increases.

An excerpt from the 2023-24 evaluation report summarizes some of the gains that the paraeducators/instructional assistants made. The report found these gains were corroborated by their teachers and school coaches.

Co-educator participants gained skills in using questioning strategies to elicit student thinking, providing manipulatives or visual ways of having students think differently about a problem, and in facilitating math games with students. Most participants reported increases in self-confidence as a result of these gains and described being better able to lead small groups and whole class lessons. They also greatly benefitted from receiving professional development targeted to their population, and the time to create a professional learning community and forge relationships with other co-educators in their districts.

Participating co-educators developed closer working relationships with their teachers, making them better able to plan for math lessons, and thereby providing two knowledgeable adults for students in math classrooms. Teachers described students’ increased engagement with co-educators around math and the value of reaching all students in the classroom and making math more enjoyable. Administrators in participating schools supported the participation of co-educators, despite the logistical challenges of classroom coverage. They saw the benefits of all staff in schools being used effectively, and how knowledgeable and confident co-educators could support teachers, and in some cases work to become teachers themselves. Math coaches who worked with co-educators closely during the program saw first-hand the changes in their confidence and ability in the classroom, and all expressed a desire to continue to offer targeted professional development to this population of educators.

Products:

  • Project Website
  • Project Video
  • Recent Presentations
    • Raymond, S., & Mutch-Jones, K. (2023). Narrative mathematical identifies of black women paraeducators. NCTM Research Conference 
    • Accepted Session: Pinet, L., Lockett, C., Heard, C., & Wright, K. (2024). Building community in the elementary math classroom: Parents, educators, and teachers. NCTM Annual Meeting & Exposition.
  • Accepted Paper: Mutch-Jones, K., & Storeygard, J. Building instructional capacity and creating opportunities for professional growth: Mathematics professional development for paraeducators. Mathematics Teacher Educator.

PMR2 Logo

Improving the Implementation of Rigorous Instructional Materials in Middle-Grades Mathematics: Developing a System of Practical Measures and Routines

PIs: June Ahn, Paul Cobb, Marsha Ing, Kara Jackson 
STEM Disciplines: Mathematics
Grade Levels: Secondary

Project Description: Our project (Practical Measures, Routines, and Representations) developed a system of practical measures (and associated routines and data representations) to support teachers, coaches, and district leaders as they implement instructional improvement strategies in secondary mathematics. In contrast to research and accountability measures, practical measures are assessments that require little time to administer and can thus be used frequently. The data can be analyzed rapidly so that teachers can receive prompt feedback on their progress, and instructional leaders can use the data to decide where to target resources to support improvement in the quality of instruction and student learning. Classroom measures assess students’ perceptions of key aspects of the mathematics classroom learning environment that prior research indicates make a difference for students’ learning. Classroom measures include student-facing surveys focused on the launch of cognitively demanding tasks, small-group work, and whole-class discussions, and a rubric designed to assess the level of cognitive demand of instructional tasks. Professional learning measures take the form of teacher-facing surveys, and assess key aspects of the professional learning environment that research indicates makes a difference for teachers’ learning. Alongside the measures, we developed a data visualization platform and routines to support the administration of the measures and productive analysis of the resulting data. Our team consisted of three research-practice partnerships in districts that were pursuing ambitious instructional improvement strategies; the tools and routines were co-designed, tested, and improved in the context of instructional improvement strategies focused on one-on-one coaching, teacher collaboration, and curriculum guides. 

Focal Practices: The Classroom Measures focus on three phases of lessons organized around cognitively demanding tasks: the launch (or introduction) of a cognitively demanding task; small group work, in which students work on solving the task; and the whole-class discussion, in which teachers orchestrate a discussion of particular strategies in order to advance students’ understandings of the key mathematical ideas. Please see the annotated measures (available via our website, https://www.pmr2.org//) for detailed description of what instructional practices are focused on in each of these phases. As an example, the whole-class discussion measure focuses on: 1) the cognitive demand of the task as implemented; 2) what students are accountable for in the discussion; 3) the extent to which discussions focus on students’ ideas; 4) opportunities for students to make sense of other students' ideas; and 5) the extent to which students want to share their ideas and feel their ideas are valued.

Innovation in Assessing Teaching Practice: The project was motivated by educators’ needs for targeted, “on the ground” information about how instructional improvement strategies aimed at advancing equity-oriented, rigorous mathematics teaching at some scale were playing out for teachers and students in a timely manner. Educators have minimal access to timely information about the impact of their teaching on students, and the impact of professional development on teachers. This lack of information makes it difficult, if not impossible, for educational leaders to revise instructional improvement strategies (e.g., collaborative PD, coaching cycles), based on an analysis of how the strategy has affected the people most impacted by the change. This system of practical measures enables educators to assess whether a change in mathematics teaching results in improvements in students’ learning opportunities, and whether a change in a support for teachers’ learning results in improvements in teachers’ learning opportunities. 

Reliability & Validity: A contribution of this project has been to conceptualize validity evidence in the context of practical measurement. The first component concerns the extent to which a practical measure can be used for the intended purpose. For example, whether the classroom measures of key aspects of the classroom learning environment can be used to determine whether an instructional change is an improvement. Multiple cycles of development and revision in collaboration with potential users were essential in establishing procedures to gather evidence that the measure could be used for the purpose of instructional improvement (see, e.g., Jackson et al., in press; Nieman et al., 2023).

The second component concerns how the measures are used by practitioners, and the processes they engaged in as they acted upon the resulting data. Based on our investigations of the use of the measures in our partner districts, we identified appropriate (and inappropriate) purposes for using for the measures, as well as key aspects of contexts and of users’ knowledge and perspectives that influence whether they are likely to act reasonably in response to the resulting data (Ing et al., 2021; Jackson et al., in press; Nieman et al., 2023).

Initial Findings:

  • Use of the Classroom Measures in Instructional Improvement Initiatives
    On the basis of our analyses of the use of the classroom practical measures in various instructional improvement initiatives (e.g., one-on-one coaching cycles, collaborative professional development, revision of curriculum guides), we have identified three distinct ways in which the practical measures can contribute to ongoing instructional improvement efforts: 1) by determining whether instructional changes are improvements; 2) by enhancing the effectiveness of supports for teachers’ learning; and 3) by enhancing the coherence of instructional improvement efforts (Ing et al., 2021; Jackson et al., in press; Kochmanski, 2020; Nieman et al., 2020). In addition, we found that teachers’ interpretations of students’ perceptions of the classroom learning environment (i.e., students’ responses to the classroom practical measures) can provide insight into their current pedagogical commitments, specifically regarding teachers’ current instructional visions and their views of the students’ capabilities. This has implications for professional development (Jackson et al., 2022).
  • Professional Development (PD) Facilitators’ Use of the Collaborative PD Measure
    On the basis of our analyses of the use of collaborative PD practical measure in various PD sites, we have shown that the measure supports PD facilitators to engage in conceptual inquiry about their practice, and to set goals and to assess changes in facilitation practice. Further, we specify characteristics of conversational routines that support conceptual inquiry, as well as orientations to practice that underpin productive use of the measure (Nieman et al.,2023; Nieman et al., under review).
  • Data Dashboard and Representations
    Members of the research team with expertise in data visualization partnered with teachers, coaches, and district mathematics specialists in the partner districts to co-design an online platform on which the surveys can be administered, and responses can be viewed and analyzed. A set of publications report on key learnings from careful study of the co-design and use of the resulting platform. One study focuses on patterns in educators’ sense-making in relation to specific features of the dashboard; and offers guidance to others designing platforms that encourage genuine inquiry (Campos et al., 2021). A second study highlights the tensions between designing for existing norms of data use, which often focus on accountability, versus challenging these norms, via a focus on improvement (Ahn et al., 2021). A third study highlights the importance of  leveraging educators' routines, values, and cultural representations into the design of the digital dashboard (Campos et al., 2024).

Products: Our project’s website, https://www.pmr2.org//, includes access to the measures, routines, and representations; as well as publications and presentations.

In addition, the PMRR measures are featured on a web-based repository of practical measures to support instructional improvement in middle-grades mathematics (funded by the Gates Foundation): https://mpm.wested.org/.

STEM STRONG logo

Investigating How Combining Intensive Professional Development and Modest Support Affects Rural, Elementary Teachers’ Science and Engineering Practice

PI: Ryan Summers | Co-PIs: Rebekah Hammack, Martha Inouye, Ashley Iveland, Cathy Ringstaff
STEM Discipline: Science & Engineering
Grade Levels: Grades 3-5

Project Description: Our STEM STRONG project aims to investigate factors that influence elementary teachers following professional learning (PL) in science and engineering, specifically exploring the impacts of PL on teachers over a 3-year intervention of modest supports. Three of our overarching research objectives consider teaching practices. These objectives include: (1) assessing the extent to which an intense 5-day PL event impacts teachers’ knowledge and self-efficacy in science and engineering; (2) observing the effectiveness of modest supports on teachers’ self-efficacy in science and engineering, on teachers’ use of inquiry-based instructional strategies, and on the sustainability of PL outcomes (e.g., instructional time in science and engineering); and (3) documenting the effectiveness of PL, followed by modest supports, on teachers’ capacities to deliver engineering instruction, and specifically their integration of engineering practices.

We are implementing the second year of the intervention during 2024-2025 with nearly 200 teachers in rural settings and distributed across four states (CA, MT, ND, WY). We will continue to examine the extent to which these modest supports individually and collectively contribute to the sustainability of PL outcomes in terms of increases in instructional time devoted to science, teacher self-efficacy in science, and teacher use of NGSS-aligned instructional strategies. We hope our research will inform future initiatives, leading to efficient teacher PL programs that positively impact K-12 classrooms.

Focal Practices: During the first year of the intervention, beginning in the Summer 2023 PL Institute, teachers were introduced to the NGSS’s three dimensions and developed their understanding of the role of phenomena in expanding opportunities to learn science and engineering. Using a pre-assigned lesson, teachers mapped where this lesson integrated phenomena and the three-dimensions. Teachers experienced how engineering can be incorporated into their classrooms through an engineering design task. Finally, teachers learned how to elicit student thinking using three-dimensional assessments. At the end of each day of this 5-day online PL, teachers reflected on their experiences, which informed the next day’s activities.

Throughout the 2023-2024 academic year, we offered a menu of modest supports to sustain teachers’ learning from the summer PL and to meet teachers’ needs in implementing authentic science and engineering lessons. Overall, we provided seven 90-minute-long online professional learning community (PLC) sessions—three whole-group sessions and four state-specific sessions. During the PLC sessions, teachers were introduced to a Culturally Relevant Engineering Design (CRED) framework and collaboratively completed a CRED-aligned lesson plan, where they were encouraged to use their community’s assets to make their lesson meaningful for students. As a response to teachers’ needs, the last two whole-group PLC meetings focused on introducing an NGSS-aligned performance-based assessment task. Like the engineering design lesson, teachers implemented this assessment task in their own classrooms. Then, they analyzed their own students’ responses, identified common trends, and brainstormed on next instructional moves. Additionally, a subgroup of five teachers per state participated in an engineering learning community (ELC) to further focus on developing their engineering teaching practices during 2023-2024.

Innovation in Assessing Teaching Practice: Our overall project aims to transform teacher practice through on-going, modest supports. Our approach to assessing practice was through intentionally designed opportunities that invited teachers to reflect on practice, celebrate their successes and the successes of peers in the group, and set goals for further improvements. Several key features of the ELC design, which involved a sub-group of the participating teachers, embody innovative approaches for assessing teacher practice. First, teachers recorded their lessons using Swivl cameras that we shipped. Next, teachers met virtually in their state-level groups, and using a success protocol, each teacher shared two brief segments of the recordings that they identified as successes to share with their cohorts. This protocol allowed teachers to share why they felt it was a success and to get additional feedback from their ELC colleagues. Teachers followed with a full ELC group virtual meeting where they shared more about their lesson outcomes, worked in small breakout groups to reflect on successes and problem-solve “sticking points” from their lessons, and engaged in a collaborative activity where they discussed what made a classroom task “science”, “engineering”, or both. The ELC experience culminated in individual interviews conducted over Zoom where teachers discussed how their engineering practice had evolved related to the PL aims and outcomes and set goals for their continued growth. Overall, our approach ensured that assessment of practice was personalized, reflective, and derived from teachers’ own perspectives through a variety of collaborative and asset-based experiences. 

Challenging Aspects of Assessing Teaching Practice & Advice for Researchers: When considering the development and adaptation of instruments for assessing teaching practice, it is important to remember that context is a key factor. Because of our focus on connecting classroom science and engineering instruction to local communities, we planned to use a modified version of the Classroom Observation Protocol for Engineering Design (COPED) to analyze teachers’ implementation of their engineering lessons that accounted for connections to culture and community. However, our rural teachers worked in such varied contexts, that there were additional aspects of teaching practice that even our modified version of the COPED did not capture. For example, a teacher working with a classroom of emergent bilingual students employed specific practices to address the language needs of students, including acquisition, vocabulary, and considerations pertaining to her students’ funds of knowledge. These teaching practices were different from those employed by a classroom teacher who was working in a multi-grade classroom with students of varying ages, simultaneously teaching students ranging from 1st through 6th grade. Our advice to other researchers is to be mindful of the classroom and community contexts teachers are working in and acknowledge that instruments may need to be altered or supplemented to account for varying contexts. Also, we recommend that researchers are prepared to bring in experts if specific contexts are outside of the research teams’ expertise (e.g., multilingual learning strategies).

Initial Findings: We have started to disseminate findings from the first year of the intervention with teachers. We are in the process of analyzing data specific to teacher practice and the ELC. Initial findings about the Summer 2023 PL Institute, based on pre- and immediate post-intervention survey responses of 111 teachers, showed positive impacts on teachers’ attitudes and efficacy for teaching engineering. These findings were shared at the ASEE Annual Conference by Hammack et al., and the proceedings are available:

Hammack, R., Robinson, J., Boz-Togu, T., Lee, M. J., Summers, R., Iveland, A., Inouye, M., Macias, M., Zaman, M., Galisky, J., Johansen, N.,  & Ringstaff, C. (2024). Supporting Elementary Engineering Instruction in Rural Contexts Through Online Professional Learning and Modest Supports. Proceedings of the 2024 ASEE Annual Conference, Portland, OR.

Products: 

Additional Projects

We invite you to explore a sample of the other recently awarded and active work with a focus on assessing teaching practice.

Further Reading

References

Ahn, J., Campos, F., Nguyen, H., Hays, M., & Morrison, J. (2021, April). Co-Designing for Privacy, Transparency, and Trust in K-12 Learning Analytics. In LAK21: 11th International Learning Analytics and Knowledge Conference (pp. 55-65). https://doi.org/10.1145/3448139.3448179

Campos, F., Ahn, J., DiGiacomo, D., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K-12 teachers and coaches react to visual analytics. Journal of Learning Analytics, Early Access, 1-21. https://doi.org/https://doi.org/10.18608/jla.2021.7113 

Campos, F., Nguyen, H., Ahn, J., & Jackson, K. (2024). Leveraging cultural forms in human-centered learning analytics design. British Journal of Educational Technology, 55(3), 769-884. https://doi.org/10.1111/bjet.13384

Dare, E. A., Keratithamkul, K., Hiwatig, B. M., & Li, F. (2021). Beyond content: Exploring the role of STEM disciplines, real-world problem 21st century skills, and STEM careers within science teachers’ conceptions of STEM education. Education Sciences, 11(11).

Dare, E. A., & Ring-Whalen, E. A. (in press). Understanding variation in integrated STEM practice as measured by the STEM Observation Protocol (STEM-OP): Comparing daily practice and curriculum unit implementation. Research in Integrated STEM Education.

Dare, E.A., Roehrig, G.H., Ellis, J.A., Rouleau, M.D., & Ring-Whalen, E.A. (2022, September). Understanding current practices of integrated STEM education in K-12 science classrooms. In Proceedings of Japan Society for Science Education.

Ing, M., Chinen, S., Jackson, K., & Smith, T. M. (2021). When should I use a measure to support instructional improvement at scale? The importance of considering both intended and actual use in validity arguments. Educational Measurement: Issues and Practice, 40(1), 92-100. https://doi.org/10.1111/emip.12393 

Jackson, K., Nieman, H., DiGiacomo, D., Campos, F., & Kochmanski, N. (2022). Mathematics teachers’ interpretations of students’ perceptions of the classroom learning environment: opportunities for inquiry and insight into pedagogical commitments. Proceedings of the 16th International Conference of the Learning Sciences - ICLS 2022 (pp. 1277-1280).

Kochmanski, N. (2020). Aspects of high-quality mathematics coaching: What coaches need to know and be able to do to support individual teachers' learning [Doctoral dissertation].

Nieman, H. J., Jackson, K., Jarry-Shore, M., Borko, H., Kazemi, E., Chinen, S., Lenges, A., Yilmaz, Z., & Haines, C. (2023). Using a practical measure to support inquiry into professional development facilitation. Mathematics Teacher Educator, 12(1), 70--83. https://doi.org/10.5951/MTE.2022.0038

Nieman, H. J., Jackson, K., & Lenges, A. (2020). Facilitators’ and school leaders’ role in establishing an inquiry-oriented professional community of mathematics teachers. In Borko, H. and Potari, D. (Ed.), Proceedings of the 25th Study Conference of the International Commission on Mathematics Instruction: Teachers of Mathematics Working and Learning in Collaborative Groups (pp. 500 - 507).

Roehrig, G.H., Dare, E.A., Ellis, J.A., & Ring-Whalen, E.A. (2022, September). Development of a framework and observation protocol for Integrated STEM. In Proceedings of Japan Society for Science Education.

Year