Conceptual Replications in STEM Education Research

Despite the abundant and frequent calls for replication studies from research communities (e.g., Shavelson & Towne, 2002) and funding agencies (e.g., IES & NSF, 2013), the number of such studies remains stubbornly small. For example, in an analysis of all articles published since 1900 in the top 10 psychological journals, Makel, Plucker, and Hegarty (2012) found that less than 1% were replication studies. Moreover, from the top 100 education journals, as ranked by 5-year impact factor, Makel and Plucker (2006) found that only 0.13% of articles were replication studies, with the majority of successful replications being authored by the same individuals who had carried out the initial studies.  At the Journal for Research in Mathematics Education (JRME), among all research articles published from JRME’s inception in 1970 through 2016, only about 3% were clearly intending to replicate prior studies (Cai et al., 2018). 

There are many reasons for the limited number of published replication studies. One of the few major reasons is a lack of clarity: What are replication studies? Why should the field of STEM education research engage in such studies?  How should reviewers, journal editors, and funding agencies review them? 

Researchers have generally agreed that there are two kinds of replication studies (Polio & Gass, 1997), exact and conceptual replications. Exact replication tests the findings of the original study by duplicating its methodology as closely as possible. This type of replication presents a challenge, especially in STEM education research, because it can be difficult or even impossible to duplicate conditions and methods in experimental situations that involve complex interactions among content, students, and teachers (Cai et al., 2018).  Given the challenge of duplicating conditions and methods, trying to conduct an exact replication diverts energy and resources from where it is most needed—understanding how and why variation and contextual factors influence the outcome—to reconfirming or disconfirming a result in the exact same context in which it was first found.

Therefore, I believe that conceptual replications are more appropriate for STEM education research than exact replications because conditions may vary for each successive replication.  In fact, a conceptual replication study can do more than investigating whether the effects of a treatment hold true under the same conditions. By being explicit about the treatment conditions of the replication and by generating hypotheses about how they might influence the outcomes, a replication study can investigate the effects of a particular treatment under different conditions or under different samples. The unique contribution of conceptual replication studies is their great potential to investigate systematically the effects of a particular treatment under different conditions or different samples.  That is, conceptual replications can shift the purpose from proving or disproving a finding to understanding more completely the conditions that explain the finding.  Therefore, well-designed conceptual replications contribute new knowledge and insights to the field.

That said, although conceptual replication studies are as important as any other studies in STEM education research, simply saying a study is a replication is insufficient justification for both carrying out a study and publishing the study. As with all studies, every part of a conceptual replication must be justified, including the research questions, theoretical framing, selection of robust methods, and interpretation of the findings (Cai et al., 2019a, 2019b, 2019c, 2019d).

References

• Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., & Hiebert, J. (2018). The role of replication studies in educational research. Journal for Research in Mathematics Education, 49(1), 2-8.
• Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., Cirillo, M., Kramer, S. L., & Hiebert, J. (2019a). Choosing and justifying robust methods for educational research. Journal for Research in Mathematics Education, 50(4), 342-348. doi:10.5951/jresematheduc.50.4.0342
• Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., Cirillo, M., Kramer, S. L., & Hiebert, J. (2019b). Posing significant research questions. Journal for Research in Mathematics Education, 50(2), 114-120. doi:10.5951/jresematheduc.50.2.0114
• Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., Cirillo, M., Kramer, S. L., & Hiebert, J. (2019c). So what? Justifying conclusions and interpretations of data. Journal for Research in Mathematics Education, 50(5), 470-477. https://doi.org/10.5951/jresematheduc.50.5.0470
• Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., Cirillo, M., Kramer, S. L., & Hiebert, J. (2019d). Theoretical framing as justifying. Journal for Research in Mathematics Education, 50(3), 218-224. doi:10.5951/jresematheduc.50.3.0218
• Institute of Education Sciences, & National Science Foundation. (2013). Common guidelines for education research and development. Washington, DC: Author.
• Makel, M. C., & Plucker, J. A. (2006). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304-316.
• Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542. doi:10.1177/1745691612460688
• Polio, C., & Gass, S. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19(4), 499-508. doi:10.1017/S027226319700404X
• Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education: Committee on scientific principles for education research. Washington, DC: National Academy Press.

---

Any opinions, findings, and conclusions or recommendations expressed are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.