External Review of DRK–12 Projects

Catherine McCulloch (CADRE PI), Education Development Center (EDC)

Thank you to evaluators Kristin Bass, senior researcher, Rockman et alKathy Haynie, director, Haynie Research and EvaluationDan Heck, vice president, Horizon Research, Inc. (HRI); and David Reider, principal partner, Education Design, Inc. for their perspectives on this topic. 

Each year when researchers are writing proposals, CADRE—the resource network for NSF’s DRK–12 program—receives requests for clarification on the use and choice of advisory boards and/or evaluators. The solicitation for the National Science Foundation’s Discovery Research K–12 (DRK–12) program states that “all DRL projects are subject to a series of external, critical reviews of their designs and activities (including their theoretical frameworks, any data collection plans, analysis plans, and reporting plans) … A proposal must describe appropriate mechanisms to assess success through project-specific external review and feedback processes. These might include an external review panel or advisory board proposed by the project or a third-party evaluator. The external critical review should be sufficiently independent and rigorous to influence the project's activities and improve the quality of its findings.”

When and how do you use one entity versus another to inform decision-making during a research project? CADRE contacted several DRK–12 evaluators to get their perspectives on this question, and this is what we heard:

There is overlap between advisory boards and external evaluators. Both can provide input and feedback about project activities. But there are differences between advisory boards and external evaluators that may help you determine which to use.

An advisory board provides nonbinding strategic input and feedback about project plans, activities, and results to a project team. While board members are typically research peers, increasingly members are drawn from target audiences (e.g., teachers) so that those perspectives are integrated into project decision-making and results are relevant to those stakeholder groups. An advisory board tends to meet once or twice per year and may be—by nature of its size (i.e., number of advisory board members), the level of compensation provided, and the frequency and duration of the members’ engagement—a less expensive option than working with an evaluator over the duration of the project.

A well-chosen advisory board can ensure the quality of a project’s findings by leveraging diverse experience and targeted expertise to help frame or situate the project’s work relative to other work in the field; review research design, methodology, or analytic approaches (e.g., recruitment and sampling strategies, data collection instruments) best suited to answering the project’s research questions in a particular context; and/or provide insights about key audiences and outlets for dissemination. In some cases, projects work with an external review panel that reviews a project’s major design elements and products, rather than providing input on project activities at the outset of the research in the way that an advisory board does. However, the use of external review panels as a wholly different entity from an advisory board is less established in education research than other fields.

The external evaluator can provide objective input and feedback throughout a project—from the early design stage through final reporting and dissemination efforts—to ensure that the project addresses its intended goals and maintains the quality and integrity of its operations. Evaluators may help to refine a project’s theory of action and logic model; frame evaluation questions; determine evidence sources; collect and analyze evidence about the projects’ processes, products, and results (which may supplement and/or critique the research the project is doing); communicate results; and make recommendations. Key to their role is their ongoing contact with the project team and engagement with the context of the project work, which allows the evaluator to straddle internal and external perspectives on the work. An external evaluator can also bring deep knowledge of evaluation within a particular disciplinary field and experience with the lessons learned by other projects. Evaluators often bring to bear experience with what has worked and what hasn’t worked for a number of other projects they have evaluated.

Projects with a small budget will need to limit the type and scope of external review. An advisory board may offer the expert advice and knowledge you need. Or, if you choose to work with an external evaluator, it may be just one person with more-limited expertise than offered by a full advisory board. Additional capacity can be added by leveraging data sources from the research and/or an internal evaluator (who alone may not offer the credible objectivity that an external reviewer does or fulfill NSF requirements for external review, but may offer an understanding of the project team and context). An external evaluator may collect data and/or share data that the project team and researchers have collected, though the evaluator’s analysis is conducted for a different end (i.e., to inform the decision-making of the project team or evaluate the value of the project processes, outputs, and outcomes).

For projects that have larger budgets (and likely, larger scopes of work), working with an advisory board and external evaluator can yield all of the advantages of both approaches and provide significant expert input, formative feedback, AND summative evaluation of the project processes, outputs, and outcomes. For instance, external evaluators may aid projects in making the best use of an advisory board by providing updates on project progress and outcomes for the advisors, reviewing or shaping questions or issues to which advisors will respond, and providing an external viewpoint on how advisors’ input and feedback might inform project decisions. An evaluator can provide a holistic view of the project since they may be familiar with the project and its development over time, and still bring an objective and outsider view, which can be beneficial to interpreting advisory board insights and recommendations.

Whichever input and review structures you choose, plan early in the project design phase how you will leverage informed advice and evidence-based feedback and recommendations to support stronger research design and implementation strategies, iterative improvements and mid-course corrections, greater intellectual merit, and broader impacts. And whether working with an advisory board or an external evaluator, consider reviewing and/or refining your theory of action together to forge common understandings. Identify key assumptions, intended actions and means of producing them, expected interim results and means of assessing them, feedback loops, and desired outcomes and impacts and means of measuring them. Your advisors/evaluators can help you anticipate barriers and create opportunities (or seize on unanticipated opportunities), and redirect efforts as needed.

More CADRE evaluation resources, including Evaluation in DRK–12 Projects: Options, are available at cadrek12.org.

CADRE is funded by the National Science Foundation, grants #0822241, 1449550, 1650648, 1743807, and 1813076. Any opinions, findings, and conclusions or recommendations expressed in these materials are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Download PDF Version of Blog