NSF Workshop on Implementation Evaluation

The Future of Implementation Evaluation:
Towards a Science of “Continuous Improvement Research in Education”
A Day-Long Workshop Sponsored by the EHR Evaluation Group

Monday, October 28
National Science Foundation
Arlington, VA
Room 375

Outcome evaluation looks at results. It measures the direct effects of project or program activities on targeted recipients. But results don’t tell the whole story. Evaluation that only focuses on outcomes is sometimes called a “black box” evaluation because it does not take process or implementation evaluation into consideration. Outcome evaluation alone won’t provide information about why a program did or didn’t work. These are implementation evaluation questions. Disappointing outcome evaluation results can frequently be illuminated by examining how the treatment was implemented. There are other equally important benefits too. A focus on implementation integrity also allows researchers to develop interventions that can withstand variation in treatment settings, to design studies that better account for such sources of variation, and to change the treatment itself so that the intervention achieves maximum effect. This workshop provided an overview of issues to consider when assessing the methodological strength of EHR proposals that could benefit from looking inside the “black box” of implementation.