A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 5: Assessing Program Impact

Key Points & Introduction

Home

Key Points

Assessing impact of a program involves decisions about experimental design, scope of the evaluation, and evaluation methodology.
Design refers to decisions that are made about comparisons groups, random assignment, and timing of assessments. A pre- and posttest, a measure of process, and a measure of the population being assessed is recommended.
Scope refers to the size of the evaluation: number of subject/sites, and depth of the assessment. Larger studies have a large number of subjects and sites, and many assessments. They may require outside assistance.
Methods refer to the techniques that are used to gather data. In this Section, qualitative techniques including open-ended interviews, focus groups, and direct observations are described.
All the above elements must be considered when approaching an impact evaluation.

In this section, how to conduct an impact evaluation of a program is described. In the past, family violence prevention programs have been criticized for their inability to demonstrate their effectiveness. At issue was the poor quality of the research or evaluation design (National Research Council, 1993). These concerns will be addressed by providing information on effective evaluation designs.

This section is intended to provide an introductory guide to elements of design that are most likely to be helpful to the evaluator. Entire books have been written about many of the single topics discussed in this section. Therefore, a list of recommended readings, referring resources containing more in-depth or detailed coverage on a number of topics is provided at the end of each sub-section.

The goal is to provide a guide that will help the evaluator select a data collection method that best suits the needs of the program under evaluation. The impact evaluation will be shaped around the important questions that need to be investigated. If it makes more sense to use a hybrid of two or more designs or methods, then that is perfectly acceptable and probably a good idea. It is not imperative that an evaluator follow any one particular design presented here exactly as it is described if it does not fit well with project goals.

The following discussion is divided into three elements of impact evaluation: design, scope, and methods. Design refers to decisions made about comparison groups, random assignment, and timing of assessments. Scope refers to the size of the evaluation: number of subject/sites, and depth of the assessment. For any of the designs and methodologies described, there are "small" and "large" versions. The final element is methods: the techniques that are used to gather data. Briefly described are both quantitative and qualitative methods, and specific instructions for three qualitative techniques. (Quantitative techniques are the focus of Section 6.) For design and scope, worksheets are also provided to help the evaluator focus in on the information needed from this section.

Home