A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 1: Conducting an Evaluability Assessment

Key Points & Introduction

Home

Key Points

Evaluability assessments provide an overall view of programs that help evaluators determine whether these programs are ready to evaluate.
Evaluability assessments can save time and money by helping evaluators to identify potential evaluation problems early, and by providing information that may prevent premature impact evaluations. Evaluability assessments help determine whether a program is mature enough to evaluate, the program is functioning as intended, or if program outcomes can be measured.
One of the first tasks is to determine who will be using the information that will be gathered, and what type of information they need.
Evaluators must determine what the target population is, what the essential program activities are, and how program goals link to program activities.
Evaluators can determine whether any of the information needed is already available by reviewing the empirical literature, reviewing existing documentation, and conducting site visits, but they may also need to collect additional data.
Typically, the evaluability assessment phase lasts an average of six to eight months.

Evaluability assessments are the preparation phase for a program evaluation, and take place before any other formal evaluation activities (Wholey, 1979). Evaluability assessments can save time and money by helping evaluators to identify potential evaluation problems early, and by providing information that may prevent premature impact evaluations. For example, an impact evaluation would be premature if the program model was still undergoing substantial change. During evaluability assessments, evaluators examine all aspects of a program in order to discover whether:

The program is mature enough to evaluate,
The program is functioning as intended, and
Program outcomes can be measured.

Evaluability assessments enable evaluators to construct a comprehensive description of a program and determine its level of maturity and stability (Smith, 1990). Rather than thinking of an evaluability assessment as a process that determines the success or failure of a program based on its evaluability status, evaluators can approach the evaluability assessment as a way to define the type and scope of evaluation activities.

Typically, the evaluability assessment phase lasts an average of six to eight months (Wholey, 1979). Serious investment in the evaluability assessment phase can prevent wasted resources on impact evaluations that are impossible to carry out, or that yield inconclusive results due to program immaturity or poor program implementation. The evaluability assessment should help the evaluator decide whether evaluation of the formative type (process) or the outcome type (impact) is appropriate. The evaluator will know it is too early to conduct a formal impact evaluation when any of these potential pitfalls are present (Wholey, 1979):

Lack of agreed-upon and/or measurable objectives.
Lack of plausible logic linking intended program activities to intended program outcomes.
Discrepancies between actual program activities and those specified in the standards or program model.
Inadequate or missing indicators of program performance.
Lack of structural support for program implementation (e.g., not enough staff to deliver program services).

Specific steps for conducting evaluability assessments of programs are described below. Lists of questions and issues to consider during an evaluability assessment are provided as well. These lists are meant to help the evaluator weigh the importance and significance of all the information collected, and make decisions on the future direction of the evaluation.

How to Conduct Evaluability Assessments

Evaluability assessments can be broken down into six steps. The order in which these steps are executed may vary. While it is important to complete all of these steps, the depth of the investigation may vary as a function of available resources and the type of information that is being collected. Evaluability assessments of community-based or primary prevention programs will generally be less involved than intervention programs that offer intensive services for families at risk.

Home