Section 1: Conducting an Evaluability Assessment

Step 3: Get an Overview of the Program

Once the audience has been identified and their needs understood, the evaluator can begin to learn more about the program under evaluation. Step 3 focuses on a general overview (additional program information will be provided at Steps 4 and 5 that follow).

In considering the program, the evaluator will need to determine what the essential program activities are, the mechanism or theory in which the program is grounded, and how program goals link to program activities. One method of getting the big picture is to develop a logic model, which is a graphic representation of linkages between program objectives and outcomes. Logic models are described in detail in Section 2 (Subsection "Specify the Prevention Mechanisms").

It must also be determined whether the entire program or just a portion of the program will be included in the evaluation. Furthermore, the evaluator will need to know the settings in which the evaluation is to be conducted (e.g., clinic, home, or other community sites). Another practical consideration is the availability of resources for the evaluation (Worthen et al., 1997). The evaluability assessment conducted by the UNH team identified "lack of resources" as a barrier to the implementation of USAF-FAP prevention programs. If program staff members are already over-burdened by their daily responsibilities, they may be less willing to participate in an evaluation, or cooperate with evaluators. Below are two issues that need to be addressed by the evaluator when preparing to conduct an evaluability assessment:

• Are there sufficient personnel and funding to conduct an evaluation?
• Are the deadlines reasonable? Is there sufficient time to conduct a thorough evaluation?

It is important to continue an evaluability assessment until enough information has been collected to make sound decisions about the evaluation possible. If important questions about the plausibility of various evaluation options have not yet been answered, the assessment should be continued until those questions have been answered (Wholey, 1979). Examples of such questions are:

• What is the program trying to accomplish? Are the goals and objectives explicitly spelled out? Are they plausible?
• Is there a program blueprint?
• How is this operationalized? Are program activities occurring as planned?
• Has the program been around long enough so that operational glitches have been resolved?
• Is there a mechanism for receiving rapid feedback on program components that are not working?

If a program blueprint exists, one way to detect areas that need improvement is by comparing the intended program model specified by the program blueprint with the actual program model (Pecora, Seelig, Zirps, & Davis, 1996). During the evaluability assessment, the evaluator should explore the blueprint as well as actual program activities to determine whether the program is operating as planned. When stakeholders involved in the evaluation reach consensus regarding the actual and intended program models (e.g., program goals, activities, and outcomes), an important milestone will have been reached. When the actual program deviates from its blueprint model, program improvements can be made, and/or the underlying model can be modified.

Below is an example from the UNH evaluability assessment of the United States Air Force Family Advocacy Program that specifies the sources of information used to get an overview of the programs under evaluation:

Thus, evaluability assessment results can be used to build and refine program models and program implementation to create clear relationships between program resources, activities, and expected outcomes (Wholey, 1987). In the following example, a program blueprint and program activities were compared during the USAF-FAP evaluability assessment conducted by the University of New Hampshire team.