Section 1: Conducting an Evaluability Assessment

Step 6: Making Sense of the Evaluability  Assessment

Once all the information described above has been gathered, a formal program model description is created. This allows comparison between the intended program and the program that is actually being used. The program description further assists the evaluator in determining whether the program is likely to achieve its goals and objectives as stated. The evaluator will then need to make a decision about whether or not this program is ready for a formal evaluation. Worthen et al. (1997) have described four steps that will assist the evaluator to determine whether he or she should proceed with the evaluation.

• The evaluator should clarify the theory or model that underlies the program (described in greater detail in Section 2, Subsection "Identify Program Goals, Objectives, and Outcomes").

Before proceeding to the next step, it is important for the evaluator to have a firm grasp of what the model is and why it is supposed to work.

• The evaluator should examine the program currently in use, and determine whether it matches the model. In addition, the evaluator will need to determine whether it is plausible for the elements of the program to achieve the specified goals. Are the program objectives well defined? Does the program match its model? Could the activities of the program reasonably be expected to help clients attain the program goals?
• The evaluator should consider whether the proposed evaluation approaches are feasible and will meet the information needs of the stakeholders. Is the proposed evaluation going to be too burdensome or difficult for the staff to implement? Have the stakeholders reached some type of consensus on the information they need and will the proposed evaluation meet that need? Are the intended uses for the evaluation results well defined?

The evaluator should determine the evaluation priorities and how the information will be used.

Who will receive a copy of the evaluation findings? Will this information be used to implement changes in the program, in personnel? How much input will stakeholders have as to the uses of the study?

Several factors may indicate when it is inappropriate to proceed with an evaluation. The evaluability assessment may reveal that stakeholders cannot agree on the program model or an evaluation plan. The program may differ significantly from its original model. The program goals may not be consistent with current program actions. Or there may not be enough resources to evaluate the program (Worthen, et al., 1997).

Other factors may indicate that the program is not ready to be evaluated. For example, the program administrator or stakeholders may have already made a decision about a program that will not be changed with evaluation results. In this case, an evaluation would waste everyone's time and is not worth pursuing. Another possibility is that the sponsors of the evaluation may be asking for something that is unrealistic in terms of both scale and timeline. For example, they may ask for results within the first month, which would not allow sufficient time for data collection. Any data collected on such short notice is likely to misrepresent the program under evaluation. These would be legitimate reasons for not going forward with an evaluation (Worthen, et al., 1997).

In Figure 1.1 at the end of this section, a worksheet is provided to assist the evaluator in assembling all the information collected, and provide guidance about how to proceed next. This worksheet refers to various sections of this guide where additional information can be found. This information-collecting worksheet is designed to help pinpoint the parts of the program that may need to be corrected before an evaluation can proceed.

Making Decisions: Process or Outcome Evaluation?

Once the evaluator has the evaluability assessment findings in hand and feels that sufficient information has been obtained, the evaluator will need to decide on the next step. Assuming the decision is to move forward on an evaluation, the evaluator will also need to decide whether to pursue a process or outcome evaluation, or some combination of both. If the program has an ongoing education or treatment component, the evaluator may also want to start a process evaluation to monitor delivery of services and ensure that all information is collected in a systematic way.

If outcome evaluation is the ultimate goal, the evaluator will have to determine whether the program is sufficiently mature, and whether the outcomes of interest can be measured in some reliable and meaningful way. Hopefully, the target program has been planned with well-defined and clearly measurable objectives, objectives that have been agreed to by program planners and administration and others involved in the implementation of the program being considered for evaluation. One should also be able to identify intended activities designed to meet the objectives, and causal hypotheses linking means and ends.

If it is discovered that the program under evaluation needs improvements in its design or implementation or that the program does not operate as it was designed, it is best to postpone an outcome evaluation. If the evaluator attempts to conduct an outcome evaluation while the program is still establishing stability, and before it has had time to accomplish longer-term objectives, the program will not have had a fair chance to demonstrate the desired outcome (Smith, 1990). In any of these circumstances, a process evaluation may be more appropriate and can provide useful feedback to improve the program.

Summary

In this section, the purposes and products of evaluability assessment have been discussed, and the steps involved in the evaluability assessment process have been outlined. A variety of questions have been presented to assist the evaluator with designing an evaluability assessment. The evaluability assessment has also been presented as a part of the evaluation process and not something that determines program success or failure in its own right.

In Section 2, the program logic model is discussed as a tool that can be used in evaluation. The steps for building a logic model are delineated, and procedures for using the logic model in evaluation work are described.