Section 1: Conducting an Evaluability Assessment

Step 2: Selecting an Evaluator

One of the first decisions to be made is who will conduct the evaluation. Should the current program staff evaluate the program, or should an outside evaluation consultant be hired? The choice depends on the needs of the program and the objectives of the evaluation. A short-term evaluation may best be handled internally while an evaluation of large programs might require the assistance of an outside evaluator. This is one of many facets of a program about which a decision must be made. The following is a list of advantages and disadvantages for using internal vs. external evaluators.

• External Evaluators: An external evaluator is any individual not directly employed by the program under evaluation.

Advantages

• Objectivity: An external evaluator is likely to be influenced by knowledge of a program's history, operation, and particular objectives. External evaluators are also more likely to be free of personal interest in the findings thereby increasing objectivity.
• Expertise in a Particular Field: External evaluators often come from a variety of fields and provide expertise that members of an in-house staff might not posses.
• Staff May Be More Honest with the External Evaluator: Sometimes program staff divide into "warring" factions. In this situation, an outsider may have more success communicating with those groups without taking sides in the dispute. It should be kept in mind that an internal evaluator who has established relationships with the staff may accomplish this as well, however.
• Better Able to Present Unpopular Information: Because external evaluators have some distance from the program, it may be easier for them to present negative or unpopular information
• More Credibility with High-Level Stakeholders: An external evaluator may have more credibility with high-level stakeholders, particularly if the evaluator is someone with stature in his or her field, and the evaluation is rigorous and scientific.

Disadvantages

• May Need More Preparation Time: External evaluators may not be familiar with the particular program under evaluation and, therefore are likely to need more time to prepare. However, internal evaluators may encounter this as well if they are new to the program or at a distant site.
• Outsider to the Organization: External evaluators may have difficulties gaining the staff's trust and cooperation during the evaluation process. Similarly, internal evaluators that are either new or unpopular might encounter the same problem.
• Increased Cost: Evaluations by external evaluators frequently cost more because of travel expenses and greater lead time. However, an experienced evaluator may be less expensive in the long run because they may need less time at the site than someone who is new to evaluation research recruited from among the staff.

• Internal Evaluators: An internal evaluator is any staff person directly involved in the program under evaluation, or in the agency in which the program is housed. For example, a program manager could function as an internal evaluator.

Advantages

• More Familiar with Stakeholder Interests: Internal evaluators may have a better grasp of stakeholders' needs. However, a skilled external evaluator may also be able to gather this information relatively quickly. A study of internal and external evaluations in the Netherlands suggests that there is a higher rate of utilization of internal researchers' findings in evaluation studies. This may be due to higher rates of communication between internal researchers and program policy makers (Rossi & Freeman, 1993). Internal evaluators may be better able to use evaluation results to effect actual change in a program because of their familiarity with the program and its administration.
• Increased Efficiency: Internal evaluators may know more about the program and its staff, history, operation, and particular objectives that mean less time spent becoming acquainted with the program during the evaluability assessment and actual evaluation. However, an internal evaluator who is new or unfamiliar with a particular program may not have this advantage.
• Better Rapport with Members of the Program/Organization: Internal evaluators may be better able to use evaluation results to effect actual change in a program because of their familiarity with the program and its administration. However, there is a risk that program staff may resist changes initiated from among their ranks, and as a result may be more willing to "hear" these suggestions from an external evaluator.

Disadvantages

• Reduced Objectivity: Internal evaluators concerned about the impact of evaluation results on their job security may not be able to conduct an objective evaluation. This includes susceptibility to pressure from other program or organizational staff. However, external evaluators are not immune from these pressures either, especially if they fear their contract will be terminated if the findings are not to the liking of those funding the evaluation.
• Insufficient Time: If staff or program managers are expected to conduct evaluations in addition to other job responsibilities, daily responsibilities and evaluation tasks could be shortchanged. Resource considerations should include a pragmatic calculation of staff availability and staff burden.

The advantages and disadvantages of internal and external evaluators listed above are not absolutes. They are simply suggestions for consideration in deciding whom to choose to conduct an evaluation. All evaluators, whether internal or external, must monitor themselves for objectivity throughout the evaluation process. Ultimately, the decision about who should evaluate a program is up to each evaluator, but it is important to be aware of the costs and benefits of both approaches. Decision making about the appropriate selection of an evaluator is a prelude to conducting the evaluability assessment.