Section 1: Conducting an Evaluability Assessment

Step 4: Determine Types and Amounts of Data Currently Available

In conducting an evaluation, it is wise to consider whether any of the information needed is already available. There are several ways to do this, including reviewing the empirical literature, reviewing existing documentation, and conducting site visits. Each of these steps is discussed below.

• Review relevant literature.

At some point early in the process, the evaluator must become familiar with the literature relevant to the program under evaluation. This will assist the evaluator to:

• Evaluate whether the program is grounded theoretically.
• Assess measures being used, including their quality and appropriateness.
• Become familiar with information on the latest evaluation efforts.

• Review program documentation

It might be possible to use information that already exists in the evaluation. Reviewing program documentation is an important early step. The evaluator can begin by collecting and reviewing documents that describe the program's mission, goals, objectives, and intended outcomes (Smith, 1989). It will be important to gather information and background on the program as well as information about individual clients and families. Descriptive information on a program may come from administrative regulations, funding proposals, administrative manuals, annual reports, reports from prior evaluations, or those of similar programs in the same organization. Previous evaluations or reports of the program should be located, if possible. Perhaps field staff or program administrators have conducted informal surveys of client groups or received client satisfaction feedback that could be made available. Since the purpose of the evaluability assessment is to obtain real insight into the program theory and practice, it is useful to look for consistent themes and patterns in data sources. Below is an example of sources and types of information the UNH team gathered from existing records.

From a thorough review of the records, the evaluator will be able to learn about the history of the program, how long it's been in existence, changes that may have taken place, as well as its staffing patterns and interactions with other organizations. Some of this information may become important in the evaluation. For example, suppose that an element of an intervention program was dropped due to funding constraints. There may be enough information available to determine whether this missing element has affected clients. For instance, for individuals and families participating in various prevention components, there may be pre- and post-class measures of these variables. Other types of program information could include articulation of program goals, description of curricula, and whether program providers can document how many classes each family attended (an indication of "dose").

For families in more intense secondary-prevention programs, data available could include prevention files, intake records, and scores on standardized instruments. The quality of this information can range from poor to excellent. Information about basic demographics (e.g., age, marital status, gender, number of children, and ethnicity) is helpful in describing the client population. The evaluator may have measures taken at intake that can be used for comparison upon completion of the program. Case reports may also be used to document and code "intensity of service," by noting how many times clients had contact with program staff.

Available documentation on families may or may not be available for the evaluation. The following issues should be considered:

• Do the records appear to cover the variables of interest?
• Has the information been collected in a non-biased way (e.g., with all eligible program families vs. only with more compliant families)?
• Are there records for most of the families in the program?
• Are the existing records complete or is information missing?

• Conduct Site Visits

If the evaluability assessment is not conducted in-house or by internal staff, the evaluator may still observe first-hand how the program operates by conducting site visits. This will allow the evaluator to integrate observations with documentation about the program and with interviews to develop an accurate description of the program (Rossi & Freeman, 1993).

During site visits, an important task is to interview program personnel about their perspectives on program goals, objectives, and implementation (Rossi & Freeman, 1993). In order to maximize the usefulness of the site visits, a variety of people should be interviewed, including key staff, board members, clients, and stakeholders. The evaluator can also observe family contacts and home visits.

Site visits frequently have a "best-foot-forward" quality to them. To moderate the latter, enough time should be allowed for observations of more typical operations and for conducting individual interviews, in addition to the special activities and meetings that can occur during site visits. By shadowing multiple staff members across repeated key activities, the evaluator will be better able to assess any difficulties that might exist in the program. Multiple visits and observations at different program sites give the truest picture of the program such as staff presenting questionable interventions as their "best practice." Furthermore, these visits will give the evaluator an opportunity to get a sense of the political climate at the site and whether personnel will cooperate with (or resent) the evaluation.

Because many sites may be involved in the program evaluation, it may be necessary to visit a variety of them to fully understand how the program is working in the field. In selecting sites to visit, diversity of geography along with variation in program emphasis and maturity are important. Many differences in program implementation will be apparent among the various sites. Below is an example of how air bases were chosen by the UNH team to participate in the USAF-FAP evaluability assessment: