A Guide To Family Intervention and Prevention Program Evaluation - Step 1: Determine the Audience for The Evaluation

Date of Publication: December 2000

Section 1: Conducting an Evaluability Assessment

Step 1: Determine the Audience for The Evaluation

Home

One of the first tasks in conducting an evaluability assessment is to determine who will be using the information that will be gathered, and what type of information is needed. Because there are usually diverse audiences for evaluation findings, each may be interested in different aspects of the program being evaluated. Below are some questions that will help the evaluator determine the audience for the evaluation.

Who are the stakeholders?

Stakeholders are individuals with a stake (vested interest) in both the program and the results of the evaluation. Stakeholders include a wide variety of people: sponsors of the evaluation, program managers and staff, program clients and their families, other agencies working with the program, interest groups, elected officials, and the general public (Worthen, Sanders, & Fitzpatrick, 1997).

Example 1.1: Community Website

One proposed project sought to establish a website with information about available resources on a wide range of subjects including domestic violence, child abuse, stress management, and parenting information. The goal of this program is community empowerment, and increasing the safety net of resources available to families.

Who will use the results of the evaluation?

Among the stakeholders, only a few may be the actual audience that will receive the evaluation results. Determining who will actually use the results will help the evaluator determine what information they need. Knowing who will use the results will also help the evaluator prepare written summaries of the findings. Many stakeholders have the predominant role of advisor. They are not involved in the day-to-day activities of the program. These stakeholders will most likely need only an executive summary of the findings. A few stakeholders, however, will play a more active role. These stakeholders, more directly involved with the various hands-on aspects of the program, will probably want a complete report of findings and specifics pertaining to the hands-on aspects of the program.

Why do the sponsors of an evaluation want one? What are they trying to accomplish?

As Worthen and colleagues describe (Worthen, et al., 1997), decisions to evaluate stem from someone's need to know. The evaluator must determine what types of information they need, and what they plan to do with this information.

Should representatives from other groups be interviewed while planning the evaluation?

Another important planning step is to determine whether members from any other groups need to be interviewed. While it may not be possible to interview "everyone," it is frequently helpful to gather information from those outside the program work with it on a regular basis, or who may be affected if changes in the program might result from the outcome of the evaluation. As in the example of the community website (see above), community agencies can also be included, as well as program clients and other consumers.

The results of an evaluability assessment may be used "in-house" to help in program development. They may also be used in progress reports to stakeholders and funding agencies, and later in articles describing program development. Below is an example of how comments and concerns of stakeholders were integrated into an evaluation of United States Air Force FAP programs.

Example 1.2: USAF Family Advocacy Programs Application

The perspectives of multiple stakeholders were incorporated into every stage of a University of New Hampshire evaluation of the three family violence prevention programs being used by the USAF Family Advocacy Program (FAP). This project was launched by conducting focus groups with field staff from three USAF Family Advocacy prevention programs selected for the evaluation assessment. The goal of the focus group interviews was to obtain a sense of Family Advocacy Program staff perceptions of their program models and procedures, and of barriers and challenges in implementing the three programs.

Next, the evaluators reviewed relevant USAF-FAP documents, and then conducted interviews with key informants in order to gain an understanding of USAF-FAP Division leadership views about Family Advocacy and the mission and goals of USAF-FAP. The evaluators also interviewed program administrators and staff during base site visits. During the implementation phase of the evaluation, the evaluation team expanded its contacts to a wider range of important stakeholders including fathers whose wives were the primary recipients of services from the First Time Parents Program, and military leadership such as commanders from various squadrons, first sergeants, chief nurses, and a USAF chaplain (also a recipient of USAF-FAP services).

A USAF-wide survey of FAP staff was administered as one of the last Phase I evaluability assessment activities conducted by the evaluators. This was done because field staff are key stakeholders and their input was considered very important. The base survey was administered near the end of the evaluability assessment so that survey questions could be grounded by information gathered from USAF-FAP staff during earlier activities (focus groups, key-informant interviews, and site visits). The survey elicited more detailed information from staff members concerning their perspectives on the goals, barriers, risk assessment and outcome measurement strategies utilized at the bases. In addition, target group information was developed by asking staff to describe family profiles characteristic of client groups currently utilizing each program. Finally, input from consultant partners was facilitated by a feedback system that included written and e-mail communication as well as periodic meetings with the entire evaluation team.

An evaluability assessment can also facilitate communication between evaluators and multiple stakeholders (Worthen, et al., 1997). For an evaluability assessment to be most effective, stakeholder interests must be an integral part of the process. When stakeholder interests are well integrated into the assessment and evaluation, chances of reaching consensus on mutually beneficial evaluation tasks and options should be greatly enhanced. In turn, this process should increase the overall usefulness of the evaluation (Rossi & Freeman, 1993). Furthermore, discussing results of the evaluability assessment with stakeholders affords them the opportunity to be engaged in follow-up decisions regarding results of the assessment. This would include all stakeholders in appropriate ways, from clients to program staff, administrators, support services, and policy makers.

Home