How To Use This Manual

In this guide, the basics of program evaluation are discussed in a sequenced format, with a nod given to family violence prevention and intervention programs. An evaluator may be called upon to evaluate many different programs. If an individual is new to evaluation, this task may seem overwhelming. This manual has been designed to guide the evaluator step-by-step through the process. The goal is to provide the evaluator with sufficient information to conduct evaluations in their own programs or participate in an evaluation conducted by others.

Throughout this manual, examples are used from the evaluation of the United States Air Force Family Advocacy prevention programs, conducted by the Family Research Laboratory at the University of New Hampshire, and the Child and Family Research Group, San Diego. This team used a multi-faceted and dynamic approach in evaluating these programs, with the ultimate goal of improving services offered to families by USAF-FAP.

Type of Programs That Are Evaluated

In approaching an evaluation, one of the first considerations must be the type of program that is being evaluated. There is a wide array of violence-prevention programs being offered. These vary substantially in complexity and intensity of service, but can be grouped into four major categories:

• Off-The-Shelf Programs. These are programs that are developed elsewhere and are purchased for implementation (or possible implementation) with program families. Examples of these types of programs include pre-packaged curricula for parent education, marriage enrichment, or stress management. These types of programs are generally developed with particular audiences in mind. They may or may not be appropriate for every population.
• Community-Focus Programs. These are programs aimed at changing attitudes within the community. These may include public service announcements, activities, and classes directed at the general public. The focus of these activities is creating awareness of family violence and promoting positive family functioning. These programs may also involve community members in designing programs and taking action on behalf of the community. The goal of these programs is to enhance the social networks of community members and connect them with helping agencies.
• Field-initiated Programs. These are programs that may have developed at a single site, designed to meet a specific community need. For example, if families live in a neighborhood with a high crime rate, these families might need to establish a neighborhood watch program. This program might be very specific to families at that location. Evaluation of programs developed to meet a specific community need can cover two issues: the program's effectiveness, and whether other sites can use this program.
• In-depth Prevention Programs. These prevention programs are for families designated as "at-risk," and have more intensive services than the programs described above. It is possible that the evaluator may need to hire outside contractors to assist in the evaluation of these programs. This manual provides sufficient information to help the evaluator determined if this is necessary and how the evaluator can remain an active participant in this process.

Three recently-evaluated USAF-FAP family violence prevention programs provide examples throughout this guide: First Time Parents (FTP), a new-parent home-based support program and its successor, the New Parent Support Program (NPSP); Home-based Opportunities Make Everyone Successful (H.O.M.E.S.), a home-based, multi-disciplinary program to promote Air Force mission readiness and reduce incidence of child maltreatment; and the Prevention and Relationship Enhancement Program (PREP), a marital enrichment program that is part of the USAF Family Advocacy Program maltreatment and family-strengthening initiative. Although these programs are frequently used as examples, the material presented can be used to evaluate other programs as well.

Unit of Analysis/Level of Focus

Another factor to consider is the unit of analysis. Is the focus on learning something about the program itself or is the goal to learn about the clients, or the community it serves? Too often, evaluators tend to focus on client/community outcomes to the exclusion of other important factors. While client or community outcomes are certainly keys to understanding the effectiveness of a program, their outcomes are by no means the only way an evaluator can develop understanding about and assess a program. Indeed, clients/communities are the unit of analysis for only one section (Section 5: Assessing Program Impact), where outcomes are the focus. Different aspects of the program itself are the focus of the other chapters. Section 1 focuses on program maturity and whether it is ready to be evaluated. Section 2 focuses on the rationale underlying the program: Is there a program model? Is the program model evidence-based? Section 3 has to do with the implementation of the actual program elements: How well does the actual program match the model developed in Section 2? Is the program running smoothly and delivering quality services? Section 4 focuses on the efficiency of the program in terms of financial resources: How much does it cost per participant? What is the cost of participating vs. the cost of not participating? Sections 6 and 7 will help the evaluator develop or gather the tools needed for both process and outcome evaluations.
If the evaluator is in the beginning phases of a project, the section on Logic Models (Section 2) might be the most helpful. If there are concerns about staffing and client satisfaction, the section on Process Evaluation (Section 3) most helpful. If a full-scale evaluation is being considered, but there is uncertainty about where to begin, the section on Evaluability Assessments (Section 1) might be the most helpful.

In this guide, seven different aspects of program evaluation are described. Below is a brief summary of each section. The manual can be read straight through, or on an as-needed basis. Each Section is designed to stand alone, but will also refer the reader to pertinent information in other sections. Below is a summary to help the reader quickly find information needed. At the beginning of each section is a "key points" box that summarizes the most important concepts. These boxes can provide further information about whether a particular section is relevant.

Section 1: Evaluability Assessment

In this section, information is provided to help the evaluator determine whether a program or individual site is ready for a formal evaluation. Indications of readiness for evaluation, potential pitfalls in evaluation, and the components of evaluability assessment are discussed.

Section 2: Logic Model Development

What is the theoretical rationale for the program? In this section, the evaluator is guided through the steps of logic model development. A logic model will help the evaluator make explicit links between program goals and specific program activities, or perhaps highlight areas where this information is still needed.

Section 3: Process Evaluation

Is the program delivering quality services? Are these consistent with the program model? Process evaluation will help the evaluator to determine this, and provide a means of on-going program monitoring.

Section 4: Cost Analysis

How much does the program cost per participant? How cost-effective is it? How is a cost-benefit analysis conducted? In an era of limited resources and funding cutbacks, focus must be kept on the bottom line.

Section 5: Assessing Program Impact

What type of design should be used for an outcome evaluation of a program? What is the scope of the evaluation? When is it appropriate to use quantitative methods? Or qualitative methods? The advantages and disadvantages of each approach are summarized in this section, and three qualitative approaches are described.

Section 6: Designing Data Collection Instruments

Once an evaluation design has been chosen, can the evaluator use existing data or does new data need to be collected? Coding data from existing records is described in this section, along with approaches to designing surveys and interviews.

Section 7: Outcome Measurement

What are appropriate benchmarks for the programs? How should outcomes be measured? This section provides detailed information on how to measure short- and long- term goals. Criteria for measurement selection are presented and domains of measurement are listed