Section 5: Assessing Program Impact

Critical Elements of Impact Evaluation

Evaluation Design

The first element described is evaluation design. The evaluator begins by specifying the general goal of the evaluation. If measurable outcomes are not present, refer to Section 2, Subsection, "Identify Program Goals, Objectives, and Outcomes", before proceeding.

• What Information is Being Sought?

It is important to formulate specific and comprehensive evaluation questions. A logic model provides a scheme that an evaluator can use to test the degree to which each intermediate and longer-term (proximal and distal) outcome has been achieved. Generally the questions take the form of how much impact the program has had on clients (e.g., how much change has occurred) either within the group of program families alone, or compared to families who received few services, no services, or some alternative services. This approach assumes that impact (or change) is measurable by some yardstick such as assessments of child well-being.

• Will There Be a Comparison Group?

One important way to show that a program has been the cause of a family's gains is to include some type of comparison group in designing the evaluation. Doing so can increase confidence in the program's efficacy. For example, it can be expected that many new or recent parents initially experience high levels of stress related to their parenting status. One might expect a decline in family members' stress level as they become more experienced in caring for a new baby. Such reductions in stress can be due to the passage of time as well as to parental and infant maturation rather than to program effects. In other words, without some comparison group of parents who did not get program services, one cannot rule out alternative explanations of change. Other than the course of time, these explanations can include many factors such as the characteristics of parents, the nature of their presenting problems, and the dimensions of the interventions provided.

The issue of comparison groups will influence the type of design that is appropriate. Each of the questions listed below represent possible design strategies.

• Will the comparison group be outside the program?
• Will group members be compared to themselves?
• Will group members be compared to those receiving services from an alternative but similar program?
• Will groups be divided based on amount of services they receive (i.e., measuring "dose" effects)?
• Will participants be randomly assigned to treatment vs. control groups?

• What Is The Timing Of Assessments?

Another design issue is the timing of assessments. Timing issues are dictated by intensity of the program, logistics of data collection such as access to families, demands of particular measures (e.g., developmental milestones), and flow of cases into program services. For example, how often will the evaluator have access to the target population (and hence, the sample)? When the population is very mobile, there needs to be a systematic means of locating families.

If families participate in primary prevention efforts, it is unlikely that there will be opportunities for assessment during the course of the program. Many of these types of programs can be completed in one day, one or two weekends (e.g., some marriage enrichment programs), or over a period of a few weeks (e.g., parenting classes, or stress-management programs). However, even with these less intense programs, the evaluator may be able to collect pre- and post-program data. It is possible to obtain data that would measure long-term impact one or two years later, provided information has also been collected to assist in locating these families.

For more intense programs, there may be opportunities to collect data not only before and after, but during the program. The evaluator must factor in the amount of time it will take to collect this information as part of an estimation of resources available and scope of the evaluation (see below). Depending on the size of the anticipated sample and case flow, each period of data collection could take anywhere from a few weeks to over a year. For example, if the evaluator wants to assess families after they have been in the program for one year, data cannot be gathered from any of the sampled families before they reach that mark. If these families were enrolled in the program over a period of six months, it will take at least six months beyond the one-year point to finish gathering all the data. Below are some basic assessment-timing categories.

• Assessment of participants at program entry, or before the program starts, to provide information on how participants perform, or what they are like, without any treatment (pretest measure).
• Assessment of participants after completing a program (posttest measure).
• Assessments that occur during the program, and/or at specified intervals after program completion beyond the posttest measure (multiple- and/or longitudinal measures).