A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 5: Assessing Program Impact

Non-Experimental Designs

Home

Non-experimental designs do not have comparison groups. These designs are used to assess program impact when there is no control or comparison group available, and usually involve time-series measurement. The soundness of these designs is affected by the number and timing of measurements. Non-experimental designs can be used for process evaluations when the primary purpose is to describe participants' experiences. One of the best ways to determine the process of change is to compare the groups performance or level of change to a baseline level established before the program began.

Types of Non-Experimental Designs

Single Group Pretest-Posttest Design. This design compares the same group of participants before and after the program. The purpose of the single group pretest-posttest design is to determine if participants improved after receiving the program. This design will not indicate, however, whether a program caused improvement in participants; there is no way to distinguish between changes over time due to other factors and effects specific to the program.
Single Group Time-Series Design. This design is similar to the time-series control group design described previously, except that no control group is used. It is said that subjects act as their own control group, in that comparisons are made between different time points for each person. This design is used to look at changes over time. The evaluator can collect data at regular intervals (e.g. daily, monthly, weekly) or at staggered time points (called "interrupted" time-series design).

Example 5.4: Non-Experiment Design for aTeen Pregnancy Prevention Community Outreach Program

The United States Air Force FAP Teen Pregnancy program uses infant and pregnancy simulators to communicate the costs of teen pregnancy and parenting. Included in the program is training in the following areas: self-esteem, values clarification, healthy relationships, intimacy without sex, and assertive communication. The goal is teen pregnancy prevention. It will be offered to youth group leaders and squadrons on air bases, and later in the schools.

This program could be evaluated using several scenarios. With a single-group posttest design, rates of teen pregnancy could be calculated for program recipients, and compared to prior year rates in the same school. Comparing pregnancy rates to those of same age teens in demographically similar community schools would be an appropriate comparison.

Pros and Cons of Non-Experimental Designs

Pro: Single group designs are easier to implement, and less expensive than experimental, or quasi-experimental designs.

Pro: These designs can serve as pilots, and help identify important variables related to success in the program.

Con: Often participants are likely to improve over time without intervention of any kind, and these changes can be mistakenly attributed to the program under evaluation.

Con: Other events can change (e.g., a new community project is offered)

Bottom line: These designs are the easiest to implement, but the most difficult to interpret. While evaluators can avoid some of the issues involved in finding a matching comparison group, it may still be difficult to account for outside factors that may be responsible for the given results (e.g., the simple passage of time). However, for certain programs, a non-experimental design may be the only option. The evaluator may find this design helpful in the beginning phases of a project to assess needs within a community, or to get a general idea of what type of program might be helpful. Again, caution must be exercised in reporting and interpreting the findings, and in avoiding undue inferences about the "effects" of the program

Home