A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 5: Assessing Program Impact

Quasi-Experimental Designs

Home

In many instances, random assignment is either not possible or not the most appropriate design to use. When that is the case, the evaluator can use a quasi-experimental design instead. Often, people are assigned to groups using a procedure called "matching." Clients receiving services are paired with people who are similar (e.g., by age, ethnicity, or marital status, etc.) but who are not receiving services. These designs are called "non-equivalent," referring to the lack of random assignment. Matching is the procedure used to create some equivalence. If people are in groups that cannot be broken up (e.g., a family), then matching can be done for groups rather than individuals. Some simple statistical analysis (e.g., Chi Square, t-test, Analysis of Variance, etc.) will help determine if sample groups are roughly equivalent, or if there are significant differences, before treatment even begins (see King, Morris, & Fitzgibbons, 1987; Mika, 1996; Reichardt & Mark, 1998).

Following are some examples of quasi-experimental designs that might be used:

Non-Equivalent Control Group Design with Pre- and Posttest.
Evaluations of new programs in multi-site agencies might be suited to this design. Sites not initially receiving the program, but which have the same primary focus and general location as the program sites, serve as controls. These control sites could be matched with program sites on several other characteristics that they have in common (e.g., stateside, near an urban area). The program sites should also be matched to control sits with families that are similar demographically. Care should be taken when using this approach, however. Families selected from different sites may differ in many ways that have nothing to do with the program. The most typical approach is to select some key demographic information (such as age range, education level, ethnicity, marital status, or number of children) to compare the two groups. Consultation with a statistician may help to determine whether the program and control groups are appropriate to compare.

An important point is that data must be collected in the same way for those receiving the program as for those in the comparison group.

Time-Series with Non-Equivalent Control Group. This design is identical to the one above, except it involves more than just one assessment at pretest and posttest. "Time series" simply means that data are collected at several (at least 3) time points before and after the program is implemented. The advantage to this approach is that the evaluator will get several "looks" at what is happening with each group before, after, and even during the program allowing for a greater understanding as to which changes are due to the program, and which changes are not.

Example 5.2: Non-Equivalent Control Group Design with Pre and Posttest

In a plan to evaluate the proposed United States Air Force FAP New Parent Support Program (NPSP), families would first be assessed for service needs and risk potential for family maltreatment. Then, each family would be identified as a "high needs" or a "low needs" family, based on several standard measures of family functioning. It would be important to design measures for both. High-needs families would be offered more intensive home visitation services, whereas low-needs families would be eligible only for more standard educational and support services. Both high-needs and low-needs families would be re-assessed at a point after which most families have terminated services (one year) as to their current level of needs, stressors and family functioning.

The primary objective for the evaluation of the NPSP would be to determine whether the program reduces risk for abuse among high-needs families. Other comparisons could also be made between high- and low-risk families, and the relative impact of the respective services offered to each. The evaluation would allow an assessment of the stability of risk (needs) levels by looking at the patterns of changes in level of risk over time (see "time-series" designs for information on repeated assessments).

This is a good example of a quasi-experimental design in that families would not be assigned randomly to groups, but rather they would be assigned services on the basis of needs level. Having data both prior to the NPSP as well as after would allow for some conclusions to be drawn about the program effects. Families deemed high-needs who decline or do not attend services may be available to be part of yet another type of comparison group.

Example 5.3: Time-Series with Non-Equivalent Control Group

One large study by the Center on Child Abuse Prevention Research (Daro, Jones, McCurdy, George, Keeton, Downs, & Thelen, 1992) aimed to investigate the relative impact of 14 child abuse prevention programs in place in the greater Philadelphia area. It would have been completely impractical, and also prohibitively complicated, to recruit individuals and randomly assign them to programs. Instead the researchers tracked clients who were already receiving one of the programs, and compared outcomes.

The researchers were trying to measure rates of child abuse behaviors. This is a difficult and complicated task, as it occurs sporadically, and family members are reluctant to report behaviors. The best assessment strategy here was to use many assessment points over time, i.e., a time-series design. In the study, every client receiving services completed the Child Abuse Potential Inventory (CAPI). They also provided information on their level of satisfaction with services and demographic information at five separate one-week periods throughout the three-year evaluation. In this way, the evaluation families became accustomed to the assessments, and researchers got a "snapshot" of client characteristics (likelihood of abuse, demographics, participation patterns) at regular intervals.

Pros and Cons of Quasi-Experimental Designs

Pro: Quasi-experiments are close approximations of true experimental designs and can be used to investigate relationships between factors.

Pro: Quasi-experiments are the best option when random assignment is not practical, or does not make sense given the target population, resource constraints, research questions or ethical considerations.

Con: Because experimental and control groups are not formed by random assignment, there may be reasons other than the program interventions that explain differences between groups.

Bottom Line: Quasi-experimental designs are amenable to many types of prevention programs. Again, the evaluator must consider some of the resource considerations described with true experiments. "Big" quasi-experimental designs are going to be more expensive and more intense than "small" quasi-experimental designs (see below). They will also be easier to implement in primary prevention programs than secondary prevention programs. For assessment of secondary prevention programs, the assistance of an outside consultant is recommended.

Home