A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 5: Assessing Program Impact

True Experimental Designs

Home

True Experimental Designs

True experimental designs compare people who have received an intervention ("treatment group") to an equivalent group who did not receive the intervention ("control group"). Subjects are randomly assigned to either the treatment or control groups; indeed, random assignment is the hallmark of the true experimental designs (a.k.a., randomized trials). In many circles, the randomized trial is the "gold standard" of quantitative research, reflecting its degree of methodological rigor.

"Rigor" refers to the degree to which evaluators can rule out alternate explanations for their findings. The true experiment, with its random assignment, allows evaluators to state with increased confidence that the intervention, and not some other factor, is responsible for the given results. What random assignment does (at least hypothetically), is to create equivalent groups, thereby controlling for the effects of other factors. Still, it will be necessary to monitor that equivalence to see if it is indeed true. This is the only design that allows researchers to specify cause-and-effect relationships (see Boruch, 1998; Fitz-Gibbons & Morris, 1987).

The true experiment, however, is not the only way to get at "truth." An example of this is in the relationship between cancer and cigarette smoking. For many years, cigarette companies hid behind the excuse that health officials could not "scientifically prove" that smoking causes cancer. In the framework of the true experiment, individuals would have to be assigned to "smoking" and "non-smoking" groups; obviously, this would have been quite unethical! Because researchers could not do this, it took multiple studies to account for all the other possible explanations (e.g., smokers may have a variety of other poor health practices that could account for higher cancer rates).

A similar problem arises in family violence research; researchers cannot assign children or adults to "abusive" and "non-abusive" families. In determining the relationship between family violence and other problems, evaluators must often account for other problems that might also be present in abusive families. In fact, a continuing controversy in child abuse research is whether the difficulties that abused children experience are due to child abuse per se, or a general level of family dysfunction that often co-occurs in abusive families (Kendall-Tackett, Williams, & Finkelhor, 1993).

True experiments can be difficult to use in family violence research. The random assignment element can also make them difficult to use in an evaluation setting. Often, administrators and other program stakeholders that seek an evaluation want to know if their on-going program is effective. Random assignment would mean starting an evaluation with a set of clients who are just beginning the program: some would be assigned to participate, and others would be assigned to a control group. This might be practical for a program that is short in duration (such as a parenting class). It is also doable for a more intense program, illustrated by the following example. However, before attempting a randomized trial of a secondary prevention program, it is strongly suggested that evaluators consult with individuals who have expertise in this area.

Example 5.1: Evaluation Using a True Experimental Design

An evaluation of a program using a true experimental design was undertaken in New Jersey, involving over 200 families (Feldman, 1991). Families who participated in the study had been screened and referred for services by state and local agencies, and were randomly assigned to either the Family Preservation Services (FPS) program group (96 families) or a comparison group consisting of 87 families receiving traditional community services.

The researchers looked at a variety of family functioning variables as indicators of program impact. A comparison of these variables after the program showed that the FPS families had fewer children placed out of home, and also that these placements occurred at a slower rate (i.e. occurred several months later, on average, after services were received, than for the control families). Using random assignment enabled the researchers to attribute these differences between the two groups to the FPS program. This is especially important in this example because the life of any family is complex and involves many events and influences that might lead to a change in functioning, regardless of program participation.

Pros and Cons of True Experimental Designs

The randomized experiment is the most powerful research design for determining whether a program is effective. Random assignment is not always possible or practical, however.

There is always the danger that the randomization procedure will not work, and that the groups are still actually different. This means that the program's impact is still unknown because scores might differ due to some other differences between the groups besides those that might be attributed to the program.
When the program in question involves health-related or prevention services, it is problematic (and sometimes unethical) to deny services to people who need them just so that they can serve as a control group. If issues are not pressing, using a wait-list control group is an acceptable and often-used solution.

Bottom Line: True experiments are likely to be quite impractical for evaluating secondary prevention programs. They can be quite expensive and difficult to implement, should not be attempted without the assistance of an outside consultant. Further, there are often easier and less expensive ways to gather the same information. However, a small-scale true experiment on a primary prevention program might be feasible.

Home