A Guide To Family Intervention and Prevention Program Evaluation

Date of Publication: December 2000

Section 3: Doing Process Evaluation

Target Population

Home

A logical place to start a process evaluation is with the target population. There are two points where the population in the program is subject to bias. The first is associated with their entry into the program (recruitment). The second refers to whether they stay in the program (retention). Recruitment and retention are described below.

Recruitment

With regard to recruitment, the first task is to determine whether there is appropriate outreach to all segments of the target population. In Section 2, the suggestion is given for the evaluator to make a list of the characteristics of the target population, and describe how the program addresses the issues that are relevant to this group. (If such a list does not exist, review Section 2 (Subsection "Specify the Target Population.") Knowledge of the target population can also help the evaluator to develop eligibility criteria. Say, for example, that the program is specifically geared for couples who are recently married. The evaluator might establish a length of time (e.g., less than three years) married as a criterion. Couples who have been married for more than that length of time would be ineligible to participate.

The makeup of the target population will vary from site to site. The evaluator will want to know if the population actually participating in matches the target population specified in the program design. Program monitoring requires that this baseline descriptive information be collected in such a form so that profiles of program participants can be compared between sites (in a multiple site program) and over time within sites. For instance, in United States Air Force prevention programs, an evaluator would need to consider the proportion of families living off-site vs. on-site. This evaluator may also need to know the proportion of families using off-site health care providers vs. the medical treatment facility for their primary sources of health care. On Worksheet 3.1 are listed some of the demographic characteristics of clients (or communities) useful in comparing the actual population to the target population. Below are some more general questions relevant to the recruitment of program participants.

Are all eligible families being approached about participating?
- Do individuals actually in the program differ significantly from those who were eligible but not recruited?
- Do all sites have specific inclusion/exclusion criteria and a protocol for the first contacts with families?

Below is an example of how a process evaluation revealed a higher level of participation among Caucasian than minority youth.

Example 3.1: Recruiting Bias in a Program for Runaway Youth

One service for runaway youth noticed that most of the program participants were caucasian and that few minority runaways used program services. Providers were concerned because a substantial percentage of runaways in this city were minorities. The program addressed this problem by hiring a more culturally diverse staff and targeting outreach specifically to these minority teens (Davis & Savas, 1996).

Thus, an important key to limiting bias in recruiting is a systematic program of outreach.

Outreach Efforts

Program outreach is the first contact that most families will have with a program. This is often where efforts to reach families break down, and is a good place to monitor for potential problems. One problem may be that program recruitment efforts are not reaching the people targeted. The example below describes how a process evaluation of program recruitment worked to improve responses to a teen parenting class.

Example 3.2: Recruitment for a Teen Parenting Education Program

Classes for pregnant or parenting teens were sparsely attended. The only attendees were women, even though the class was specifically intended for fathers. The concerned program manager initiated a process evaluation, and discovered that the advertising flyers were placed only in the women's restrooms. The manager changed the flyers to reflect a more male orientation and posted them in the men's restrooms and locker rooms (Aultman-Bettridge, 1998).

When thinking about outreach, the evaluator will need to consider whether the program is accessible to the target population. Families may be lost at this very first step unless attention is paid to issues of accessibility and addressed appropriately. Has some element of the program made it difficult or impossible for certain families to participate? Below is a partial list of potential barriers to accessibility.

Language
Does the intended population have a limited grasp of English, or is English a
second language? If so, are program materials available in other languages? Is the level of language in flyers and other recruitment materials written at too difficult a level for some clients (even native English speakers)?
Finances
Does participation in the program require a fee? Could this be waived if families
were unable to afford the program? What types of costs do families incur when they participate (e.g., parking/transportation, child care, hours lost from work)?
Transportation
Is there access to public transit? Do families have alternate means of participating (e.g., interviews at home, assessments at different times)? Is the program offered in the local neighborhoods? Are clients reimbursed for parking and other transport expenses? Do families live away from the site need to come to the site when others have left for the day? Transportation is often a significant barrier (and can be a significant program expense), especially after families leave the area but are being followed for assessment of long-term effects.
Child Care
Another potential barrier is lack of access to affordable child care. Can the program provide child care on site? Can the program reimburse clients for child care expenses incurred while the client participates in the program?
Rapport/Cultural Sensitivity Issues
Is the recruiter pleasant or obnoxious? Does the recruiter interact with the family in a way that was respectful of cultural or class differences? Is the program designed to encompass cultural differences in an appropriate way? Do clients perceive that the program is only for a certain type of individual? Or officers? Is there a stigma attached to participating in the program? For community programs, are there barriers associated with neighborhood and location? Are some members of the target population unlikely to attend an event in a certain neighborhood? Is there a neutral meeting place?
Program Marketing
Was the program marketed well? Are recruiting and program materials attractive? Do they describe for clients the benefits they might expect if they participate? Are materials sent, or distributed, to all eligible families? Do families have multiple opportunities to learn about the program?

In the example below, accessibility issues were addressed, leading to greater father participation in a program.

Example 3.3: Process Evaluation and Program Accessibility in a Home Visiting Program

In a Massachusetts early intervention program, a questionnaire for parents indicated that many fathers were interested in participating in home visits, but needed the visits to take place at night. In response to this information, the program introduced flexible working hours for the staff, and fathers did become more involved in the program (National Center for Clinical Infant Programs, 1987).

Accessibility issues can also affect participation in community programs. One field-initiated program was considered for development of a Web site with information on available resources for families. Such a program might be very effective, but it makes a number of assumptions that may or may not be true. For example, it assumes that families have access to a computer. It also assumes that the adults in the family are comfortable with computers in general, and with "surfing" the web in particular. These are not safe assumptions. Many working-class (and some middle-class) families do not have a computer in the home, and may not use one at work. And even if they do, there is a strong possibility that the children know much more about how to work it than do their parents. In this situation, it is unlikely that the parents would ask children to find referrals for family violence prevention.

Refusals

Once a family has been approached and the program has been presented, they can decide whether or not to participate. When families refuse, it would be helpful to find out why, as well as record some basic demographic information that will help to determine if they differ significantly from participating families. Of course, the evaluator may not be able to collect any of this information, and it goes without saying that representatives of the program should never hound people. If many families mention the same problem, it might be a barrier to participation that should be addressed. The rapport/cultural sensitivity issue described above may also influence whether people decide to participate. Even if the problem cannot be fixed, the information gathered will be useful in interpreting the results of the outcome evaluation.

While some potential participants may state directly that they do not want to participate, others may opt for a more indirect approach. These families may express that they do not want to participate by never coming to program activities (e.g., classes, clinic visits), by never returning telephone calls, or by ducking home visits. The evaluator and program staff must make a decision about how to handle these "passive refusals." Unfortunately, these behaviors are often typical of high-risk chaotic families (see below), and the evaluator might need to be diligent with these families. On the other hand, these behaviors can also waste a great deal of time for program providers whose efforts might be better spent on families who appear to want services. One strategy is to set a limit (e.g., six attempts to contact), after which all attempts to contact are stopped. Worksheet 3.2 is a sample form for monitoring these attempted contacts.

Below, a study on the marketing of a marriage enrichment program is presented. In this study, the researchers found that program marketing influenced the decision to participate in the program.

Example 3.4: Program Perception among Non-participants in a Marriage Enrichment Program

In a study of marketing a "marriage enrichment" program, perception of the program differed significantly among participants vs. non-participants. In a follow-up survey, non-participants were more likely to consider the program "therapeutic," a program for couples having severe marital difficulties. They also expressed concern that the program would invade the privacy of their marriages. On the other hand, those who participated (and the program providers) considered the primary purpose to be "educational," designed to enhance an already effective marriage. Evaluators of this program recommended that advertising, and program materials, specifically emphasize the program's educational aspects (Roberts & Morris, 1998).

Retention

The other side of the recruitment/retention equation reflects a program's ability to maintain their families' involvement. This issue is not relevant to programs that are one-time-only, but is very much an issue when families have multiple contacts with program staff, and contacts are extended over a period of time. Some families may overtly quit the program. Others will simply "drift away," and be lost to program follow-up.

Studies that attend to recruitment success (or failure) are few. Even rarer are published reports analyzing program retention. Retention is very important both from an evaluation standpoint, and from a service perspective. If significant numbers of families do not complete the program, interpreting the results of an outcome evaluation is difficult. Do families who stayed look better than they normally would have because the really troubled families dropped out? Of concern is that multi-problem chaotic families are the most likely to drop out of intervention programs. This means that the families most in need of the program are the least likely to hang around long enough to receive any benefit. Below is an example from an early intervention program in California that used information from a process evaluation to retain especially high-risk parents.

Example 3.5: High-Risk Target Population in a Secondary Prevention Program

An early intervention program worked with particularly high-risk parents, whose children were in "grave danger" of child abuse and neglect. The home environments of these families were often so chaotic that they could never keep office appointments. Further, many of these families were difficult to engage in the program activities. When reviewing detailed case notes, the staff of this program discovered that visits were more successful when the workers placed heavy emphasis on interpreting negative feelings of the parent. Visits were less successful if this emphasis was lacking, and families were less likely to stay (National Center for Clinical Infant Programs, 1987).

To maintain families in high-intensity programs, there are several helpful strategies that personnel at specific sites could consider integrating into their programs. One is to offer incentives for families to continue to participate. Some appreciate the extra help and support. Others like to feel that they are involved in something "bigger," and that their participation will eventually help other people. Keep these families informed about the goals of the program, and why it's important. At each contact (or certain designated contacts), offer small tokens of appreciation such as certificates or other memorabilia. If children are involved, offer small gifts to them such as books or T-shirts. In the box below is a description of the Infant Health and Development Program (IHDP). The population of this study was high-risk in a number of ways, including many young, unmarried, and poor mothers who had premature babies. Families enrolled in similar studies had high drop-out rates (e.g., 50 to 60%). The IHDP study was unique in that the retention rate was over 90% at the end of the three-year study. This was achieved because of extraordinary and diligent efforts of staff at each of the eight sites. While this level of diligence may not be practical--or even possible--for the programs, the IHDP approach might offer some suggestions that the programs can include.

Example 3.6: Sample Retention in a Secondary Prevention Program

Constantine and associates (1993) reported on process data specific to the Infant Health and Development Program (IHDP), a three-year multi-site, randomized clinical trial testing the effectiveness of early intervention in reducing health and developmental problems of low birth weight premature infants. The authors indicate that all low birth weight infants were logged at birth, and screened for eligibility.

Forms were then completed that documented reasons for inclusion or exclusion, and sites were required to report weekly to a national coordinating center on the status of each infant. Infants were seen at the study clinic for eight visits lasting 1.5 to 3 hours each during the three year period. Families in the intervention group also received home visits during the first year, and the children participated in a center-based educational program during the second and third years.

Efforts to minimize attrition from the study included extensive tracking procedures, mandated contacts between visits, and incentives such as transportation to the clinic, compensation for assessments, photos of the children, t-shirts, books, toys, and other birthday and holiday cards. Statistical analysis was conducted to determine if the sample recruited was biased in any way (whether enrolled mothers differed from non-enrolled mothers), and a similar analysis was conducted on information about retention bias (whether mothers leaving the program differed from those who remained). This study is important because it demonstrates how careful program monitoring can improve the program's ability to deliver services to those most in need, and to keep them engaged in the program. In addition, the monitoring procedures allowed the investigators to ensure comparability of groups at each site.

Another helpful strategy is to ask for locator information at each contact with families. An example of a locator question is: "What's the name and telephone number of someone who will always know how to reach you?" This type of information can help families keep in touch with the program and minimize the number who are lost to follow-up. This approach is especially helpful if there is an extended period of time between contacts, or the program families are transient. The evaluator can also keep family information up to date by sending holiday cards, appointment reminder cards, and other types of information that will go through the mail, with forwarding information requested. The post office will notify if there is a change of address. Finally, credit organizations such as Equifax or TRW are sometimes willing to help organizations locate individuals for study purposes, but this can be costly.

Even for families not identified as at-risk, it is important that they receive all (or most) of the services described in the program model. If they do not, these families can skew results of an outcome evaluation, or even render them meaningless.

Another aspect of retention has to do with "customer satisfaction." Do families perceive that the program is helping them? This issue is relevant to all levels of prevention programs. Clients who are happy with a program are more likely to stick with it until the end. Further, word-of-mouth can influence whether other families will be easily recruited into the program. This is particularly important with primary prevention programs, where interest is recruiting families who are not at-risk, and who may have little incentive to participate in a program.

Some ways to assess client satisfaction are through focus groups (see Section 5, Subsection "Focus groups") or through telephone or mail surveys (see Section 6, Subsection "Self-administered Questionnaires"). The evaluator needs to seek out and include those who were unhappy with the program, or those who dropped out. Families who completed the program are more likely to only describe its positive aspects, whereas families who quit may be more willing to provide information about where the program can be improved. For instance, clients may bring up accessibility issues described earlier. The evaluator may also notice that some aspects of the program do not transfer well to a particular population (Davis & Savas, 1996).

There is one caution with regard to customer satisfaction. While it can be a helpful measure of program effectiveness, it should not be the only measure-especially for programs that work with families at risk. These more intense programs may need to apply some "tough love," which may not always make participants happy, even when this approach is necessary and will be helpful in the long run.

Tracking Retention

The complexity of monitoring retention is directly related to the complexity and intensity of the program under evaluation. For programs where clients participate for one day only, or over a weekend (e.g., a marriage enrichment program), retention can be monitored by having participants sign-in at the beginning of the program, and out at the end (similar to the procedure for continuing education units). For classes or activities that take place over a few weeks, retention can be monitored by taking attendance at each session.

More intense programs will have to monitor all contacts with families: those specified by the program, and those initiated by the families. Some families in these programs will have dozens of contacts with program staff. Others may not even have the specified number. As described earlier, the phenomenon of the "passive refusal" appears in retention as well. Some families are difficult to maintain in the program from the beginning. Others may become difficult later in the program. There are a variety of explanations for "passive dropouts." The family may have had a disagreement with the service provider. Their family circumstances may have changed (e.g., birth of a new baby, change in marital status due to death or divorce, or relocation). As with refusals, it is important to track attempts to contact and set a limit on failed contacts (see Worksheet 3.2). Please note that Forms 3.2 and 3.3 at the end of this section also have places to indicate whether the client was a "no show" (Form 3.2) or did not receive services (Form 3.3). After a certain number of attempts to contact, the program staff should cease efforts to contact the family.

If not done so already, the evaluator could develop a form to help track the program's rate of retention. Worksheet 3.3 will assist evaluators in doing this.

Home