Section 3: Doing Process Evaluation

Program Activities

The other portion of process evaluation described in this section has to do with the delivery of services.

Monitoring Delivery of Services

So that the quality of the program delivery can be monitored, its components are first specified, as described in Section 2, Subsection "List Services and Activities". In prevention programs that offer classes as the primary modality, the evaluator will need information on the course curriculum, learning objectives, and whether the classes are taught in a consistent way.

For primary prevention programs designed for communities, the evaluator should monitor such process variables as inter-agency cooperation, the extent to that clients have been informed of services available, and whether the program is being implemented in a consistent way. Outcomes of these types of programs include global measures such as crime rates, or measures of community resiliency, as operationally defined.

For more intense prevention programs, the evaluator will need information on activities, procedures, routines, timing of events, duration of the total program, and timing between discrete phases. Information on delivery of services can be collected from a variety of sources including program staff, consumer interviews, and program documentation (e.g., grant proposals, minutes from planning meetings, FAP standards, and policy manuals).

Monitoring services can address several different questions. Some of the most important are considered below.

• Implemented as planned?

A question raised at several points in the evaluation is whether the program model matches the program that is actually being delivered. The model may specify contact that occurs at specific intervals (e.g., once a week), and lasts for a certain amount of time (e.g., two hours), during which certain activities take place (e.g., teaching basic stress management).

To decide on the best way to monitor the program, the evaluator should first refer to the logic model. The points at which families are contacted will suggest when to collect data. For example, in intensive programs, families may be seen every two months. When tracking these contacts, the evaluator should determine whether families are receiving all (or most) of the specified contacts.

In programs that involve a teaching component (in either primary or secondary prevention programs), the best way to monitor these programs is through observations. This can be accomplished by tape recording sessions or through direct observation. For large programs, in-person observations may be impractical. One option is to conduct a small pilot study to monitor the program. In-person monitoring, especially initially, may help to spot problems that the evaluator might not be able to observe via audio tape. Later, when the program is at multiple sites, the evaluator can monitor audio tapes at a random selection of sites. When the program is just getting started at a site, the evaluator may want to observe, or listen to tape recordings, of most sessions. As the program matures, the evaluator may decide to monitor the program intermittently (such as every second or third session).

During the observations, the evaluator should focus on the content of the program. Are instructors/program providers covering the agreed-upon topics? Are they adding or deleting information? Does some of the program content appear to be problematic?

If it is discovered that the program in use differs from the planned program, the evaluator should try to determine why, and be specific about ways in which the two programs differ. Was the planned program not feasible? Were there insufficient staff and other resources for carrying out the program? Since programs may be implemented at a large number of sites, there may be significant variations in how the program is administered in different locations. The evaluator should gather enough information about these differences so that problems can be described and program differences explained to others at a later date. For example, an off-the-shelf curriculum for marriage enrichment may be more feasible at a larger site than a smaller site. Moreover, a program that helps families tap into community resources is going to be more effective in an area that has community resources. For example, a site near an urban area may have more community resources than a site in a rural area. Likewise, clients may not be able to avail themselves of community resources because of language or cultural barriers that prevent them from participating.

• Timing

Timing includes several variables associated with the implemented program. Is the timing appropriate for what the program is trying to influence? For example, a new-parent program would be most effective if administered in the first few weeks after birth. Does the timing of the program interventions and/or assessments coincide with major developmental shifts? Is there enough time to administer all the program interventions? Is the timing consistent with "best practice" recommendations?

Another issue with regard to timing is efficiency. Are clients processed through the program at a steady rate or are there times when many clients are seen all at once? Is staffing flexible and are staff able to handle changes in their work flow? Can timing of the administration of the program components be changed to ensure a more steady work flow?

• Consistency

Consistency refers to whether the program is delivered in a standard way. Do participants at one site (or with one instructor) receive approximately the same program as participants at other sites? Standard protocols and monitoring delivery of services are two ways to increase consistency across providers and sites.

The first step to improve consistency is to specify the particulars of the program. If this has not already been done, refer to Section 2 (Subsection "List Services and Activities") for suggestions about how to list this information. For programs whose primary activity is classroom instruction, is there a specific curriculum? This question is relevant for programs that are purchased off-the-shelf and those that are field initiated. In both of these cases, the evaluator is taking a program that was developed in one situation and applying it to another. Does the off-the-shelf program transfer well to a particular population of client? For example, is the implementation of the program flexible enough to accommodate clients who are not available for some of the program activities? For field-initiated programs, it will be important to know whether a program that was designed to address a specific issue in one community is relevant to sites at other locations.

The evaluator might decide to try different programs to address a specific problem. To determine which one is most effective, it will be important to examine both process issues and outcomes. When comparing two or more programs, the evaluator can measure outcomes with both qualitative (see Section 5, Subsection "Types of Qualitative Approaches") or quantitative approaches (see Section 6, Subsection "Collecting New Data"). A quasi-experimental design will allow for a comparison of outcomes for alternative types of program services using the same measures across programs (see Section 5, Subsection "Quasi-experimental Designs" and Section 7, Subsection "Outcome Measurement").

Another factor that influences consistency is staff training. Are appropriate providers being trained? Are providers trained to administer the program, and how often is this training provided? Are providers following the protocol? Is there a mechanism to insure that training occurs in a timely manner to take into account sporadic new hires? Some differences in program administration are quite legitimate. For example, instructors may have had to adapt classes to meet the needs of the clientele at a particular site. Preferably, these changes will be documented so that they can account for differences at that site, or even lead to changes in the program at sites using it. On the other hand, some issues around implementation of the program are not legitimate. Individual instructors may be disorganized, and can never seem to follow a standard curriculum. Others may not "like" the program. Rather than communicating with other program staff, they may unilaterally change it.

While there needs to be flexibility in the implementation of any program, delivering the program with at least a modicum of consistency is something all program staff should strive toward. To get a true sense of consistency of delivery of services, site visits and on-going monitoring of service provision should be provided, such as randomly audio taping class sessions or by using specific data collection methods.

• Quality

The final monitoring variable is more subjective: the quality of the program or services. Reference has already made to quality issues more broadly, and at many points, in the sections above. However, in regard to making judgements about whether staff are engaged in providing the "best" services possible, measuring quality is more amorphous. It must be judged against some yardstick, or benchmark such as the "best practices" discussed below. Quality assurance can include case reviews, supervision of staff, and staff continuing education. It can include client demographics, client satisfaction, service utilization, and case disposition (Pecora et al., 1995). In order to make judgments about quality, the standard of comparison must first be specified and variables chosen to be used as quality indicators. For example, in health care settings, productivity, work processes, resource availability, staffing patterns, and leadership are considered to be important input variables in quality assessment (Dimond & Roca, 1997). The evaluator can make quality judgments using, for example, one or all of the following standards.

• Smooth Implementation

Are there enough resources? Are families cycled through the program efficiently? Are appointments spread evenly throughout the day or week? Are clients keeping their appointments? Are clients kept waiting for more than 15 minutes? How often does that happen?

What could be done to improve the way the program operates? Is the paperwork easy to complete? Is program staff keeping up with it? Is the office neat and orderly? Are the staff happy? Are there high rates of staff turnover or burnout? Which activities were particularly successful?

• Client Satisfaction

Do clients seem happy with the program or are they hostile? Are they participating regularly or do they tend to miss appointments or meetings? Do they have any specific suggestions? Have they identified a component that they especially like? Dislike? Is information on client satisfaction collected from all clients and analyzed on a systematic basis? Many of the same kinds of questions discussed in relation to target populations and retention are also relevant to examining issues of client satisfaction. For example, it may be important to know if satisfaction differs by client demographics, the nature of family problems, or other characteristics of the client.

• Best Standards

How does the program compare to established "best standards" in the field? What are these standards? How does the population differ from the one these standards were based on? Does the program differ from established guidelines? Why? One example is comparing frequency and intensity of service provision with the criteria specified (see Section 2, Subsection "Volume and Timing of Service Delivery"). The evaluator might also consider whether a program is based on theory and sound research evidence.

If the program is new, there may not be "best standards" that are already established. In that case, these could be developed based on theory and the components specified in the logic model for the program. The evaluator should distinguish between sufficient vs. optimal levels of service when establishing performance criteria for measuring the quality of services being provided to families.

Monitoring Dose Effects

An additional reason to monitor service delivery is to enable the evaluator to profile families in terms of their utilization of prevention program services. In many prevention programs, there are usually families who receive more services and some who receive much less. Determining who these families are is quite simple with a good service delivery monitoring system. Profiling high-dose and low-dose families usually leads to an understanding of optimal utilization rates as well as provides basis for program modifications where under and over utilization is seen to be a problem relative to the original program model.

There are three types of program failures that can occur because of delivery system errors. In programs not showing positive outcomes, the real problem may be in the failure to deliver the interventions in the ways specified in the program design. There may be a problem when families receive no treatment, the wrong treatment, or treatment that varies across sites.

Many program manuals provide lengthy standards on the core elements of the "treatment" to be delivered. From a strict evaluation standpoint, fidelity to the program's blueprint is paramount. However, practitioners may balk at providing "cookie cutter" as opposed to individualized intervention. They may view some standards as not adaptable to their setting or, possibly, as providing too little guidance on nuts-and-bolts issues. Some compromise may be needed so practitioners feel they have the necessary flexibility to maximize outcomes for clients, yet areas of critical importance are consistent across families and sites. One approach would be to document the achievement of particular benchmarks (e.g., immunizations, height and weight, developmental assessment results) in data that were collected at regular intervals. Healthy Families America (NCPCA, 1997), for one example, identifies 12 critical elements in program service delivery (See Section 2, Subsection "Volume and Timing of Service Delivery").

Monitoring service delivery can be done in many ways, ranging from daily recording of service activities for each family to a cumulative accounting at the end of the service episode. For service delivery monitoring to be most useful in ongoing program management, including quality assurance and quality improvement, "real time" data collection is optimal. However, this type of monitoring requires a non-labor intensive method for providers of service to record their activities and a reporting system that provides service utilization profiles on a timely schedule by family and usually by provider.

Forms for Recording Service Activity

In this section, some sample forms to help record service activities are discussed. Form 3.1 is an example of forms that can be used with educational or community-based programs. The others are examples of forms currently in use with other high-intensity service programs. For more information on how to develop forms, please refer to Section 6 (Subsection "Collecting New Data").

As described in Form 3.1, designing a form to assess participation in a less intense program is straightforward. What the evaluators using this form are particularly interested in is how much exposure did each client receive to prevention information.

The other forms are examples of relatively "low-provider-burden" recording forms for service activity. The examples differ in the time frame used, with one using a cumulative end-of-service episode frame and another a cumulative monthly frame. Still another example illustrates a protocol that requires recording of service activity on a daily basis. All of the forms generate information that can be used to profile individual families. The monthly and daily recording forms are examples of those used in quality assurance activities. These reflect the principle that service monitoring should be part of the provider routines during rather than after an episode of service delivery in order to be useful for ongoing quality assurance activities, especially those that involve a supervisor and a provider.

The forms also indicate the comprehensive character of the service delivery monitoring that can be achieved. The forms collect information on who was served, when and where they were served, and what the nature of the service was. In addition, Forms 3.2 and 3.3 indicate that both units of service and amount of service in terms of time can be collected. Furthermore, Form 3.3 shows that the home visitation activity can include referrals to adjunct services as well.

It should be noted that all of the forms are used for recording on hard copy and then transferring this information to computerized files for easier profiling of the service delivery patterns. However, computer technology now permits easy recording of prevention service activity in the field directly into computerized databases. This will make it possible for individual providers, such as home visitors, to easily record service delivery on a daily basis and also receive immediate feed-back in the form of computer driven profiling reports.

In summary, monitoring the program is important to quality assurance and provides information about how the program can be improved. The information can also aid decisions about whether to continue, expand, or alter existing programs. The extent to which program specifications are actually met in the delivery of the intervention obviously must be fully documented for policy making and resource allocation-the topic of Section 4.