Section 2: Developing/Assessing Logic Models

Developing the Logic Model

Although developing a logic model might seem far removed from providing services to families, this process is simply a "re-framing" of what providers do every day. Service organizations have a mission to serve certain populations. Providers and managers of these programs identify problems and make decisions about what type of service to provide. They select interventions that draw upon existing strengths and address areas that need improvement. They also monitor the results of their interventions, trying to determine if their clients are "getting better." The logic model can aid in this process by helping the evaluator to articulate the design more clearly (Stovell, 1998).

Logic models can be derived from experience with the population, past studies, or a combination of both. One advantage of logic models is that all information is summarized concisely. This enables the evaluator to see interconnections between variables, and identify at once the type of data needed for collection. Further, logic models can identify any gaps in planning. At a later date, the evaluator can use logic models to explain the program to others in presentations and written reports. Finally, logic models help in writing proposals for funding because the goals and their theoretical underpinnings are concise and clear.

The steps described below have been loosely adapted from the Evaluation Guidebook (Burt, 1997), developed for the Urban Institute. They are staged in order from the program goals (right side of the graphic logic model) to characteristics of the target population (left side of the graphic logic model). Although these steps are portrayed in a linear fashion, it might make more sense to think of this process as circular. Program outcomes also contribute to feedback to improve a program. With a circular model, one can enter at any point, which also makes it easier to develop a model for an already existing program. It is often easier to start from the program's outcomes and then describe the activities and target population, working right-to-left.

Logic model development overlaps considerably with evaluability assessment. During either (or both) phases, the evaluator might discover that the program goals are not clearly articulated, or that the measures do not relate to the outcomes. This information helps the evaluator to refine the program and facilitate program evaluation. If it is decided to skip this step, the evaluator will run the risk of having an unclear rationale behind the program. This could become a problem for both potential funders and other stakeholders (Stovell, 1998). The first two steps in developing a program logic model involve identifying the people who will take part in its development. The last few steps have to do with the various components of the model, and how these fit on the schematic design.

• Determine Who Will Be Involved In The Development Process

The choice of participants revolves around the issue of stakeholders. If a program has clearly determined who the stakeholders are, this step may be relatively simple. If the program has not yet been developed, the participant pool should include representatives of the key stakeholder groups as well as the program evaluation team, if a program evaluation is intended. Of course, it may not be possible to get representation of all potential stakeholders. If the program is multi-site and has already been implemented, the participants should include sufficient representatives of the service providers that knowledge of the variation in programs across sites is available for development of the logic model. In general, representative providers should be chosen from among those with the greatest experience and the capacity to articulate in a group setting their perspectives and what they had learned. Finally, the development participant group should be small enough (6 to 8) to facilitate positive interaction.

• Develop A Common Language

The evaluator begins this step by distributing literature and materials about the logic model development process to all participants. This should take place before the first session. These materials could include this section of the evaluation guide as well as literature about logic model development. Examples of this literature are noted in the selected references at the end of this chapter (e.g. Burt, 1997, for an example of developing a spouse abuse program model, and Olds, Eckenrode, Henderson, Kitzman, Powers, Cole, Sidora, Morris, Pettitt & Luckey, 1997, for an example of a logic model related to child abuse prevention). These materials provide a common base for the opening discussions and also provide the common language about logic models that can then be used immediately upon commencing the group process. All relevant parties should write down a description of what the program is trying to accomplish. Although most will agree with the more general goals (e.g., "prevent child abuse"), this exercise will highlight differences in perceptions on how these goals are actually accomplished. Once these differences are known, the group can work toward consensus (Stovell, 1998).

• Identify Program Goals, Objectives, and Outcomes

As described in Section 1 (Subsection "Questions about Program Model"), goals are distinct from objectives. Goals are broad and not directly measurable. They reflect the mandate of the program, and usually address the nature of the social problem being addressed. Unrau (1993) gives the following example of a formal goal for a family preservation program:

"To preserve family units where children are at-risk for out-of-home placements due to problems with physical abuse. The program aims to strengthen interpersonal functioning of family members through intensive home?based services" (p.122).

To be measurable, goals should be specified into objectives or outcomes. Outcomes specify the targeted changes that the program is trying to accomplish. On Figure 2.1, both goals and outcomes are listed. A logic model is flexible enough to include both types of information.

Outcomes can be divided into immediate (right after completion of the program) and longer term. The reason for separating the outcomes into early and later is directly related to the idea of mechanisms. A good program logic model uses program theory. Specifically, this indicates the mechanisms by which goals will be achieved by the program activities for the specific target population. Proximal and distal describe the "distance" from the immediate goals. Distance can be temporal: proximal outcomes will be measured before distal ones (e.g., proximal outcomes are measured at one year, and distal outcomes are measured at three years after the program). Proximal/distal can also refer to an outcome's theoretical or ideological "closeness" to the immediate outcomes: the further the outcome from the immediate assessment, the more general it will be. For example, the distal outcome on Figure 2.1 (enhance mission readiness) is quite far from the immediate outcome of increasing parent-child attachment.

Community variables can be also substituted for individual client variables. The section on program outcome would be identical to a program where the clients are the unit of analysis. The evaluator may be interested in community cohesion as a broad program goal. This can be operationalized as "increased contact with neighbors," "greater use of community resources," or other measurable outcomes.

Determining when the outcomes are most likely to manifest themselves can also aid the evaluator in deciding which outcomes are immediate and that are longer term. Generally, concrete and specific outcomes can be immediate (e.g., increased knowledge in child development upon completion of a parenting class). Other outcomes are more general and must develop and be evaluated over time so that the new skill can be fully integrated into client behaviors. For example, "decreased social isolation" may take time to establish. The client may have left the program with improved social skills (something that could be measured immediately), but it takes time before a client can apply these skills to people they know and gradually form an effective social network. The proximal outcomes are typically those more directly related to the program goals: decrease child maltreatment, decrease wife abuse, etc… The amount of time that needs to elapse before these outcomes are measured is somewhat arbitrary. The evaluator can receive some guidance from what previous programs have done and/or general knowledge about the target population. The distal goals are generally more global and broad in focus. They could include outcomes such as "increased community resiliency," or "increased health." Again, the timing of this measurement is somewhat arbitrary, but may be measurable even years after the initial intervention.

Below are questions to assist evaluators in considering program outcomes (see Worksheet 2.1). Once this worksheet has been completed, then all sections of column A on the blank logic-model sheet (Figure 2.3) can be completed also.

• What are the program goals?
• How are these goals operationalized? List specific objectives for each goal.
• What are the immediate outcomes? What do "successful" clients look like at this stage?
• What are the short- and long-term outcomes? What do "successful" clients look like at this stage?

• List Services and Activities

It is in this section that the abstract goal of "reducing family violence" becomes concrete action. This section should be very specific, with special thought given to how each part of the program relates to outcomes. Below are some general questions about activities.

• Which client problems is the program supposed to address?
• What activities occur within each program component?
• What is the rationale for the program?

The evaluator should group the program activities according to how they accomplish the short?term and longer?term outcomes (some might do both). Wherever possible, program activities and services should have sufficient detail so that the scope, volume, intensity, and length of services are specified.

The results of past studies may help the evaluator to decide what program services to offer. What has worked in the past with similar populations? The evaluator may also make decisions about services through clinical observations, or by conducting short-term studies with current clients (e.g., focus groups, client interviews).

In addition to listing all program activities, the evaluator should consider how the major components of the program fit together, from intake to discharge. As "client X" enters the program, what is the sequence of activities he or she might experience. It might be best to think in terms of a timeline, dividing the program into three components: beginning/intake, program activities, and end/discharge. Which activities are present at each point? Below is an example to help the evaluator identify activities that might be considered (see also Worksheet 2.2).

• What are all the program activities?
• Which activities occur at intake/the beginning?
• Which activities occur upon completion?
• How long are program activities administered?

Other ways to describe program activities are in terms of provider characteristics, and the volume and intensity of service.