Section 5: Assessing Program Impact

Evaluation Methodology

This final portion of Section 5 focuses on the types of evaluation methodology used in collecting data. Either quantitative or qualitative techniques may be selected, or a combination of both. Quantitative analyses focus on testing hypotheses, and use structured designs and statistical methods to analyze data. This type of information needs standardization, precision, objectivity and reliability of measurement. Quantitative techniques are described more fully in Section 6.

Qualitative approaches, in contrast, gather data in a more open-ended fashion. Data collection usually occurs in natural settings, and focuses more on experiential or subjective aspects of a program. These data can include narrative accounts and may employ multiple data collection techniques (Worthen, Sanders & Fitzpatrick, 1997). Evaluators use qualitative designs to help them understand and describe program implementation, rather than to demonstrate statistically significant effects. With quantitative designs, the evaluator uses direct contact and experience with program families in order to gain in-depth and detailed understanding of the program. Qualitative designs use naturalistic methods of information-gathering (e.g., observation, interviews, case studies). These methods allow evaluators to explore experiences of the program for different individuals, in contrast to providing them an overall impression of its effects.

As Worthen and colleagues have indicated, "an unfortunate debate" (1997, p.71) has arisen about which of these approaches is "best." This debate is not productive. A combination of approaches frequently yields the most useful information. For example, when first approaching an evaluation, an open-ended approach can provide information that will assist the evaluator in developing more quantitative measures (such as a survey). Qualitative measures will provide illustrations and examples that stakeholders may find helpful in understanding the effect of the program on individuals. Further, stakeholders, and members of the general public tend to remember and be moved by case illustrations that describe an individual's experience.

On the other hand, data that are entirely qualitative can quickly become unwieldy. Processing this information is very labor intensive and may be well beyond the resources available. Quantitative data summarize information from large numbers of people in a concise way. Furthermore, the greater the methodological rigor, the greater the certainty that it is the program, and not some other factor, that accounts for the differences identified in family functioning. Rigor can be much more easily determined with quantitative methods.

Although a distinction has been drawn between these two methodological types, in the real world, these lines are often blurry. For example, in Section 6, an overview of survey research is presented. Many would consider this a "quantitative" technique. However, if the questions on the survey are open-ended rather than closed-ended, they are really more qualitative. Further, questionnaires quite often contain both types of questions. Moreover, focus groups, usually considered a "qualitative" approach, frequently yield information that can be coded and quantified.

Below, three qualitative techniques are described that are easy to use and valuable at all stages of program evaluation. These techniques are also handy because they allow the evaluator to "take a quick pulse" of the program, and make decisions about how it is functioning.

Types of Qualitative Approaches

• Open-Ended Interviews
  Rather than using closed-ended questions, open-ended interviews have several advantages:
  • Clarity: The interviewer can clarify questions, and avoid a problem such as illiteracy or language barrier.
  • Richness: The interviewer can pose more complex questions and make observations about the respondent's appearance and behavior.
  • Completeness: The interviewer can minimize missing and inappropriate responses.
  • Control: The interviewer can control the order of questions.

Strategies For Involving Respondents
Those planning to conduct open-ended interviews should investigate various factors that will help to increase the response rate, such as the best time of day to call or visit. Before conducting in-person interviews, an advance letter could be sent to the sample respondents, explaining the purpose of the survey and that an interviewer will be calling soon. Respondents must be informed of the voluntary nature of the interview and how the answers are to be used before they agree to participate.

Interviewer Training
Interviewers must be carefully trained in the techniques of interviewing such as making initial contacts, listening skills, and avoiding influencing or biasing responses. It is possible for an inexperienced or improperly trained interviewer to skip certain questions that he or she is uncomfortable with, or to not listen carefully to respondents. Practice interviews are necessary before the interviewer conducts any interviews. The interviewer needs to be fully familiar with each interview question, to be able to assist respondents who might need questions clarified. The interviewer also needs to be well-versed in the ethics of interview research. This includes maintaining confidentiality and not coercing or pressuring participants to participate, or answer specific questions.

• Focus Groups
  Focus groups are very much like individual interviews except that they are done with a group of people. They involve discussion of a topic that is the "focus" of the conversation, and typically last 1 ½ to 2 ½ hours.

Purpose
The most common purpose of a focus group is to help the evaluator learn more about a little-known topic. It is especially helpful for providing information about how people think or feel about products or services, and is frequently used in marketing research. This same technique can be used to help the evaluator to learn more about the needs of a particular population, or how these individuals reacted to a prevention program. In addition, focus groups are useful in conjunction with a quantitative study in explanations of results. One should conduct 3 to 12 focus groups on a topic, because various combinations of participants will yield different overall responses to questions and issues.

Selecting the Participants
Focus groups usually contain 4 to 12 members to allow ample opportunity for expression, and to enable the leader to reasonably manage and document the discussion. Ideally, the participants should regard each other as equals so that they feel free to speak what is on their minds. However, depending on the resources available it may not be possible to constitute groups in a singular manner (e.g., low-income fathers). Often groups are selected to get opinions from a cross-section of people (e.g., active duty military mothers and non-military mothers). Unlike regular survey research, random samples are not a priority. In terms of group composition, group members are selected who are knowledgeable about or have experience with a particular topic. It is not necessary to limit focus groups to program clients. Informants can be used to problem solve with staff members, or to assess a need in a specific community. Informants should be selected on the basis of their knowledge and experience with the topic being explored. Another factor to consider is the impact of group members on each other. For example, women are often less talkative in groups with men than they are in groups that are all women.

Setting the Ground Rules
The facilitator's job is to keep the discussion focused, and allow everyone to express their opinions. Focus group facilitators must understand the cultures of the group members, and must make sure that each participant has a chance to speak. Moreover, they must let participants know that there are no right or wrong answers. All comments, both positive and negative, should be welcome. Also let participants know that they do not have to feel "on the spot" to answer something that they are uncomfortable about (one of their rights as research participants). Ground rules should include telling participants that they may simply say "pass" if asked a question they do not want to answer. Ground rules should also include getting consent to tape if this is part of the plan, and explaining reasons for taping or taking notes. Assurances of confidentiality are important and should include telling group members that no names are attached to individual comments, and that tapes will be destroyed after they are transcribed.

The Questions
The interviewer/facilitator uses pre-written interview questions developed as part of an interview guide, asks the group to respond, and helps to promote dialogue among the group. The interview guide is usually no more than 12 to 15 questions. Note that in some cases it may not be possible to fully cover more than a few questions, but it's nice to have some extras, especially if some questions "fall flat." The facilitator must be able to listen carefully and use appropriate non-leading probes to elicit more information from participants.

Taping and Analyzing the Focus Groups
Focus group sessions are usually tape recorded so that the discussion can be transcribed after the session when it is convenient. The analysis of focus group data can cover a wide range of specificity. In some cases, all that is needed is a brief summary of what was discussed in the group. For more rigorous approaches, the first step is transcription of the group tape. The tape should be transcribed verbatim. The evaluator then examines the written transcribed comments for common themes and insights, within and between groups. Direct quotations can be used as part of the presentation of findings.

In formal content analysis, coders search for themes in the transcripts. Each transcript should be independently reviewed by two different analysts, who then compare their reviews for content and consistency. They must agree on the content and meaning of the transcripts and notes. Analysts look for themes, key terms and concepts, and contrasts in the data. Points that are not clear are then checked, and verified by reviewing tapes and notes, and through discussion with the facilitators. Coders need extensive training on how to code focus group (or other qualitative) data. Formal content analysis can be very labor intensive. This process can be helped by computer programs designed to follow "data-making" rules established by the investigator, but can still be demanding.

Evaluators should not rely solely on this type of data. The lack of representativeness in the samples can be misleading. On the other hand, focus groups can highlight issues that evaluators may have missed and can add richness to quantitative data that are collected.

• Direct Observation
  Using direct observation for assessing families' interaction has advantages, especially when trying to determine whether child maltreatment is occurring. Direct observation of interactions between children and parents is useful for assessing the basic elements of a child's health, safety, and physical development, while eliminating possible biases from the children's caretakers. Direct observations may be conducted in the home, or the clinic, with each setting providing its own type of information.

Observations in a clinic-type setting are useful when observing behavior between individuals, such as between parents and children. Behavior in the office is often assumed to be a client's "best." Abusive behavior observed in the clinic (when families know that they are being observed) often leads staff to conclude that the situation is worse in private.

Home observations are useful for understanding the whole scope of influences on a family. Home visits may help pinpoint problems quickly in a way that traditional interviews cannot. For example, a breastfeeding baby with slow weight gain may be distracted from nursing at home because of the chaotic or noisy environment.

Pros and Cons of the qualitative measurement technique

Pro: Qualitative techniques work well for exploratory research where pre-determined response categories may not need to be anticipated or developed (see also Section 6).

Pro: Qualitative techniques yield in-depth, detailed, and rich information.

Con: Analysis of qualitative data can be difficult and time-consuming because responses are neither systematic nor standardized.

Con: Qualitative data collection is often more labor-intensive; it requires more time and effort to interview or observe than to administer questionnaires to a large group.

Bottom Line for Qualitative Methods: The qualitative techniques that have been described above will be helpful during many phases of the evaluation. These techniques, and the survey techniques described in Section 6, will yield the most comprehensive data. (In Section 6, further information is provided to guide the evaluator through an impact evaluation. Both data coding from existing records and designing surveys are described.)