Section 6: Designing Data Collection Instruments

Collecting New Data

Once the type of information still needed has been determined, a measure that will collect it must be located or developed. In this section, the technique most likely to be used, survey research, is described. A survey is simply a series of questions. Researchers give surveys to individuals who answer the questions and then return the survey to the evaluator. The evaluator then analyzes the responses to the survey questions, and hopefully draws meaningful conclusions from those answers about the program under evaluation (Fowler, 1998; Mangione, 1998).

The basic survey design is a skill that takes time to master. As the evaluator approaches the task of developing a new questionnaire, allow sufficient time to pre-test items. Learning what others have done can be very helpful. Using questions developed by others can speed the process considerably, even if the evaluator has to modify them (Sudman & Bradburn, 1982). Sometimes a pre-existing measure can be used (e.g., a depression inventory) as part of the survey, and then filled in with other items the evaluator has developed. Some of the basics of good survey questions are described below.

Designing Survey Questions

The evaluator may decide to write the survey questions when an appropriate data collection tool cannot be found (Sudman & Bradburn, 1982). Good questions are those that:

• ask for specific information
• address only one topic per question
• are not too general or vague
• do not contain hidden biases that might influence the respondent's answer in any way
• are necessary to ask, not just interesting to know about
• are clearly worded so that they cannot be misinterpreted.

Some questions are written to ask about respondents' opinions and attitudes ("How did you feel about…"?), while other questions are concerned with facts or behaviors (people's health, income, housing). Below are three basic types of survey questions (Fowler, 1998).

• Demographic/Other Content
  Surveys frequently include questions that ask about the respondents themselves (demographic information). These questions help the survey researcher classify their responses (such as age, gender, marital status, occupation, and place of residence).
• Opinions and Attitudes
  The evaluator may want to ask respondents what their opinion is on an issue, how strongly they feel and why, their interest in the issue or other questions that try to get at a person's feelings. For example "How did you feel about having a visiting nurse come to your home"? "Do you feel that the program was helpful to you"? There are no right or wrong answers to these types of questions.
• Knowledge
  These questions assess respondents' knowledge on a particular topic, such as knowledge of appropriate developmental milestones. Unlike opinion questions, there are right and wrong answers.
• Behavior
  Questions about behavior ask respondents to indicate whether or not they have participated in a certain activity, and how often they have done it. These types of questions are important for assessing potential or actual family violence, for example. However, respondents may under- or over-report their behaviors because they really do not remember, or because they want to appear favorable to the interviewer or evaluator.

Question Format

In the previous section, information about the content of questions was provided. It will now be important to describe how these questions can be formatted. Survey questions may be open-ended or closed-ended (forced-choice). Most questionnaires will probably contain both types of questions. The following examples are taken from a form developed by the USAF Family Advocacy Program.

• Open-ended Questions

Open-ended questions allow the respondent to answer questions in an unstructured format; there are no pre-coded response options. Open-ended questions are typically structured as a question followed by blank space for the respondent to write out an answer. Similarly, in a spoken interview, the interviewer asks a question, the respondent answers, and the interviewer records the response. An example is listed below.

What was most helpful among the services you received from your FAN (Family Advocacy Nurse)?
_______________________________________________
_______________________________________________

One of the uses of the open-ended format is in pre-testing and questionnaire development. Open-ended questionnaires allow the evaluator to collect information that can be used to develop closed-ended categories for a future assessment. Answers, or themes, that occur frequently are good candidates for closed-ended response categories (Sudman & Bradburn, 1982).

Open-ended question formats have other advantages and disadvantages:

Advantages

• The respondent uses his/her own words to answer the question, without being confined to narrow options provided on the survey.
• They give the evaluator an idea of exactly how the respondent feels or thinks.
• They provide specific information to be used to illustrate overall themes among many respondents.

Disadvantages

• They do not provide information that is easily summarized into numerical data.
• They are more difficult and time-consuming to analyze.
• In a self-administered format, the responses depend on the reading and writing ability of the respondent.
• They can be more time-consuming for the respondent to answer.

Closed-ended Questions

Closed-ended questions come with a set of pre-coded response options. There are several types of closed-ended responses. These include rating scales, true/false or yes/no questions, or questions where respondents must circle the appropriate code. The following are examples of closed-ended questions.

Rating Scales

Questions that use a rating-scale allow the respondent to choose only one option. Typically, this is a 3, 5 or 7-point scale. A question or statement is presented, and the respondent is asked to circle a value on a numerical scale that corresponds with the best answer, such as in the following question (Fowler, 1998).

On a scale of 1 to 7, how would you describe your labor and delivery, with "1" being "very easy" and "7" being "very difficult"? (please circle)

7    6    5    4    3    2    1

Check the Answer Questions

These questions present the respondent with a question or statement, and then a set of possible responses. The respondent simply places a check-mark next to the response (or responses) that apply.

How old are you? (Please check one)
15 to 19____
20 to 29____
30 to 39_____
40 to 49_____
50 to 59______
60 or older_____

Did you experience any of the following complications with your delivery or afterwards? (Circle all that apply)

Fill-in-the-blank Questions

Some questions ask the respondent to fill in an answer:

In all, about how many years have you and your spouse/partner lived together?

years_____

True or False Questions

For the following statements, the respondent is asked to circle either T (True) or F (False).

Yes or No Questions

For the following items, the respondent is asked to circle either "Yes" or "No."

Closed-ended questions have other advantages and disadvantages in comparison to open-ended questions (Sudman & Bradburn, 1982):

Advantages

• Closed-ended questions help the respondent to answer the questions more easily.
• They help the evaluator to collect and summarize responses more efficiently.

Disadvantages

• Closed-ended responses limit the respondents' answers.
• They inhibit the researcher's ability to understand what the respondent really means.

Putting Questions Together

Once the questions have been designed, the next step is to put them together in a survey. The evaluator can weave together newly developed questions, questions from other surveys, and questions from standardized instruments. Guidelines about how to put a survey together are provided below, including a rationale for why question order has the potential to influence respondents' answers.

Question Order and Transitions

It is important to consider the order in which questions are placed on the survey. Question order can affect the responses to survey questions. The first few questions that are asked can set the tone for the survey. If respondents become bored or feel threatened by sensitive questions early in the survey, they may not complete it, or they may not answer the questions seriously. The answer a respondent gives to a question on a survey or interview can affect the way the respondent then answers the next questions. In this section, general guidelines on question order are provided.

Surveys begin with an introduction. Below is an example of an introduction to an interview, but something similar could be written at the beginning of a self-administered questionnaire.

"In this survey, I will be asking you some questions about what's been happening in your life since the baby was born. Before we begin, I want to remind you that all your answers are confidential and you can refuse to answer any question that makes you uncomfortable or that you do not want to answer. You can also stop the interview at any time. Do you have any questions?"

The evaluator might begin with some basic demographic questions such as age, marital status, number and ages of children, and duty status. The most accepted strategy is to begin the survey with the least sensitive questions, and gradually build to the most sensitive questions. Similar items should be grouped together. Transitions between sections help the respondent to move into the different parts of the survey. For example:

"Now I want to ask you some questions about some stressors in your life. Please answer yes or no to let me know if any of these things have happened to you in the past six months."

Asking questions about sensitive topics is also an art. There are ways to do it that gather the desired information, and yet are respectful of the respondent's feelings. Suppose the evaluator wants to know if the respondent has a history of childhood sexual abuse. The questions need to start gently, while reminding respondents that they have a right to refuse to answer. This type of approach works well with many kinds of sensitive topics and can be applied to both interviews and self-administered forms.

"In this section, I'm going to ask you some questions about some things that may have happened to you in childhood. Before we start, I'd like to remind you that you can refuse to answer any question and all of your answers are kept strictly confidential."

A yes/no format, where a list of items (such as a list of sexual acts) are read, and respondents then asked to indicate "yes" if this ever happened to them, works well and seems to be less invasive than to ask them to recount their experience in an open-ended fashion. This can also work well for a listing of other highly sensitive topics such as criminal activities or drug use.

Household income is another sensitive subject. For instance, asking for an income range rather than asking respondents to openly state their income is a more sensitive method of gathering information about income. Respondents interviewed in-person should be handed a card and asked to pick a letter that corresponds with the range of their income level. In a telephone interview, the list of income categories is read and respondents are asked to pick one. Below is a sample question that asks about income. The income ranges are adjusted to meet the needs of the evaluation.

What is the total income in 1999 for you and the members of this household who are related to you, before taxes and other deductions. Just give me the letter.

A. $15,000 or less
B. $15,000 to 17,500
C. $17,500 to 20,000
D. $20,000 to 25,000
E. $25,000 to 30,000
F. $30,000 to 35,000
G. $35,000 to 45,000
H. $45,000 to 60,000
I. $60,000 or more

The evaluator will also need to be aware that question order can affect answers respondents give (Schuman & Presser, 1981). Question-order effects seem especially likely if asking questions about sensitive topics. The following is an example of how question order influenced responses to two questions about partner violence:

Pretesting

Pretesting is a way to gauge the appropriateness of questions or the survey as a whole (Suen & Ary, 1989). Before any formal pretest, the questions aloud should be read aloud to a colleague. Many errors can be spotted this way, including typos, awkward wording, and response categories that make no sense. The data collection tool used must be appropriate to the age, education level, and cultural background of the participant. If multiple sites are participating in the evaluation, it is important to have a pretest group that is similar to all evaluation subjects. Staff at other sites (especially those with an atypical group of clients) could also pretest the questionnaire. To make certain that the tool is appropriate, a small sample of participants (6-10) who are similar to the target population should be selected and administered the survey. Their responses can then be examined for possible problems with question order, wording, and and their feedback on the questions gathered. Pilot test participants can also be asked if the questions were clear, and the response format made sense to them. In addition, pilot testing allows the researcher to test the survey administration procedures (time limit, type of location, instructions for completing). The evaluator may then change the questions/survey according to pilot participants' feedback and suggestions.

This data should not be used as part of the evaluation study per se; the purpose of the pilot test is to check the measurement instruments and data collection procedures before any actual evaluation data are collected.

Survey Format

Another factor to keep in mind is the format of the survey. Are the questionnaires going to be self-administered or will they be administered by an interviewer? Interviews can take place either in person or by telephone. Interviews tend to be more expensive to administer than mail surveys, but compliance rates (percentage of people who agree to participate) are generally better, with less missing data. Self-administered questionnaires are substantially less labor intensive to administer (questionnaires are simply handed or mailed to respondents), but there may be difficulties if some respondents find the questionnaire difficult to understand or confusing or if they do not read at the level of the questionnaire. Further, self-administered questionnaires are easier to "turn down" than a questionnaire with a real person at the other end. These issues are discussed below.

• Self-Administered Questionnaires

Self-administered surveys can be administered in-person (such as when clients come into the clinic) or via mail. Mail surveys can be relatively low in cost, and may be the technique most likely to be used. As with any other survey, problems exist in their use when insufficient attention is given to getting high levels of cooperation. If only a small percentage of the families approached agree to complete a form, data may be biased in that only the more compliant respondents participated. This group may differ in some unknown way from the group who refused to participate (Schuman & Presser, 1981). The closer the compliance rate is to 100%, the better the quality of the information. A minimum of 60 to 70% compliance is typically acceptable (Fink & Kosecoff, 1985).

Increasing Compliance

There are several ways to increase respondent cooperation. First, respondents can be contacted before they are mailed the questionnaires, either by letter or telephone, to let them know a survey is on the way. The form itself should arrive with a cover letter, and a self-addressed stamped envelope for returning the completed form.

The cover letter should describe the purpose of the study and explain that all answers are confidential. There should be a phone number that respondents can call with questions, and instructions about what to do with the completed form. If the respondent will receive any type of compensation or token of appreciation, this should be described as well (Mangione, 1998).

The questionnaire should be as concise as possible. The practice of grouping multiple items together in order to save paper should be avoided if at all possible. If the form looks like it will be difficult to complete, the respondents are less likely to participate. To the extent that it is possible, try to have the form pre-coded since this format is faster and easier for the coder, and respondents have only to circle or check a response rather than writing out an answer (Sudman & Bradburn, 1982). During pre-testing of the form, the length of time it will take a respondent to complete the form can be determined. Trim as necessary.

Good follow-up will also increase compliance. After a pre-determined amount of time has elapsed since mailing the questionnaires (e.g., 3 to 4 weeks), reminder post cards or letters can be sent to respondents who have not yet returned their surveys. Also, additional survey packets can be sent (even though this increases the cost). This can be repeated two or three times, with the compliance rate increasing with each mailing. Follow-up telephone calls can also be used, if feasible (Mangione, 1998). Below is an example of a USAF program that was evaluated via a mail survey.

In reviewing the above example, one might wonder whether all the additional effort of Bowen was excessive. Nevertheless, the decision about the amount of follow-up of respondents is an important one that will need to be made. In almost all cases the compliance rate is more important than raw numbers. The information gathered is likely to be more accurate if there is a higher compliance rate. For example, if the evaluator were to send out 1,000 questionnaires with 700 returned (compliance rate=70%), this would be preferable to 700 questionnaires returned out of 2,000 (compliance rate=35%). The money saved by not sending out the additional 1,000 questionnaires could be used to send reminders.

Bottom Line: If financial resources are limited, the best strategy is to send fewer questionnaires with good follow up (increasing the compliance rate).

Self-administered questionnaires can also be completed in person. Questionnaires can be distributed to clients when they come into the office for appointments, or when a home visitor sees a client in his or her own home. The compliance rate for these types of questionnaires is likely to be higher than when the questionnaire is mailed, but the form will still have to be concise so that it can be completed in a brief amount of time. The main concern is distributing the questionnaires in a non-biased way. The evaluator will need to develop a protocol for distributing questionnaires and tracking refusals. For example, the evaluator may decide that all clients who come into the office on Tuesday and Thursday will complete a questionnaire, or all clients who are seen between April and June. Avoid approaching only those clients who appear "nice," thereby potentially biasing the sample. It is important to track refusals so that a compliance rate can be calculated, and if possible, to obtain some basic demographic data on those who refused.

• Interviews

The second major type of survey is the interview. Interviews can be conducted in-person or over the telephone. The interviews described in this section are generally not the open-ended kind described in Section 5. This section refers to a standardized interview, where the questionnaire form functions like a script (Rosenthal & Rosnow, 1984). The questions are to be read to the respondent exactly as worded. As previously described, question-order effects are a distinct possibility. Only by administering the same questionnaire to everyone can the evaluator control for this type of effect. When question order is the same, differences observed between subjects are not due to question order but are due to some other factor (such as the intervention).

In-person interviews are the most labor intensive of all survey methods, especially if the interviewer must travel to the respondent's home. But they often yield the most complete data. In-person interviews are most feasible when combined with services (such as interviews administered by the home visitor). However, it may not be practical to combine service-provision with interviewing because of time-constraints and/or lack of training for the service provider in standardized interviewing techniques. Whether in-person or on the telephone, interviewers need to be trained in how to handle non-responses and how to keep respondents on-task. They also need practice in using and accurately administering the questionnaire (especially when the questions are complicated or have multiple parts). For most evaluators' needs, telephone interviews may be the most practical. These are described next.

• Telephone Interviews

Telephone interviews are an efficient method of collecting some types of data. They are equivalent to in-person interviews and are often a reasonable alternative (Bradburn, 1983). They offer several advantages (Lavrakas, 1998). The first is quality control. An interviewer administers the questionnaires so that there are fewer missing data. They are cost-effective, especially when compared to in-person interviewing. They are also best when speed is required (e.g., information needed right away in order to complete a report).

As with in-person interviews, the interviewer needs to read the questionnaire like a script. Questions and response categories need to be as simple as possible since the respondent will not be able to see the question. For example, a question with seven possible response categories would be fairly simple to answer in a self-administered format. But the evaluator may find that limiting responses to three categories works better over the phone (Lavrakas, 1998; Sudman & Bradburn, 1982).

Interviews of both types tend to have higher compliance rates than mail surveys (Fink & Kosecoff, 1985). However, the evaluator may want to follow some of the same guidelines described above to increase participation rates. A post card or letter can be sent to the respondent describing the study, and letting them know that an interviewer may be calling. The evaluator could make a pre-interview contact to find out when a good time to call would be, and to explain the purposes of the survey and that all responses will be confidential. Respondents can also be told about any incentives that they will be offered. It is important to develop a protocol for when a family will be considered a refusal (e.g., six unreturned telephone calls), and to track the number of clients/families who refuse to participate. The evaluator might also decide to conduct some interviews in person, especially among those who are impossible to interview by telephone (Lavrakas, 1998).

• Who Should Be Interviewed?

One of the first steps in survey approaches is to identify a sampling frame. The sampling frame is the operationalization of the target population-a list of potential subjects to be surveyed. Should the evaluator sample from the general population of families that program services are directed towards (such as services that might be considered for a primary prevention program)? Or would it be better to sample from an already identified population-such as families being served in an existing program? A sampling frame list can be generated from directories or membership lists (e.g., involvement in community groups). Another technique unique to telephone surveys is Random Digit Dialing, where a computer generates telephone numbers. Random Digit Dialing may not be appropriate for some populations. For instance, Random Digit Dialing may not be appropriate if it would draw in many individuals who have the same prefix but do not represent families in the program. On the other hand, it might be effective if most of the families the evaluator is interested in live within certain prefixes. (For detailed information on telephone surveys, please see Lavrakas, 1998). A technique that goes hand-in-hand with telephone interviews-computerized interviews-is described next.

• Computerized Interviews

Until recently, interviewers completed questionnaires that were printed on paper, and eventually coded into computerized databases. Many large-scale telephone surveys are now conducted using Computer-Assisted Telephone Interviews (CATI). With CATI, the interviewers use a computer terminal to administer and record survey questions and responses. The survey questions appear on a computer screen, the interviewer reads the questions to the respondent, and then uses the keyboard to directly enter the respondents' replies as they are given (Lavrakas, 1998).

One advantage of CATI is increased quality of data. The CATI interviewer works from a computer screen, which is programmed to show questions in a planned order. Therefore, it is more difficult for the interviewer to accidentally omit questions or ask them out of sequence. Often, answers to some questions require using "skip patterns" (e.g., "if the answer is 'yes' skip to question 10"). The correct branching can be done automatically with CATI. In telephone interviewing without the use of CATI, following skip patterns incorrectly can be a source of error.

An additional advantage is improved monitoring of the data collection process. Because data are instantly computerized, a data set can be created and data can be analyzed early in the process. This can allow early detection of problems with the process, including problems with questions or interviewers.

Cost savings may not be a benefit for small, non-repeated surveys, due to the cost and labor involved in programming the questionnaire. However, the cost per interview with CATI decreases as sample size increases -- so in large and/or repeated surveys, CATI becomes a more cost-efficient survey method, competitive with conventional telephone methods. As this technology becomes more widely available, computerized questionnaires may become an option from which to choose.

• Incentives

To thank respondents for their time, the evaluator may decide to give them a token of appreciation. This could include a certificate indicating their participation in the study. If the program involves children, the evaluator might consider offering gifts for them (e.g., small toys or books, T-shirts). Some other ideas include gift certificates for local restaurants or fast-food establishments, or movie tickets, or vouchers for other types of goods or services.

• Survey Etiquette

These etiquette and ethics issues are common to all surveys. First, the interviewer should always thank the respondent for their willingness to participate. Second, out of respect for their time, the survey should be kept as short as possible and include only what is absolutely necessary. The interviewer must be sensitive to the needs of respondents at all times during the interview, and alert for any sign that they are uncomfortable. If respondents are completing the survey in person, refreshments could be made available. Respondents may also need something to distract and entertain their children while they are answering the questions. To aid in future follow-up, it would be appropriate to ask respondents if they could be contacted again, and locator information collected, as described in Section 3 (Subsection "Retention"). At both the beginning and end of the interview, the interviewer must make certain that the respondent understands that his or her responses will be confidential. This confidentiality reminder is also helpful just before questions are asked on sensitive topics. Respondents need to be informed about how their data will be used, who will have access to it, and that it will only be reported in aggregate form (e.g., a summary of answers across respondents). At the end of the survey interviewers can close on a positive note by thanking respondents again, and giving any incentives that were promised.

In this section, information on how to develop specific, tailored survey instruments was provided. Standard measures may also be added as part of the data collection for outcome assessment.