Section 4: Cost Analysis

Potential Problems (and Solutions) in Cost Analysis

Even in the planning stages of a cost analysis it is common to encounter resistance to cost analysis and to experience some of its limitations and liabilities. These problems are likely to present fewer and less serious difficulties if they are acknowledged and dealt with from the very beginning.

Perhaps the greatest fear of program management and staff is that detailed financial information may invite unjustified and misinformed comparisons. To avoid such problems and allay fears about misuse it is important to involve program staff and other stakeholders in the design, implementation, and interpretation of the cost analysis. One way to do this is to form a steering committee to guide the cost analysis through each step. With regard to program evaluation, the involvement of representatives, both from those expected to use the results as well as those who will be substantially affected by their use, is essential for producing a useful analysis. It is also essential for preventing potential mistakes that could occur in its use.

A distinction that may be useful for addressing fears about misuse of cost data is between the use of cost data for monitoring and auditing, and the use of cost data to reflect on practice patterns. Generally, these two uses should not overlap. If program management and staff perceive that the cost data will be used for monitoring and auditing, their cooperation in using the data for reflection on practice patterns will be limited. Using cost data to reflect on practice will usually mean identifying where there is uncertainty (such as the frequency and length of home visits) and using the data to reduce that uncertainty.

Other potential problems and possible solutions in cost analysis are:

Problem: The detailed analysis of how staff members use their time, an important part of cost analysis, can be threatening, and this may lead staff to provide inaccurate information.

Solution: The evaluator can inform staff well in advance what will be done and why. If alternative ways of collecting time-use information are considered, the staff can be asked for their ideas. Staff might be asked to keep a time log for a week before actual collection of data for use in analysis. This will allow staff to develop habits for recalling and recording the information that may increase accuracy, and it provides an opportunity for feedback on how well the system works. They should not be asked for more detail than is really needed. Asking for less detail will be less burdensome, less threatening, and will produce more accurate reports.

Problem: Like everything else, cost analysis takes time and may compete with other program activities.

Solution: The users must weigh the potential benefits of cost analysis against the potential benefits of other activities, recognizing that the goal of cost analysis is to help develop and operate the best program possible. Involvement of staff and their representatives in the planning will help them understand the importance of the data collection.

Problem: Program costs may be too complex for analysis without outside assistance.

Solution: Evaluators of most programs will be able to conduct their own cost analysis with the aid of this chapter. When considering a cost analysis, this section can be scanned to determine what parts of the analysis might require assistance. If the evaluator is considering a more ambitious undertaking such as a cost-effectiveness or cost-benefit analysis, it is more likely that assistance will be needed. Someone with extensive experience in the desired type of analysis is preferred, since economists specialize in many different types of research and study.

Problem: Cost data may encourage decision makers to rely too heavily on cost factors when making decisions about meeting human needs.

Solution: The evaluator can counter this tendency by providing decision makers with more complete information. At the very least, the evaluator should provide them descriptions of the program, children, families, and staff involved so that they can see the 'human faces' behind the numbers. Every attempt should be made to provide the information needed to provide understanding of the consequences of funding decisions for operation and outcomes.

Combining Cost and Outcome Data

Although cost analysis can be quite useful by itself, it can be even more useful when it is combined with outcome data from an evaluation. There are a number of different ways in which cost and outcome data can be combined, the most common being cost-effectiveness analysis and cost-benefit analysis. Cost-effectiveness analysis generally is quite straightforward once one has both cost and outcome data are available. Cost-benefit analysis is much more complex and the reader is referred to other sources for conducting such analyses.

Cost-effectiveness Analysis

Typically, cost-effectiveness analysis is used to compare two or more alternative ways of achieving the same goal. In this type of economic analysis, the net monetary value of the resources used to produce a program or policy is estimated and compared with the outcomes that are not valued monetarily. For example, cost-effectiveness analysis has been used to compare the effectiveness of home visit programs that differ in the frequency with which they provide home visits. Clearly, it is the case that more frequent visits are also more expensive. However, are they necessarily more effective? Furthermore, is there a point at which additional visits yield little or no improvement in the desired program outcomes or perhaps are even counter-productive? Simple experiments can be designed to answer such questions. For example, one study that evaluated the effects of home visits on child development found that biweekly visits resulted in no diminution of effects on the cognitive ability of children at risk for poor developmental outcomes and were much less costly than weekly visits. A change to biweekly visits made it possible to serve nearly twice as many children and families with the same resources.

Unfortunately, cost-effectiveness analysis has limitations such as when an option is both more costly and more effective or when there are multiple outcomes. In such situations, it is essential that those responsible for the program have a clear understanding of the relative importance of cost savings and possible outcomes. The table below presents hypothetical cost-effectiveness results for alternative intervention programs that seek to improve birth outcomes and prevent repeat pregnancies for teenagers. Are Programs B and C worth the roughly $2,000 increase in cost over Program A? The answer depends on the priority assigned to the outcomes. Both programs have fewer low birth weight babies and repeat births than Program A, but is the margin of difference worth the additional $2,000 per child? If Programs B and C are both worth the extra money, which should be chosen? Program B has a larger effect on low birth weight but a smaller effect on repeat births. Which outcome is more important? Answering these questions requires that one go beyond cost-effectiveness analysis. It should be noted that this problem is not peculiar to cost-effectiveness analysis; efficacy studies have the same problem. When programs have multiple objectives, effectiveness has no clear measure.

• Cost-benefit Analysis

Cost-benefit analysis is an attempt to directly address the limitations of efficacy and cost-effectiveness studies by translating all costs and effects into a common unit of measure-money. This type of analysis attempts to value all of a program's outcomes, as well as its costs, in monetary terms. It can be used to study a single program (a program, no-program comparison), to compare two or more programs with the same goals, or to compare programs with entirely different goals. The greatest problem for cost-benefit analysis is that it can be very difficult, if not impossible, to estimate a monetary value for some outcomes. It is important to note, however, that some potential outcomes of prevention and family support programs are easily valuated in monetary terms. One example of such an outcome is increased productivity in the work place for clients. In some cases, looking at the dollar value of just a few outcomes can reveal that an intervention pays for itself (the benefits exceed the costs). There is a great potential for estimating the value of many of the benefits of family support, mental health, and other prevention-oriented programs. However, the lack of previous work in this area means that there is little to build on at present, and pioneering efforts will be required. Those wishing to pursue the topic further are referred to a detailed example of cost-benefit analysis applied to an early education program by Barnett (1993) and to classic texts by Mishan (1976) and Thompson (1980).