7.5. General Accounting Office Review
The General Accounting Office (GAO) has also studied the RIA issue. The GAO study focused on EPA regulations although other agencies also produce RIAs. GAO reviewed three RIAs and the report largely reflects conclusions drawn from that effort although it is not a set of case studies:
- "We performed a detailed review of three EPA analyses which EPA and/or OMB identified as state of the art. These analyses were of the effluent limitation guidelines for the iron and steel industry, and the organic chemicals, plastics, and the synthetic fibers industry, and the ambient air quality standard for particulate matter. EPA and OMB indicated that other EPA analyses included only a very rudimentary calculation of benefits, and we therefore did not include them in this review." (p. 4)
Early in their critique, GAO categorizes their comments on the assessments as follows:
- "We addressed three factors ... These three factors involve the adequacy of the data used to determine the state of the art of cost-benefit analysis as applied to environmental problems, statutes which prohibit or limit the use of cost-benefit analysis results in environmental rulemaking, and problems that EPA has encountered in implementing EO 12291." (p. 4)
GAO devotes a section of the report to each of the categories. The first category as described in the summary statement above is entitled “Data Gaps Prevent Cost-Benefit Analysis From Providing Exact Answers.” However, the noted gaps are not only in data but also in technique in both science and economics.
GAO offers a few examples of data gaps that are more science than economics:
- "In analyzing the effluent limitation guidelines for the iron and steel industry, EPA noted a number of basic data weaknesses. For example, knowing how much cleaner the water will be from lower discharges of pollutants was described as “... one of the weaker links in benefit analyses of water pollution controls.” (p. 8)
- "In analyzing the effluent limitation guidelines for the organic chemicals, plastics, and synthetic fibers industry, EPA quantified health benefits using methods in which health risks to humans from low-level doses of pollution were based on high-dose animal experiments. EPA’s contractor described these methods as debatable and referred to the estimates as the '... best guess of the unknown.' (p. 8)
- "In analyzing the benefits of an air quality standard for particulate matter, EPA noted that the underlying theory of how particulate matter affects human mortality is not known. As a result, different epidemiological studies EPA reviewed came to conflicting conclusions about the dangers of such pollution." (p. 8)
GAO also has reservations about the economics side of cost-benefit analyses. As to benefits, they question the reliability of both property value and wage studies based on both quality and quantity of data on the one hand and on the difficulty of distinguishing pollution’s effect from other factors such as smoking. GAO, op. cit., p. 9. Next, GAO expresses skepticism about contingent valuation, “surveys” in their parlance, based on methodological grounds. GAO, op. cit., p. 10. After disparaging the various methods for valuing benefits, they conjecture that cost-effectiveness might be better! GAO, op. cit., p. 10. However, those comments are followed by several paragraphs questioning cost-side estimates on grounds of weak abatement data and modeling of physical and economic relationships. GAO, op. cit., pp. 10-11.
GAO is wary of cost and benefit estimates. They indicate that EPA should address this uncertainty:
- "Despite data weaknesses, cost-benefit analysis can identify ranges of estimates where a single estimate is not very accurate. These estimates can be useful if they represent a range of likely values associated with a particular benefit or cost. The analysis can also trace the effect that a particular estimate chosen from such a range will have on the net benefits and desirability of a given regulatory alternative. This approach can reduce uncertainty to more manageable proportions because a decisionmaker can select a particular alternative without having to rely solely on guesswork." (p. 11)
Although risk analysis is appropriate, GAO wisely seems to stop short of claiming that it can remedy data deficiencies. GAO notes that:
- "In dealing with uncertainty caused by data weaknesses, two of the three cost-benefit analyses we reviewed emphasized single values rather than ranges of dollar estimates for costs and benefits. However, EPA’s particulate matter analysis is a good example that shows how net benefits – and the resulting ranking of regulatory alternatives – change depending on what health studies are used to calculate benefits." (p. 11)
Finally, in concluding their discussion of risk and uncertainty, they suggest that Monte Carlo simulation might be an appropriate technique to use. GAO, op. cit., p. 12.
The GAO document turns next to the second of the three categories of concerns, laws that prohibit or limit the use of cost-benefit analysis. There is a cursory enumeration of statutes that do and don’t limit such analysis. Provisions of the Clean Air Act and the Occupational Health and Safety Act seem to preclude it whereas the Clean Water Act is permissive. However, regarding the latter, the Act seems to require industry-by-industry limits which tend to vitiate economically efficient forms of regulation. GAO, op. cit., pp. 15-17.
Although various statutes preclude or limit the use of cost-benefit analyses in regulation, their preparation is still required under the executive orders. Regarding such analyses, GAO concludes:
- "We believe that cost-benefit analyses that are ... not used because of legal restrictions should be sent ... to the Congress in order to assist it in carrying out its oversight responsibilities.
- "We also believe ... Congress may want to reconsider legislative provisions which prohibit or limit the use of cost-benefit analysis ...." (p. 21)
If GAO’s first area of concern could be called state-of-the-art problems, then its third deals with more mundane, correctable defects it has observed, particularly in the three assessments it reviewed in depth.
GAO asserts that EPA has ignored new source compliance costs. For some rulemakings, the effect may have been to avoid the requirement to prepare a cost-benefit analysis which applies to rules with annual costs exceeding $100 million. GAO offers examples:
- "EPA has determined that 16 of 18 proposed effluent limitation guidelines regulations we reviewed were not major rules, as defined in Executive Order 12291, because they did not have annual effects of $100 million or more ... In all but one of these guideline regulations, new source compliance costs were not included ... Had EPA always included compliance costs of new sources ..., then 2 of the 16 regulations ... could likely have cost more than $100 million annually ...
- "EPA published ... a proposed effluent limitation guidelines regulation for the electroplating and metal-finishing industry. In the proposal, EPA estimated that the regulation, if promulgated, would have an annual cost effect of $92 million; therefore, EPA did not consider it a major rule [requiring a cost-benefit analysis].
- "... EPA did not include annual compliance cost estimates for new sources. Yet, EPA recognized that the regulation would affect both new and existing sources. Because EPA did not include new source compliance costs in the annual cost estimates, EPA could not be certain that the proposed regulation would cost less than $100 million." (p. 23)
GAO’s next set of criticisms pertain to the choice of alternatives evaluated. The first of these is that EPA has not considered enough alternatives.
- "EPA considered only one regulatory alternative in each of the two effluent limitation guidelines analyses.
- "In the cost-benefit analyses prepared for the iron and steel industries and the organic chemicals, plastics, and synthetic fibers industries, EPA performed an in-depth cost-benefit analysis only on one alternative to the existing regulation. ... in the iron and steel analysis, EPA compared the costs of industrial compliance for different treatment scenarios with the corresponding reductions in pollution. Having done this, EPA chose to estimate dollar costs and benefits for only one of these scenarios.
- "The organic chemicals, plastics, and synthetic fibers cost-benefit analysis also estimated the dollar costs and dollar benefits of only one regulatory strategy. That analysis stated that other alternatives, such as the “bubble” concept, could have been considered, but they were not made part of the analysis." (pp. 25-26)
Closely related the foregoing criticism is the comment that the alternative yielding maximal net benefits was not properly identified by bracketing it with inferior studied alternatives:
- "The cost-benefit analysis EPA prepared for the ambient air quality standard on particulate matter examined 10 alternative approaches, which is more in keeping with the intent of Executive Order 12291....
- "However, the analysis concluded that a wider range of alternatives was needed to identify the most efficient regulatory level because the most stringent alternative considered ... showed the greatest net benefits. This raises the question as to whether net benefits would have continued to increase had additional, more stringent alternatives been considered ... EPA needs to identify the regulatory alternative where net benefits peak ..." (pp. 26-27)
Among the last of GAO’s implementation comments is one concerning presentation. They would indicate uncertainties more clearly, particularly in executive summaries often read by decision makers.
"Two of the three cost-benefit analyses we assessed included single point estimates rather than a range of estimates of either costs or benefits in the executive summary section. ... [Distributions] appeared in other EPA or contractor documents, making it difficult for decision-makers to appreciate the range and significance of uncertainty." (p. 27)
Finally, GAO levels the complaint that the RIAs ignore various categories of costs and benefits and do so inconsistently across the three they studied. GAO, op. cit., pp. 28-29. One example of a neglected category previously cited is new source compliance cost. Another neglected category is administrative monitoring and enforcement costs.
In sum, the GAO report categorizes problems into three areas: the state-of-the-art, which EPA can improve over the longer run; statutory restrictions, about which EPA has little control; and implementation problems that are largely under EPA’s control. In its conclusion, GAO states, seemingly with reference to the third class of problems:
- "The problems we found in EPA’s cost-benefit analyses can, in our judgment, be readily corrected by following EPA’s recently adopted guidelines. To improve the usefulness of EPA’s cost-benefit analyses as decisionmaking and informational tools, however, EPA needs to include and prominently document, in executive summary form, all cost and benefit categories, areas of uncertainty, and all of the most promising alternatives in future analyses." (p. 31)