The OIRA Module: Published Research

From: RegBlog

Cary Coglianese

Over  the past few decades, governments around the world have established procedures to try to analyze the impacts of new regulatory proposals before they are adopted.  By contrast, they have paid remarkably little attention to analyzing regulations after adoption.  Admittedly, some countries have begun to undertake modest efforts to examine the impacts of regulation retrospectively.  Under the Obama Administration, for example, agencies across the United States government have undertaken a formal “look back” review to identify and remove outdated regulations. But by and large these kinds of look-back efforts, however valuable they may be for tidying up the law books, fall far short of the kind of rigorous evaluation research needed to determine which regulations are causing positive outcomes and which ones are not.

To  be able implement better evaluation efforts, and thereby better inform future policy decision making, government officials need to adhere to a framework of best practices in ex post evaluation of regulation. They need to follow a methodological roadmap for institutionalizing the kind of systematic evaluation research that will generate a much improved understanding of the actual effects of regulation. Such a roadmap will include both relevant and reliable indicators to measure relevant outcomes of concern as well as solid research designs to support credible inferences about the extent to which a regulation has actually caused any change in the measured outcomes.

Indicators for evaluation should focus on the specific problems addressed by the regulation under evaluation, focusing whenever possible on the ultimate problem or concern.  Indicators can be grouped into three main types:

1.      Impact (changes in the problem or other outcomes of concern);

2.      Cost-effectiveness (costs for a given level of impact); and

3.      Net Benefits (all beneficial impacts minus all costly impacts).

Generally speaking, a net benefits measure will make the best indicator as it seeks to capture and incorporate into one unit all the impacts of a regulation, both positive and negative.  However, if the benefits of a regulation cannot be placed into monetary terms to facilitate a calculation of net benefits, the evaluator can rely next on cost-effectiveness, which measures the cost per nonmonetary unit of benefit.  Should cost-effectiveness not be feasible, an evaluation can simply focus on discrete impacts, such as health, environmental quality, or security.

In addition to selecting appropriate indicators, policymakers and public managers need to support evaluations based on careful research designs. In order to be able to attribute with confidence any changes in indicators to the regulation under evaluation, research designs should aim to emulate conditions in a laboratory experiment.  The best research design will be the randomized experiment, which could be used much more extensively than it is at present in measuring progress about many issues of public policy.  When randomized experiments are not feasible, evaluations can be based on observational studies which use a variety of statistical methods to isolate the effects that can be causally attributed to the policy under evaluation.  If quantitative observational studies are not feasible, evaluators can rely on qualitative studies, such as matched case studies, that seek to control for other influences as much as possible.

To be able to improve regulatory performance, policymakers first need to know what difference regulations actually make, whether for good or ill. Yet at present governments around the world have been operating too much in the dark. They need to devote greater attention to selecting reliable indicators and appropriate research designs in order to conduct more and better ex post regulatory evaluation.  Institutionalizing practices of rigorous ex post evaluation will help ensure more informed decision making in the future about regulation, as techniques of ex ante regulatory analysis can only be validated when we know better ex post how regulations actually work in practice. Equipped with a sound methodological roadmap for improving the ex post evaluation of regulations, governments should move forward to establish institutionalized evaluation practices for their major regulations, creating better data systems and fostering the kind of independent research bodies that will undertake this much-needed work.