CRE Homepage About The CRE Advisory Board Newsletter Search Links Representation Comments/Ideas
Data Access
Data Quality
Regulation by Litigation
Regulation by Appropriation
Special Projects
CRE Watch List
OMB Papers
Abstracts and Reviews
Regulatory Review
Voluntary Standards Program
CRE Report Card
Public Docket Preparation
Consumer Response Service
Site Search

Enter keyword(s) to search

®: CRE Regulatory Action of the Week

CRE Submits Comments to EPA on the Agency's Data Quality Guidelines
EPA, in a precedent-setting move, has requested public comments prior to proposing the agency's conforming Data Quality guidelines. CRE applauds this step designed to promote early public involvement and input, and urges other agencies to adopt this approach. CRE has taken this opportunity to address a number of issues which should be considered by other federal agencies in drafting Data Quality guidelines. CRE is making its comments available immediately so as to be useful in the context of the Data Quality Workshop being hosted by the National Academy of Sciences on March 21-22. CRE encourages all interested parties to submit their input on CRE's comments.

  • Read review CRE's comments to EPA on the agency's Data Quality guidelines.
  • Submit a comment to the EPA
  • Submit a comment to CRE
  • CRE Regulatory Services

    March 22, 2002

    Ms. Evangeline Tsibris Cummings
    Office of Environmental Information
    Environmental Protection Agency
    Ariel Rios Building
    1200 Pennsylvania Avenue, N.W.
    Washington, D.C. 20460

    Re: Comments on EPA Data Quality Guidelines

    Dear Ms. Cummings:

    Pursuant to EPA's announcement in the March 14, 2002 Federal Register (67 Fed. Reg. 11483), I am writing on behalf of the Center for Regulatory Effectiveness (CRE) to provide the Center's comments to the Agency as it seeks to develop its Data Quality guidelines, pursuant to Section 515 of the FY 2001 Treasury and General Government Appropriations Act (Pub. Law No. 106-554, H.R. 5658). I would like to start by commending the Agency for seeking broad public input during the early development phase of its guidelines, even before proposed guidelines are issued for public comment. CRE believes that such involvement will ultimately lead to a better product and is a model which other agencies should follow.

    CRE has been actively involved in the Data Quality issue (see the Data Quality section of CRE's website for details on the Center's involvement), and CRE is committed to helping federal agencies achieve implementation of the Data Quality Act by meeting the objectives of Congress, as interpreted by OMB. CRE acknowledges that EPA has been a leader in terms of its efforts to promote information quality at the Agency, not the least of which has been establishment of the Office of Environmental Information.

    The Center also believes that EPA's development of Data Quality guidelines, as required by statute, will provide an additional opportunity to foster strong data and information management at the Agency. To this end and as the following comments explain, CRE believes that EPA should make statements regarding both the short-term and long-term goals of its Data Quality Program.


    CRE Recommendations for EPA Data Quality Guidelines

    CRE is pleased to offer the following recommendations to EPA, which will be supplemented once the Agency issues proposed guidelines in the Federal Register and requests further public comments.

    (1) EPA should make clear that the Data Quality Program, as encompassed in its Data Quality guidelines, is an evolutionary process which will involve ongoing review and improvement by the Agency.

    (2) EPA should formally commit to the development of a long-range plan for improving Data Quality at the Agency. In this context, CRE urges consideration of the following goals:

    Short-term Goals

    Completeness of Data

    • In its guidelines, EPA should provide for the inclusion of all relevant and reliable scientific information, from both animal and human studies. If there are inconsistencies, these should be explained, along with a discussion of how they might be reconciled.
      • Such approach is demanded by the OMB directive to adopt or adapt the quality provisions of the Safe Drinking Water Act Amendments of 1996 and the definition of "objectivity" as requiring completeness.
    • The Data Quality Act and OMB's guidelines preclude EPA's categorical ban on the use of third-party clinical human test data.
    • EPA should adopt, not adapt, the Data Quality standards set forth in the Safe Drinking Water Act as part of its Data Quality guidelines.

    Use of Third-Party Proprietary Models

    • Federal agencies often use various models developed by third parties (often government contractors) to formulate policies based upon influential scientific information. These models are sometimes asserted to be confidential and proprietary.
    • CRE's recommendations regarding third-party proprietary models are expressed in more detail in the attached policy paper along with suggestions on implementation, but the crux of the Center's recommendations in this context are:
      • EPA should adopt a general prohibition against third-party proprietary models.
      • Use of third-party proprietary models conflicts with the goals and intent of the Data Quality Act (e.g. related to reproducibility).
      • Public disclosure of third-party proprietary models should be required in all but the most unusual circumstances.
      • If EPA believes it must use third-party proprietary models in order to carry out its regulatory duties and functions, then EPA should have the burden of demonstrating to OMB, before entering into a contract to use the model, that no other option is available.
      • EPA's proposed and final Data Quality Act guidelines should explain in detail what "especially rigorous robustness checks" will be applied to third-party proprietary models that EPA and OMB agree must be used. EPA's guidelines should further explain how the public will be informed of these "robustness checks."

    Risk Assessments

    • EPA should specifically address risk assessments as a special type of information disseminated by the Agency. The OMB guidelines address risk information as such a special category.
    • As noted above, the OMB rules require that risk information be objective and unbiased. To the extent EPA uses "conservatively" biased policy assumptions, uncertainty factors, and data selection in risk assessments/analyses (as opposed to risk management analyses), the possible conflict between the OMB guidelines and these actions should be reconciled. Such reconciliation may be achieved by clearly separating an objective scientific analysis of what is know and not known about adverse effects and the circumstances under which they occur, from a risk management options analysis that discusses whether or not to incorporate policy assumptions or decisions for any regulatory decision that might be made on the basis of the risk assessment.
    • EPA should also modify its use and dissemination of hazard information, given the policies set forth in the OMB guidelines. The statute and OMB rules require agencies to ensure and maximize the utility of information. More specifically, hazard information is often disseminated without appropriate caveats that risk B as opposed to hazard B depends on level of exposure, thereby neglecting context that is essential to the apprizing the public of the utility of the information. 
    • An EPA cumulative risk assessment is only as good as the risk assessments of the individual substances that underlie the cumulative risk assessment. Consequently, EPA's guidelines should emphasize that the Data Quality Act standards must be met first by the individual risk assessments before any cumulative risk assessment can be performed. Moreover, if more recent and better data are available than were used in the individual assessments, then those data should be used for the cumulative risk assessment. Standardized data are scientifically necessary for conducting cumulative risk assessments. EPA's guidelines should also require that an approved and properly validated methodology be used to conduct cumulative risk assessments. Finally, all models used for cumulative or individual risk assessments should be publicly available, and adequate time for comment and peer review allowed, before the models are used. Proprietary model issues are also addressed elsewhere in these comments.

    Information Related to Fees

    • EPA is permitted by statute to charge fees for various services it provides to the regulated community. Economic analyses are routinely conducted by the agency to determine the level of such fees, although such calculations are not always fully transparent.
    • These economic analyses related to fees are subject to the Data Quality Act, and OMB's guidelines state that analytical methods (models) should be publically available to facilitate the reproducibility of such information by qualified third parties. In this context, CRE recommends that EPA disclose the underlying data, models, and assumptions used in determining the fees set by the agency. Such action would not only serve the goals of Data Quality, but would increase transparency as well.

    Applicability of the Data Quality Guidelines to Third-Party Information Submitted to the Agency

    • Because the Data Quality Act applies to essentially all information which the Agency disseminates to the public, the same standards will apply to information developed by third-parties and submitted to EPA for Agency publication or related regulatory action. EPA should make a clear statement that information submitted to the Agency must comply with the Data Quality guidelines, if:
      • A third party petitions for correction or withdrawal of the information; or
      • The Agency itself decides to disseminate this information or to adopt its substance and take regulatory action.

    Definition of the Term "Unbiased"

    • Scientists and the regulated community have complained for decades that EPA risk assessments are unduly biased by policy assumptions. The OMB guidelines require that risk assessments be unbiased. Thus, CRE believes that EPA should bifurcate its risk analysis into: (1) an objective consideration of the science and (2) a review of policy decisions that might be used to make risk management decisions.

    Deadlines for Agency Action on Data Quality Petitions

    • In accordance with the OMB guidelines, the agency should set a date specific for action on Data Quality petitions once they are received by the Agency.
    • However, as part of the procedural process for Data Quality petitions, the responsible Agency official should be responsible for reviewing each petition filed to determine those that are relatively simple or uncontroversial and, therefore, can be decided in an expedient fashion prior to the deadline for action.
    • In addition, EPA's Data Quality petition procedures should establish a process for accelerated review of petitions and describe the circumstances when accelerated review is appropriate.

    Publication for Rules of Procedure for Data Quality Petitions

    • In addition to accelerated review procedures, CRE believes that the Agency and stakeholders would benefit from EPA's development of a set of overall written procedures for the Data Quality petition process.
    • These procedures should be published in the Federal Register in proposed form for public notice and comment.

    Availability of Administrative Appeals

    • CRE believes that appeals under the Data Quality petition process should only be available to the Petitioner. The Agency should not be allowed to challenge positive action on the petition at a lower level within the Agency. 

    Standard of Care for Agency Information Prior to Issuance

    • EPA should specify the process the Agency intends to establish to ensure quality of the Agency's information and compliance with the Data Quality Act prior to the issuance and dissemination of such information.
      • This process should identify which official will be conducting such review and the steps that will taken in considering compliance with the Act, OMB's guidelines, and EPA's own guidelines.

    Higher Level of Quality for "Influential" Information

    • The OMB Data Quality guidelines mandate a higher level of quality for "influential scientific, financial, or statistical information." Consequently, the Agency should clearly and explicitly address all relevant aspects of what will constitute a higher level of quality for such influential information.

    Decisionmaking in the Petition Process

    • EPA must ensure that the decisionmaker(s) in the administrative petition process B at both the initial consideration stage and the administrative appeals level - is competent to handle the substance of the petition and sufficiently independent to make a decision on the petition.
      • CRE believes that the petition should initially be reviewed and decided within the relevant Program Office.
      • CRE believes that if a petition is denied, the appeal should be reviewed and decided by an EPA official outside the Program Office in question. Consideration might even be given to establishing an independent office within the Agency for the purpose of deciding Data Quality Act appeals.

    Treatment of Inaccurate Data During the Course of Administrative Proceedings

    • If the designated EPA decisionmaker becomes aware of an inaccuracy in data during the course of a petition review, that official should have the authority to suspend or withdraw dissemination of the inaccurate information until such time as it can be amended or clarified.

    Long-term Goals

    • Building upon its Agency-wide Data Quality guidelines, EPA should develop and adopt specific Data Quality guidelines for each of the Agency's major Operating Offices. This approach would provide additional clarity and consistency to both EPA staff and stakeholders by addressing data issues in a program-specific fashion.
    • More specifically, EPA should announce in a proposed rule a plan for the development of Data Quality guidelines for each Operating Office. The proposal should be published in the Federal Register for public comment and should address, at a minimum, the following elements:
      • List of Operating Offices for which targeted Data Quality guidelines would be developed;
      • Articulation of a transparent agency process for development of Data Quality guidelines for EPA Operating Offices, including ample opportunities for public comment; and
      • Timeframe for Agency action.

    Again, the Center appreciates the opportunity to provide EPA with its comments at this early stage in the agency's Data Quality guidelines development process. CRE strongly urges the Agency to make the comments it receives in this round publicly available, preferably through the EPA website. Such step would permit interested parties to consider the views of other thoughtful respondents, thereby raising the level of debate on this issue and the overall quality of input provided to the Agency.

    Finally, CRE looks forward to reviewing EPA's draft guidelines once published and providing further input at that time. Should you have any questions regarding CRE's comments, please feel free to contact me at (202) 265-2383.


    Jim J. Tozzi
    Member, CRE Board of Advisors




    Federal agencies often use various models developed by third parties (often government contractors) to formulate policies based upon influential scientific information. The third-party models are sometimes asserted to be confidential and proprietary.

    This issue does not involve the concerns that arise when regulated entities are required to submit confidential or proprietary data to an agency pursuant to a regulatory program. Instead, this issue is limited to situations where an agency and a contractor agree to use a model on a proprietary basis to develop influential scientific information.

    OMB's guidelines implementing the Data Quality Act, 44 U.S.C. § 3516 note, require that influential scientific information be reproducible. This reproducibility standard generally requires that the models used to develop such information be publicly available. The OMB guidelines further explain that when public access to models is impossible for "privacy, trade secrets, intellectual property, and other confidentiality protections," an agency "shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken." 67 FR 369, 374 (Jan. 3, 2002).

    EPA must propose its own guidelines implementing the Data Quality Act by May 2001, and promulgate final guidelines by October 1, 2002. EPA's guidelines must be consistent with OMB's. The issue raised by this paper should be addressed in EPA's guidelines.


    General Policy

    • EPA should adopt a general prohibition against use of third-party proprietary models.
    • Use of third-party proprietary models conflicts with the goals and intent of the Data Quality Act.
    • Public disclosure of third-party proprietary models is required in all but the most unusual circumstances.
    • If EPA believes it must use third-party proprietary models in order to carry out its regulatory duties and functions, then EPA should have the burden of demonstrating to OMB, before entering into a contract to use the model, that no other option is available.
    • EPA's proposed and final Data Quality Act guidelines should explain in detail what "especially rigorous robustness checks" will be applied to third-party proprietary models that EPA and OMB agree must be used; and explain how the public will be informed of these "robustness checks."

    Implementation of the General Policy

    Prospective Implementation: EPA proposes and promulgates Data Quality Act guidelines declaring the general policy on this issue as described above. EPA's guidelines should further state that, before EPA agrees to use a third-party non-public, proprietary model, it will provide OMB a written justification why the agency has no other option, and await OMB's views before entering into a contract that utilizes an allegedly proprietary model. The written justification to OMB should describe why the agency cannot:

    • Use an existing public model;
    • Enter into a contract to develop a new public model;
    • Reimburse a contractor so as to convert a proprietary model into a public model.

    This prospective implementation policy should be in effect as of the date EPA proposes Data Quality Act guidelines.

    Retroactive Implementation: If EPA has already agreed to use a third-party proprietary model before EPA proposes Data Quality Act guidelines as described above, then EPA should, within 45 days of the date EPA proposes its Data Quality Act guidelines:

    • Provide OMB with a written identification of what third-party proprietary models are being used by the agency;
    • Provide OMB with a written explanation of why the agency cannot reimburse the contractors so as to convert third-party proprietary models into public models, or enter into a contract to develop a public