March 22, 2002
Ms. Evangeline Tsibris Cummings
Office of Environmental Information
Environmental Protection Agency
Ariel Rios Building
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460
Re: Comments on EPA Data Quality Guidelines
Dear Ms. Cummings:
Pursuant to EPA's
announcement in the March 14, 2002 Federal Register (67 Fed. Reg.
11483), I am writing on behalf of the Center for Regulatory Effectiveness (CRE)
to provide the Center's comments to the Agency as it seeks to develop its Data Quality guidelines, pursuant to Section
515 of the FY 2001 Treasury and General Government Appropriations Act (Pub.
Law No. 106-554, H.R. 5658). I would like to start by commending the Agency
for seeking broad public input during the early development phase of its guidelines,
even before proposed guidelines are issued for public comment. CRE believes
that such involvement will ultimately lead to a better product and is a model
which other agencies should follow.
CRE has been actively involved in the Data Quality issue (see
the Data Quality section of CRE's website www.TheCRE.com for details on the
Center's involvement), and CRE is committed to helping federal agencies achieve implementation of the Data Quality
Act by meeting the objectives of Congress, as interpreted by OMB. CRE acknowledges
that EPA has been a leader in terms of its efforts to promote information quality
at the Agency, not the least of which has been establishment of the Office of
The Center also believes that EPA's
development of Data Quality guidelines, as required by statute, will provide
an additional opportunity to foster strong data and information management at
the Agency. To this end and as the following comments explain, CRE believes
that EPA should make statements regarding both the short-term and long-term
goals of its Data Quality Program.
CRE Recommendations for EPA Data Quality Guidelines
CRE is pleased to offer the following recommendations to EPA,
which will be supplemented once the Agency issues proposed guidelines in the
Federal Register and requests further public comments.
(1) EPA should make clear that the Data Quality Program,
as encompassed in its Data Quality guidelines, is an evolutionary process
which will involve ongoing review and improvement by the Agency.
(2) EPA should formally commit to the development
of a long-range plan for improving Data Quality at the Agency. In this context,
CRE urges consideration of the following goals:
Completeness of Data
- In its
guidelines, EPA should provide for the inclusion of all relevant and
reliable scientific information, from both animal and human studies.
If there are inconsistencies, these should be explained, along with
a discussion of how they might be reconciled.
approach is demanded by the OMB directive to adopt or adapt the
quality provisions of the Safe Drinking Water Act Amendments of
1996 and the definition of "objectivity" as requiring completeness.
- The Data Quality Act and OMB's
guidelines preclude EPA's categorical ban on the use of
third-party clinical human test data.
- EPA should adopt, not adapt, the Data Quality
standards set forth in the Safe Drinking Water Act as part of its Data Quality
Use of Third-Party Proprietary Models
agencies often use various models developed by third parties (often
government contractors) to formulate policies based upon influential
scientific information. These models are sometimes asserted to be confidential
recommendations regarding third-party proprietary models are expressed
in more detail in the attached policy paper along with suggestions on
implementation, but the crux of the Center's
recommendations in this context are:
should adopt a general prohibition against third-party proprietary
of third-party proprietary models conflicts with the goals and intent
of the Data Quality Act (e.g. related to reproducibility).
disclosure of third-party proprietary models should be required
in all but the most unusual circumstances.
EPA believes it must use third-party proprietary models in order
to carry out its regulatory duties and functions, then EPA should
have the burden of demonstrating to OMB, before entering into a
contract to use the model, that no other option is available.
proposed and final Data Quality Act guidelines should explain in
detail what "especially rigorous robustness checks" will be applied
to third-party proprietary models that EPA and OMB agree must be used. EPA's
guidelines should further explain how the public will be informed
of these "robustness checks."
should specifically address risk assessments as a special type of information
disseminated by the Agency. The OMB guidelines address risk information
as such a special category.
- As noted above, the OMB rules require that risk
information be objective and unbiased. To the extent EPA uses "conservatively"
biased policy assumptions, uncertainty factors, and data selection in risk
assessments/analyses (as opposed to risk management analyses), the possible
conflict between the OMB guidelines and these actions should be reconciled.
Such reconciliation may be achieved by clearly separating an objective
scientific analysis of what is know and not known about adverse effects and
the circumstances under which they occur, from a risk management options
analysis that discusses whether or not to incorporate policy assumptions or
decisions for any regulatory decision that might be made on the basis of the
should also modify its use and dissemination of hazard information,
given the policies set forth in the OMB guidelines. The statute and
OMB rules require agencies to ensure and maximize the utility of information.
More specifically, hazard information is often disseminated without
appropriate caveats that risk B
as opposed to hazard B depends
on level of exposure, thereby neglecting context that is essential to
the apprizing the public of the utility of the information.
- An EPA
cumulative risk assessment is only as good as the risk assessments of
the individual substances that underlie the cumulative risk assessment.
Consequently, EPA's guidelines
should emphasize that the Data Quality Act standards must be met first
by the individual risk assessments before any cumulative risk assessment
can be performed. Moreover, if more recent and better data are available
than were used in the individual assessments, then those data should
be used for the cumulative risk assessment. Standardized data are scientifically
necessary for conducting cumulative risk assessments. EPA's
guidelines should also require that an approved and properly validated
methodology be used to conduct cumulative risk assessments. Finally,
all models used for cumulative or individual risk assessments should
be publicly available, and adequate time for comment and peer review
allowed, before the models are used. Proprietary model issues are also
addressed elsewhere in these comments.
Information Related to Fees
is permitted by statute to charge fees for various services it provides
to the regulated community. Economic analyses are routinely conducted
by the agency to determine the level of such fees, although such calculations
are not always fully transparent.
economic analyses related to fees are subject to the Data Quality Act,
and OMB's guidelines state
that analytical methods (models) should be publically available to facilitate
the reproducibility of such information by qualified third parties.
In this context, CRE recommends that EPA disclose the underlying data,
models, and assumptions used in determining the fees set by the agency.
Such action would not only serve the goals of Data Quality, but would
increase transparency as well.
Applicability of the Data Quality Guidelines to Third-Party
Information Submitted to the Agency
the Data Quality Act applies to essentially all information which the
Agency disseminates to the public, the same standards will apply to
information developed by third-parties and submitted to EPA for Agency
publication or related regulatory action. EPA should make a clear statement
that information submitted to the Agency must comply with the Data Quality
- A third party petitions for correction or
withdrawal of the information; or
- The Agency itself decides to disseminate this
information or to adopt its substance and take regulatory action.
Definition of the Term "Unbiased"
and the regulated community have complained for decades that EPA risk
assessments are unduly biased by policy assumptions. The OMB guidelines
require that risk assessments be unbiased. Thus, CRE believes that EPA
should bifurcate its risk analysis into: (1) an objective consideration
of the science and (2) a review of policy decisions that might be used
to make risk management decisions.
Deadlines for Agency Action on Data Quality Petitions
- In accordance with the OMB guidelines,
the agency should set a date specific for action
on Data Quality petitions once they are received by the Agency.
as part of the procedural process for Data Quality petitions, the responsible
Agency official should be responsible for reviewing each petition filed
to determine those that are relatively simple or uncontroversial and,
therefore, can be decided in an expedient fashion prior to the deadline
- In addition, EPA's Data Quality petition
procedures should establish a process for accelerated review of petitions
and describe the circumstances when accelerated review is appropriate.
Publication for Rules of Procedure for Data Quality Petitions
- In addition to accelerated review procedures,
CRE believes that the Agency and stakeholders would benefit from EPA's development
of a set of overall written procedures for the Data Quality petition process.
- These procedures should be published in the
Federal Register in proposed form for public notice and comment.
Availability of Administrative Appeals
- CRE believes that appeals under the Data Quality
petition process should only be available to the Petitioner. The Agency should
not be allowed to challenge positive action on the petition at a lower level within
Standard of Care for Agency Information Prior to Issuance
- EPA should specify the process the Agency
intends to establish to ensure quality of the Agency's information
and compliance with the Data Quality Act prior to the issuance and dissemination
of such information.
- This process should identify which official
will be conducting such review and the steps that will taken in considering
compliance with the Act, OMB's guidelines, and EPA's own guidelines.
Higher Level of Quality for "Influential" Information
- The OMB Data Quality guidelines mandate a higher
level of quality for "influential scientific, financial, or statistical information."
Consequently, the Agency should clearly and explicitly address all relevant
aspects of what will constitute a higher level of quality for such influential
Decisionmaking in the Petition Process
- EPA must ensure that the decisionmaker(s) in the
administrative petition process B at both the initial
consideration stage and the administrative appeals level -
is competent to handle the substance of the petition and sufficiently
independent to make a decision on the petition.
believes that the petition should initially be reviewed and decided
within the relevant Program Office.
believes that if a petition is denied, the appeal should be reviewed
and decided by an EPA official outside the Program Office in question.
Consideration might even be given to establishing an independent
office within the Agency for the purpose of deciding Data Quality
Treatment of Inaccurate Data During the Course of Administrative
- If the
designated EPA decisionmaker becomes aware of an inaccuracy in data
during the course of a petition review, that official should have the
authority to suspend or withdraw dissemination of the inaccurate information
until such time as it can be amended or clarified.
upon its Agency-wide Data Quality guidelines, EPA should develop and
adopt specific Data Quality guidelines for each of the Agency's
major Operating Offices. This approach would provide additional clarity
and consistency to both EPA staff and stakeholders by addressing data
issues in a program-specific fashion.
specifically, EPA should announce in a proposed rule a plan for the
development of Data Quality guidelines for each Operating Office. The
proposal should be published in the Federal Register for public
comment and should address, at a minimum, the following elements:
of Operating Offices for which targeted Data Quality guidelines
would be developed;
of a transparent agency process for development of Data Quality
guidelines for EPA Operating Offices, including ample opportunities
for public comment; and
for Agency action.
Again, the Center appreciates
the opportunity to provide EPA with its comments at this early stage in the agency's
Data Quality guidelines development process. CRE strongly urges the Agency
to make the comments it receives in this round publicly available, preferably
through the EPA website. Such step would permit interested parties to consider
the views of other thoughtful respondents, thereby raising the level of debate
on this issue and the overall quality of input provided to the Agency.
Finally, CRE looks forward to reviewing EPA's
draft guidelines once published and providing further input at that time. Should
you have any questions regarding CRE's comments, please feel free to contact me at
Jim J. Tozzi
Member, CRE Board of Advisors
EPA'S USE OF THIRD PARTY PROPRIETARY MODELS
CONFLICTS WITH THE DATA QUALITY ACT
Federal agencies often use various models developed
by third parties (often government contractors) to formulate policies
based upon influential scientific information. The third-party models
are sometimes asserted to be confidential and proprietary.
This issue does not involve the concerns that arise
when regulated entities are required to submit confidential or proprietary
data to an agency pursuant to a regulatory program. Instead, this issue
is limited to situations where an agency and a contractor agree to use
a model on a proprietary basis to develop influential scientific information.
OMB's guidelines implementing the
Data Quality Act, 44 U.S.C. § 3516 note, require that influential scientific
information be reproducible. This reproducibility standard generally requires
that the models used to develop such information be publicly available. The OMB
guidelines further explain that when public access to models is impossible for
"privacy, trade secrets, intellectual property, and other confidentiality protections,"
an agency "shall apply especially rigorous robustness checks to analytic results and
document what checks were undertaken." 67 FR 369, 374 (Jan. 3, 2002).
EPA must propose its own guidelines implementing
the Data Quality Act by May 2001, and promulgate final guidelines by
October 1, 2002. EPA's guidelines must be consistent with OMB's.
The issue raised by this paper should be addressed in EPA's guidelines.
should adopt a general prohibition against use of third-party proprietary
of third-party proprietary models conflicts with the goals and intent
of the Data Quality Act.
disclosure of third-party proprietary models is required in all
but the most unusual circumstances.
- If EPA believes it must use third-party
proprietary models in order to carry out its regulatory duties and
functions, then EPA should have the burden of demonstrating to OMB,
before entering into a contract to use the model, that no other option
- EPA's proposed and final Data Quality Act
guidelines should explain in detail what "especially rigorous robustness checks"
will be applied to third-party proprietary models that EPA and OMB
agree must be used; and explain how the public will be informed
of these "robustness checks."
Implementation of the General Policy
EPA proposes and promulgates Data Quality Act guidelines declaring the general policy
on this issue as described above. EPA's guidelines should further
state that, before EPA agrees to use a third-party non-public, proprietary
model, it will provide OMB a written justification why the agency has
no other option, and await OMB's views before entering into a contract
that utilizes an allegedly proprietary model. The written justification
to OMB should describe why the agency cannot:
- Use an existing public model;
- Enter into a contract to develop a
new public model;
- Reimburse a contractor so as to convert a
proprietary model into a public model.
This prospective implementation policy
should be in effect as of the date EPA proposes Data Quality Act
Retroactive Implementation: If
EPA has already agreed to use a third-party proprietary model before
EPA proposes Data Quality Act guidelines as described above, then
EPA should, within 45 days of the date EPA proposes its Data Quality
- Provide OMB with a written identification
of what third-party proprietary models are being used by the agency;
OMB with a written explanation of why the agency cannot reimburse
the contractors so as to convert third-party proprietary models
into public models, or enter into a contract to develop a public