®: CRE Regulatory Action of the Week
CRE Submits Comments on "Reproducibility" Standard in OMB's Data Quality Guidelines
October 26, 2001
Ms. Brooke J. Dickson
Re: Comments on OMBís Interim Final Data Quality Guidelines
Dear Ms. Dickson:
The Center for Regulatory Effectiveness (CRE) has maintained a significant and ongoing interest in the area of Data Quality, so we would like to compliment OMB on the timely issuance of efficacious Data Quality guidelines that will help ensure high standards for information that the government disseminates to the public. Consequently, the Center is pleased to offer the following comments in response to OMBís request for additional public input on the term "capable of being substantially reproduced" in OMBís "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies," 66 Fed. Reg. 49718 (September 28, 2001).
From the outset, I would like to express CREís strong support for OMBís Data Quality guidelines, particularly the requirement that analytic results be "capable of being substantially reproduced." In the comments that follow, CRE provides its recommendations for further refinements and clarifications to the guidelines.
Reproducibility/Replicability of Results Across Testing Environments
(1) When an Agency Uses Data From Validated Tests
- The disseminating agency should be able to presume that the test data satisfy inter-laboratory consistency concerns for all the end points encompassed by the validation and demonstration.
(2) When an Agency Uses Data From Non-Validated Tests
Ė The disseminating agency should be required to demonstrate and document that the tests will generate substantially the same data at any laboratory conducting them properly for all the involved end points before the agency disseminates information containing or based on the data.
(3) When an Outside Party Submits Contradictory Information
- The disseminating agency should be required to consider this data and resolve this discrepancy in accordance with sound scientific principles before the agency disseminates the relevant information.
- If subsequent attempts to replicate the original experiment or observations by other qualified persons demonstrate that the originally reported results are not, in fact, substantially reproducible, those original results must be regarded as unreliable.
Need for Validated Models
- CRE recommends that the definition of "reproducibility" should also include tests of robustness so as to exclude results which may not be reliable, even if, because they are model-driven, they appear to be reproducible.
Minimum Peer Review Requirements
- CRE recommends that the Data Quality guidelines incorporate, either directly or by reference, the attachment's discussion of peer review. The standards set for peer review by this document would enhance the objectivity and transparency of the peer review process.
- The presumption of objectivity based on peer review, which is currently stated in the guidelines, should be clarified to require that agencies follow the peer review recommendations.
Data Quality Guidelines Linkage to the Paperwork Reduction Act
Ė CRE recommends that the linkage between the guidelines and the PRA be strengthened so as to clarify the role of guidelines vis-a-vis the PRA, and, thus, reduce any uncertainty regarding standards to be set for federal agencies. Specifically, we recommend that the OMB guidance describe the PRA as containing the original and fundamental statutory authority and directives, with the statutory language in the Appropriations Act being supplemental to the PRA.
Objectivity of Formal Peer Review
- CRE recommends that, if outside parties are able to demonstrate that the peer review was not objective (i.e. that the process was not independent, external, or was driven by policy rather than scientific considerations), agencies must consider such evidence and, if warranted, not accept the objectivity of the technical information.
Safe Drinking Water Act Language
- The Safe Drinking Water Act provides a standard to which all government information dissemination should be held. CRE strongly supports continued use of the SDWA language.
Influential Scientific or Statistical Information
Ė CRE supports OMBís definition of "influential scientific or statistical information" as well as the high standards set for information in this category.
CRE appreciates this opportunity to comment, and we would be pleased to further assist the agency regarding Data Quality Act implementation, as appropriate. Please feel free to contact me, should you have any questions or require further clarification.