Why Federal Science Agencies Should Share More Data

I originally wrote this piece a year ago, and then parts of it turned into this co-authored piece at Issues in Science and Technology. But there were several thousand words that didn’t end up being published. I thought it was worth resurrecting here.

Introduction

The National Institutes of Health budget was $49 billion in fiscal year 2023, while the National Science Foundation’s budget was nearly $10 billion. The overall goal is to improve scientific advancement. How are NIH and NSF doing with some $59 billion in public funds?

It might be surprising, but no one actually knows very much about a wide range of important questions, such as:

• Is it better to give more funding to “the person not the project”? “The team not the project”? “The institution not the project”?

• Is it better to give 8- or 10-year grants as opposed to 3-4 year grants?

• Is it better to change peer review so that a single positive vote might get something funded?

• Should we “red team” scientific fields where the funded grants mostly seem to follow one particular theory?

• Would it be better to give program officers more discretion to overrule or even bypass peer review? 

• Is it better to fund a lone assistant professor or to fund the 51^st person to join a large lab?

• What would happen if we demanded that all scientific grants publicly report at least one “failure”? Would we encourage more risk-taking and/or truth-telling?

• What types of research proposals do peer reviewers tend to reject? What is the difference between those proposals and the ones that get approved?

• How often is high-impact research turned down in initial review?

• How often do proposals below the payline nonetheless get funded? Who makes those decisions, and what is their rationale? Over time, how do those grants perform compared to grants above the payline?

• How does the NIH intramural program compare to the extramural program? 

Why don’t we know the answers to these questions?

In part, it’s because we need more experimentation with different funding models. But many of these questions could be at least partially answered by existing data that NIH and NSF usually keep hidden from scholars, despite having a slim legal basis (if any) for doing so. And even if we did more experimentation in federal science funding, external scholars would still need access to the basic underlying data in order to help out.

For example, very few scholars have ever been able to access data on successful versus unsuccessful proposals, how they fared in the peer review process, the internal reviews from agency staff and peer reviewers, etc. There are many top scholars who could make use of this data and who are frustrated by agency responsiveness (or lack thereof), but who do not want to be named in any way whatsoever for fear that NIH or NSF might somehow retaliate against them.¹

***

The above is just my list of interesting metascience questions. But agencies themselves have been coming up with their own lists. The NSF’s 2022-26 Learning Agenda includes research questions such as:

• “What are the impacts of NSF policies and programs on the diversity of the STEM workforce and the participation of the most underrepresented groups?”

• “How do EPSCoR [Established Program to Stimulate Competitive Research] program funding strategies (infrastructure, co-funding, and outreach) contribute to increasing academic research competitiveness across jurisdictions?”

• “What are the benefits of receiving an award from a program supported by a partnership? How do these differ from benefits associated with awards from programs not supported by a partnership?”

• “What are the characteristics of proposals evaluated through the merit review process? Are these characteristics (of individual investigators, teams, institutions, or proposed projects) associated with different review or funding outcomes?”

• “What outcomes are associated with the adoption of a no-deadlines proposal submission process?”

Similarly, NIH posted a list of evaluation questions on the OMB Portal. Some of its questions include: 

• “How can NIH improve on identifying desired outcomes and measuring impact related to its mission?” 

• “What measures can NIH use to capture both incremental knowledge gains and failures that ultimately contribute to scientific success?” 

• “What approaches can NIH use to measure impact of different categories of science (e.g., basic, translation, clinical) and the technology and operations used to support the science?”

• “Are there methods that NIH can use to better predict and identify scientific opportunities (e.g., the emergence of gene editing technology)?”

• “Are there approaches that could inform NIH funding decisions by measuring scientific quality, rigor, and reproducibility?”

• “What evidence does NIH need to improve the clinical research ecosystem? What would inform a re-envisioning of the clinical trials system to maximize quality, participant experience, accessibility, timeliness, and impact on clinical care?”

For all of these questions and more, science funding agencies would benefit from working with external evaluators and scholars and gaining from all of their additional expertise. In fact, the agency would probably learn far more from allowing external scholars to have access to the necessary data, than from trying to analyze all of these questions in-house (where there is a serious conflict of interest).

POLICY RECOMMENDATION: If we want to know whether any government agency is well-functioning and successful, independent scholars and evaluators need access to comprehensive data on the agency’s operations and outcomes. Thus, NIH and NSF (and more) should make their internal data available to outside researchers by default given a good faith request via the Standard Application Process at ResearchDataGov, and to do so on a streamlined basis with narrow exceptions only when necessary. If agencies need extra appropriations for personnel to prepare data, Congress should step in. Moreover, agencies should not be allowed to veto publications except in the narrowest of circumstances (e.g., malfeasance).

The Evidence Act

A major theme of the Foundations for Evidence-Based Policymaking Act of 2018, more commonly known as the Evidence Act, is that federal agencies should be engaged in regular evidence-building and evaluation as to how their efforts are working.

As well, the Evidence Act included a section titled the “OPEN Government Data Act,” with OPEN standing for “Open, Public, Electronic, and Necessary.” [You might be reminded of the joke that the B in Benoit B. Mandelbrot stands for “Benoit B. Mandelbrot.”]

The point was to require federal agencies to “make data open by default” (see section 202(c) here). The statute requires all federal agencies both to “make each data asset of the agency available in an open format” and to “make each public data asset [i.e., subject to FOIA] of the agency available as an open government data asset; and under an open license.” 44 U.S.C. § 3506(b)(6). In other words, as a Data Foundation report put it, “For the first time, OPEN codifies a presumption that non-sensitive government data should be open by default.”

But agencies are sidestepping the Evidence Act as well as what other laws (such as FOIA and the Privacy Act) actually entail. Contrary to NIH and NSF’s approach, federal appellate caselaw has interpreted FOIA to actually require the release of information about unfunded proposals, and under the Evidence Act, that information should be automatically available as an open data asset.

Moreover, contrary to these agencies’ representations, the Privacy Act isn’t an automatic fallback reason to protect such information (as it has a clear exception for “statistical research”). In other words, agencies can and must do more under current law to make data available to outside researchers. Section II below discusses current law in more detail.

Another difficulty is that agencies see themselves as trapped in an outdated and binary legal regime wherein the two choices are: 1) Release data to (potentially) the entire world under FOIA, or else 2) Treat the data as protected under the Privacy Act.

We need more of an in-between legal regime for handling data that might be slightly private and that should be released to some people but not everyone. An example is peer review scores, which might be embarrassing if released to the entire world with names attached, but should most definitely be made available to independent scholars for analysis.

The last section therefore makes the case for an intermediate legal regime that would allow NIH and NSF to comply with the spirit of the Evidence Act. The lack of legal clarity here creates an environment wherein agency lawyers (who are understandably prone to be overly protective) err on the side of refusing to allow the release of data to external researchers. As a result, we are lacking as to independent analyses of how NIH, NSF, etc., are doing.

The memo concludes with several specific suggestions for how the White House could take action in order to point scientific research agencies in the right direction, as well as a bill that Congress could adopt to further clarify the agencies’ responsibilities to make data available.

Release of Research Data Under FOIA and the Privacy Act: More Is Possible or Even Required Than Agencies Admit

A.    FOIA

The Freedom of Information Act (FOIA) in 5 U.S.C. § 552 broadly requires the federal government to share information and data with the public on request. Similarly, the Government in the Sunshine Act (5 U.S.C. §552b) requires government meetings to be open to the public, with a few exceptions that are copied from FOIA.

The NIH treats FOIA as the governing law here. As the NIH Grants Policy Statement says in Section 2.3.11.2.1, if anyone other than the researcher requests data on proposals, “such requests are processed under FOIA.” Then, in Section 2.3.11.2.2, “The Freedom of Information Act … require[s] NIH to release certain grant documents and records requested by members of the public, regardless of the intended use of the information. These policies and regulations apply to information in the possession of NIH.”

So, there’s an initial question to be answered: Does FOIA actually require this sort of data (e.g., proposals and peer review scores) to be made public? Yes, at least some of the time.

***

FOIA generally requires government data to be provided to the public, with several exceptions—three of which are usually mentioned here:

Exception 4: Trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential.”

Exception 5: Inter-agency or intra-agency “memorandums or letters that would not be available by law to a party other than an agency in litigation with the agency”

Exception 6: “Personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.”

Also worth noting: FOIA’s exceptions just mean that an agency isn’t required to release the information in question, but the agency may still have the discretion to do so. See Chrysler Corp. v. Brown, 441 U.S. 281, 291 n. 11 (1979) (noting that FOIA exemption 4 is “an exception to the disclosure mandate of the FOIA, and not a limitation on agency discretion”).

NIH explains which types of information it will release in response to a FOIA request, and which types it will not (NIH Grants Policy Statement, Section 2.3.11.2.2):

In other words, the NIH does release funded applications pursuant to a FOIA request, along with progress reports and the like. But it won’t release information about pending or unfunded applications, nor will it release site visit reports or peer review statements.

The NIH doesn’t explain why it has this interpretation of FOIA. Indeed, you might note that its refusal to release unfunded applications and peer review scores (which are not FOIA exceptions) is separate and apart from its refusal to release “trade secrets,” “valuable commercial rights,” or other “competitive” information (which would be FOIA exceptions).

Let’s consider the FOIA exceptions in turn:

Exception 4: Trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential.”

Some might argue that scientific research proposals should be considered as a “trade secret” that is “privileged or confidential.” After all, if a researcher thinks up a new idea or technique to explore in future research, it might be unfair if someone else got access to that unfunded idea, and was somehow able to scoop the initial research.

This is not a correct interpretation of FOIA, however. Not all proposals are a trade secret with proprietary information, and not even NIH thinks so. Applicants have to go out of their way to specifically identify which information might be proprietary. The NIH Grants Policy Statement notes that “applicants are instructed to identify proprietary information at the time of submission [NOTE: NOT AT THE TIME OF APPROVAL] of an application. . . . If an applicant fails to identify proprietary information at the time of submission as instructed in the application guide, a significant substantive justification will be required to withhold the information if requested under FOIA.”

In other words, even by NIH’s own interpretation of FOIA, grant proposals aren’t “proprietary information” unless the scholar provides some “significant substantive justification.”

As well, under the limited caselaw available, grant proposals and progress reports are not protected by Exception 4 of FOIA. In Washington Research Project, Inc. v. Dept. of Health, Education and Welfare, 504 F.2d 238 (D.C. Cir. 1974), the Washington Research Project sued to get access to “eleven specifically identified research projects that had been approved and funded by the National Institute of Mental Health.” This information included the grant application, a site visit report from the agency, and a summary report on the application.

One issue was whether the “research designs” were trade secrets or commercial information, in that “ideas are a researcher’s stock-in-trade.”

The court ridiculed this claim: “the government has been at some pains to argue that biomedical researchers are really a mean-spirited lot who pursue self-interest as ruthlessly as the Barbary pirates did in their own chosen field. . . . It is clear enough that a non-commercial scientist’s research design is not literally a trade secret or item of commercial information, for it defies common sense to pretend that the scientist is engaged in trade or commerce.”

The court therefore held that “research designs submitted in grant applications are not exempt from disclosure under the Act.” Moreover, “this holding extends to all types of applications — initial, continuation, supplemental, and renewal — and to progress reports made by grantees as part of the last three kinds of applications.”

This case has been reaffirmed more recently in Physicians Committee for Responsible Medicine v. National Institutes of Health, 326 F. Supp. 2d 19 (D.D.C. 2004). In that case, a group of animal welfare activists sued NIH over a “research project that involves giving amphetamines to cats infected with the feline equivalent of HIV.” The court did not agree with NIH’s claim that the information was a trade secret, or that it was confidential commercial information. While the researcher in question (a Dr. Podell) had claimed he was trying to develop potential treatments for AIDS, the court said, “He was not involved in trade or commerce when his research design was developed. . . . The fact that Dr. Podell was engaged in research for the university renders the possibility of a trade interest in his research design remote.”

Indeed, think about the disparity between how the NIH treats unfunded and funded proposals. When a proposal has been funded, NIH can release it under FOIA, whereafter everything can be broadcast to the world (with only a few exceptions). Here's an example of such an application that the NIH has posted publicly. As you can see, the NIH does black out occasional pieces of information, such as phone numbers and addresses. But the rest of the proposal isn't sensitive or confidential at all: it’s a description of proposed work on the innate immune system, a bunch of citations, a description of the lab space, and mostly people’s long CVs (which are usually publicly posted on their own websites anyway).

But the thing to note is that unfunded and funded proposals have literally the same content. No one retroactively changes the information in a proposal that happens to get funded. Occasionally, some information might be proprietary or private, but that information is the same both before and after funding.

So, in legal terms, there is no reason to treat unfunded proposals and funded proposals differently. Proprietary information might be funded (and need to be protected), but non-proprietary information often goes unfunded and doesn’t qualify for legal protection just because it was unfunded. FOIA in this case is about proprietary vs. non-proprietary, not funded vs. unfunded.

Why do NIH and NSF nonetheless treat this kind of information as sensitive/confidential/proprietary just because the proposal was unfunded? Sure, some scientists don't want to get scooped or lose intellectual property before being able to actually do a planned line of work, and therefore don’t want their unfunded proposals publicly available. But that doesn’t mean those proposals fit the legal definition of “trade secret” under FOIA. A proposal can be a trade secret whether funded or unfunded, and it can be a non-trade secret in either case as well. Funded-vs-unfunded and trade-secret-vs-not-trade-secret are two completely orthogonal issues. 

In short, under longstanding caselaw, NIH grant applications and progress reports cannot be generally treated as “trade secrets” that are “private or confidential" (unless identified and approved as such ahead of time). Neither FOIA itself nor any caselaw that I could find even hints that whether a proposal is funded or not has anything to do with whether the information is a trade secret or confidential business information.

Exception 5: Inter-agency or intra-agency “memorandums or letters that would not be available by law to a party other than an agency in litigation with the agency”

Peer review scores and similar data are typically treated as falling under Exception 5 to FOIA. That is, a number of cases have interpreted the so-called “deliberative process privilege” as covering advice from peer reviewers. See Formaldehyde Inst. v. Dep't of Health & Human Servs., 889 F.2d 1118, 1121 (D.C. Cir. 1989); Judicial Watch, Inc. v. U.S. Dept. of Commerce, No. 15-cv-2088 (D.D.C. Aug. 21, 2017); Washington Research Project, Inc. v. Dept. of Health, Education and Welfare, 504 F.2d 238 (D.C. Cir. 1974) (discussed above).

However, as discussed below, peer reviewer ratings and comments are hardly all that confidential (these, after all, are votes on how to distribute government funds). NIH/NSF/etc. could perfectly well work to anonymize and release these data to independent scholars even if not strictly required to do so by FOIA. Remember, FOIA exemptions just mean that the agency isn’t required to release the data to anyone, not that the agency is forbidden to release the data in its discretion.

There is a closely-related legal issue from a law that basically copies much of FOIA’S language, namely the Government in the Sunshine Act. HHS regulations refer to that Act in saying that meetings of NIH study sections are “closed to the public.” See 42 C.F.R. § 52h.6:

“Meetings of peer review groups reviewing grant applications or contract proposals are closed to the public in accordance with sections 552b(c)(4) and 552b(c)(6) of the Government in the Sunshine Act, as amended (5 U.S.C. 552b(c)(4) and 552b(c)(6)).”

What do those sections of the Government in the Sunshine Act say? That government meetings and deliberations have to be open to the public, with some exceptions that are basically copied from FOIA. I.e., government meetings need not be open and transparent if they involve “trade secrets and commercial or financial information obtained from a person and privileged and confidential“ ((c)(4)), or if they “disclose information of a personal nature where disclosure would constitute a clearly unwarranted invasion of personal privacy“ ((c)(6)).

But while peer review discussions could logically be treated as part of the “deliberative process,” it’s not clear why they would be automatically treated as involving “trade secrets” that are “privileged and confidential” information, let alone information that creates a “clearly unwarranted invasion of personal privacy” akin to private medical records. I can't find any relevant caselaw or legal articles explaining why concepts like “clearly unwarranted invasion of personal privacy” or “trade secrets” that are “privileged and confidential” would apply to the NIH's peer review process in all cases.

In other words, the HHS regulations purport to implement the Government in the Sunshine Act, but in fact are considerably more restrictive and conservative than what the law actually says. 

As an ex-lawyer, I understand the impulse to opt for the most restrictive and conservative interpretation possible, because the goal is to protect the agency at all costs. But that doesn’t mean it’s the right interpretation if one has the larger public interest in mind. The White House could issue an Executive Order requiring HHS (or any other agency with a similarly restrictive interpretation) to re-examine the interplay between the Government in the Sunshine Act, FOIA, scientific peer review, and the need for outside researchers to engage in research and evaluation pursuant to the Evidence Act of 2018. 

Exception 6: “Personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.”

Finally, there’s Exception 6. The main cases here held that despite FOIA Exception 6, NIH was required to release information about applicants who went unfunded!

In Kurzon v. Dept. of Health and Human Services, 649 F.2d 65 (1st Cir. 1981), George Kurzon “wanted to test his theory that the peer review method by which the National Institutes of Health (NIH) evaluate grant applications is biased against unorthodox proposals.” After unsuccessfully trying to get the data, he filed a FOIA lawsuit asking that the National Cancer Institute turn over “names and addresses of unsuccessful applicants for research grants.”

The lower court held that NIH didn’t have to turn over data, because of FOIA Exemption 6: “personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.” The court thought that this information “would be a serious unwarranted invasion of privacy and might reflect opinions about the competence of the applicant or his professional qualifications.”

The First Circuit disagreed and overturned the lower court’s decision. It noted that under most cases, the sorts of files at issue in Exemption 6 had “intimate details” of a “highly personal nature,” and that the lower court shouldn’t have merely asked whether the information was “personal” but whether it was “of the same magnitude—as highly personal or as intimate in nature—as that at stake in personnel and medical records.”

But here, the court said, Kurzon was just seeking names and addresses of rejected applicants—i.e., only “slight informational content,” and the loss of privacy would be “minimal.” Nor would there be a risk of embarrassment, since “twice as many applications are rejected as are not.” (That figure would be higher today.)

The First Circuit also emphasized the public nature of this data:

“Finally, federal grant applicants cannot reasonably expect that their efforts to secure government funds, especially in a field so much in the public eye as cancer research, will remain purely private matters. There is an obvious public element to the process and the results, as recognized in the NIH practice of releasing both the applications and identities of funded grant applicants.”

This makes perfect sense. It is unreasonable to ask for money from the government and the taxpayers in order to support your research, and yet also demand that everything about that process remain confidential.

The Kurzon case did allude to the possibility of a “promise of anonymity” on the part of NIH. The court held, however, that there “was no such promise,” and that “the best the government can do is to assert a general implied promise of confidentiality based on its policy statement, published in the Federal Register, that ‘[i]nitial research or [a] research training grant application on which award is not made’ is ‘generally not available’ to the public.” Kurzon, 649 F.2d at 69-70 (quoting 45 C.F.R. Part 5, App. (1980)).

Moreover, even if an agency (such as NIH) promised confidentiality to applicants, that cannot be a good reason to override FOIA. Otherwise, agencies could always evade future FOIA requirements just by unilaterally promising to keep certain documents secret.

In short, the only federal appellate case on this issue held that data on unfunded applications must be released under FOIA.²

Other cases have followed this precedent. For example, here’s a 2001 case in which the federal district court for New Hampshire required the National Institute of Mental Health to release the names and addresses of unfunded researchers, again to George Kurzon (the aspiring metascience researcher who wanted to ask unfunded researchers about whether they thought peer review “does not sufficiently recognize the value of innovative cutting-edge research”). In that case, the court noted that HHS’s argument that “confidentiality permits scientists to avoid or hide their application records with impunity is not a persuasive argument.”

Why don’t NIH and NSF take these cases more seriously in their daily dealings with outside researchers? That’s a good question. The White House should demand that agencies regularly work with outside researchers to let them do research and evaluation on the content and disposition of unfunded proposals, without anyone needing to file future FOIA lawsuits.

B.    The Privacy Act

Whenever NIH and NSF are not strictly required to release data to the public under FOIA, they typically find reasons to fall back on the Privacy Act, which protects individual privacy as to data held by the government. For example, I asked both NIH and NSF lawyers for any memo or explanation as to why they restrict access to data on unfunded proposals and peer review. I never heard back from NIH, but NSF personnel reached out to say the following:

“Most of the data that NSF collects at the individual person-level as part of the merit review process is protected by the Privacy Act, which generally prohibits disclosure without consent of the individual. Like other federal agencies, NSF publishes Privacy Act System of Records Notices (SORNs) in the Federal Register that explain what Privacy Act protected data the agency is collecting and how it may be disclosed. You can find links to all of NSF’s SORNs here: https://www.nsf.gov/privacy/.”

In another email, the NSF lawyers told me:

The statutory language of the Privacy Act describes a “system of records” that is covered by the Act as “…a group of any records under the control of any agency from which information is retrieved by the name of the individual or by some identifying number, symbol, or other identifying particular assigned to the individual…” 5 U.S.C. 552(a)(5).

Because NSF routinely retrieves proposal information by the name of the Principal Investigator who submitted the proposal, the proposals are maintained in a system of records covered by the Privacy Act. NSF peer reviewer information is handled similarly. In fact, NSF has prevailed in litigation on this issue: https://www.nsf.gov/news/news_summ.jsp?cntn_id=100876. NSF does not publicly disclose data sets where there is a reasonable risk of re-identification of individuals.

As an initial matter, does the Privacy Act actually require NSF or NIH to protect any and all information relating to unfunded proposals and/or peer review scores?

No.

The baseline requirement is that individuals have a right to expect the government to keep their personal information private, with a number of exceptions. That is: “No agency shall disclose any record . . . by any means of communication to any person, or to another agency, except pursuant to a written request by, or with the prior written consent of, the individual to whom the record pertains . . . .” 5 U.S.C. § 552a(b).

But there are two issues here: 1) Exactly what information gets Privacy Act protection? 2) What exceptions are there?

First, let’s talk about what sort of information is at issue. The law talks about “records,” a term defined as “any item, collection, or grouping of information about an individual that is maintained by an agency, including, but not limited to, his education, financial transactions, medical history, and criminal or employment history and that contains his name, or the identifying number, symbol, or other identifying particular assigned to the individual, such as a finger or voice print or a photograph.” 5 U.S.C. § 552a(a)(4).

Right from the start, you may notice that “records” have to contain someone’s “name” or some identifying information. If you took the names and other identifying information out of an NIH/NSF proposal or its peer review scores, it wouldn’t be a “record” under the Privacy Act in the first place!

Another point is to note the kind of records at issue – someone’s education, financial status, medical history, criminal history, etc. All of this amounts to personal information (in some cases, highly personal information). The mere fact of applying for an NIH/NSF grant, or the peer review scores given to the application, don’t resemble someone’s medical or financial history in terms of the privacy interests involved.

Yes, the statute says that “records” include but are “not limited to” those examples. But by traditional principles of statutory construction, words in a list typically are given a related meaning. The list isn’t for nothing—it serves to demonstrate the kinds of things at issue.

For example, if a statute refers to a park being off limit for “vehicles, including but not limited to cars, trucks, motorcycles, and electric scooters,” that list is telling us something about the word “vehicles.” A child’s tricycle would probably not be treated as a “vehicle” to be excluded from the park. After all, the purpose of the list seems to be excluding motorized vehicles that might pose a danger to children or pedestrians. A child’s tricycle isn’t anything like that at all, even if it might technically be a “vehicle.” So too, data on peer review scores for a research proposal does not resemble any of the statutory examples of personal data.

But to clinch the case, even if the definition of “records” stretches broadly enough to include proposals and peer review scores, there’s a key exception:

Agencies ARE allowed to disclose records to a “recipient who has provided the agency with advance adequate written assurance that the record will be used solely as a statistical research or reporting record, and the record is to be transferred in a form that is not individually identifiable.” 5 U.S.C. § 552a(b)(5).

In other words, if an outside researcher is merely looking to perform “statistical research” on data that “is not individually identifiable,” then the Privacy Act provides no reason for NIH or NSF to decline to provide that data.

There might be a debate about what “individually identifiable” means, and some people might argue that pretty much any information is individually identifiable. In technical, statistical terms, that might be true. If there’s a grant application on subject X submitted at a particular time by someone at a university of a particular size, etc., etc., etc., a clever data scientist or an AI tool could probably identify the person who submitted that application.

That said, the law doesn’t define “individually identifiable” in such an expansive fashion, and we shouldn’t worry about that sort of sophisticated effort at reidentification. Simply put, no political scientist, economist or policy scholar would undertake the trouble of getting approval to work with internal NIH or NSF data, all for the purpose of an underhanded attempt to figure out whether it was John Doe at the University of Wisconsin or Jane Doe at UCLA who submitted a particular application as to cell biology. Outside scholars are interested in broader macro questions, not in reidentifying anyone (as if such reidentification would be harmful in any event).  

Moreover, a general rule for statutory interpretation is that statutory provisions should not be interpreted so as to be a nullity. If “individually identifiable” was interpreted so broadly as to apply to virtually any release of data under any circumstances, then there would be no point for Congress to have included an exception for “statistical research” in the first place. Congress clearly intended that “statistical research” should be allowed, and that necessarily means that Congress did not intend for all data to be treated as “individually identifiable” so as to preclude the possibility of all statistical research.

Now, what about the NSF’s rationale as provided to me? I think it doesn’t quite add up. As a reminder, NSF told me that under the Privacy Act, a “system of records” is anything where information “is retrieved by the name of the individual,” 5 U.S.C. 552(a)(5), and that NSF “routinely retrieves proposal information by the name of the Principal Investigator.”

But this merely shows that a “system of records” exists at some point. It does nothing to show why that information remains a protected “record” if the individual names are stripped out, let alone if the data is used for “statistical research” under the Privacy Act’s exception. To use a legal cliché, the tail would be wagging the dog if we allowed the mere fact that an agency stores the person’s name somewhere to make it impossible to release anonymized data in ways that the Privacy Act expressly allows (and that FOIA often requires).

Moreover, NSF claimed to me that “NSF peer reviewer information is handled similarly,” and that “NSF has prevailed in litigation on this issue.” This isn’t quite right. Here, the NSF is referring to Henke v. U.S. Department of Commerce, 83 F.3d 1445 (D.C. Cir. 1996), a case in which someone who failed to win an NSF grant as well as a Dept. of Commerce grant (Henke) sued NSF demanding to know the identities of the peer reviewers who rejected the proposal.

The court noted that under the Privacy Act, individuals (such as applicants) have the right to see information kept about themselves, but there is an exception for “investigatory material“ related to grant eligibility if provided “under an express promise that the identity of the source would be held in confidence.” The court also noted that NSF had told reviewers that “the identity of reviewers will be kept confidential to the maximum extent possible.”

As the court said, the identity of peer reviewers needs to be confidential so that they will be candid without fear of repercussion. That is, they need the ability to be honest without worrying that they might “offend a colleague whom they may like as a person, or hope to work with in the future.”

All of that may be true, but has nothing to do with the case of independent scholars who wish to see anonymized peer review scores so as to evaluate how agencies are funding research. It is hard to see how such research would be threatening to peer reviewers, or cause them to temper their true beliefs. Moreover, the statutory exception applies “only to the extent that the disclosure of such material would reveal the identity of a source . . .” If no peer reviewer identity is revealed, then that term of the Privacy Act doesn’t apply in the first place.

Indeed, the NSF’s own official guidance on data regarding proposals (including peer review and later reports) notes that “information may be disclosed to” researchers who “carry out statistical studies for or otherwise assist NSF with program management, evaluation, or reporting. . . . Disclosures are made only after scrutiny of research protocols and with appropriate controls. The results of such studies are statistical in nature and do not identify individuals.”

In short, there is no statutory reason under the Privacy Act (or the accompanying caselaw) for NIH or NSF to have a blanket refusal to provide deidentified data on proposals and peer review scores to outside researchers who are merely trying to do “statistical research.” That position is, in fact, contrary to both the plain text and the spirit of the law.

As for the NIH, that agency’s policy (or System of Records Notice) under the Privacy Act seems oddly narrow. When it comes to data on proposals (including peer review scores, other reports, etc.), the NIH says that it will provide data:

To a party for a research purpose when NIH:

(A) Has determined that the use or disclosure does not violate legal or policy limitations under which the record was provided, collected, or obtained;

(B) has determined that the research purpose (1) cannot be reasonably accomplished unless the record is provided in individually identifiable form, and (2) warrants the risk to the privacy of the individual;

(C) has required the recipient to (1) establish reasonable administrative, technical, and physical safeguards to prevent unauthorized use or disclosure of the record, (2) remove or destroy the information that identifies the individual at the earliest time at which removal or destruction can be accomplished consistent with the purpose of the research project, unless the recipient has presented adequate justification of the research, and (3) makes no further use or disclosure of the record except when required by law, and reports results of the research in de-identified or aggregate form . . . .

As the bolded phrases indicate, this paragraph seems to be talking only about individually identifiable data. And in that case, it might make sense to be fairly restrictive.

But as discussed above, when identifiers are stripped out of the data, the data is arguably not subject to the Privacy Act at all. At a minimum, the data would be subject to an exception if the researcher merely provides “advance adequate written assurance that the record will be used solely as a statistical research or reporting record.”

Thus, if the NIH takes out identifiable information, it would be completely superfluous to ask researchers to prove that their research “cannot be reasonably accomplished unless the record is provided in individually identifiable form.” Those researchers aren’t even asking for the individually identifiable form of the data in the first place!

In other words, it is quite an odd oversight that the NIH’s Privacy Act notification doesn’t even contemplate releasing non-identifiable information under the “statistical research” exception from the Privacy Act itself.

Conclusion: The White House should direct NIH and NSF (and perhaps other research-funding agencies as well) to rewrite their policies to be more in accordance with the Privacy Act, which directly allows sharing of internal data for “statistical research.”

Everything Is Too Dichotomous: We Need a Better Legal Regime for Letting Scholars Work With Slightly-Private Internal Data

As we’ve seen above, agencies regularly interpret the law in an overly conservative fashion—they ignore FOIA cases that require the release of data on unfunded proposals, and they point to the Privacy Act as a reason never to release data even though the Privacy Act has a straightforward exception for “statistical research.” Even under current law, agencies can and should do much more to release data to independent researchers.

But part of a bigger problem is this: Our current legal regime is arguably too dichotomous or binary. If data is treated as subject to release under FOIA, then in theory it could be released to anyone at all, and could be posted on a public website. On the other hand, if data is treated as subject to a FOIA exception (and/or as protected under the Privacy Act), then no external researchers get the data at all.

That all-or-nothing approach doesn’t fit well with the kinds of data we are talking about. Consider peer review comments. We might not want the full text of reviews to be posted on a public website, at least not without consent— it might be embarrassing to applicants, and might compromise the integrity of the process. Same goes for unfunded applications, which scientists would often want to be kept private in case someone else scoops their idea.

That is why in some of the cases described above, agencies like NIH and NSF are struggling to figure out how to avoid FOIA—they don’t want to be put in the position of having to release all of this data to the entire world.

But outside researchers (e.g., economists and political scientists) aren’t engaged in the usual FOIA-type activity. Plastering information on a public website is not what these researchers want to do at all. Instead, they want to perform academic research that, if necessary, treats any given unfunded application or peer review score as confidential information, and that just analyzes overall questions such as, “Are more innovative proposals doing worse in peer review, and if so, why?”

In other words, these types of data are slightly private. They are not fully public, per se—you wouldn’t necessarily want the data to be published in fully identifiable form on the Internet. But neither are they fully private, in the sense that they need provable statistical anonymization (such as differential privacy) and accessibility only on computers walled off from the Internet.

The latter would be overkill. Remember, these materials all concern requests for public funding, and should by default be as freely and openly available as possible. As a federal appellate court mentioned above held:

“Federal grant applicants cannot reasonably expect that their efforts to secure government funds, especially in a field so much in the public eye as cancer research, will remain purely private matters. There is an obvious public element to the process and the results, as recognized in the NIH practice of releasing both the applications and identities of funded grant applicants.”

There are a number of existing options for handling this kind of slightly private data. In many or most cases, it is likely sufficient just for NIH and NSF to require IRB approval and a signed confidentiality agreement that states, “No personally identifiable information will be published.”

Another alternative is for NSF and NIH to engage with the Institute for Research on Innovation in Science (IRIS), hosted at the University of Michigan. IRIS is a coalition of major universities that contribute “data on 580,000 funded awards worth $192 billion, payments to more than 1.2 million vendors ($35 billion), and wages to approximately 985,000 employees.” It provides access to that data on a privacy-protected basis, with a virtual data enclave supplied to researchers who have IRB approval and who also get approval for their research plan. NIH and NSF data would be a great resource to integrate with the university-level data.

Yet another alternative is that NIH and NSF could try to work with the OMB’s new data resource for “restricted microdata from federal statistical agencies,” namely, ResearchDataGov.org. This website is a platform for getting access to confidential data from, among others, the Census Bureau, the Bureau of Labor Statistics, and the IRS Statistics of Income Division. This was the result of a large, federal-wide effort to make a streamlined platform for accessing such confidential data from over 15 agencies. NIH and NSF could easily piggyback on this existing work (although their data might be more useful if combined with IRIS data above).

In any event, even if you think that NIH/NSF proposals and peer review scores need some sort of top-level protection for confidentiality – which is illogical and contrary to the caselaw on the issue – there’s no reason not to use IRIS or other existing systems to allow outside researchers to do their independent evaluations.

But in any event, it might help if Congress passed a version of “Evidence Act 2.0.” Such an Act could help clarify the options for how federal agencies should handle slightly private data that needs to be widely accessible to independent researchers without going as far as FOIA. We’d have more and better research on how NIH and NSF perform with upwards of $59 billion a year.

FUTURE ACTIONS

The White House:

The White House could, by executive order, ask federal agencies that fund research to:

• “Make data open by default” (see section 202(c) here) to independent researchers with approval by a qualified IRB, including data on internal operations such as peer review scores, disposition of research proposals (included unfunded proposals), and perhaps even analyses by program officers and other agency officials (if anonymized);

• Stop invoking FOIA Exception 4 (trade secrets and commercial/financial information that is obtained from a person and that is “privileged or confidential”) as to any of the above information, unless that exception has been specifically invoked and demonstrated ahead of time by the researcher in question;

• Re-examine the Government in the Sunshine Act and any rules thereunder, so as to enable outside researchers to have full access to any meetings where decisions are made on research funding;

• Reassess all data access policies so as to allow to much more streamlined research and evaluation pursuant to the Evidence Act of 2018;

• Rewrite their policies (as necessary) to be more in accordance with the Privacy Act’s express allowance for “statistical research.”

Congress:

Congress could enact legislation like the Grant Reform and NIH Transparency (GRANT) Act, a bill introduced by Rep. Huizenga. This bill would require NIH to take substantial steps to share its internal data with qualified external researchers. This legislation could expanded to all federal agencies that fund scientific research.

Much thanks to Katy Rother, Julia Lane, and an anonymous White House official for their comments and advice, although all opinions in this brief remain entirely my own.

1. Without jeopardizing anonymity, I’ve heard from top scholars that federal agencies are known to retain veto power over scholarly articles whose findings are displeasing, and agencies have been known to exercise that veto power for no defensible reason. No one ever wants to talk about this sort of sordid activity publicly, but that makes it all the more important that we have a public debate about the external evaluation of federal agencies (and how often the agencies can thwart that evaluation).

2. Note that under the Privacy Act, there is an exception for data that has to be released under FOIA, 5 U.S.C. § 552a(b)(2), so the Privacy Act is not a fallback reason to protect such data.