Part 2: Research Conservatism
A survey conducted by Fast Grants found that, if bioscience researchers had complete research freedom, 78.5% would change their current work “a lot” and 20.3% would change their current work “a little.” Only 1.2% wouldn’t change their current research.19
Furthermore, 81% of respondents said they would make their research more ambitious if they could, 62% would prefer to pursue research outside their field, and 44% said they would pursue counterintuitive hypotheses.20
Generally, I think there is good evidence that the NIH systematically promotes conservative research and is overly biased against ambitious research.
At least one interviewee, for instance, said they believed that the vast majority of NIH-funded research is worthless, at least compared to optimized, alternative research funding methods. However, the more moderate position, expressed by a vast majority of interviewees, is that current scientific progress is slower than it could be due to NIH research conservatism.
Aversion to High-Risk Research
One interviewee said that they had a graduate school professor whose final assignment was to write an R01-type grant proposal (the standard NIH grant). After everyone had turned in their projects, the professor told the class that their proposals were excellent, innovative, and he would love to see many of them come to fruition… but the NIH would never actually fund these proposals. They were too high-risk.
Interviewee estimates of how much high-risk research was funded by NIH extramural grants varied widely, from almost none, to one-third of grants.
How Does the NIH Do So Much Basic Research If It’s So Conservative?
Nearly all interviewees said that NIH research is too conservative. And while most interviewees agreed that the NIH is doing less basic research than it should, it still does lots of basic research (51% of grants). But isn’t basic research, which tends to ask fundamental questions about biology without clear answers, inherently not conservative?
So how can the NIH simultaneously do so much conservative and basic research?
The answer, as far as I can tell, is that the NIH’s structures seem to incentivize uniquely conservative, basic research.
The NIH’s structures seem to incentivize uniquely conservative, basic research.
Basically, the NIH pushes a significant portion of its extramural grant applicants into open-ended, basic research paths. But the NIH punishes researchers with high-risk, ambitious proposals within these open-ended paths. Researchers often start with moderately open-ended research goals, then secretly (i.e. outside grant boundaries) conduct experiments to get preliminary data. As the data comes in, they narrow the scope and goals of the research to a more marginal goal with clearer “deliverables.” Then, they apply for a basic research NIH grant with this narrow, marginal goal.
In practice, this means that most NIH grants, even basic research ones, are quite narrowly focused. Even if an exciting and open-ended research path has dozens of grant applicants working within it, each one is aimed at a highly-likely result, which will yield incremental advancements in the field. Open-ended, innovative projects that could either make great breakthroughs or yield null results are largely ignored by grant review panels. The most innovative projects are secretly pursued outside of grant boundaries, but are usually abandoned for safer, narrower scopes as time and funding wind down and the lab needs to apply for a fresh grant.
How is High-Risk Research Discouraged?
This is a more complicated question than it might seem. There is no high-level mandate to move the NIH away from open-ended projects. Extramural research grants are awarded by study sections whose leaders are appointed by the NIH, but whose membership consists of dozens of scientists from universities. The conservatism of NIH grant distribution can best be attributed to the structure and evaluation parameters of study sections, as well as the hegemonic position of the NIH in the bioscience research industry.
Averaging of Research Proposals
The study section process will be discussed at length in Part 4, but a crucial factor that may bias their outcomes towards conservatism is that they choose to award grants based on a consensus process determined by averaging.
Study sections generally consist of between 10 and 30 reviewers who are selected due to their expertise in a particular domain. These reviewers, in many cases, are working on research projects that are related to the very grants they are assigned to review.
Each reviewer gives a score (1 to 9, with 1 being the best) to every grant proposal, these scores are averaged, and then the average is converted into a percentile against all other grant applications in the cohort. All grants below a certain percentile (known as the “payline”) are funded.
Given the scoring system, an NIH grant application is more likely to get approved if it’s liked by everyone rather than loved by some and hated by others. Hence, researchers often design and submit projects that have broad appeal and aren’t too novel or ambitious, and which won’t lead to the invalidation of the work of anyone sitting on the review panel (which often happens as science advances). This is incrementalism: Instead of discovering something truly novel or demonstrating that some of our knowledge was wrong, approval by committees of active scientists means that researchers must build on what’s already there.
“Deliverables” Restrict Research
Many interviewees blamed the lack of high-risk research on study sections emphasizing “outcomes,” “hypothesis-driven research” or “deliverables.”
Basically, the NIH wants its extramural grants to have clear research goals with obvious practical applications. This seems to be the result of a relatively recent push, and was not as much of an issue 20+ years ago.
The demand for clear “deliverables” might seem reasonable, but it counterintuitively punishes ambitious proposals. High-risk projects tend to be more open-ended and unpredictable, and thus are heavily penalized in an evaluation process that asks researchers to enumerate specific hypotheses and likely outcomes which can be translated into practical, public health benefits. In contrast, narrow, conservative projects, with less interesting but more predictable results, are rewarded.
Some interviewees felt that this narrowing of scientific parameters ran counter to the nature of scientific progress. Discovery is just as often unpredictable or random as it is planned.
One interviewee told me an anecdote about microbiologist Dr. Thomas Brock. 21
In 1964, Brock visited Yellowstone National Park on vacation and was intrigued by the greenish-blue coloring of the hot springs. A park ranger told him that it was algae, which Brock didn’t think was possible since nothing could live in waters that hovered near 200 degrees Fahrenheit and sometimes boiled.
Brock spent six years analyzing samples from the hot springs and found that, contrary to established scientific wisdom, bacteria could survive at these high temperatures, including a microbe called Thermus aquaticus. The discovery was a paradigm shift in bioscience and microbiology.
A decade later, biochemist Kary Mullis was working on a process to rapidly amplify and copy DNA. He couldn’t find an enzyme that could tolerate extremely high temperatures, until he stumbled across Brock’s Thermus aquaticus. In 1993, Mullis was co-awarded the Nobel Prize in Chemistry for inventing the PCR process, a foundational tool for modern genetic research and treatment development, and more recently used as a primary method for COVID-19 testing.
Note that Brock’s random discovery enabled an entirely unexpected and unrelated major advance in science. First, a scientist randomly stumbled upon something interesting while on vacation, ushering in a paradigm shift in the scientific understanding of the limits and adaptability of life. That breakthrough unexpectedly identified the perfect component to insert into a process that rapidly manufactures DNA.
This is not to say that the NIH should fund any random curiosity that pops into a researcher’s mind, nor that the NIH should ignore broader scientific objectives (like curing particular diseases). But many researchers believe that the NIH’s structure hinders genuine curiosity-driven research, which often starts with tangential questions and end with broadly useful results.
Preliminary Data, Unspoken Requisite for Funding
Another much-maligned promoter of conservatism is an emphasis on preliminary data. Study sections want to see pre-existing work, either from the applying researcher or other sources, which demonstrate the viability of their hypothesis. High-risk projects are far less likely to have pre-existing data because, if they are truly novel, no one will have done anything like them before.
However, after hearing complaints about preliminary data requirements by maybe a dozen interviewees, I was surprised to learn that it is not a requirement at all, at least not officially. That is, the NIH study section criteria does not officially require preliminary data. Rather, having preliminary data is basically an informal norm for current grant evaluation standards.
Study Section Leadership
Some interviewees suggested that the NIH’s managers are pushing for more conservative research through the purposeful selection of the study section leaders, or Senior Research Officers (SROs). Whether by design or accident, the NIH has supposedly been choosing more conservative SROs, who in turn pick conservative study section reviewers, who then select conservative grants. Study section leaders tend to be older and more well-established in their fields; there may be some selection bias at play.
Most Grants Propose “Safe” Aims
Finally, grant applications are supposed to have clear goals, and if those goals aren’t met, the grantee is unlikely to get funding in the future. In the words of one interviewee, “if you make a big leap and you miss, you’re done for.”
This plays into recent calls across the scientific community for more tolerance for research that yields null results (see [Nature]* and [Inside Science]*). Scientists make empirical observations, devise a hypothesis about a causal explanation for the observation, and then design experiments to isolate variables that determine whether a single variable is causal; but there is no default reason to expect that the hypothesis is true. And the more ambitious and speculative the research, the less likely the hypothesis is to be true.
Pushback Against Claims of NIH Conservatism
Upon showing this subsection to researchers, some pushed back against my argument. They did so not because they thought NIH-funded research was sufficiently ambitious, but because they questioned whether such a broad and substantial assessment could be made about the tens of thousands of grants given out by the NIH each year, and based upon the approval of thousands of reviewers, by interviewing 41 individuals and scouring journal articles.
That’s a fair critique. I couldn’t find any good studies comparing the ambitiousness of NIH-funded research to other institutions, except in narrow cases, like how the NIH’s handful of specially designed high-risk grants stack up against Howard Hughes Medical Institute researchers. Indeed, the size of the NIH makes good comparisons impossible, since all other organizations can afford to be more selective.
There’s also the lingering caveat that interviewee testimonies could represent biased opinions. Many of my interviewees relayed personal stories of grants they considered excellent (usually their own) getting shot down, and it’s possible that they just had a difference of opinion from more sober minds.
But I still think that the NIH extramural research program is biased in favor of conservative research.
For one, this opinion was held by almost every interviewee I asked, albeit to varying degrees. These interviewees vastly ranged in terms of age and career level. Postdoctoral fellows still learning about the system, as well as established professors with decades of NIH-funded research projects, agreed on this point.
Second, I think the structure of the study sections and their parameters create clear incentives that lead to conservatism. These incentives could be counteracted by other stronger incentives, but as will be elaborated upon more in Part 4, I don’t think these incentives are strong enough.
Third, as I’ll explain in the rest of Part 2, I think there are strong incentives for the NIH leadership to encourage this conservatism, both given the unique incentives of the NIH, and due to the default incentives of any large bureaucracy.
The “Benevolent Ponzi Scheme”
Many NIH-funded labs follow a form of the “Benevolent Ponzi Scheme” strategy. Two young graduate student interviewees said the system was already explained to them by superiors, while nearly all older interviewees confirmed familiarity with some form of it. Here’s how it works:
When a researcher becomes a faculty member and gets his own lab and staff, they are typically given a funding package of a few million dollars by the university to kick off new research. The researcher will come up with a project. Then, they will use their starter funds to complete “80-90%” of the project. Then they will apply for an NIH grant for the project, even though it’s 80-90% done.
Once the grant is in-hand, they will use a small portion of the funds to complete the project.
Then, they will use the bulk of the funds to get started on the next project. When that project is 80-90% done, they will apply for an NIH grant for that project. Then they will finish that project, and use the bulk of the funds to get started on the next project. And so on.
Why Do Researchers Use This Scheme?
The Benevolent Ponzi Scheme optimizes both grant acceptance rates and research freedom within the current NIH system.
Two of the most important criteria for grant approval are to have clear “deliverables” and pre-existing data. The easiest way to get both of those is to basically do most of the project before it’s approved. A fresh, bold, novel, high-risk project will have uncertain endpoints and no preliminary data. A project that is 80-90% done will have plenty of preliminary data and an obvious hypothesis that the data is pointing toward. Plus, since the researcher has already been working on the project for a while, they probably had plenty of time to think about applications for whatever result they have already pre-guaranteed.
That might sound utterly cynical, but there’s a more optimistic upside. Since the Benevolent Ponzi Scheme researcher can finish their official project proposal quickly and at little cost, they are free to pursue more interesting, higher-risk research with the leftover funds and time. If they make genuine discoveries, they can just claim that they decided to change research directions on grant maintenance reports. If they don’t make discoveries, then they can just turn in their original project.
Do the Study Sections Know About This?
I’m not sure, as I’ve gotten different answers from interviewees.
Some interviewees say that everyone knows about the Benevolent Ponzi Scheme. Any decent study section reviewer will assume that the grant applications they are reviewing is at least partially based on lies or omissions. Maybe the grant applicants will have conducted more experiments than they are letting on, or maybe they are pretending their hypotheses aren’t, for the most part, already confirmed by the data. It doesn’t really matter.
But some interviewees say that the Benevolent Ponzi scheme is kind of a trade secret and plenty of researchers don’t figure it out until later in their careers. Other researchers, they say, don’t end up using this strategy because they’re too confident, competent, or honest.
Could The NIH be Slowing Research? The Gravity Well Effect
A few interviewees suggested that the NIH was slowing the entire bioscience research industry by acting as a gravity well and spreading conservatism. Here’s how this could work.
If the NIH tends to fund conservative researchers, and NIH grants are the basis of the entire American bioscience research industry, then research institutions are strongly incentivized to hire conservative researchers. These conservative researchers then teach younger researchers conservative techniques, and move into the NIH study sections where they will judge grant applicants by conservative standards. And so on.
One interviewee says this incentive structure “has corrupted everything” – young researchers, career orientations, the hiring process, the universities, and even academic journals. For young researchers in particular, they claimed it was causing a brain drain, whereby the most ambitious researchers were fleeing academia and basic research in favor of biotech and pharmaceutical companies, leaving more conservative researchers behind for the remnants of basic research and the bulk of NIH funds.
How Much Real Research Freedom Is There?
A researcher’s freedom to pursue a broad range of hypotheses is stronger than their grants would suggest, because many researchers lie about their true grant parameters. Still, many people don’t have that much freedom because they need some level of fidelity on their grant applications and they need to report research progress to the NIH. How well they balance these factors is up to their grant application and management skills, which in turn is dependent upon their training.
One interviewee suggested that the more NIH money a researcher gets, the less control they have over their research. That is, with more money from their university (starter funds) and private foundations, they will tend to have more latitude in their research. But the more they rely on NIH funds, the more they will be tethered to their grant applications and updates to grant administrators.
I would think that well-established researchers would have more latitude, but even that has its limits. One interviewee related stories of two instances when their grants were rejected because they involved technology not ongoing in their lab and, thus, there was no preliminary data. In the latter case, the interviewee had been receiving NIH grants for over forty years, they had served as an editor on a major journal, and had been an advisor for an NIH institute. All that clout and history wasn’t enough to get the grant approved. While such ability to withstand political forces is impressive, the reason the grant wasn’t approved was that the interviewee never worked in the field in which they applied for the grant.
Fortunately, the proposal later caught the attention of a prominent non-profit. The interviewee submitted a one-page application and they “nearly fell off [their] chair” a few months later when they got full approval at a higher funding amount than expected. Their project has since yielded “transformational” progress in the field, and though the interviewee is extremely positive about the NIH overall, they are concerned about the lack of risk-tolerance in study sections.
Another interviewee said they once applied for a grant that involved mouse studies. The grant wasn’t approved. On their feedback forms, they found that they lost a lot of points because they didn’t yet have possession of the mice. “Well of course I didn’t… I wasn’t funded yet.”
A few researchers had an interesting take on an unintended consequence of this system: the NIH is biased against “super nerds.”
Navigating the “benevolent ponzi scheme” requires anticipating the judgments of colleagues, knowing the right people to talk to for advice, plotting out how to stagger grant timing and explain results that diverge from official grant applications. These are all skills correlated with extraversion, networking, and sociability. They are not the typical traits of a socially awkward scientist who loves to spend hours going through data sets and discussing abstract theories, rather than figuring out how to game complex bureaucratic systems.
This is not to say that a researcher can’t be both a great scientist and a skilled player of the game. But there are certainly researchers who are uncomfortable with the system, and who wish they could spend more of their time on the science and less on figuring out how to get to do the science.
On the other hand, there are some researchers who are probably a bit too comfortable with complexities embedded in the grant system. Whether by design or happenstance, some lab leaders gain reputations at being so good at getting grants that they focus most of their energy on getting resources and then leave the actual science to their staff. Then again, maybe a bunch of super nerds working for a master grant-getter is the ideal lab structure?
The NIH Does Fund Some High-Risk Grants
Despite all of the above, the NIH has a few grant programs specifically designed for high-risk research, all of which garnered praise from interviewees.
The Pioneer Award is arguably the single-most prestigious grant at the NIH. It is open to applicants from all career stages and requires no preliminary data, and it must be for an entirely novel, scientific proposal. Grants are for at least $700,000 per year for five years. I can’t find an acceptance rate, but the NIH aims to give only seven per year.22
According to the NIH’s own research (from 2013), 23 Pioneer Awards produce more citations, more publications in high quality journals, and are more innovative per-dollar than the standard R01 grants. Somewhat comparable Howard Hughes Medical Institute researchers generate more publications in top journals, but otherwise produce similar results, at least within the smaller timeframe of the Pioneer Awards.
The New Innovator Award is basically a smaller version of the Pioneer Award, but for new researchers who just completed their terminal degree. Funding is $1.5 million split across “two multi-year segments,” and the NIH gives out about 35 per year.24
The Early Independence Award is for current or recently graduated doctoral students (a group that receives vanishingly little funding through direct grants from the NIH). The NIH aims to give out 10 awards worth $250,000 per year for up to five years.25
Another interesting grant is the R35, which is structured similarly to the Howard Hughes Medical Institute’s model of funding “people over projects.” The grant requires no specific aims, but provides a steady (albeit relatively low) level of funding for eight years.
The Transformative Research Award is for “inherently risky and untested” projects, and is more oriented to big teams. In 2022, the NIH will dole out seven awards, worth a total of $8 million.26
Many interviewees said that these grants are underfunded. Even former NIH Director, Francis Collins, has advocated for their expansion.27 Going by the NIH’s guideline numbers, the organization will spend only tens of millions per year on new grants in these categories. The total extramural budget, recall, surpassed $30 billion last year.
A grant-giving non-profit, Open Philanthropy (OP), has an interesting take on the Transformative Research Award. In 2017, OP worked with the NIH on their Second Chance program. They invited researchers who had been rejected from the NIH’s Transformative Research Award to reapply for OP funding. For non-profits, piggybacking off the NIH’s infrastructure seems like a no-brainer.
OP received 120 proposals in three weeks, and funded four grants for a total of $10.8 million.
OP found “little correlation between our evaluations of the 120 proposals and the NIH peer review panel’s evaluation of these proposals.” They suggested this might be because they “[look] to support projects that could have the largest humanitarian impact, regardless of whether they ‘overturn fundamental paradigms,’ as the NIH phrases it.”
However, OP also said: “we considered many of the submitted proposals to be a bit on the conventional side. This surprised us given the ‘transformative’ premise and focus of the TRA program. We speculate that this may be due to the constraints within which applicants feel they must work to get through panel reviews.”
Basic Vs. Applied Research
In a 2017 Congressional testimony, then-NIH Director Francis Collins said:28
“The core of our mission remains basic biomedical science. Given the exploratory and, hence, unpredictable nature of fundamental discovery, basic science is generally not supported in the private sector – but it provides the critical foundation for advances in disease diagnosis, treatment, and prevention through future clinical applications.”
The consensus among interviewees, particularly those who had worked with the NIH for more than two decades, is that the organization has shifted away from basic research and more toward applied research. Admittedly, the line between these concepts can be fuzzy.
“the organization has shifted away from basic research and more toward applied research”
Nevertheless, there is a pervasive sense that there has been a struggle between these two research categories and that the latter is winning out.
The NIH currently considers about 51% of its research to be oriented toward basic science;29 based on their own categorization, basic research has slightly declined from 56% of grants in 2003.30
Basic research is the foundation of science; its insights are far less predetermined, but have far more potential value in downstream applications. Typically, applied science is considered the domain of private industry, where medicines, treatments, and biotech can be packaged and sold for profit. The NIH and National Science Foundation favor basic science precisely because private industry allegedly lacks the incentives to do so.
The NIH’s applied research aims have expanded far beyond the gaps left unfilled by private industry, though, and the agency is now funding research into treatments for a slew of major diseases. My speculation is that this trend started with President Richard Nixon’s “War on Cancer” and the 1971 National Cancer Act, which massively expanded the NIH’s National Cancer Institute (NCI) and cemented its status as the highest-funded institute in the agency.
While most of the interviewees lamented this trend, some were positive about it. Some interviewees asserted that the NIH has done too much research in the past to fill up academic journals, without enough practical results.
Ultimately, the NIH was never intended to fund solely basic research, like the NSF. And, given the enormous scope of the NIH’s work, I’m sure there are some domains where it could be more productive with applied science. But overall, most researchers were concerned that the NIH is drifting away from its core function.
Translational Research
The drift away from basic research largely manifests as a new emphasis on “translational” research, or research that translates basic research into applied research. One interviewee summarized the support for translational research with something he overheard in a study section: “It’s the National Institutes of Health, not the national institutes of cool tricks you can do with genetic mouse models.”
There is a concerted effort to push translational research both within and outside the NIH. One interviewee argued that the NIH has produced a significant amount of useful work which has yet to yield real-world results because of a lack of translational efforts either by the government or private industry.
Why Is the NIH Moving Away from Basic and High-Risk Research?
The best explanation I heard was from an interviewee who said the NIH’s overarching goal is to attain “slow and steady progress on all fronts.”
The NIH’s budget is determined by annual appropriations doled out by Congress, and thus the NIH is highly conscious of how its operations are perceived by the public. Whether it is optimal for the NIH from a public health perspective to pursue more basic or applied research isn’t especially relevant here. What is relevant is how the public perceives these methods. If anything, there is a bias in the general public in favor of applied research because it’s more tangible.
The public is not an expert on scientific matters, and thus it is far easier for the NIH to maintain and increase its budget by promoting itself as an agency devoted to directly solving major health problems through applied research, rather than by laying the foundations for other organizations to create treatments for these health problems through NIH-funded basic science research.
This trend has only intensified in the modern era as the NIH has come under attack by a faction of conservative Republicans who accuse it of exacerbating big government waste. Republican Senator Rand Paul publishes a “Waste Report” where he often identifies NIH projects of allegedly dubious value. In 2019, he singled out an $874,000 project to see how cocaine impacts the sex drives of Japanese quails, and a $1.2 million joint study between the NIH and NSF to study online dating habits (the NSF was also targeted for a $467,000 study on Panamanian frog mating calls).31 He has also attacked a $500,000 NIH study on how much people like spicy food,32 and a $380,000 study on why college freshman tend to gain weight.33
Senator Paul’s tactics are arguably nothing more than cheap political jabs. As the Washington Post pointed out, Senator Paul got his figures wrong on the cocaine quail sex study (it only cost $357,000), and the study was an attempt to better understand how cocaine impacts sexual behavior for the sake of drug addiction treatment (the quails were used because of their highly consistent mating patterns).34
I don’t know enough about Japanese quails or the impact of cocaine on sex patterns to judge the study, but neither does almost anyone else. That’s the point. Maybe it was just scientists pulling a fast one on the agency by studying something completely meaningless and pretending that it could be useful for our understanding of drug addiction. Or, maybe it was really done because of its direct application to the problem. The sheer absurdity of the study, from the layman’s perspective, makes it an effective punchline for criticisms of government waste. Instead of spending more money on educating children or curing cancer, the government is buying cocaine for birds and watching them have sex.
I can’t say for sure that Senator Paul’s criticisms or anyone else’s are what’s pushing the NIH away from basic research, but the sentiment of numerous interviewees is that it’s having an impact. I personally don’t see any substantial, organized opposition to the NIH, so if this is triggering the “fortress mentality,” I think the NIH leadership and its benefactors are being overly defensive.
