The NIH Needs a System to Evaluate NCATS, its New Center for Translational Research

Medical institutions have increasingly focused on translational research. The purpose of such research is to accelerate the application of basic scientific discoveries to practical therapies; in other words, to transport discoveries from the researcher’s bench to the patient’s bedside.

The NIH is following the lead of non-governmental research organizations by creating a center devoted to this field, the National Center for Advancing Translational Sciences (NCATS). The NIH plans to spend $575 million on NCATS in fiscal year 2012.

Yet the annual appropriation of $30 billion for the total NIH budget is under intense scrutiny. Because the NIH is a “non-defense discretionary” item in the budget, the agency could be politically susceptible to sharp budget cuts in the next round of negotiations in Congress.

In the past, NIH has given grants for translational research to a broad array of academic and non-profit institutions. These grants were approved through a rigorous process of independent peer reviews.

By contrast, NCATS will perform much of this translational research with its own staff and facilities. As a result, internal projects for translational research will be selected by senior NIH staff without the benefit of independent peer reviews.

Given the tight financial context and internal selection process, the NIH needs a systematic framework to assess the success of translational research projects done at NCATS. In particular, the assessment framework should compare the performance of NCATS to that of grant-receiving institutions along the relevant dimensions.

In a recent article (abstract here), Heather Kline and I proposed seven dimensions for assessing the performance of translational research at institutions such as NCATS: Funding, talent, creation, validation, dissemination, external uptake, and collaboration. For each dimension, the NIH should establish key performance indicators, or KPIs, each of which should have at least one quantifiable metric. For instance, along the “creation” dimension, a KPI could be the drug pipeline for a specific gene therapy, and its metric could be the number of drugs to reach a certain stage of testing.

Given the vagaries of scientific discovery, it is not realistic to simply judge NCATS on whether it produced a cure for a major disease; that could mostly be a measure of good fortune rather than effective management. On other hand, it is not enough for scientists to say: “We are smart and hard working, so continue to give us taxpayer money.”

Here are the seven dimensions, in more detail.

  • Funding: As with most government programs, the funding level of NCATS may be influenced heavily by politics rather than results. But the funding dimension also includes measures of efficiency. For instance, what portion of expenditures goes to administrative overhead instead of directly supporting research?
  • Talent: Recruiting the most talented researchers leads to more effective research. Performance indicators should include the number and qualifications of NCATS researchers as well as the recruitment yield in response to its offers.

Conversely, NCATS should keep track of scientists who leave and why. Non-performing scientists should be encouraged to find other jobs.

  • Creation: This dimension is critical because it focuses on the research’s concrete contribution to patient treatment. It measures volume of an organization’s pipeline and its speed of progress toward milestones.

In addition to treatments, the creation dimension should encompass tools for applied medical research, such as animal models. Although tools do not directly improve patient care, they help other research organizations develop therapies.

  • Validation: This indicates the impact of an organization’s output; it is usually measured by articles published in peer-reviewed journals and citations to such articles.
  • Dissemination: This reflects the promulgation of findings to relevant audiences. Findings are disseminated to the general public through websites or newsletters.

Dissemination to the scientific community goes further than articles; it encompasses releasing data to other researchers, and licensing patents to them on reasonable terms.

  • External uptake: This key dimension gauges the extent to which third parties invest in the products of the NCATS’ applied research. Third parities include venture capital, drug companies and public-private partnerships.

These third parties are critical to the success of applied medical research. They may be the only organizations with the expertise and resources to advance through human trials a concept with promising pre-clinical results.

  • Collaboration: Many characteristics of academia, such as promotion criteria, inhibit collaboration between researchers and clinicians. The lowering of these barriers should be a key priority for NCATS. Performance indicators for collaboration include co-authorships, data sharing and use of shared equipment.

If Congress adopts this seven-dimensional framework for assessing the performance of NCATS, it can evaluate whether its $575 million appropriation for translational medicine is well spent. In particular, Congress can judge whether a more centralized approach to translational research embodied in NCATS is better than a more diversified array of research grants to academic institutions subject to independent peer reviews.

Bob Pozen is a Senior Lecturer at Harvard Business School and a Senior Fellow at the Brookings Institution. His latest book, Extreme Productivity, is now available at your favorite local or online bookstore.

One thought on “The NIH Needs a System to Evaluate NCATS, its New Center for Translational Research