The FDA approved 27 new drugs in 2013. Is this a downward trend? Is it an upward trend? Does it suggest that the pharmaceutical industry is failing, or that genomics is finally paying dividends? We revisit Darrell Huff’s 1954 classic How to Lie with Statistics for insights into these pressing questions.
The headline of a recent Associated Press release read “The Big Story, New drug approvals from FDA declined in 2013.” (1) Indeed, the FDA approved only 27 New Molecular Entities (NMEs) in 2013, compared to 39 in 2012. This discouraging news was widely reported in mainstream and business media.
The AP release quotes FDA officials as saying that “the tally of innovative medications approved last year is in line with the historical trend.” The official FDA report was more circumspect, stating that “CDER approved 27 NMEs in 2013, which is similar to average totals of other years from this time period. For instance from 2004 through 2012, CDER has averaged about 26 NME approvals per year. In 2012, CDER approved 39 NMEs, but this was an unusually high number….” (2).
The Associated Press repeated the FDA’s analysis, but then states that “Experts attribute the recent uptick to a combination of factors: a stable, well-funded FDA and a newly established research model among drug makers…” A commentator on the web site of the National Venture Capital Association, extolling the importance of regulatory reforms championed by the Association, echoed this view, writing: “…we believe that the upward trend in drug approvals over the past several years has increased investor confidence in the FDA's review and approval process” (3).
Reading the conflicting interpretations of the FDA data, I was reminded of Darrell Huff’s classic book How to Lie with Statistics (W.W. Norton, 1954) and his statement that “The secret language of statistics, so appealing in a fact-minded culture, is employed to sensationalize, inflate, confuse, and oversimplify.” Huff’s book illustrated the subjectivity that could be introduced into analysis by selective data analysis, graphical representation, and the association of correlations with trends or causation. For the record, the FDA has reported historical data (4) on the number of new drugs (New Chemical Entities and Biologicals) as shown in the attached figures. The top panel shows the raw data with three possible “trends” that could be inferred from selectively chosen linear regressions. The first, 1970-2013, might be interpreted as representing the “post-thalidomide” era of drug development. The second, 1996-2013, might be interpreted as representing the post-PDUFA (Prescription Drug User Fee Act of 1993) era of the FDA; and the others, 2005-2013, or the 2004-2013 time frame described by the FDA, might be used to assess recent trends. In the tradition of “How to Lie with Statistics,” each of these arbitrary analyses provides the basis for empirical, though not statistically significant, narratives suggesting that drug development trends are either positive (green) or negative (red).
Using standard regression models to evaluate long term trends is even more confusing. The bottom panel shows the application of various analytical formulations including linear regression (purple), 5, 10, 12, 15, and 20 year moving averages (blue), or polynomial (4th, 5th order) (orange) curve fits. The 10 year moving average is in bold with +/- SD of the residual shaded. It is apparent that analysis of drug approval data alone does not conveniently support either a “declining” or “upward” narrative.
Can anything be learned from the FDA data?
These data are seemingly consistent with Jurgen Drews’ 1995 prediction (5), that the number of new pharmaceutical products in development at that time was not sufficient to replace the number coming off patent and sustain continued growth of the biopharmaceutical industry. While each of the regression models in the bottom panel appear to show growth in the number of NMEs approved though the 1970s and early 1990s, this trend seems to be absent after 1995.
Less defensible is the statement in the AP release that “FDA drug approvals are watched closely by analysts as both a barometer of industry innovation and the federal government's efficiency in reviewing new therapies.” On the surface, it seems obvious to make a causal connection between innovation in industry, the efficiency of the review process, and the number of NMEs that result. Indeed, Drews’ concern about the inadequacy of the product pipeline is often referred to as the “innovation gap.”
Yet, as Huff warned, “flaws in assumptions of causality are not always so easy to spot, especially when the relationship seems to make a lot of sense or when it pleases a popular prejudice.” Is the assumption of causality in this instance flawed? We think it might be.
It is recognized that not all technological innovations have a sensible, sustaining effect on product development. Many technological innovations are incompatible with the capabilities and conventions of established industries. Such innovations may initially be disruptive to product development processes and markets until businesses adapt to new technological requirements and commercial opportunities. Disruptive innovations may require substantial periods of time to mature before they contribute to the development of competitive products.
In this context, short term changes in number of NME’s reflects only incremental or sustaining innovations, and not the type of radical and disruptive innovations embodied in genomics and other “omic” technologies that have emerged in recent decades. In fact, a 2001 report titled Fruits of Genomics predicted that genomic innovations in the pharmaceutical industry would initially have a negative impact on the timelines and cost of drug development (6). Their model predicted that it would not be until after 2010 that these innovations would begin to produce increasing numbers of NMEs.
Is there evidence to support the prediction that this “newly established research model” is finally producing increasing numbers of NMEs? I would suggest that you first read “How to Lie with Statistics,” then come to your own conclusions.
Fred Ledley is Director of the Center for Integration of Science and Industry at Bentley University, and Professor of Natural & Applied Science and Management.
1 - http://bigstory.ap.org/article/new-drug-approvals-fda-declined-2013
2 - http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/DrugInnova...
3 - http://nvcaccess.nvca.org/index.php/topics/research-and-trends/406.html?...
4 - http://www.fda.gov/AboutFDA/WhatWeDo/History/ProductRegulation/Summaryof...
5 - Drews, Jürgen, and Stefan Ryser. "Innovation deficit in the pharmaceutical industry." Drug Information Journal 30.1 (1996): 97-108.
6 – Brothers, Lehman. "The fruits of genomics." Lehman Brothers: New York (2001).
