BMJ 2020; 371 doi: https://doi.org/10.1136/bmj.m3434 (Published 07 October 2020)Cite this as: BMJ 2020;371:m3434
Thomas J Hwang, researcher1 2, Joseph S Ross, professor3 4 5, Kerstin N Vokinger, assistant professor2 6, Aaron S Kesselheim, professor1
Correspondence to: T Hwang
Objective To characterize the therapeutic value of new drugs approved by the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) and the association between these ratings and regulatory approval through expedited programs.
Design Retrospective cohort study.
Setting New drugs approved by the FDA and EMA between 2007 and 2017, with follow-up through 1 April 2020.
Data sources Therapeutic value was measured using ratings of new drugs by five independent organizations (Prescrire and health authorities of Canada, France, Germany, and Italy).
Main outcome measures Proportion of new drugs rated as having high therapeutic value; association between high therapeutic value rating and expedited status.
Results From 2007 through 2017, the FDA and EMA approved 320 and 268 new drugs, respectively, of which 181 (57%) and 39 (15%) qualified for least one expedited program. Among 267 new drugs with a therapeutic value rating, 84 (31%) were rated as having high therapeutic value by at least one organization. Compared with non-expedited drugs, a greater proportion of expedited drugs were rated as having high therapeutic value among both FDA approvals (45% (69/153) v 13% (15/114); P<0.001) and EMA approvals (67% (18/27) v 27% (65/240); P<0.001). The sensitivity and specificity of expedited program for a drug being independently rated as having high therapeutic value were 82% (95% confidence interval 72% to 90%) and 54% (47% to 62%), respectively, for the FDA, compared with 25.3% (16.4% to 36.0%) and 90.2% (85.0% to 94.1%) for the EMA.
Conclusions Less than a third of new drugs approved by the FDA and EMA over the past decade were rated as having high therapeutic value by at least one of five independent organizations. Although expedited drugs were more likely than non-expedited drugs to be highly rated, most expedited drugs approved by the FDA but not the EMA were rated as having low therapeutic value.
Most novel medicines that are introduced in clinical practice globally are first approved by the US Food and Drug Administration (FDA) and European Medicines Agency (EMA).12 Over the past two decades, both regulatory agencies have established programs to expedite drug development and regulatory review for serious conditions. The FDA has four main expedited programs: fast track (introduced in 1987), accelerated approval (1992), priority review (1992), and breakthrough therapy (2012).3 The EMA has two such programs: accelerated assessment (2005) and conditional marketing authorisation (2006).45 The EMA launched a third program—the priority medicines scheme (PRIME)—in 2016.6 These expedited programs, which are intended to prioritize the most important medicines for faster access by patients, are increasingly the route by which most new medicines are approved.78 The FDA approved 60% of new drugs through at least one expedited program in 2019; by contrast, only 34% of drug approvals in 2000 were expedited.9
FDA and EMA guidance indicate that expedited programs generally should be reserved for drugs that are expected to provide an improvement over available therapies (table 1).34 However, neither the FDA nor the EMA specifically requires data on, or makes regulatory approval contingent on, comparative effectiveness; most new drugs are approved on the basis of placebo controlled trials or single arm studies.10 By contrast, after approval, health systems and payers generally decide to reimburse new drugs on the basis of their added benefit compared with existing therapies (table 2).11
The therapeutic value of medicines benefiting from the FDA and EMA expedited programs is therefore uncertain. Using ratings of therapeutic value published by health authorities in four countries (Canada, France, Germany, and Italy) and an independent non-profit organization (Prescrire), we evaluated the association between expedited programs and ratings of therapeutic value for all new drugs approved by the FDA and EMA from 2007 through 2017.
We used publicly available FDA (https://www.accessdata.fda.gov/scripts/cder/daf/) and EMA (https://www.ema.europa.eu/en/search/search/ema_group_types/ema_medicine) databases to identify all new drugs and biologic agents approved by the FDA and EMA, respectively, between 1 January 2007 and 31 December 2017, excluding generic, biosimilar, diagnostic, contrast, and imaging agents. We chose this study timeframe to allow at least two calendar years of follow-up from the date of approval for an assessment of drugs’ therapeutic value. All data were updated through 1 April 2020.
Expedited programs and regulatory status
Using methods that we described previously,12 we extracted key information from the FDA and EMA databases on each drug in our study cohort, including generic and brand names, therapeutic area (the World Health Organization’s Anatomic Therapeutic Classification system), date of approval, indication, expedited program(s), and orphan designation (rare disease, defined in the US (Orphan Drug Act) as affecting fewer than 200 000 people and in the EU (Orphan Regulation) as a prevalence of fewer than five in 10 000). Expedited programs were priority review, accelerated approval, fast track, and breakthrough therapy designation for the FDA and accelerated assessment and conditional marketing authorisation for the EMA. Drugs may qualify for more than one expedited program. As the PRIME program was established in 2016, no approvals had yet been made through this program by the end of 2017. We manually cross checked expedited program designations with publicly available review dossiers, reports, and annual summaries of new drug approvals.1314
Ratings of therapeutic value
We searched for ratings of therapeutic value published by drug regulatory, public health, or health technology assessment agencies in the US, Europe, and Canada. No rating agencies were identified in the US. In Europe, several health technology assessment agencies publish only a decision on coverage, without providing a rating of the level of added benefit, and were therefore excluded (for example, the National Institute for Health and Care Excellence).15 We identified four national organizations that publish therapeutic value ratings in Canada (Human Drug Advisory Panel),16 France (Ministry of Health),17 Germany (Federal Joint Committee),18 and Italy (Italian Medicines Agency).19 We also included a non-profit organization, Prescrire, which publishes a monthly medical journal that reviews new treatments for healthcare professionals.20 This resulted in five organizations that assess new drugs according to the level of added benefit and public health relevance and subsequently publish ratings of therapeutic value (table 2). All included organizations did not consider cost or cost effectiveness in their ratings and were independent from the pharmaceutical industry.
We defined ratings of moderate or greater therapeutic value as “high” and the rest (that is, minor, possible, not quantified, and no/slight benefit) as “low,” consistent with cut-off values for favorable reimbursement and coverage decisions by the national authorities. We focused on the rating for the indication at the time of first FDA or EMA approval. When multiple ratings were provided by an organization for a single drug (for example, for different subgroups or disease stages), we used the most favorable rating received at the time of the drug’s approval for any subpopulation or clinical setting included in the approved indication. The primary outcome was the highest rating provided by any organization, which was chosen to provide the most generous estimate of the number of drugs rated highly. We also included a more restrictive definition—drugs rated highly by more than one organization—as a secondary outcome.
We used descriptive statistics to analyze the proportion of new drugs rated highly by any organization and logistic regression models with approval year and an indicator variable for expedited status to examine trends in new highly rated drugs. To identify factors associated with being rated highly, we used multivariable logistic regression models that included indicator variables for therapeutic area, a linear term for approval year, separate indicator variables for each expedited program, and an indicator variable for orphan drug designation. We also assessed the sensitivity, specificity, and area under receiver operating characteristic curves of any expedited program and each expedited program separately for a high rating. In a sensitivity analysis, we repeated our unadjusted and adjusted analyses examining the association between expedited program and high therapeutic value rating in the subgroup of drugs assessed by more than one organization. We used Stata version 12 for all analyses and considered two tailed P values under 0.05 to be statistically significant.
Patient and public involvement
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. However, we plan to disseminate results directly to patients through interactive online tools.
From 2007 through 2017, the FDA approved 320 new medicines and the EMA approved 268 new medicines (table 3). As of 1 April 2020, 239 (75%) of the FDA’s new drug approvals were also approved by the EMA, and an additional 15 (5%) were approved through national approval procedures. Most new drugs approved by the FDA and EMA were indicated to treat oncologic (25% and 31%, respectively), alimentary and metabolic (13% and 13%), or blood and cardiovascular (12% and 11%) disorders. Among the 320 new drugs approved by the FDA, 181 (57%) qualified for at least one expedited program, and roughly half (163; 51%) qualified for priority review. By contrast, 39 (15%) of the 268 new drugs approved by the EMA qualified for either accelerated assessment or conditional marketing authorisation.
Ratings of therapeutic value
Ratings of therapeutic value by one of the organizations included in the study were available for 267 drugs (83% of FDA approvals; all but one of EMA approvals). Overall, 31% (84/267) of FDA drug approvals and 31% (83/267) of EMA drug approvals were rated as having high therapeutic value by at least one organization. Drugs that were approved by the FDA but not the EMA were less likely than drugs approved by both regulators to be rated as having high therapeutic value (14% v 33%; odds ratio 3.02, 95% confidence interval 1.01 to 9.02; P=0.04). The relative proportion of drug approvals rated as having high therapeutic value did not change over the study period for either the FDA (odds ratio for time trend 1.00, 0.92 to 1.08; P=0.92) (fig 1) or the EMA (1.00, 0.92 to 1.09; P=0.92).
Association between FDA’s expedited programs and high ratings
Among FDA approved drugs with at least one available therapeutic value rating, 45% (69/153) of expedited drugs (qualified for at least one expedited program) were rated as having high therapeutic value, compared with 13% (15/114) of non-expedited drugs (P<0.001). The number of qualifying expedited programs was also associated with the proportion of drugs rated as having high therapeutic value: 49% (29/59) of drugs qualifying for two and 65% (26/40) of drugs qualifying for three or more expedited programs were rated as having high therapeutic value, compared with 26% (14/54) qualifying for only one expedited program (P<0.001) (fig 2).
In multivariable logistic regression analysis adjusting for therapeutic area, orphan designation, time trends, and each expedited program separately, drugs qualifying for priority review (odds ratio 3.93, 1.73 to 8.91; P=0.001), fast track (3.09, 1.51 to 6.33; P=0.002), and breakthrough therapy designation (4.34, 1.57 to 12.0; P=0.005) were associated with greater odds of being rated as having high therapeutic value than drugs not qualifying for those programs (table 4). Accelerated approval was not associated with greater odds of being rated as having high therapeutic value (odds ratio 1.02, 0.40 to 2.61; P=0.96).
Overall, the sensitivity and specificity of qualifying for any FDA expedited program for a drug being independently rated as having high therapeutic value were 82% (95% confidence interval 72% to 90%) and 54% (47% to 62%), respectively (table 5). Expedited program was not a strong predictor of high therapeutic value rating overall (area under receiver operating characteristic curve 0.68, 95% confidence interval 0.63 to 0.74) and similarly for each expedited program separately: 0.70 (0.65 to 0.76) for priority review, 0.67 (0.61 to 0.73) for fast track, 0.55 (0.50 to 0.59) for accelerated approval, and 0.70 (0.62 to 0.78) for breakthrough therapy designation.
Association between EMA’s expedited programs and high ratings
Among EMA approved drugs with at least one available rating, a greater proportion of drugs qualifying than not qualifying for accelerated assessment were rated as having high therapeutic value (67% (18/27) v 27% (65/240); P<0.001). This was not the case for conditional marketing authorisation (31% (4/13) v 31% (79/254); P=0.98).
In multivariable logistic regression analysis adjusting for therapeutic area, orphan designation, time trends, accelerated assessment, and conditional marketing authorisation, drugs qualifying for accelerated assessment (odds ratio 3.73, 1.49 to 9.31; P=0.005) and orphan designation (2.31, 1.09 to 4.86; P=0.03) were associated with greater odds of being rated as having high therapeutic value than drugs not qualifying for those programs (table 4). Conditional marketing authorisation was not associated with greater odds of being rated as having high therapeutic value (odds ratio 0.54, 0.17 to 1.68; P=0.29).
The sensitivity and specificity of qualifying for any EMA expedited program (accelerated assessment or conditional marketing authorisation) for a drug being independently rated as having high therapeutic value were 25% (16% to 36%) and 90% (85% to 94%), respectively. The sensitivity, specificity, and area under receiver operating characteristic curve of accelerated assessment for a high therapeutic value rating were 22% (13% to 32%), 95% (91% to 98%), and 0.58 (0.54% to 0.63); for conditional marketing authorization, they were 5% (1% to 12%), 95% (91% to 98%), and 0.50 (0.47% to 0.53), respectively (table 5).
Of 267 drugs with any available therapeutic value rating, 245 (92%) were assessed by more than one organization; of these, 50 (20%) were rated as having high therapeutic value by more than one organization. Nearly all of them (48/50; 96%) qualified for an expedited program. We obtained substantively similar results when we repeated the unadjusted and adjusted analyses with this secondary outcome of drugs rated highly by more than one organization and with the primary outcome in the subgroup of drugs with ratings available from more than one organization.
We found that less than a third of all new drugs approved by the FDA and EMA were rated by any of five independent organizations as having high therapeutic value—that is, providing moderate or better improvement in clinical outcomes for patients. Most of the increase in the number of new drug approvals over the past decade was driven by drugs rated as having low therapeutic value, which calls into question the common practice of using simple counts of new drug approvals as a measure of innovation.212223 Rather, a more nuanced view of innovation is needed that takes into account the clinical benefits and relevance to patients of new medicines.
Expedited drugs were more likely than non-expedited drugs to be rated as having high therapeutic value. Stratified by expedited program, drugs qualifying for priority review, fast track, and breakthrough therapy designation by the FDA or accelerated assessment by the EMA were more likely than drugs not qualifying for those programs to be rated highly. Few non-expedited drugs were highly rated. However, in absolute terms, most drugs in the FDA’s expedited programs were rated as having low therapeutic value—even for breakthrough designated therapies and those that qualified for priority review, which is intended for drugs that provide “significant improvement.” By contrast, few drugs qualified for the EMA’s accelerated assessment, but most of them were rated highly.
Implications of findings
The study findings, which are consistent with a previous study of FDA designated breakthrough cancer drugs and an analysis stratifying new drug approvals by novelty of mechanism of action,242526 suggest a widening gap between regulatory approval and the clinical and public health priorities of health systems, payers, and patients after approval. Contributing to this may be the varying quality of clinical trial evidence available at the time of approval and resulting uncertainty around the extent of clinical benefit.272829 As more evidence accumulates after approval, the assessment of therapeutic value may also evolve.
These data emphasize the importance of robust postmarketing evaluation for expedited drugs. Such an evaluation would confirm early evidence of efficacy and help to elucidate findings from several previous studies suggesting that accelerated approval, priority review, and fast track drugs were associated with increased safety related reports or labeling changes.30313233 Greater assurance of timely completion of postmarketing study requirements for expedited drugs could be achieved by requiring that certain mandated studies begin enrolling patients before the FDA or EMA grants approval. In addition, regulators should account for the high expectations that physicians and patients have of the therapeutic value of drugs qualifying for expedited programs.34 In one national survey of board certified internists and specialists in the US, 77% of respondents believed that high quality evidence shows that a drug is more effective than existing treatments if it was designated as a “breakthrough” by the FDA, and 94% preferred a hypothetical cancer drug described as a “breakthrough” over an equally effective alternative without that descriptor.35 Regulatory agencies should explore whether additional explanations or disclaimers are necessary, such as in product labeling, press releases, or approval documents, to provide more realistic expectations of benefit for expedited drug approvals by patients and clinicians, as they already do in labels for drugs with accelerated approval.
Our study findings have implications for the ongoing controversy around drug prices. In the US, the largest public payers are required by law to cover most (Medicaid) or a substantial number (Medicare) of drugs approved by the FDA, regardless of the quality of the evidence supporting their approval or their therapeutic value.3637 Previous studies of cancer drugs have found no association between clinical benefit and drug prices and reimbursement.38394041 Recent proposals to allow public payers to prioritize coverage of high value products, or to vary reimbursement rates based on effectiveness,42 could help to stretch limited budgets and attenuate incentives for the development of, and investment in, medical technologies providing marginal or unknown benefits.434445 The US could learn from the experience of countries in Europe that have integrated assessment of clinical benefit into their reimbursement decisions. Canada, France, and Germany generally permit higher prices for highly rated drugs, while subjecting drugs with low ratings to reference pricing (that is, priced no higher than comparators) or possible formulary exclusion.
Limitations of study
The study had several limitations. Firstly, we focused on initial approval, as this represents the critical clinical and coverage entry point of new therapies; the therapeutic value of supplemental and unapproved uses may differ. Secondly, we did not assess expedited program designations before approval, which were not publicly available. Thirdly, 17% of FDA approved drugs did not have an available rating, mainly because they were not approved in Europe or Canada and therefore not subject to assessment of benefit. Given our finding that drugs that were approved by the FDA but not the EMA were less likely than drugs approved by both regulators to be rated highly, our estimates may overestimate the proportion of drug approvals considered high therapeutic value. Finally, although the criteria and frameworks for benefit assessments were broadly similar across rating organizations, the methods and scoring system for individual organizations can also be influenced by country specific factors and assumptions.46 To be conservative, we focused on the primary outcome of the highest rating provided by any rating organization. We also did sensitivity analyses limited to drugs rated by multiple organizations as having high therapeutic value, which confirmed the primary results.
The FDA and EMA are increasingly using expedited programs to facilitate the development of new drugs, and drugs approved through expedited programs were more likely than non-expedited drugs to be rated as having high therapeutic value. However, the absolute number of highly rated drugs approved by the FDA and EMA over the past decade was low, and the proportion of drugs rated as having high therapeutic value did not change over time. Policy makers and regulators could explore implementing therapeutic value ratings more broadly for new drug approvals, aligning the evidentiary needs of regulatory approval and reimbursement decisions, and informing patients and physicians about the benefits and risks of new drugs, especially those approved via expedited programs.
What is already known on this topic
New drugs are being approved by the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) in greater quantity and with greater speed than ever before
This is due in part to programs designed to expedite drug development and approval
Some countries have independent organizations that help to determine which new drugs offer clinical benefits to patients over existing treatments
What this study adds
Less than a third of new drugs approved by the FDA and EMA were rated by any organization as having high therapeutic value
Expedited drug approvals were more likely to be rated as having high therapeutic value than were drugs not qualifying for any such expedited program
Policy makers and regulators should explore ways to better inform patients and physicians about the results of the rating process and limitations of new drug approvals
Contributors: All authors contributed to the conception and design of the study. TJH and KNV extracted the data, and TJH did the statistical analysis. All authors contributed to interpretation of the results. All authors read and critically revised the manuscript for important intellectual content. All authors made a significant contribution to the research and manuscript, and approved the final version for publication. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. TJH is the guarantor.
Funding: TJH and KNV were supported by the Swiss Cancer Research Foundation. ASK received grant funding from the Kaiser Permanente Institute for Health Policy, as well as Arnold Ventures and the Harvard-MIT Center for Regulatory Science. The funders had no role in considering the study design; in the collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit the article for publication.
Competing interests: All authors have completed the ICMJE form at http://www.icmje.org/coi_disclosure.pdf and declare: this study was supported by the Swiss Cancer Research Foundation, the Kaiser Permanente Institute for Health Policy, Arnold Ventures, and the Harvard-MIT Center for Regulatory Science; TJH has received unrelated consulting fees from the Urban Institute; ASK has received grants from the FDA Office of Generic Drugs and Division of Health Communication outside the submitted work (2013-16); JSR has received unrelated grant funding from the FDA, Johnson and Johnson, Medtronic, Blue Cross Blue Shield Association, Centers for Medicare and Medicaid Services, Agency for Healthcare Research and Quality, and the National Heart, Lung, and Blood Institute of the National Institutes of Health; KNV has received unrelated grant funding from the Swiss National Foundation; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not needed, as this was a study of publicly available summary regulatory information.
The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study was reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Data sharing: Data are publicly available and will be shared after publication.
Provenance and peer review: Not commissioned; externally peer reviewed.
Dissemination to participants and related patient and public communities: The authors plan to disseminate results directly to patients through interactive online tools.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
(Image: Wesley Fryer/Flickr)