- General overview of the value framework
- Final Specific Updates
- Ultra-orphan Drugs
- Delivery System Interventions
- Conceptual structure of the ICER value framework
- Comparative Clinical Effectiveness
- Incremental cost-effectiveness analysis
- Other Benefits or Disadvantages and Contextual Considerations
- Potential Budget Impact Analysis
- Report development
- Patient engagement
- Identification of low-value services as part of evidence review process
- Economic model transparency
This paper presents final updates to the ICER value assessment framework, including refinements of its conceptual structure and modifications to the specific methods used to gather and assess evidence of different types. The process for stakeholder engagement during report development and public meetings is also being updated and will be described briefly here; greater detail will be available in separate guides to engagement for patients and manufacturers.
This update to the ICER value assessment framework follows upon several years of experience using the current framework in the evaluation of drugs, devices, tests, and delivery system innovations. During that time ICER has actively sought the input of all stakeholders and made iterative changes to its methods and overall procedures to enhance their transparency and to seek to improve the ability of all parties to participate meaningfully in the process. ICER has also benefitted from two separate periods of national public comment. The first, in late 2016, resulted in the submission of over 300 pages of comments and suggestions from more than 50 organizations and individuals. Following posting of specific proposals in January 2017, a second public comment phase saw submissions of over 260 pages from 49 organizations. ICER wishes to thank all of these commenters for the time and effort they put into these comments, and the many thoughtful contributions they have made.
This paper reflects this combined experience, public input, and many additional discussions with stakeholders in various settings. This finalized update to the ICER value framework and associated methods will be in place to guide reports launched during the two year period of July 2017- June 2019, with the next formal update cycle scheduled to begin in 2019.
Overarching purpose and principles of the ICER value assessment framework
For more than 10 years ICER has been active in developing methods for evidence assessment. Evidence assessment, however, is only one component of ICER’s broader effort to provide mechanisms through which all stakeholders and the general public can engage in discussions on how best to use evidence as the foundation for a more effective and sustainable health care system. A formal effort was undertaken between 2014-2015 to gain input through a multi-stakeholder advisory group on ways to define with greater detail the conceptual and methodological underpinnings of ICER reports. Ultimately, the purpose of the value framework is to form the backbone of rigorous, transparent evidence reports that, within a broader mechanism of stakeholder and public engagement, will help the United States evolve toward a health care system that provides sustainable access to high-value care for all patients.
In this effort ICER is guided by several key underlying principles. One is that we act with respect for all, in concordance with a presumption of good will on the part of all participants and stakeholders in the health care system. ICER does not intend to target any particular interest group or organization. There are many areas in which the US health system fails to serve patients well, in which access to care is suboptimal, waste and inefficiency pose major problems, and costs to patients and the health system fail to align with added value. ICER believes that only through collaborative efforts, built upon a foundation of civil discourse and honest consideration of evidence on effectiveness and value, can lasting progress be made on behalf of patients today and those of the future.
The ethical vision inherent in ICER’s work recognizes that many choices that are made in health care – choices in clinical care, insurance coverage, pricing, payment, and allocation of resources within health systems – must address the basic reality that societal resources for health care are not unlimited, and that there will always be trade-offs and dilemmas over how to organize and pay for the services provided within a health system. Too often, these decisions are made without rigorous evidence and with little transparency. Too often there is little chance for reflection or public engagement in managing the tensions that can arise between innovation, access, and costs. ICER’s value assessment framework seeks to place scientific methods of evidence analysis at the heart of a clearer and more transparent process. The value framework reflects our strong underlying belief that rigorous thinking about evidence can prevent the kind of waste that strains our ability to provide patient-centered care. The framework also is intended to support discussions about the best way to align prices for health services with their true added value for patients. While considering value and linking it to pricing and insurance coverage cannot solve every dilemma, nor satisfy every need, ICER believes it offers the best hope of avoiding rationing of care by the ability of patients to pay for care, and that it can promote a more dynamic, innovative health care system that will make the best use of available resources in caring for all patients.
The population perspective and intended uses of the ICER value framework
The ICER value framework describes the conceptual framework and set of associated methods that guide the development of ICER evidence reports. ICER reports are intended to support deliberation on medical policies related to health services (e.g. tests or treatments) and delivery system interventions (e.g. preventive programs, changes to the organization of medical personnel). To inform these kinds of medical policies the ICER value framework takes a “population” level perspective as opposed to trying to serve as a shared decision-making tool to be used by individual patients and their clinicians. Taking a population perspective implies that the ICER value framework seeks to analyze evidence in a way that supports population-level decisions and policies, such as broad guidelines on appropriate care, pricing, insurance coverage determinations, and payment mechanisms. A value framework intended to support decisions about the care of individual patients requires a structure that invites weighting of benefits, harms, and costs from the individual patient’s perspective. There is a very important need for better evidence-based guides for individual decision-making, and ICER reports may be helpful in providing some of the substrate for these kinds of decision guides, but this is not the primary intended purpose of the ICER value framework or of ICER reports.
Even with its population-level focus, however, the ICER value framework seeks to encompass and reflect the experiences and values of patients. Representing the diversity of patient outcomes and values in a population-level framework is difficult because there will always be an inherent tension between average findings in clinical studies and the uniqueness of every patient. There will also always be diversity in the way that patients view the balance of risks and benefits of different treatment options. The ICER value framework does not solve these tensions, but neither does it obscure them. Population-level decisions and policies have always been made by life science companies, insurers, and clinical organizations looking at evidence in the same general way. One important goal of the ICER value framework is to provide an evidence report that does a better job of analyzing the strengths and limitations of the available evidence, including what is or is not known about the variation in response to different treatments among patients with different personal and clinical characteristics. The ICER value framework also creates an explicit place and role for consideration of elements of value that are important to individual patients but that fall outside traditional clinical measures. Guided by the value framework, ICER reports analyze evidence and incorporate input from patients and other stakeholders to help explore the potential tension between population-level policies and the perception of value by individual patients.
General overview of the value framework
As shown in the new structure of the ICER value framework, it seeks to inform decisions that are aimed at achieving sustainable access to high-value care for all patients (see Figure 1 below). This goal requires consideration of two general concepts: “long-term value for money” and “short-term affordability.”
Figure 1. New conceptual structure of the ICER value assessment framework
Long-term value for money
Long-term value for money serves as the primary anchor of the ICER value framework. It is itself a concept that is comprised of multiple domains: 1) comparative clinical effectiveness; 2) incremental cost-effectiveness; 3) other benefits or disadvantages; and 4) contextual considerations. Updates to how these domains are measured and integrated into an ultimate judgment of long-term value for money will be described in detail in the “updates” section of this paper. There are several high-level points about this element of the value framework that bear highlighting here:
- Long-term perspective.
Even though most of the clinical data available on health care services come from studies of relatively short duration, the grounding of any evaluation of value should recognize the long-term perspective on both outcomes for patients and costs. The ICER value framework recognizes this principle by grounding the methods of incremental cost-effectiveness analysis in simulations that estimate outcomes and costs at the longest feasible time horizon, usually the full lifetime of patients. Benefits for patients and potential cost offsets for new treatments that might take many years to be seen are therefore estimated and included as a core element of the value framework.
- Foundation in the evaluation of evidence on comparative clinical effectiveness.
The ICER value framework is rooted in an objective evaluation of the evidence on the comparative clinical effectiveness of different care or care delivery options. This element of the framework serves as the primary source of information to inform cost-effectiveness analysis, and includes a systematic review of available evidence performed according to the highest academic methodological standards. As part of the evaluation of comparative clinical effectiveness, ICER reports include a clear description of the sources of evidence, the strengths and limitations of individual studies, and a summary judgment of the net health benefit of different care options along with a statement explaining the relative certainty that the body of evidence is able to provide. The detailed methods used by ICER in its evaluation of comparative clinical effectiveness are available on the ICER website (https://icer-review.org/methodology/icers-methods/). The ICER rating system for evidence has been published in a peer-reviewed journal and has been endorsed by the AMCP-NPC-ISPOR Comparative Effectiveness Research Collaborative.,
- Acceptance of multiple forms of evidence.
Patients, clinicians, and policymakers are most interested in evidence on the comparative clinical effectiveness of care options, but this does not mean that ICER’s value framework limits the type of evidence to be considered to the results of randomized controlled trials (RCTs). When available, high-quality RCTs provide evidence that is least susceptible to certain scientific biases, but the best evidence on longer-term benefits and harms often comes from other sources, such as observational analyses based on cohort studies, patient-reported data, and long-term registries. When head-to-head trials have not been performed, rigorous insights into comparative clinical effectiveness also often require indirect comparisons through the method of formal network meta-analysis. In short, ICER has a flexible and ecumenical approach to sources of evidence and, while stressing the importance of the rigor of clinical trial data in any assessment, the value framework and ICER’s methods incorporate multiple sources and types of evidence, seeking the evidence that is most helpful in understanding the long-term net health benefits for patients of different care options.
- Recognition that what matters to patients is not limited to measured “clinical” outcomes.
The inclusion of an explicit domain of value labeled “other benefits or disadvantages” demonstrates that the ICER value framework fully acknowledges that all too often what matters most to patients is poorly captured in the available clinical trial data. Sometimes this occurs because surrogate outcome measures do not reflect true patient-centered outcomes; but even when trials do capture the clinical outcomes that matter most to patients, there are other aspects of value related to the complexity of the treatment regimen or the impact of care options on the ability to return to work, on family and caregiver burden, on public health, or on other aspects of the health system or society. The ICER value framework identifies these “other benefits or disadvantages” as important elements of any overall judgment on long-term value for money, and all ICER reports have separate sections in which evidence and information pertaining to these elements are presented. As part of this final update we will describe later in the paper a method for making the integration of this domain of value in the public deliberation and voting process at ICER meetings even more tangible.
- Acknowledgment of the role of contextual considerations.
Decisions about the value of care options do not happen in a vacuum. There may be broader contextual issues related to the severity of the condition, whether other treatments are available or soon will be, and ethical, legal or other societal priorities that are important to acknowledge as part of any discussion on value. The ICER value framework includes this element and it is explored in a separate section of each ICER report. In addition, contextual considerations often feature prominently in the deliberation on value between independent expert committees and all stakeholders that is a central feature of the public meetings convened by ICER on each report. Linked to the updates for “other benefits or disadvantages,” revisions to the methods used to integrate contextual considerations into the ICER meeting and voting process are described later in this paper.
 For further insight and examples a useful resource is the FasterCures and Avalere Health work on “Integrating the Patient Perspective into the Development of Value Frameworks” available at http://www.fastercures.org/assets/Uploads/value-coverage-framework-March-2016.pdf
With long-term value for money being the dominant element in considerations of value, a complementary perspective is provided by an evaluation of short-term affordability. The ICER value framework includes an explicit evaluation of the short-term affordability of different care options by analyzing the potential short-term budget impact of changes in health expenditures with the introduction of a new test, treatment, or delivery system process. Among the final updates presented later in this paper there are several related to potential budget impact analyses, but here it is important to note two high-level aspects of this part of the ICER value framework.
- Potential budget impact analyses estimate the net budget impact across all elements of the health system.
ICER’s methods have never sought to estimate the potential budget impact of treatments within “silos” of a payer budget, such as the expenses only on pharmaceuticals, devices, or hospital costs. It remains a core principle of ICER’s value framework that it should evaluate both short and long-term costs across the entire health system, so that care options that might increase spending for one type of service (e.g. drugs) while reducing other spending (e.g. hospital costs) receive full credit for cost offsets and are not penalized in any way.
- At 5 years, the time frame for considering “short-term” affordability is stretched as far as possible without losing relevance for identifying new care options that may require special measures – in pricing, payment mechanisms, coverage criteria, or budgeting – to maintain patient access without serious financial strain throughout the health care system.
Public insurers in the US, such as Medicaid, and private insurers manage health care use and expenses within relatively short 1-2 year budget timeframes. Any new added costs to the health system must therefore be estimated and planned for within that timeframe. At one time ICER reports used 1-2 year budget timeframes, but with feedback from stakeholders an iterative change was introduced in 2015 to push the time frame out to 5 years. Doing so reduces the utility of the analysis for most insurers but helps accommodate some of the important potential clinical benefits and cost offsets that may not occur immediately with the adoption of a new care option. With the primary anchor of the ICER value framework being the long-term perspective represented by long-term value for money, the time horizon for short-term affordability has been extended as far as it seems possible in order for it to serve the important purpose of providing information to inform discussions on whether special efforts need to be taken to manage the introduction of a new care option so that access and affordability can both be maintained.
Justification for short-term affordability as part of the ICER value framework
Critics of including budget impact assessment as part of a value framework often ignore a key question: What type of economic perspective and information currently influences decision-making by insurers in the US? Without a doubt—and insurers are the first to admit this—it is budget impact. Considerations regarding budget impact, and not measures of long-term cost-effectiveness, have dominated the way that insurers assess the economic impact of all health care services, not just drugs. There is a logic to this perspective but also obvious perverse outcomes.
Budget impact is a reasonable consideration because insurers work in rapid cycles with purchasers and individual subscribers, translating short-term cost projections into planned insurance premiums for the coming year. Rapid cost growth in the short-term, especially when it increases beyond anticipated inflation rates, pushes quickly upstream to purchasers and policymakers who have to make their own short-term decisions about how to find the needed resources. This may lead to decisions to increase deductibles or otherwise reduce health benefits for employees; for example, state governments might need to consider reducing next year’s education budget to find the funds to keep a Medicaid program afloat.
In addition, for provider groups that bear financial risk, budget impact analyses inform very real short-term decisions about how to allocate resources to maximize the quality of health care within a given budget. A rapid increase in costs resulting from the significant budget impact of a new drug might lead to decisions to forgo hiring of needed new staff, or delay of introduction of other new services. Quite simply: budget impact, and not long-term cost-effectiveness, determines how affordable health care insurance will be in coming years and shapes what health care can be provided with the resources available.
And yet, the perverse influence of an undiluted focus on budget impact cannot be overstated. A narrow short-term perspective blinds policy makers, insurers, and providers to the need to forge efforts to reshape the delivery system and reframe payment mechanisms to “make room” for new, and potentially expensive interventions that will help patients and pay off in the end. Therefore, if an economic analysis of new interventions is focused only on the short term, relying solely on budget impact estimates, patients and the health care system will be the ultimate losers.
But our nation cannot make advances in managing the potential tension between long-term value for money and short-term budget impact by keeping budget impact assessment out of value frameworks and value assessment reports. The idea that having analyses of long-term value and budget impact in the same report will somehow taint decisions can only be imagined if budget impact were not already dominating the playing field. The ICER value framework includes budget impact analysis because leaving it out would only reinforce its silent power over too many decisions.
Thus, ICER’s value framework represents the conviction that keeping budget impact considerations off the table, to be factored in only post-hoc by insurers or provider groups in ways unknown, would be a mistake. It would rob our nation of the chance to bring the public directly into the critical discussions about health care and health insurance that we need to have if we are going to achieve sustainable access for all patients to the kind of innovative new tests, treatments, and delivery system interventions that add value to their lives.
 Ollendorf DA, Pearson SD. An integrated evidence rating to frame comparative effectiveness assessments for decision makers. Med Care. 2010 Jun;48(6 Suppl):S145-52.
Final Specific Updates
Following a second phase of public comment, what follows below are specific updates that have been confirmed for the ICER value assessment framework and associated methods of evidence analysis and stakeholder engagement during the coming two year cycle of reports 2017-2019. These updates are generally applicable to ICER reviews of all health care services, including tests, drugs, devices/procedures, and delivery system interventions such as prevention programs, alterations in the organization and composition of health care teams, etc. However, please note several important exceptions and modifications relevant to specific types of interventions:
|Public Comment and Final Update:
Many comments emphasized the importance of distinguishing methods for assessments of treatments for rare conditions. ICER hosted an all-stakeholder meeting on May 31, 2017 to gather further input on whether and how to adapt its standard assessment methods and will be posting a set of proposals sometime in July 2017 for a 60-day public comment period.
There are reasons to consider the possibility of modifications to the evaluation of comparative clinical effectiveness and value of treatments for ultra-rare conditions, given certain limitations in the ability to field large clinical trials and given the tension between usual cost-effectiveness thresholds, the need for innovators to receive adequate rewards for risk, and the research and development costs necessary to develop drugs for very small patient populations. Other questions exist about whether treatments for ultra-rare conditions have distinctive “other benefits or disadvantages” and “contextual considerations” that need to be addressed in a value assessment. ICER has produced a technical brief on these issues and held a multi-stakeholder meeting to begin gaining input on these questions.
|Public Comment and Final Update:
Comments received from device manufacturers emphasized their view of the importance of distinguishing types of evidence and the many nuances in device evolution, learning curve of practitioners, etc. in value assessment. Comments also highlighted the difficulty of using the approach proposed for potential budget impact thresholds for new drugs to new devices. ICER will continue to use the same general value framework when doing assessments of devices and will highlight these important contextual issues related to the type and strength of evidence available. In addition, ICER will calculate potential budget impact for new devices but will not attempt to calculate a potential budget impact threshold against which to judge whether there should be an affordability and access alert for a new device.
There are many important, unique aspects to the development, early evaluation, regulatory approval, and patterns of use and iterative evidence generation for devices. Therefore, although the conceptual elements of the ICER value framework remain the same for any health care intervention, the specific methods for incorporating and judging evidence will differ for devices. For example, ICER methods acknowledge the practical and ethical considerations that may make it impossible to use RCTs in the early evaluation of clinical effectiveness, while iterative changes to devices, along with the learning curve for practitioners, also raise special considerations about how to judge the available evidence. Evaluations of long-term cost-effectiveness are made challenging because of the potential for evolution of devices and the attendant changes in cost, effectiveness, and the types of patients who will be treated. These complexities are also relevant to estimations of potential budget impact, and, as noted in sections below, it is very difficult to identify the current baseline costs of all device use in the US health care system in order to calculate a growth target for a budget impact threshold. For these reasons the conceptual elements of the ICER value framework remain relevant for devices but within that framework ICER will continue to incorporate specific approaches to evidence evaluation for devices that reflect their unique features.
 See for example the discussion of strength of evidence for devices in the ICER reviews on proton beam therapy, depression, prostate cancer, migraine management, and congestive heart failure (https://icer-review.org/topics/).
|Public Comment and Final Update:
Very few public comments were received that addressed ICER’s methods for assessing diagnostic tests. As stated in the original update proposal, ICER will continue to recognize the different hierarchy of evidence that is fit-for-purpose for assessments of tests, and ICER will not develop a specific potential budget impact threshold.
Similarly, different approaches to evidence evaluation are required for diagnostic interventions and tests used to monitor patients or provide information on disease prognosis. For example, the general hierarchy in the types and strength of evidence for tests is different than that for therapeutic interventions. As with devices, tests will continue to be evaluated using the overall conceptual approach of the ICER value framework but there will be important modifications based on the distinctive nature of the evidence and the system for the development, evaluation, and use of diagnostic interventions. Further work will be needed to develop a method for estimating a threshold for potential budget impact that should trigger additional policy maker consideration of short-term affordability.
Delivery System Interventions
|Public Comment and Final Update:
No public comments were received on ICER’s methods for assessing delivery system interventions. As stated above, ICER will use the same general value assessment framework to guide its reviews of delivery system interventions, but as with devices and tests, some of the specific methods for judging evidence and for determining thresholds for potential budget impact analysis will reflect the unique nature of these kinds of health service innovations.
There are also many distinctive challenges to evaluating the evidence on the effectiveness and value of delivery system interventions. Chief among these is that in most cases a delivery system intervention will be highly variable in its implemented form across different settings, raising great questions about the generalizability of results from studies of one institution or one system of care. RCTs can be difficult to perform, increasing concerns about the internal validity of study findings. ICER will use the same general value assessment framework to guide its reviews of delivery system interventions, but as with devices and tests, some of the specific methods for judging evidence and for determining thresholds for potential budget impact analysis will reflect the unique nature of these kinds of health service innovations.
 See for example the discussion of the Fryback and Thornbury evidentiary model used as part of the ICER review on cardiac nuclear imaging, coronary computed tomographic angiography, CT colonography, breast cancer screening, and diagnostic tests for Alzheimer’s disease (https://icer-review.org/topics/).
 See for example ICER reviews on Community Health Workers, Integration of Behavioral Health Care into Primary Care, Management of Opioid Addiction, and Diabetes Prevention Programs, and Outpatient Palliative Care (https://icer-review.org/topics/).
1. Conceptual structure of the ICER value framework
|Public Comment and Final Update:
Public comments supported the previously adopted changes in the terminology of the different elements in the ICER value framework that are described below. Comments on mechanisms for integrating the different elements of the framework in an assessment will be discussed in sections below, as will comments on the role of potential budget impact analysis and affordability in a value assessment.
1.1 As shown in Figure 1 earlier in this paper, ICER has already moved to replace the conceptual term “care value” with “long-term value for money” as the term for the summary judgment of value that incorporates evaluation of comparative clinical effectiveness, incremental cost-effectiveness, other benefits or disadvantages, and contextual considerations.
1.2 The original value framework included a second framing of value, “provisional health system value,” to reflect the formal integration of “care value” with consideration of potential short-term budget impact. Interim changes had already dispensed with a vote on provisional health system value, and with this final update ICER confirms its elimination of this term. The ICER value framework is now structured (and depicted) without a formal integration of long-term value for money and short-term affordability. Instead, the value framework suggests that consideration of these two elements is necessary to inform decisions seeking “sustainable access to high-value care for all patients.”
|Public Comment and Final Update:
Comments received appreciated the re-emphasis on the inclusion of evidence from multiple sources, not just RCTs. Some comments encouraged ICER to remain vigilant in applying a rigorous analysis to the risks for bias in different forms of evidence. Clarification was sought on the methods that ICER will use to evaluate the heterogeneity of treatment effect for key clinical outcomes. New language has been added to section 2.2 to address this issue.
2.1 ICER re-states its intent to evaluate evidence arising from multiple sources, not just randomized controlled trials (RCTs), that can be useful in judging the comparative clinical effectiveness of different care options.
RCT data and results of rigorous systematic reviews of high-quality evidence will remain critical elements of ICER evidence reviews, but evidence from non-RCT sources will continue to be included according to transparent quality stipulations (e.g. size, duration, eligibility criteria, etc.) in the scoping document and final evidence report. ICER also continues to include evidence from the “grey literature” as per its stated criteria available at https://icer-review.org/methodology/icers-methods/icer-value-assessment-framework/grey-literature-policy/.
2.2 Whenever possible from available data or data provided by manufacturers, ICER will include an evaluation of the heterogeneity of treatment effect for key clinical outcomes. This will be accomplished through two efforts: 1) by continuing to emphasize in discussions with clinical experts, patient groups, and manufacturers that we seek to identify clinical subpopulations for which data are available to determine relative effectiveness; and 2) by presenting within the evidence report information complete distributional statistics for key outcomes, beyond means and medians, whenever the data are available.
3. Incremental cost-effectiveness analysis
|Public Comment and Final Update:
The most common comments were on the following two issues:
1. Bertram MY, Lauer JA, De Joncheere K, Edejer T, Hutubessy R, Kieny MP, Hill SR. Cost-effectiveness thresholds: pros and cons. Bull World Health Organ. 2016 Dec 1;94(12):925-930. Epub 2016 Sep 19.
2. Karlsberg Schaffer, S., Cubi-Molla, P., Devlin, N. and Towse, A. Shaping the Research Agenda to Estimate Cost-effectiveness Thresholds for Decision Making. Office of Health Economics. 2016. Available at: https://www.ohe.org/publications/shaping-research-agenda-estimate-cost-effectiveness-thresholds-decision-making.
3. Woods B, Revill P, Sculpher M, Claxton K. Country-Level Cost-Effectiveness Thresholds: Initial Estimates and the Need for Further Research. Value Health. 2016 Dec;19(8):929-935.
3.1 The primary measure by which the incremental cost-effectiveness of different care options will be compared will remain the cost per quality-adjusted life year (QALY).
The QALY is the established benchmark for capturing benefits for patients through lengthening life and/or improving the quality of life, and it is the standard used by academics, manufacturers, patient groups, and governments around the world. ICER participates in the global dialogue around the best methods for evaluating the value of health services and is always attuned to new developments that might provide a better and fairer system of measuring benefits across different kinds of interventions and patients. Several refinements to ICER’s use of the QALY are described below.
3.2 The range of incremental cost-effectiveness ratios used by ICER for several years in its calculation of value-based price benchmarks has been $100,000 to $150,000 per QALY. Current benchmarks for cost-effectiveness thresholds are frequently justified by estimates of “societal willingness to pay,” which, based on earlier consensus efforts at the World Health Organization have commonly been cited as approximately 1-3 times the per capita GDP of the country per additional QALY. For the US this range is now approximately $57,000 to $171,000. Among others organizations, the American College of Cardiology has adopted a range of $50,000-$150,000 per QALY for its methods of incorporating value judgments in clinical guidelines.Studies of individual willingness to pay (by trading off salary for additional years of life) have widely varying results but many are in the range of two times the individual’s salary. Given the mean personal income in the US in 2015 was $44,510, this would suggest a threshold of approximately $90,000 per QALY. The third, and in many ways most relevant information to guide the setting of cost-effectiveness thresholds is information on the true opportunity cost at the margin of health spending. Recently, empiric studies have been performed in upper- and medium-income countries in Europe and Latin America that have found that to reflect true opportunity costs the cost-effectiveness threshold should be set lower than 1 times the per capita national GDP (approximately $24,000-$40,000 per QALY by extrapolation for the US).
Given this information and after reflection on public comment, ICER has decided to adopt the following for its methods over the next report cycle:
- Maintain the $100,000-$150,000 range for the ICER value-based price benchmark.
- At ICER public meetings we will have the independent CTAF and CEPAC voting panels vote on “long-term value for money” when the base case incremental cost-effectiveness ratio is between $50,000 per QALY and $175,000 per QALY. At incremental cost-effectiveness ratios below $50,000 per QALY there will be a presumption of “high value”; at ratios above $175,000 the intervention will be deemed “low value” without formal voting by the committee. The lower bound of $50,000 per QALY we feel is justified by opportunity cost estimates. The upper bound within which voting will be held is extended to $175,000 per QALY despite the move away from this high a relative threshold in other countries, and despite reservations expressed by the WHO itself about using 3x per capita GDP as an upper bound, because this approach will allow for more explicit consideration of uncertainty around a base case estimate near $150,000 per QALY, and $175,000 is just slightly higher than 3 times the per capita GDP in the United States for 2016.
- As will be described later in greater detail, the $100,000-$150,000 range for the ICER value-based price benchmark will not be shifted according to votes on “other benefits or disadvantages” and “contextual considerations” or on “long-term value for money” by the independent appraisal committees.
 See ACC/AHA Statement on Cost/Value Methodology in Clinical Practice Guidelines and Performance Measures. Available at http://circ.ahajournals.org/content/129/22/2329.
 Figures based on “Selected Characteristics of People 15 Years and Over, by Total Money Income, Work Experience, Race, Hispanic Origin, and Sex”. US Census Bureau. 26 August 2016. Available at https://www.census.gov/data/tables/time-series/demo/income-poverty/cps-pinc/pinc-01.html.
 Woods B, Revill P, Sculpher M, Claxton K. Country-Level Cost-Effectiveness Thresholds: Initial Estimates and the Need for Further Research. Value Health. 2016 Dec;19(8):929-935.
3.3 ICER will include cost per life-year gained and additional measures of clinical benefit in the presentation of the results of incremental cost-effectiveness analyses.In order to increase the transparency to the cost per QALY estimates, ICER will include in its reports analyses of cost per life-year gained and certain other “cost per consequences” as a core part of every report, seeking input from patients, clinical experts, payers, and manufacturers on what outcome(s) will be most important for this comparison. For example, treatments intended to prevent strokes might be compared by a “cost per stroke averted” analysis as a complement to the cost per QALY analysis.
3.4 ICER will perform scenario analyses that examine the influence of lower utilities for individuals with chronic severe conditions on cost per QALY findings.When the major impact of a treatment is to extend the life of patients with a chronic and severe condition, the lower utility (quality of life) assigned to these health states will diminish the overall QALY gain relative to the QALY gain that would be calculated for the same extension of life for patients with a higher baseline quality of life. Under these conditions ICER will actively compare the cost per life-year-gained with the cost per QALY and will also perform scenario analyses in which any life-year gained in less than perfect health will count as one whole year as long as the health state in question is good enough for the person question to consider it “livable,” i.e. preferable to dying. (See Nord E. Cost-value analysis of health interventions: Introduction and update on methods and preference data. PharmacoEconomics (2015) 33:89-95.) When the impact of lower utilities for baseline chronic conditions makes a substantive difference in the calculated incremental cost-effectiveness ratios for treatments, ICER will seek patient input and public comment on which scenario analysis should serve as the base case within the ICER report.
3.5 ICER will continue using estimates of prices net of discounts, rebates, and other price concessions as the base case input for prices used in cost-effectiveness and potential budget impact analyses. Analyses using WAC prices will also be included for context. ICER has long sought a benchmark for pricing that could reliably and with relative transparency provide an estimate for net prices in the US market. With the recent report on treatments for psoriasis ICER has begun a collaboration with SSR Health L.L.C., a consultancy which combines data on net US dollar sales with information on unit sales to derive net pricing at the unit level across all payer types. Further details on the mechanism used to estimate net prices are available in the methods section of each ICER report.
3.6 ICER will continue to calculate incremental cost-effectiveness from the health system perspective as its base case, but will perform a scenario analysis including work productivity when feasible, and will use a new template to give greater detail and transparency to the perspective taken in all cost-effectiveness analyses.ICER will continue to use the health system perspective as the “base case” of its cost-effectiveness analyses for two reasons. First, full consideration of the societal perspective often requires inclusion of broad and uncertain assumptions regarding the impact of health care on productivity, income tax generation, educational outcomes, the criminal justice system, and disability and social security benefits. Finding reliable estimates for long-term effects of health interventions on these broader outcomes is usually not possible. Second, the health system perspective is the most relevant for decision-making by public and private insurers, risk-bearing provider groups, and health care policymakers. Whenever feasible, however, the relative impact of different care options on work productivity will be estimated in the ICER report and considered by ICER independent public appraisal committees as part of their weighing of “other potential benefits and disadvantages” as described later in this paper. Work productivity may also be included in a scenario analysis of the cost-effectiveness model when the scoping discussions suggest that different interventions under review may have notably different impacts on work productivity. To emphasize the important distinctions among health system and societal perspectives, ICER will use the template recently promulgated by the Second Panel on Cost-Effectiveness to describe the elements of health system and societal perspectives included in ICER cost-effectiveness analyses.
3.7 ICER’s cost-effectiveness analyses will not routinely make estimates of price changes across comparator treatments linked to patent and exclusivity time horizons. However, when high likelihood of a major change to pricing can be anticipated within 12-24 months, a scenario analysis may be developed to explore the impact of price changes on long-term cost-effectiveness ratios.Including assumptions about price changes is not currently the standard in academic or health technology assessment agency cost-effectiveness analyses. In part this is because it is very difficult to predict the pricing landscape many years into the future, and even when patents and exclusivity expire prices may not drop substantially. When they do it may be because prices have increased rapidly in the preceding few years, and prices of relevant comparator drugs may drop as well. Changes in the competitive landscape anticipated within 12-24 months, however, may be considered if there are consistent historical findings of price changes that can be applied to the topic under review.
|Public Comment and Final Update:
ICER’s proposed mechanism for capturing and weighing “other benefits or disadvantages” and “contextual considerations” sought to respond to the first round of open public comment and continue our own efforts to increase the explicitness of consideration of value elements outside of comparative effectiveness and incremental cost-effectiveness in the ICER report and in the deliberation of voting of the independent appraisal committees. While public comment praised ICER for the general thrust of its proposal, nearly all comments advised ICER not to move forward with this approach, expressing concern that there are no “validated” or “consensus” methods to integrate these factors into overall judgments of value, and that there were therefore risks of poor calibration across appraisal committees and other unintended consequences.
Most health technology assessment groups around the world do not attempt to quantify these domains of value, believing that their relative weight in any overall judgment of value should be left qualitative and subject to public discussion. Given the comment from various stakeholders, however, ICER has considered several methodological options that could enhance the transparency and explicit integration of these considerations.
Formal multi-criteria decision analysis (MCDA) has been considered but rejected because we do not believe that the methods for weighting individual elements are robust enough to add to reliability of value judgments. ICER has attempted formal MCDA with its independent committees on several occasions in the past and found the technique too complicated for reliable use.
Other approaches were explored for quantifying some considerations within “other benefits or disadvantages” and “contextual considerations.” For example, ICER considered use of proportional and absolute QALY shortfall methods to measure “burden of illness” and create a weighted QALY. However, we have decided not to adopt these methodologies given the risk of unintended consequences that favor some conditions/treatments over others and in light of the lack of consensus on these techniques in the academic community and among national health technology assessment agencies worldwide.
We also explored recently proposed methods in Norway in which broader societal considerations, including burden of illness, would be used to assign a treatment to one of 3 step-wise cost-effectiveness thresholds. Our own investigation, including discussion with multiple patient groups and other stakeholders, suggests that this approach would be likely to create what many would view as ethically problematic distinctions between types of patients and conditions.
ICER could continue with a general approach that identifies other benefits or disadvantages and contextual considerations and invites independent committees to consider these factors in overall votes on what will now be called “long-term value for money.” After the public comment described above, and further reflection within ICER, we will adopt the following approach for the next one year, treating this element of ICER reports and meeting deliberation as a specific learning exercise from which we hope ICER and all stakeholders will gain further insights into best options for integrating these considerations.
4.1 Based on ICER’s existing value framework, discussions with stakeholders, benchmarking other value frameworks around the world, and this most recent round of public comment, ICER reports will explicitly delineate 7 potential “other benefits or disadvantages” and 5 “contextual considerations” as shown in the tables below:
 Ottersen T, Førde R, Kakad M, et al. A new proposal for priority setting in Norway: Open and fair. Health Policy. 2016 Mar;120(3):246-51.
Other Benefits or Disadvantages:
4.2 ICER will seek input on other benefits and contextual considerations from all stakeholders, especially patients and patient groups, beginning during the Open Input phase of each report and continuing throughout report development. In its reports ICER will synthesize this information and include an analysis of any data related to these elements. During the public meeting, a summary of the ICER report findings will be presented, and the appraisal committee will discuss all potential other benefits and contextual considerations following the public comment phase of the meeting. This process will culminate with a vote in which each member of the appraisal committee will vote yes, no, or uncertain on each of the other benefits and each of the contextual considerations. We are exploring the technical possibilities of allowing patient representatives and clinical experts involved in the panel discussion, and even all those in attendance, whether in person or on the web, to vote as well, with non-appraisal committee votes displayed separately.
4.3 The voting on other benefits and contextual considerations will be a topic of discussion during the policy roundtable that concludes the meeting. The results of the voting will also be included in the final ICER report and report-at-a-glance summaries. But during this upcoming 2-year report cycle, and until ICER and all stakeholders have had the chance to exchange perspectives on the lessons learned from this approach over the coming 2 years, the results will not be used to suggest a specific relevant incremental cost-effectiveness ratio or associated value-based price for the intervention(s) being considered. The final report will continue to include the base case cost-effectiveness analysis findings and value-based price ranges extending from $100,000/QALY to $150,000 per QALY. During the trial of this approach to voting on other benefits and contextual considerations ICER will continue to welcome dialogue with all stakeholders and may decide based on experience and input to revise the list of elements and/or how the votes are conducted. Ultimately, the goal will be to engage all stakeholders in a shared process of learning whether this or another approach can offer a transparent, reliable approach to support more specific integration into assigning cost-effectiveness thresholds and value-based prices.
5. Potential Budget Impact Analysis
|Public Comment and Final Update:
Comments received ranged from ardent support for ICER’s current approach to potential budget impact analysis to equally passionate arguments that budget impact analysis should be excluded entirely from ICER drug assessment reports. Two important specific concerns/suggestions were:
2. “As budgets are annual, it is essential we have a perspective on both the short-term and long-term impacts a medication or medication class is expected to have on the pharmacy benefit and on broader insurance expenditures. Pharmacy benefit premiums are set a least a year and frequently two years prior to activation. Therefore, the potential short-term impacts of a new medication or medication class are especially critical to understand. We are concerned with the short-term time frame change from two years to five years. We believe five years is stretched too far and would prefer ICER remain with a two year time frame.” ICER’s response is that we acknowledge that 5 years is far beyond the usual short-term budget horizon for PBMs, insurers, and employers and other plan sponsors. We moved to 5 years in part to stretch the time horizon as far as it seemed feasible to capture some of the potential longer-term cost-offsets provided by new treatments. We also believed that 5 years was a better time frame within which to capture the full uptake of a new drug, given that for many drugs the uptake continues to rise up and through year 5 after launch. Given our new potential budget impact graph, it should be possible for stakeholders to make a close estimate of a 2-year budget impact if they use their own assumptions of uptake at 2 years and the price they will pay. What would be lost are any cost-offsets that kick in only after 2 years of treatment.
5.1 As previously described, potential budget impact will no longer be included conceptually as one component of “provisional health system value.”
5.2 The new approach for performing and presenting an analysis of potential budget impact for new drugs is shown in the Figure on the following page and described subsequently.
Figure 2. Updated approach to potential budget impact analysis. The black line indicates when the potential budget impact threshold is reached at each combination of price and percent uptake among eligible patients at 5 years.
5.3 ICER will no longer attempt to estimate the uptake of a new intervention. The intention of ICER’s initial methods was to estimate the potential uptake of a new intervention if insurers and provider groups exercise no restraint on utilization. This “unmanaged uptake” assumption has proven difficult to convey given the natural tendency to view an uptake estimation as an estimate of what will actually happen in the marketplace.
5.4 Rather than try to estimate real-world uptake, ICER will present information that will allow stakeholders to ascertain the potential budget impact of a new service according to a wide range of assumptions on price and uptake. Prices modeled in the potential budget impact analysis will include: WAC, estimated net price from SSR data, and prices to achieve cost-effectiveness thresholds of $50,000, $100,000, and $150,000 per QALY. As part of this analysis for new drugs, ICER will continue to present information allowing stakeholders to know when the combination of price and uptake at the national level would lead to a potential budget impact that would meet a threshold linked to a growth target for the overall health system.
5.5 This potential budget impact threshold for new drugs will continue to be calculated as double the average net budget impact for new drugs that would contribute to overall health care cost growth beyond the anticipated growth in national GDP plus an additional 1%. Extensive discussions with stakeholders have affirmed the relevance of linking the potential budget impact threshold to national GDP growth.
5.6 For services other than new drugs ICER has explored different ways to calculate a potential budget impact threshold linked to growth in existing use and overall growth targets, but has not arrived at an approach in which we have full confidence. After discussion with stakeholders, for devices and other services a net budget impact threshold will not be used.
5.7 New calculations updating the potential budget impact threshold have been done for 2017-2018. The updated potential budget impact threshold is $915 million per year, annualized over 5 years. Full calculations with sources for data are shown in the Figure following this section. The two most notable changes from the previous calculation are the estimates of total personal medical health care spending and the contribution of drug spending to total health care spending. For the former, personal health care less dental care is now used as a more relevant figure for health spending. For the latter, updated figures that capture drugs administered in hospital and physician offices are now used. The combined effect of these two changes, in combination with a lower projected GDP growth rate for the US, leaves the overall threshold for potential budget impact for drugs little changed from 2015-2016.
5.8 ICER will note in its reports the percent uptake of a new intervention, at its net price level, that would produce a potential budget impact that exceeds this threshold. Note that it is possible that the new intervention will not exceed the threshold regardless of uptake level; this will be noted as well.
5.9 As described earlier, the goal of integrating considerations of potential budget impact with appraisal of an intervention’s “long-term value for money” will be clarified in the value framework as being: “sustainable access to high-value care for all patients.” Given this goal, ICER will include as part of its final report an “affordability and access alert” if discussion among stakeholders at the meeting of ICER’s independent appraisal committees suggests that utilization driven by clinical need, at estimated net pricing, would exceed the budget impact threshold without active intervention by insurers and others to limit access to the treatment. The purpose of an ICER affordability and access alert will be to signal to stakeholders and policy makers that the amount of added health care costs associated with a new service may be difficult for the health system to absorb over the short term without displacing other needed services or contributing to rapid growth in health care insurance costs that threaten sustainable access to high-value care for all patients.
6. Report development
|Public Comment and Final Update:
Comments generally praised the evolution of ICER’s methods for report development, including the specific changes noted in the update proposal. There were a large number of additional comments, some general, some very specific, regarding ICER’s process for report development, procedures related to the selection and training of independent appraisal committee members, and the management of the public meeting deliberations. The Methodology section of the ICER website has much of the information requested in several public comments, including how topics are selected, how we interact with manufacturers and patients, and how the public comment processes work. The Methodology section of our website will be undergoing a significant re-organization in the coming months, and many of the comments we received have already started to inform our thinking about how to make information about what and how we do more accessible. Two key issues/ comments that we would highlight include:
1. Expertise of the appraisal committee members: “It is important for multiple voting panel members to have clinical expertise in the disease area under discussion to improve the clinical accuracy of their determinations. The recent inclusion of patient and clinician representatives alongside the panel is helpful for broadening the panel perspective, but these representatives do not actually get to vote alongside the panel. Transparency around how panel members and the patient/clinician representatives are evaluated and selected is needed.” ICER’s response is that we disagree that all voting panel members have to be patients with that condition or clinical experts in the field under discussion. We believe we can have fully adequate input from clinical experts without involving them as voting panel members, a process that would embed intellectual and financial conflicts of interest in the process. We have clearly defined selection criteria for panel selection and work closely with industry, patient groups, and other stakeholders to identify patient/clinician representatives.
2. Updates to ICER reports: “ICER should develop a clear process for managing the evolution of evidence and that assessments be revised as significant new evidence becomes available. If an assessment is no longer reflective of current evidence and ICER does not have the bandwidth to update the report, it will be important to flag the assessment as out of date and remove the report from the website.” ICER’s response is that we acknowledge that reconsideration of evidence on comparative clinical effectiveness and value is important for all stakeholders. ICER does not have the capacity currently to do routine updates to its reports but has performed one “Brief Evidence Update” as a model of what we will do on an ad hoc basis in the near future. We anticipate with future growth that we will be able to launch a more systematic approach to updates. Given the critical early decisions that are made regarding pricing, coverage, and use of new technologies, in its reports on tests, drugs, and other treatments ICER will continue to focus its evaluations to inform policy decisions at or near the time of regulatory approval.
Along with these final updates for 2017-2019, ICER has posted updated versions of guides for patient and manufacturer engagement in the report development and meeting process. These engagement guides provide improved templates and suggestions to help the patient community participate in the scoping of the topic and contribute data and other information to the report. Timelines have all been reconsidered, with overall length of report development extended and every effort made to provide suitable response time to draft documents. Some of the highlights from these materials include the following changes, some of which have already been instituted in recent reports and meetings.
6.1 Preliminary report findings from the systematic review and economic modeling are now discussed with manufacturers and patient groups prior to posting of the first draft review for broader public comment.
6.2 Patient representative(s) and clinical experts now join the independent committee for the entire meeting, being available for questions and able to make comments during the presentation of the evidence and deliberation prior to voting.
6.3 Patient groups are now being given the opportunity to present the results of their own evidence generation through patient-reported outcomes and surveys on other benefits or disadvantages.
6.4 At our meeting in February 2017 we successfully introduced a change to our meeting agenda that creates an expanded opportunity for manufacturer comment and discussion immediately following the presentation of the summary of the evidence review. We will continue with this approach.
6.6 All reports will include improved transparency in the listing of patient groups, clinical experts, and policy experts who have been consulted as part of the report development process. Peer review of the draft report by 1-2 clinical experts will be noted.
7. Patient engagement
|Public Comment and Final Update:
Many of the comments received on patient engagement highlighted a need for more clarity around our processes. We will be editing the Patient Engagement Guide to provide clearer guidance, and are currently in the process of updating the Methodology section of our website. We hope to use that section to provide more in-depth explanations of key areas of concern, such as how we identify patient groups, how patient comments are incorporated into our reports, how decisions are made regarding public comment speaking slots during meetings, and more.
Below, we have provided a brief summary of the comments received related to our patient engagement methods.
Several comments related to ICER’s outreach to patients and patient groups. Patient groups expressed concern that the patient role can feel reactionary, with few opportunities to engage other than responding to a posted document, and that ICER should be more proactive in identifying patient groups to participate in the process.
ICER strives to provide multiple opportunities for patients to share their perspectives. The open input period gives patients and patient groups opportunity to provide any information in any format to inform the report, and ICER frequently engages in phone conversations with patient groups, and individual patients as appropriate, to learn more about the patient perspective. Information from open input, public comment, and stakeholder calls give insights into the “other benefits or disadvantages” of a treatment or intervention, which factor heavily into the value assessment.
ICER continues to work to improve its outreach to patient groups at the outset of each project. We currently rely on internal searches, recommendations from other patient groups and stakeholders, and outreach through larger patient networks to identify key organizations in each topic area. We will continue to consider new ways to identify patient groups and increase visibility of our reports to ensure that all interested groups are aware of the review early in the process.
Comment Timeline and Format of Comments
Numerous comments received expressed concern with the timeline and required format for public comment submissions. ICER adheres to tight timelines for each report in order to balance timing of expected drug approvals with decision makers’ need for timely information to inform policy and practice. As such, comment periods must be limited to ensure time for ICER staff to review comments and incorporate them into the report.
We recognize that short timelines can be difficult, especially for organizations with fewer resources and staff members to provide comment. In an effort to allow for preparation, the timing of all public comment periods are announced at the beginning of each report process. We would be happy to receive feedback on how we can provide useful guidance to groups submitting public comments to ensure that the process can be as efficient as possible for these organizations. We will continue to consider new strategies to support patient groups in that process.
Some comments noted concerns with formatting requirements that may present barriers to some patients. In cases where these requirements are a burden, ICER is happy to discuss accommodations on a case-by-case basis. Other comments noted that page limitations made it difficult to provide patient analyses and other information. All data and analyses can be submitted as background information, and do not count towards the page limit.
Finally, comments noted confusion over how and when comments would be incorporated into the report, and how the comments are addressed. A revised explanation of our methodology will further clarify how information is used throughout the process, and we will continue to provide summaries in each report of how patient input in particular was incorporated throughout the process. All public comments received on scoping documents or draft reports are posted to ICER’s website and remain publicly available alongside the report. ICER also provides a detailed response to public comments received.
Types of Information Collected
Some comments pointed to a greater need for transparency around the types of data included in reviews and how patient groups can be involved in submitting their own data sources. This information will be covered more thoroughly in the updated Methodology section of our website.
Public Meetings, Oral Comments, and Policy Roundtable
Several comments noted a lack of clarity around how patients are selected to provide public comment during meetings, concern with the balance of patient representatives on the policy roundtable, and barriers to attending an in-person meeting.
Public commenters are selected to ensure a balance of perspectives; spaces are reserved for patients or patient advocates, clinicians, and other stakeholders. Priority among patients requesting a slot is given to those living with the condition, or representatives from organizations whose work relates directly to that condition. Since time is limited, and we cannot guarantee speaking slots for all interested speakers, we do encourage all stakeholders to submit written comments on the report, which will be publicly posted online and made available to the voting council at least two weeks in advance of the meeting.
Comments also noted a need for patient representation on policy roundtables during public meetings. Each roundtable includes representatives of numerous stakeholder groups, including patients, clinicians, payers, and drug manufacturers. In most cases, we include 1-2 representatives from each of these groups in order to create a balanced discussion. Updated methodology documents will provide further clarity around selection of patients for both public comment and roundtable participation.
Finally, comments highlighting financial or physical barriers to meeting participation suggested having an option for patients to participate in meetings remotely. ICER is looking into this possibility, and we look forward to providing more information in the coming months.
Other comments received called for more patient-friendly descriptions of key report findings and conclusions. Our Report-at-a-Glance provides a high-level summary of the report, and we will continue to identify new ways to present our findings in ways that are accessible and actionable for all stakeholders.
8. Identification of low-value services as part of evidence review process
9. Economic model transparency
|Public Comment and Final Update:
We received many comments encouraging ICER to seek additional ways to make its models more transparent and open to critique. Provision of executable models was desired by some; others thought that some more limited access to the model might suffice.
ICER has been working on this issue for a long time and acknowledges the legitimate interest among all stakeholders to be able to examine (and perhaps later apply) ICER’s economic models. Over the course of the public comment periods we have continued our dialogue with academic colleagues to maximize the degree to which our (their) work can be replicated during the review process. We already make much more information publicly available prior to the meeting than is the norm for academic peer review. All parameter inputs and the model structure are shared as early as possible with manufacturers. We are pleased that an increasing number of companies have told us that they have been able to replicate our results. But two roadblocks exist to making available the executable model. First, top flight academic health economists and their academic institutions require that they retain the intellectual property to the executable model and have the ability to use it for future academic purposes. It is possible that some kind of confidentiality agreement could be constructed to address this issue, and we will continue our efforts to explore this option. Second, however, is the very real practical barrier that it is not possible to simply hand over a model and expect someone, even someone very skilled, to know how to dissect or run the model without extensive help from the model developer. We have explored this issue with our academic modeling network and received consistent guidance that it is not feasible for them to assist all stakeholders in this effort during the development of the model. Therefore, we will continue to emphasize all possible efforts to share the model inputs and model structure with stakeholders in a way that will enable them to understand and critique the model, often through replication. When manufacturers or other stakeholders are having difficulty replicating the ICER model results we will use our multiple touchpoints to seek to dissect the reasons why and give them the information that they need to succeed in replication. Secondly, we will continue to discuss with our modeling collaborators the possibility of release of additional model information after a suitable “embargo” period to allow for academic publication. Further details on this approach will require more discussion among ICER’s network of academic health economists but we highlight this issue as one that requires significant further attention from ICER and we have prioritized this area for more rapid update within one year. Our current practice is described in the language on the following pages that is also available on the ICER website.
To try to make the model assumptions and outputs as transparent and reproducible as possible, ICER has adopted the following general statement and approach.
|The Institute for Clinical and Economic Review (ICER) is committed to open and transparent engagement with all stakeholders that have an interest in each of its evidence reviews. This commitment to transparency extends to the development and/or modification of economic models. Such transparency helps to increase the public’s confidence in model results. Without detailed descriptions of model structure and processes as well as estimates used, economic models run the risk of being considered “black boxes,” with no way to evaluate the validity of model processes or accuracy of model inputs. Explicit delineation of model structure and flow gives stakeholders the ability to evaluate the model’s face validity. Details on the point estimates and ranges used in sensitivity analyses allow for the explicit testing of alternative assumptions and model inputs, provide insight into the drivers of specific results, and allow other interested parties to replicate or extend analyses conducted by ICER and its collaborators.
Our general approach to model transparency is based on the Modeling Good Research Practices Task Force report on “Model Transparency and Validation” jointly produced by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision-Making (SMDM). Our aim is to provide information on the model structure and processes, all major inputs and sources for data, and key assumptions used in our economic analyses, so that readers can judge their confidence in the results while preserving the intellectual property rights of those we collaborate with.
All model documents will note that funding for ICER’s analyses is unrestricted and publicly disclosed. In addition, ICER develops economic models in collaboration with academic researchers who are free from financial conflicts on any given project. In addition, ICER maintains a strict conflict-of-interest policy for its own employees, which can be accessed at: http://www.icer-review.org/policy-on-interactions-with-external-partners-and-potential-influences-on-judgment/.
For each report, the interventions selected for study will be specified in detail. ICER and its collaborators will provide model documentation, including model structure, comparators, and specifications. When existing models are being used, ICER will provide references to prior publications that provide further details on the model. When new models are developed, this information will be provided as part of the technical report.
Following the publication of a final scope for each topic, ICER and its external collaborators will publish a modeling analysis plan with detailed specifications for the expected conduct of the work. The plan will be published on a public website used to share collaborative research known as the Open Science Framework (https://osf.io/7awvd/), approximately 10-12 weeks after a topic is publicly announced. Stakeholders will be notified when the analysis plan is posted. The plan may be updated following review of additional data sources, discussions with stakeholders, and other activities, and so is intended to be considered a “living document.” Detailed elements of the analysis plan will include:
Sources for model inputs, risk equations, etc. will be provided as part of the documentation. In general, ICER’s analyses will use data sources and information from published or publicly available sources, including peer-reviewed journals, supplementary appendices, briefing documents used by regulatory authorities, and conference proceedings. In specific instances, valid analyses may require the use of unpublished information, such as manufacturers’ data on file. In such circumstances, explicit requests will be made to affected parties, and any reasonable documentation to protect patient and/or stakeholder confidentiality will be provided. The final version of the modeling analysis plan will be used in conducting the ICER’s “long-term value for money” analyses.
Importantly, the modeling analysis plan is intended to provide enough information for an experienced researcher to be able to replicate the economic model and analyses. Actual executable models and associated computer code will not be provided as part of the deliverable, as such an effort would unduly compromise the intellectual property rights of ICER’s external collaborators. As the ISPOR-SMDM Task Force has pointed out, without such protections, “the incentives and resources to build and maintain complex models could disappear.”
Additionally, ICER and its collaborators will provide a summary of the results of these analyses in a model technical summary. This 10- to 15-page summary will be part of a larger report that ICER will produce that will include information on the available clinical evidence, current guidelines and payer coverage policies, and other relevant topics. The model summary will consist of the following sections:
Methods, including key assumptions and key model inputs
Summary and Comment, including limitations and comparison to other published models on the topic of interest
The model results become sections of an ICER-published report on the comparative clinical effectiveness, cost-effectiveness and budget impact of the specific interventions being evaluated. The initial draft report will be posted for a public comment period of four weeks, after which it may be revised. The revised draft report is then presented as part of a public assessment meeting. The modeling sections of the report are intended to provide enough information to evaluate the economic analysis, but not necessarily all of the information that would be required to replicate the analysis.
ICER endeavors to follow recommended best practices throughout our evaluations. By following the process outlined above, we hope to make our economic models and associated analyses more transparent and useful to the health care community.
 Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, on behalf of the ISPOR−SMDM Modeling Good Research Practices Task Force. Model transparency and validation: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health 2012;15:843-850.