Statement of ICER’s Commitment to Economic Model Transparency

INTRODUCTION

The Institute for Clinical and Economic Review (ICER) is committed to open and transparent engagement with all stakeholders that have an interest in each of its evidence reviews.  This commitment to transparency extends to the development and/or modification of economic models. Such transparency helps to increase the public’s confidence in model results. Without detailed descriptions of model structure and processes as well as estimates used, economic models run the risk of being considered “black boxes,” with no way to evaluate the validity of model processes or accuracy of model inputs. Explicit delineation of model structure and flow gives stakeholders the ability to evaluate the model’s face validity. Details on the point estimates and ranges used in sensitivity analyses allow for the explicit testing of alternative assumptions and model inputs, provide insight into the drivers of specific results, and allow other interested parties to replicate or extend analyses conducted by ICER and its collaborators.

GENERAL APPROACH

Our general approach to model transparency is based on the Modeling Good Research Practices Task Force report on “Model Transparency and Validation” jointly produced by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision-Making (SMDM).[1]  Our aim is to provide information on the model structure and processes, all major inputs and sources for data, and key assumptions used in our economic analyses, so that readers can judge their confidence in the results while preserving the intellectual property rights of those we collaborate with.

All model documents will note that funding for ICER’s analyses is unrestricted and publicly disclosed.  In addition, ICER develops economic models in collaboration with academic researchers who are free from financial conflicts on any given project.  In addition, ICER maintains a strict conflict-of-interest policy for its own employees, which can be accessed at: http://www.icer-review.org/policy-on-interactions-with-external-partners-and-potential-influences-on-judgment/.

POLICY

For each report, the interventions selected for study will be specified in detail. ICER and its collaborators will provide model documentation, including model structure, comparators, and specifications. When existing models are being used, ICER will provide references to prior publications that provide further details on the model. When new models are developed, this information will be provided as part of the technical report.

Following the publication of a final scope for each topic, ICER and its external collaborators will publish a modeling analysis plan with detailed specifications for the expected conduct of the work.  The plan will be published on a public website used to share collaborative research known as the Open Science Framework (https://osf.io/7awvd/), approximately 15 weeks after a topic is publicly announced.  Stakeholders will be notified when the analysis plan is posted.  The plan may be updated following review of additional data sources, discussions with stakeholders, and other activities, and so is intended to be considered a “living document.” Detailed elements of the analysis plan will include:

    • Analytic objectives
    • Model structure, including a textual and/or graphic depiction of the model structure, process, and outputs
    • Descriptions of interventions and comparators
    • Perspective (generally health care system)
    • Time horizon (generally lifetime)
    • Discount rate
    • Key assumptions to be used in the model
    • Model input values, ranges, and sources of data
    • Other variables crucial to understanding model transition and flow (e.g., risk equations for downstream events)

Sources for model inputs, risk equations, etc. will be provided as part of the documentation. In general, ICER’s analyses will use data sources and information from published or publicly available sources, including peer-reviewed journals, supplementary appendices, briefing documents used by regulatory authorities, and conference proceedings. In specific instances, valid analyses may require the use of unpublished information, such as manufacturers’ data on file. In such circumstances, explicit requests will be made to affected parties, and any reasonable documentation to protect patient and/or stakeholder confidentiality will be provided.  The final version of the modeling analysis plan will be used in conducting the ICER’s “care value” analyses.

Importantly, the modeling analysis plan is intended to provide enough information for an experienced researcher to be able to replicate the economic model and analyses. Actual executable models and associated computer code will not be provided as part of the deliverable, as such an effort would unduly compromise the intellectual property rights of ICER’s external collaborators. As the ISPOR-SMDM Task Force has pointed out, without such protections, “the incentives and resources to build and maintain complex models could disappear.”1

Additionally, ICER and its collaborators will provide a summary of the results of these analyses in a model technical summary. This 10- to 15-page summary will be part of a larger report that ICER will produce that will include information on the available clinical evidence, current guidelines and payer coverage policies, and other relevant topics.  The model summary will consist of the following sections:

    • Methods, including key assumptions and key model inputs
      1. Overview, including description of model structure
      2. Perspective
      3. Patient Population
      4. Costs
      5. Quality of Life/Utility
      6. Primary, Alternative, and Sensitivity Analyses
      7. Budget Impact Analysis
      8. Appendices, including other assumptions and model inputs
    • Results
      1. Primary (Base-Case) Analysis Results
      2. Alternative and Sensitivity Analysis Results, including tornado diagram
      3. Budget Impact Analysis Results
      4. Appendices, including supporting tables/figures summarized in main text
    • Summary and Comment, including limitations and comparison to other published models on the topic of interest

The model results become sections of an ICER-published report on the comparative clinical effectiveness, cost-effectiveness and budget impact of the specific interventions being evaluated. The initial draft report will be posted for a public comment period of four weeks, after which it may be revised.  The revised draft report is then presented as part of a public assessment meeting. The modeling sections of the report are intended to provide enough information to evaluate the economic analysis, but not necessarily all of the information that would be required to replicate the analysis.

Finally, following the public meeting, ICER and its collaborators will publish a detailed technical monograph that represents a transparent description of relevant aspects of the model (also to be posted on the Open Science Framework website: https://osf.io/7awvd/). This technical monograph will provide details of the economic model in a more structured, comprehensive format, using a standard template. The monograph will be posted no later than 6 months after the public meeting.

Detailed monograph components to be included are listed below:

  • Background and purpose
  • Model overview
  • Model type and structure, including a textual and/or graphic depiction of the modeling process and outputs
  • Health states and possible patient flow
  • Descriptions of interventions and comparators
  • Perspective (generally health care system)
  • Time horizon (generally lifetime)
  • Discount rate
  • Model assumptions and rationale
  • Input parameters and risk equations, with sources for each
  • Types of model outputs
  • Results from all relevant base case and sensitivity analyses
  • Tables and graphs representing model results
  • Relevant citations (published and unpublished)

ICER endeavors to follow recommended best practices throughout our evaluations. By following the process outlined above, we hope to make our economic models and associated analyses more transparent and useful to the health care community.

 

 

[1] Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, on behalf of the ISPOR−SMDM Modeling Good Research Practices Task Force. Model transparency and validation: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health 2012;15:843-850.