Policy on Inclusion of Grey Literature in Evidence Reviews

ICER is frequently asked by various stakeholders to consider evidence for its reviews beyond that found in formally published, peer-reviewed literature sources. Such evidence, collectively known as “grey” literature, may include conference proceedings and/or abstracts, manufacturer submissions to regulators, technical briefs, and other online reports. Use of the grey literature is commonplace in evidence reviews to identify potential publication or other reporting biases (i.e., studies presented publicly that have not been published). However, explicit synthesis of evidence from grey literature sources alongside data from published studies may be problematic, as there is no guarantee of any adjudication or review of the authenticity of information available in grey literature sources.

In response to these requests, ICER has developed the following policy, to be applied to its work for the California Technology Assessment Forum (CTAF), New England Comparative Effectiveness Public Advisory Council (NE CEPAC), Midwest Comparative Effectiveness Public Advisory Council (MW CEPAC),  and other programs.

  1. ICER’s general policy is to evaluate the grey literature as part of its assessment of the potential for publication or reporting bias, but not to include such sources in its synthesis of the available evidence. Exceptions will be made to this policy under certain circumstances, as below:
    • The evidence base is deemed to be “rapidly evolving” such that grey literature represents a significant portion of the available evidence. For example, a drug or device could be approved by regulators using an accelerated pathway; the review timeline in such a pathway may be shorter than the publication backlog for key clinical studies.
    • Certain outcomes deemed to be of primary interest by clinical experts, ICER’s review panels, or other influential bodies are available only in the grey literature. Examples might include detailed subgroup information from manufacturer submissions to regulators or long-term data on durability of treatment effects beyond the timeframe of key clinical studies.
    • Data from an individual study deemed to be pivotal for ICER’s review is currently available only in the grey literature. A common example is availability of data presented at clinical conferences that also resides in a manuscript currently undergoing peer review. Note that studies that have completed peer review but are not yet published (i.e., “in press”) will be considered on par with published studies, as they have already undergone peer review and any necessary revision.
  2. If any of the above circumstances exist, ICER will provide a rationale for inclusion of grey literature in its review, and explicitly describe the methods of searching, screening, and synthesizing evidence derived from it.
  3. In addition, ICER will only consider evidence from sources with a clearly described and formal submission process, such as conference presentations and manufacturer submissions to regulatory agencies. Technical reports from recognized governmental authorities such as regulators and health technology assessment agencies will also be considered acceptable. Information from unqualified sources such as blog posts, social media interactions, and reports from commercial entities are not eligible for consideration.
  4. If ICER finds the inclusion of grey literature evidence to be appropriate, qualitative findings from grey literature will always be presented separately from data available in peer-reviewed published studies, so stakeholders will clearly understand what has and has not undergone peer review. In some circumstances, it may be necessary to combine findings from grey literature and published sources in any quantitative synthesis (i.e., meta-analysis), however. If such an analysis is performed, sensitivity analyses will be conducted where feasible that limit the meta-analyzed studies to the published literature only.
  5. If data are available from both peer-reviewed publications and grey literature sources, information will always be abstracted from peer-reviewed published studies alone unless one of the exceptions described above is identified.