Slides – Emma McManus – Open Research Conference 2024

Slides used by Emma McManus for the University of Manchester Open Research Conference 2024

Title: Exploring the replicability of health economic decision models

Abstract: Health economic decision models are an analytic methodology that account for events over time and across populations to assess the cost-effectiveness of health interventions. Their results are used to inform policy and determine whether interventions are made available within the NHS.

These models are often termed ‘black boxes’ and there have been calls for heightened transparency in their reporting. They are also vulnerable to manipulation, as researchers have many freedoms in their development.

This study aimed to explore the current reporting transparency of decision models by attempting to exactly replicate five published models. In doing so, it also sought to identify common barriers and facilitators to replication and assess how well a commonly used reporting checklist for decision models was at determining replicability.

Each replication was conducted by a different replicator, using information from the manuscript and supplementary materials. The replicator attempted to reproduce the key results detailed in the paper.

The results of the replications showed that no model was completely replicable, with replicated costs often varying more than health outcomes. Across the case studies, the variation between original and replicated results ranged from −4.54% to 108.00% for costs and −3.81% to 0.40% for outcomes. It also found that a reporting checklist for decision models was a poor indicator of model replicability, with all of the studies seeming to fulfil most criteria. Common barriers and facilitators to replication were also identified, such as conflicting information being presented and not all parameters being reported.

This study demonstrated that although models may appear to be comprehensively reported, it is often not enough to facilitate a precise replication. Further work is needed to understand how to improve the transparency and reporting of models to ensure their replicability and to explore incentives for researchers to engage in such practices.