What makes a high-quality budget impact analysis?

Introduction

Budget impact analysis has quietly become one of the most consequential methods in market access. As health expenditure continues to outpace economic growth, health technology assessment (HTA) bodies in many jurisdictions now expect a budget impact analysis (BIA) alongside a cost-effectiveness analysis before a new technology is funded. Other payers, such as hospitals and local commissioners, also expect to see this type of analysis before they commit to funding a new treatment or intervention. Where cost-effectiveness asks whether a technology offers good value, a BIA asks a blunter question: can this budget holder afford to adopt it, and what will it do to their spending over the next few years?

Which raises an obvious follow-up. If budget impact analyses are shaping real funding decisions, how good are the ones actually being published? And how many achieve the methodological standards set by the ISPOR good practice guidelines?

Case study: budget impact analyses of orphan drugs

In 2021, Khadidja Abdallah, Isabelle Huys, Kathleen Claes and Steven Simoens at KU Leuven published a systematic review in Frontiers in Pharmacology that put the question to the test. They searched PubMed, Embase and the ISPOR conference abstracts, screening 1,960 records and including 90 studies that analysed the budget impact of orphan drugs: 69 examining individual drugs and 21 examining orphan drugs in combination.

Orphan drugs are a demanding test case, and a revealing one. Patient numbers are small, data are scarce and prices are high, so the penalty for methodological shortcuts is severe. If the field’s methods hold up anywhere, they need to hold up here.

Each study was assessed against the ISPOR good practice guidelines for budget impact analysis – the closest thing the discipline has to an agreed standard – across parameters including perspective, target population, data sources, time horizon, scope of costs, assumptions, sensitivity analysis, discounting and validation.

What most studies got right

The picture was not uniformly bleak. Most studies adopted a third-party payer perspective, as ISPOR recommends. Time horizons typically ran from one to five years, with budget impacts reported periodically rather than as a single aggregate figure. Discounting was rarely applied – correctly, since a BIA is meant to reflect the budget holder’s actual expected cash flows rather than a discounted present value.

Where the analyses fell short

The first major gap concerned the eligible population itself. ISPOR recommends modelling an open population, with patients flowing in as they are newly diagnosed and flowing out as they die, recover or discontinue treatment. Among the 69 individual-drug analyses, only 17 (25%) modelled the population dynamically. Thirty-seven (54%) treated it as static and a further 14 (20%) did not report on population dynamics at all. The reviewers acknowledge that a static population can be defensible where the eligible group is very tightly defined, but as a default it runs contrary to good practice.

The second gap concerned assumptions – specifically, where they cluster. The single most assumed input across the review was not cost: it was the size of the target population, assumed in 56 (81%) of studies, ahead of assumptions about the intervention or comparator (47 studies, or 68%) and far ahead of assumptions about costs (just 7 studies, or 10%). In other words, the number these models depend on most heavily is the number most often asserted rather than sourced.

Assumptions are not, in themselves, a flaw. Every analysis requires them, and orphan drug research is genuinely data-poor – the authors describe resorting to assumptions as “disadvantageous but inevitable”. The problem is what happened next, which in most cases was nothing. Thirty-seven (54%) studies reported no sensitivity analysis of any kind, and only one conducted a probabilistic sensitivity analysis. The input carrying the greatest uncertainty was, in the majority of published analyses, never tested.

Validation completed the pattern: 65 studies (94%) made no attempt to corroborate their results against stakeholder opinion, comparable analyses or any other external reference.

There were narrower findings too. Across the review as a whole, only around half of the analyses costed anything beyond the drugs themselves, leaving out administration, adverse events and other condition-related costs that can materially change a budget holder’s true exposure.

A fair reading

It would be easy to read all this as an indictment against the authors of these BIAs. But that would be wrong. Eighty-three per cent of the individual-drug analyses were conference abstracts, a format with little room to report sensitivity analyses even when they were performed. Rare disease data are scarce by definition, and the review found that full-text publications generally did better than abstracts.

The fair criticism is narrower and more useful: not that assumptions were made, but that they were so rarely justified and so rarely tested. That is a discipline problem – and disciplines can be taught.

What good practice looks like

The reviewers’ recommendations track the ISPOR guidelines closely. Model the population as it actually behaves, with influx and efflux over the time horizon. Cost more than the drug. Justify every material assumption and subject each one to sensitivity analysis, so the reader can see how much weight the conclusion places on it. Validate the results – against budget holders, against comparable analyses, or ideally both – and have inputs and formulas checked by a second modeller. And motivate methodological choices explicitly, so the analysis can be appraised rather than merely believed.

Underneath all of that sits one transferable principle. The inputs you are least certain about need to be the main focus of your sensitivity analysis.

Building budget impact models that meet the standard

These are precisely the skills HEOR Institute’s budget impact analysis course is designed to teach: a practical, hands-on course on how to design, build and present budget impact models for health technology assessments, business cases and value propositions, using Excel. Participants work through the full arc of a BIA – framing the analysis, estimating the eligible population, modelling the treatment mix, costing the comparison and testing the result – to the methodological standards required by national and ISPOR good practice guidelines.

Visit the Budget Impact Analysis course page to discover more.