Failing to Figure
Civitas, 18 June 2009
On the Daily Telegraph’s blog, Richard Preston writes about Civitas’ latest publication, Failing to Figure by Mervyn Stone, emeritus professor of statistics at UCL. Half-way through, Preston makes the wry observation:
‘His biggest case study is the immensely complicated, deceitful and deluded means by which the formula was arrived at that determines how much of the NHS budget goes to each of the 152 primary care trusts in England. It’s almost proof of his case that you need to be a professional statistician to follow his argument: the chances of politicians understanding the formula, and therefore assessing whether it was fair, were virtually nil.’
True, it’s complicated. But here’s something of an idiots’ guide (something also that Nigel Hawkes has written about in a typically eloquent way in the British Medical Journal):
• Decisions about what the NHS spends its money on in England is devolved to 152 primary care trust (PCTs); regional ‘commissioners’ of health care that buy health care on behalf of their patient population from numerous providers. PCTs are geographically constrained and are typically co-terminus with local authorities.
• The Department of Health (DH) must decide how much money to allocate each PCT.
• Different PCTs have different levels of need among their populations for health care, so to split funds between them merely per capita would leave many short-changed. Fair enough, and Prof Stone has no qualms with this principle.
• As a result the DH have introduced an extremely complex formula (runs to roughly a page; the explanation to eighteen!) that attempts to decide allocations by weighting for population and numerous proxies for need (call this the target index).
This sets the scene. It is the formula, specifically the target index it uses and how it is derived, that is systematically torn to pieces by Prof Stone’s analysis.
As he shows, the method behind the formula is false and the results it produces extreme (Tower Hamlets PCT, for example, gets a target index exactly twice that of Wokingham PCT… does it have twice the health needs?):
• First, the methodology use is statistically unsound.
A team from Glasgow University used a set of socio-economic indicators to act as a proxy for direct measurement of healthcare need. They reasoned this by using linear modelling (a means of selecting from a large collection of potentially explanatory variables those that may be judged to have a reasonably true relationship). This is deeply flawed.
First, which socio-economic indicators to look at is a subjective judgement. Second, linear modelling can identify interesting relationships, but cannot imply causal inference (i.e. that the proxy socio-economic indicator actually translates to greater health need). In all the variables computed by the Glasgow team there was a large unexplained residual; i.e. plenty of room for computing a different variable with very different financial consequences.
• Second, the DH ignored the Glasgow team’s own concession that the method – particularly regarding the use of morbidity indicators used – had very real limitations.
The team said: ‘Given the time constraints and other aims of the project, our analysis has been essentially exploratory but in view of the promising results obtained we recommend that further work on augmenting the set of needs indices be undertaken.’ (their emphasis)
Whether it was exploratory or not didn’t seem to matter, though. According to Prof Stone’s analysis, the DH essentially took the expressions they wanted. It was a ‘do-it-yourself’ construction that ultimate determined the make-up of our page-long allocation formula.
Does it matter? Yes, hugely. And why? Because, ultimately, we can have little guarantee that PCTs receive funds according to the true health needs of their populations. There is a large, essentially arbitrary, element that’s as much political gerrymandering as proper needs assessment. And as the government tinkers, budgets and organisations are destabilised.
What we really need, as Prof Stone argues, is less reliance on false proxies, more of an open, honest and direct assessment of need. In techie language: ‘Contact with individuals in a project to measure healthcare need from a large stratified random sample of GP patients would sideline the econometric assumptions and pretensions that undermine the present approach’.
The government prefers more of the same, however. Just take a look at this table, an annex to the NHS Operating Framework 2009/10 that introduces the latest version of the formula to ‘target funds at the places with the worst health outcomes’ and ‘a new needs formula which enables need according to age and other factors to be assessed together for the first time’ (p.34). You wouldn’t want to be in Barnsley, Bassetlaw, Leicester City or Wakefield.
As Nigel Hawkes pithily put it in the BMJ, ‘it’s a jungle out there’.