Any Data Lake will fail; there is an alternative

We’ve added some new words to our front page.

Any attempt to solve problems of records following patients along a care pathway that involves putting all those records into a big pile, will either fail – or first breach the Hippocratic Oath, and then fail.

A Data Lake does not satisfy the need for doctors to reassure their patients (e.g. false positive tests), does not satisfy the need for doctors to hold information confidentially from others (e.g. in the case of Gillick competency, or on the request of a patient), or when institutions cannot tell doctors relevant details, e.g. in situations where there is “too much data, but no clear information”.

From the NHS’ national perspective, micromanagers at NHS England will get to reach into any consultation room and read the notes – especially in the most controversial cases. They might be trying to help, and while members of Jeremy Hunt’s Office itself might not reach in (to be fair, they probably wouldn’t), do you believe the culture at NHS England is such that some NHS middle-manager wouldn’t think that is what they were expected to do, urgently, under the pressure of a crisis?

This is also why any ‘blockchain approach’ to health (specifically) will fail. Such technologies don’t satisfy the clinical and moral need to be opaque – deniability is not a user need of your bank statement.

Just as every civil servant recognises aspects of Sir Humphrey in their colleagues, it is the eternal hope of the administrator – however skilled, and especially when more so – that if a complex system worked just as they think it should, everything would be eternally perfect.

Such a belief, whether held by NHS England, DH, or the Cabinet Office is demonstrable folly. If you build a better mousetrap, the system will evolve a better mouse; everything degrades over time.

It was a President of the Royal Statistical Society who talked about “eternal vigilance”. This is why, and it also provides the solution.

As we’ve outlined before, the alternate approach to a leaky Data Lake is to add accountability to the flow of data along a care pathway.

The system already measures how many patients are at each stage, and their physical transfers; it should give the same scrutiny to measuring how many records follow electronically. Where the patient goes, but their data doesn’t, should be as clear to patients as statistics on clinical outcomes – because access to accurate data is necessary for good clinical outcomes.

Interoperability of systems, in a manner that is monitored, is already being delivered by care providers up and down the country. Creating lakes of records is simply an administrator’s distraction from what we already know works for better care.


medConfidential takes donations