medConfidential spoke about the Framework for Data Processing by Government at the All Party Parliamentary Group on the Rule of Law. The topic of the APPG provides a useful perspective for much work on data in the public sector, and the wider use of AI by anyone. The meeting was on the same day as the launch of the AI Select Committee Report, which addresses similar key issues of  ‘data ethics’.

The ‘Rule of Law’ is defined in 8 principles as identified by Lord Bingham. The principles are not themselves law, but rather describe the process that must be followed for the Rule of Law to be respected.

Public bodies must already follow that process, and also be able to show how that process has been followed. As a result, those developing AIs (and data processing tools) for use by public bodies must also show how these processes have been followed. This is necessary to satisfy the lawful obligations of the bodies to which they are trying to sell services.

The principles identified by Lord Bingham are a model for testing whether an explanation of an AI and its output, or a data model, is sufficient for use by a public body.

While debates on ethics and society, and on politics and policy, focus on whether a technology should be used – the Rule of Law is about the evidence for and integrity of that debate. As Departments implement the Framework for data processing, to deliver on their obligations under the Rule of Law, it must be compliant with the Principles identified by Lord Bingham – not just the ethics and policies of the Minister in charge that day.

Public bodies are already bound by these rules – unless Parliament legislates to escape them. The principles are widely understood, they are testable, and they are implementable in a meaningful way by all necessary parties, with significant expertise available to aid understanding.

Companies and other non-public bodies

Companies (i.e. non-public bodies) are not subject to the same legal framework as public bodies. A Public Body must be able to cite in law the powers it uses; a Private Body may do (almost) anything that is not prohibited by law. This is why facebook’s terms and conditions are so vague and let it get away with almost anything – such a data model does not apply to the tax office.

Some of those looking to make money – to “move fast and break things” – would like the standard to be ethics, and ethics alone. There are currently many groups and centres having money poured into them, with names involving ‘data and society’, ‘ethics and society’, and DCMS’s own ‘Centre for Data Ethics’. The latter is led by a Minister in a Government that will always have political priorities, and – given recent revelations about Facebook – the consequences of incentives to lower standards should be very clear.

Ethics may contribute to whether something should be done – but they are not binding on how it is done, and they offer no actual accountability. After all, no tyrant ever failed to justify their actions; it is the rule of law that ultimately holds them accountable, and leads to justice for those harmed. Ethics alone do not suffice, as facebook and others have recently shown.

There is a great deal more work to do in this area. But unlike other AI ‘ethics’ standards which seek to create something so weak no-one opposes it, the existing standards and conventions of the Rule of Law are well known and well understood, and provide real and meaningful scrutiny of decisions – assuming an entity believes in the Rule of Law.

The question to companies and public bodies alike is therefore simple: Do you believe in the Rule of Law?

[notes from APPG talk]
[medConfidential (updated) portion of the APPG briefing]

The Health Select Committee has published a report on data sharing which “raises serious concerns about NHS Digital’s ability to protect patient data” under the headline “NHS Digital failing to uphold patient interest”.  The Home Office is “treating GP patient data like the Yellow Pages” according to the RCGP.

The NHS has been trying to rebuild trustworthiness around data since the last big NHS data project collapsed in 2014. This report shows that all promises can be undermined by the narrow minded view of one office in Whitehall

The Health Select Committee is clear that NHS Digital has again failed in its statutory duties, and has put patients at risk by the processes it has adopted and refuses to change.

HSCIC rebranded into NHS Digital in an attempt to avoid the history of past failures, but this report shows actions are unchanged…

We submitted written evidence to the inquiry.

Last September, a company which helps institutions understand data started a new project. What their client wanted, was to tell whether one category of videos could be distinguished from some others. The project was successful on a test dataset, and they produced a demo. The very happy client forwarded this to their boss, who sent it to their boss, and so on, until then the Home Secretary went on TV to say that the Home Office had better technology than Google for blocking ISIS videos. At no point, was there a need to test or explain whether the demo worked beyond the test data. That seems to be the standard for AI – is data processing a place where the rule of law doesn’t matter?

The Department of Health also launched guidance on “decision making” AIs for use in the NHS. Innovations have always come to healthcare and spread based on understanding – there is no need to throw all past lessons out because a PR budget gets spent. Separately, the “Malicious AI report” is worth reading – but already feels dated as the risks are both real and timely, and political imperatives are rarely spun as malicious.

Given the option for a quick headline and a cheap political point, politicians will choose to score it. With digital systems of any kind, there is a temptation to take a shortcut and claim victory.  The Home Office claimed to have an AI which did what it wanted – by ignoring any real world requirements that made things harder. This is not the greatest of precedents for public bodies using AI tools to make decisions, especially on groups who do not command easy political support.

“Explainability” is just putting in the extra time to test models and understand how they work – rather than selling the first thing that seems to meet the goal. That faster approach may have short term financial benefits, but it can be more widely toxic as an outcome generally best avoided. The Home Office can make this claim for this AI, as it as the first Department to do so; the next claim will be treated with the greater scepticism that it deserves. We’ve put the Home Office statements into the Government’s ethics framework – which again shows the failures of that framework.

‘Trustworthy technology’ needs to address systemic harms. The first mover advantage on AI will go to those with the least moral concerns about real world effectiveness, until there is a clear reputational harm for continuing to work on systems which are known to be damaging. This is why the ‘most successful’ public sector AI project in the UK is from the Home Office – harms to others are something they have never bothered to avoid.

What started out as a legitimate technology project – can AI help identify ISIS videos? –  demonstrating potential, was spun as something else. Had explainability been a prerequisite of that project as being considered a success as it was claimed, (rather than simply a stage of a trial). Where an entity refuses to follow such processes, as in the drug discovery arena, reputable actors should simply refuse to deal with them as that should be one of the requirements of being seen as a reputable actor.  The Partnership on AI was supposed to consider how to address such issues – but companies outside the partnership aren’t bound by their rules… But many of the staff of those outside would not wish to be barred from working there due to other associations (there, of course, must be a way to demonstrate lessons have been learnt)…

The AI guidance from the NHS contains a checklist, written by “industry” and the Wellcome Trust, which is so vague it barely addresses previous problems, let alone handling future questions. There are no considerations of the principles of ‘trustworthy technology’ by developers, nor any references to equivalent protocols for decision making AIs as we have for determining doctors are trained or new medicines are safe. Claiming you have a phase 0 success is one thing (whether a drug or AI), claiming you have a phase 3 success is quite another – and so it should be with machine learning tools that need to be explained.

Many of the greatest failures of the Home Office are due to technical ineptitude. While their policy can not correctly distinguish an arse from an elbow, technology has moved on sufficiently to do it for them, letting Marsham Street ignore the details while delivering the opposite of human flourishing.

Does HMG wish to be permanently excluded from buying from working with partnership members because it chased a cheap headline?  Does the partnership have the willingness to ensure members deliver “responsible” AI? It is the public headlines and narrative that matters, and the biggest headline about AI in Government is that of the Home Office wanting to choose content purely based on a single suprious claim. Government acts as a single customer to their suppliers; and the reverse must be true for AI & ethics.

Data Protection and Datasets of National Significance

Second reading of the Data Protection Bill is in a week – and Government has still not explained the effects of their proposals to centralise data policy in the periphery of Whitehall. As DCMS struggle with a politically led centre for AI data and ethics, announcements like the one from the Home Office will grow. Not because they have solved any problems, but they have done something which redefines the problem as sufficient for the box to get ticked, political claims to be made, and someone else to pick up the pieces. The Home Office does not care about DCMS politics or policy, but which way will Google DeepMind be lobbying DCMS on this?

Lord Mitchell amended the Data Protection Bill to require public bodies estimate the worth of their “datasets of national significance”. Lord Darzi is thinking along similar lines about a new deal with patients. While both good and worthy initiatives that are deserving of time, there is a risk other policies will make them irrelevant.

Lord Mitchell’s amendment mandates an assessment that should be written down, but under current rules, what NHS England or any public body will be forced to write by HM Treasury is that giving data to private companies that employ UK staff, will create new tax revenues from those staff (since company profits go offshore). One NHS trust working with Google might create a nice deal for themselves from some data – but the rest of the NHS will still have to pay a much higher rate.

What will happen when the public understand that this is how their data gets used, and where the money goes?

Even if the Government take the clause out of the Data Protection Bill, whether UK data should be flowing to tax havens is likely an increasingly important question for public debate.  This question is not going away – NHS Digital already do some checks that they’re not dealing with an empty shell company (PHE’s only meaningful step is to check that the fees are in their bank account). Does Government wish to ignore an issue that will resonate with the public on data, or leave in place the small and sensible steps Lord Mitchell added to the Bill?

Enclosed:

• Home Office Data Science Ethics Framework form for their project
• Google DeepMind / RFH and the Wellcome Trust AI checklist
• Other comments on the AI guidance

NHS Digital has published some information on the new “National Data Opt Out Programme”. The information for patients is still missing entirely, and everything else is subject to change.

The new model

The new consent model has the opportunity to do good – and it appears far clearer than the old. But if the loopholes and failures of the past continue to be included in the new model, it will fail in terms of public confidence, just like care.data.

We continue to be promised that every patient who has opted out will get a letter describing the new arrangements, although patients who didn’t opt out may be left in the dark about how their data will be used.

As Dame Fiona Caldicott was asked to create, the new opt out will be a simple question:

“You have the right to opt out of your personal confidential information being used for [these other] purposes beyond your direct care” (Deck A)

While we feared there may be new non-statutory ‘special purposes’ that override the opt out created, there seem to be none. While NHS England, and Public Health England wanted their desire to use data to be more important than any patient’s ability to dissent – the slides currently say that the Department of Health disagrees with them. This is good.

Data ‘anonymised in line with the ICO’s anonymisation Code of Practice’ will be excluded from the opt out, but we’ll come onto that later. The embedding of the list of exclusions in the Deck A list (as above) is probably just going to cause problems for all sides in future years – the language has to work irrespective of how the NHS reorganises next week, and we hope the words in square brackets above are not in the language that patients see.

It was the interactions and nuances between what the public were told, and what public bodies wanted to actually do in practice, which led to the collapses and repeated failure to restart care.data, and which led to its ultimate demise. DH hopes that avoiding sharing details too early will mean no one spots problems until it is too late. That is a theory unlikely to survive contact with the real world – which it has not yet attempted.

A trustworthiness question from medConfidential

What should the response be when a public authority is found to have misled the public about how it used data? What happens when the release register is agreed to be wrong, in a way which is demonstrably misleading? What should a trustworthy organisation do?

The remainder of this blog post is a slightly technical look at various specifics. When the patient information is published, we will write a patient perspective on where things are.

NHS England has started by issuing a tender for suppliers to ACOs and STPs. That doesn’t seem like the ideal place to start.

The loophole that remains

Data that is “anonymised in line with the ICO code” remains outside the dissent from ‘information being used for purposes beyond direct care’.

This is purely a political choice by the Department – while they may feel they have no requirement to cover it, the opt out is a gift of the Secretary of State, and he may decide what it covers, and it is only a minor implementation change to go from one to the other.

Previously, the formulation for the opt out was ‘dissent from data leaving the HSCIC” (or GP) – the focus was on data as it left an institution. And in that, it was possible to argue that it was no longer personal data when it was moving (emphasis on possible).

Now, the formulation is about “use” of data for “purposes beyond direct care”, and is undoubtedly clearer in many ways, but it looks at ‘use’. But someone is going to have to justify to the public how ‘anonymisation’ is a direct care purpose. Less than a week after DH told the Health Select Committee that it believes patients have no privacy right over non-clinical data held by NHS Digital, this looks to be a fundamental flaw, resulting from a fundamental change.

Moving to the formulation to be about use also reduces the scope for the purposes of the “promotion of health”. DH has likely missed (again) the opportunity to explain to patients whether or not “promotion of health” can include promoting McDonald’s salads or alternate tobacco products.

There are clearly cases where 100% of data is needed; but the vast majority of research projects would come to exactly the same answer if they used data for which dissent had been honoured. The Confidentiality Advisory Group was placed on a statutory footing and was deliberately given the ability to give advice to NHS Digital on these questions – but the Government has never got around to even starting that process.

No data recipient is currently asked to justify why receiving 98% of patient data is insufficient – data recipients are not asked whether they wish to receive data on people who dissent from their data being used for purposes beyond direct care. Many academic research projects outsource some of their ethical obligations to the data provider – who in this case, will be ignoring them. Again, the obligations change because of a new focus on ‘use’, and it is unlikely that NHS Digital (and PHE) not asking that question will simply allow ethical researchers to ignore it.

The Department of Health ignoring a problem just means everyone else has to deal with it. That has also not gone well elsewhere (for everyone else, DH may disagree).

The Disease Registries (including cancer)

According to the slide decks, the new opt out will not apply to the cancer registry or other data held by PHE (although after last week, it is expected that this should change – and that has consequences for the system).

If the new opt out model does cover PHE datasets like the cancer registry, this will need to be clearly explained in the letter that goes to patients.

Either way, the ‘cancer charities’ should explain (in a fair way) both the benefits of the new consent model, and the choices that it offers cancer patients (and also what those choices will not protect their community against).

Unlike some other health conditions, those who fundraise for the cancer charities often end up being encouraged to publish sensitive personal data about their diagnosis (such as the type, and date), when fundraising for charities. Where those fields are released unprotected, (whether opt outs are disregarded or not), that may change the institutional conceit around inadequate protection. (But this probably won’t happen until after something goes very badly for a someone to whom they owe a duty of care…).

If the text in the slide decks remains accurate, and there are two divergent dissent systems, it is unlikely that the cancer registry could ever integrate into the NHS and remain a backwater of unlinked and idiosyncratic data sources. That is not a good outcome for those who wish to see a cure for cancer.

We have no information that isn’t public or above – so it may be we’ve misunderstood things. This has happened before. Feel free to get in touch if your reading is different to ours ..

Update (16/1): PHE consider tobacco companies to be doing ‘medical research’

The Daily Telegraph reports today (15/1/2018) that Public Health England gave the medical histories of many cancer patients to a company for a “trends in … lung cancer” study – a company that works for a tobacco company, while that tobacco company was taking the Department of Health to court over “plain packaging”.

PHE believe NHS rules should not apply to them – this includes other opt outs or rules on data handling.

We expect more details to emerge in coming days, as the entries in the PHE release register are scrutinised, and many more companies that don’t appear on the register come to light. (if you’d like to help/watch, there’s a communal google doc for the PHE register here, plus a second one for all NHS data releases)

Press quote: Sam Smith, a coordinator at medConfidential said:

“This is a system that relies on public trust; a system so flawed that it’s not yet clear whether PHE broke any rules.

“The release of this data relies upon loopholes in the Data Protection Act – and unless Public Health England satisfied each technicality they’re arguing about, then they may have breached 3 different laws by giving sensitive personal data on the treatment of cancer patients to a US company that works for a tobacco firm.

“But even if tighter NHS rules had been followed, the Department of Health still argue any patient opt outs wouldn’t have applied in this case. What are patients supposed to do?

“It is vital for public trust that uses of patient level data for purposes beyond direct are covered by the new opt out.

medConfidential supports an NHS cancer registry that follows all NHS data rules, and where the new opt out model applies to all patient level data for purposes beyond direct care. Every data flow in the NHS should be consensual, safe, and transparent, and that includes a well run cancer registry inside the NHS.

If you may be affected, what can you do?

If you are a patient with cancer do not make any treatment changes based on this news; but if you have any questions, talk to your nurses or support structures.

If you may be affected (ie you have had a cancer diagnosis), a request to the cancer registry to opt out will result in your data being deleted, which they admit will potentially harm your direct care in the future. We can not recommend you do this, but it is the only current option. Given the situation, we suggest you write to your MP, and ask them to ask the Department of Health what they are doing to fix this. 

You may wish to:

• Say why/how this affects you or your family – in as much/little detail as you wish.
• Ask your MP to, on your behalf, ask the Department of Health why your only choices are to have your cancer diagnosis given to companies working for tobacco firms, or to have your treatment history be deleted affecting the future care of you or your family. No part of that choice is appropriate.
• In short, why does data taken from the NHS not have to follow NHS rules?

PHE’s past failures are what they are – there are limited things that can be done to fix them. However, they can be brought in line with the NHS, and catch up to all the improvements that the HSCIC/NHSDigital has made since 2014 (although this failure could still have happened to them, and opt outs would still not have been respected)

Please also join our mailing list for additional information as this evolves over the next few weeks:

As the House of Lords Select Committee on AI looks at health data, it is only 2 weeks shy of the 20th anniversary the 1997 ‘Caldicott Review’. In retrospect, it understood the world that was coming then, and the review still holds up well for the future that is still coming now. It said:

“Increasing adherence to the principles will reassure patients and those treating them that confidentiality is safeguarded. Such progress should be monitored and appropriately identified, and individuals held to account wherever patient-identifiable data is present in the Service. We believe that the principles outlined here should also be applied to information identifiable to individual patients concerned with their clinical care, and medical research. It is clear that patients expect nothing less.”

20 years on, patients still expect nothing less.

The Review could have said one other thing – that data hygiene should have been treated as a part of clinical hygiene, and thereby integrating information into clinical governance. It would have avoided a great deal of problems over the last few years, but may also have made the the original creation of Caldicott Guardians effectively impossible. It is a step operationalised by Caldicott 3. While it could have led to a very much more Digital NHS today, it is all too easy to forget that hospital hygiene, under the oversight of clinical governance, has had problems from failed incentives around outsourcing, infected by the profit motive.

Today, the demand from profit seeking technical startups is even greater, the desire to skirt the rules intense, and modern startups push a “lobbyist viable product” onto a cash strapped NHS – selling patients’ data as an asset when a parent company the patient may not have heard of inevitably gets bought.

Whatsapp’s sharing of phone numbers to facebook would breach the rules around patient confidentiality; but then lobbyists went to see NHS England, who changed the rules and put the burden on each clinician to choose…

Whatsapp still shares its data to facebook to offer it’s “patients you may know” feature, and shadow profiles, and the other creepiness. Any officially sanctioned messenger could never do this, but NHS England doesn’t care – it has taken a problem off its desk for free, and dumped it onto every clinician in the country.

As NHS Digital takes egregious actions in the name of “burden reduction”, NHS England increases a burden yet further, because it’s reduces the burden on them from lobbyists wanting a change.

Messaging in the NHS remains a problem unsolved at scale – so the lobbyists now swarm offering their solutions to similar but different problems. NHS England, not running any hospitals, caved. Pagers work because they are used only for one thing – when they go ding, the doctor is needed for something at a level of urgency that has been triaged; but also the doctor can ignore it while dealing with something else. Doctor judgement is supreme over the tools – the pager is a busy doctor’s bullshit blocker – ignore that feature at your peril.  End-to-end properly encrypted messaging is not hard, but the easiest of the tasks.

If NHS England commissioned a messaging system, institutionally it would abuse it the same way it does email, and destroy any benefits due to its own worst institutional micromanaging impulses – a problem non-existent in the pager world. But those who would change the system would include changes that distract clinicians, so the status quo continues. Such perverse incentives were well understood by Dame Fiona Caldicott when she was writing 20 years ago.

As a result, Caldicott Reviews 1, 2 and 3, are all still relevant. The history is still relevant for designing consent, and designing out unethical and harmful behaviours. That NHSE abdicates any political responsibility does not mean NHS Digital may do so when designing technical systems.

For example, the “GP at hand” service, a rebadged “Babylon Health” product invested in by DeepMind’s founders, says it has the right to sell the medical records it holds as an asset of the company when babylon get bought (which it must in order to pay back the investors). This is a model that Caldicott 1 and GMC/BMA guidance has previously made clear is not appropriate. But as with the dodgy deal with the Royal Free, it is the NHS institutions left holding the bag as the company takes what it wants. Why did NHS England approve that service with those conditions?

Companies will change their rules for profit, and use their public relations machines to argue the NHS “harms patients” by walking away from a disturbing deal the companies only offer on a take it or leave it basis.

Such a business model may be fine for a profit focussed business with no sense of public purpose or accountability to anything beyond their bottom line, but the AI companies claim a higher standard… but also, they don’t:

Q52 Lord Swinfen: In your view, do investors have a duty to ensure that artificial intelligence is developed in an ethical and responsible way? If so, how should they do this? Should such development be regulated?

Eileen Burbidge: I thought this was an incredibly insightful question when I saw it on the papers for the session. The stark and objective answer, strictly speaking, is that I do not think investors have a duty to ensure ethical and moral behaviour. Most investors sign up to a code of conduct and are authorised persons by the FCA because they have fiduciary responsibilities as a first and foremost point. That is simply the objective current situation.
…
To be quite clear, our obligation to our investors is to generate as strong a financial difference as possible.
…
It will be more social pressure and market pressure.

If the companies also refuse any social or market pressure from the NHS, then they have learnt no lessons at all.

It was with that insight that Caldicott 1 laid down strong and sustainable guidance for handling of patient data. Every failure of the last 20 years has come because that guidance was ignored.

Dame Fiona Caldicott was precient in her first report. That first report from 1997 can be applied to AI and genomics – 2 ideas that would have been almost inconceivable when it was written – and we are confident it will apply to whatever comes after AI and genomics.

Let us also hope that there doesn’t need to be a fourth part in this trilogy…