Tag Archives: DeepMindRFH

More DeepMind secrecy – What the lawyers didn’t look at

The Royal Free has been recommended by ‘independent’ lawyers to terminate its ‘Memorandum of Understanding’ with DeepMind (page 68, second bullet from bottom)

If the “research” agreement with DeepMind – the MoU covering “the use of AI to develop better algorithms” – isn’t terminated, the deliberate exclusions from the legal opinion can only be interpreted as an attempt to mislead the public, once again.

What is the legal basis for continuing to copy 8 years of data on every patient in the hospital? While DeepMind claims the “vital interest” of patients, it still keeps the data of over a million past patients whose interests it will never serve, because RFH’s systems cannot provide “live data” (para 26.1) – despite the report saying that is only temporary (para 15.1).

When RFH completes its move to “fully digital”, will the excessive data be deleted?

The biggest question raised by the Information Commissioner and the National Data Guardian appears to be missing – instead, the report excludes a “historical review of issues arising prior to the date of our appointment” (page 9, para 8.4, 5th bullet, and page 17, para 5,bullet 7).

The report claims the ‘vital interests’ (i.e. remaining alive) of patients is justification to protect against an “event [that] might only occur in the future or not occur at all” (page 43, para 23.2). The only ‘vital interest’ protected here is Google’s, and its desire to hoard medical records it was told were unlawfully collected. The vital interests of a hypothetical patient are not vital interests of an actual data subject (and the GDPR tests are demonstrably unmet).

The ICO and NDG asked the Royal Free to justify the collection of 1.6 million patient records, and this legal opinion explicitly provides no answer to that question (page 75, para 5, final bullet).

The lawyers do say (page 23, para 12.1) “…we do not think the concepts underpinning Streams are particularly ground-breaking.” In Streams, DeepMind has built little more than a user-friendly iPhone app – under scrutiny, its repeated claims of innovation are at best misleading.

But Google DeepMind clearly still thinks it is above the law; it tries to defend all of the data it has by pointing at different justifications each time. Is this the ‘ethical’ ‘accountable’ approach we must accept from the company that wants to build dangerous AIs?

-ends-

Background to the long running saga.

Google now tries to blames Doctors and Snapchat for its unlawful behaviour

Responding to Google’s claims that doctors “use” Snapchat to send photos for a second opinion, coordinator of medConfidential Phil Booth said: “Had Google managed to buy Snapchat, they wouldn’t have said anything about this. The Report blames doctors for hygiene, and the hospital for it’s IT systems, everyone but Google. Now they’re blaming doctors for their choice of secure messaging apps to care for patients with whom they have a direct care relationship – something Google clearly fails to understand.”

If the assertions are based on evidence acquired in the Review, that should have been reported to CQC – unless there was a see no wrong, hear no wrong policy in place. Google provided no evidence that Doctors actually do this, just that they could install an app. They could also use any google messaging tool (except no one uses any of them). We fully expect DeepMind will “surprisingly” come out with a messaging app for doctors, which will be no better than email, and so solve none of the widely understood problems that mean fax machines are still useful. 

Doctors are responsible for safely caring for their patients, and it’s up to them which safe and lawful tool to use. The only reason DeepMind care is they have an tool to sell; and they’re still in denial that they way they built it was unlawful.

We’re mostly surprised that Google didn’t use this to kick Facebook; but perhaps they didn’t want to criticise another member of the Partnership on AI…

Original press release here: https://medconfidential.org/2017/medconfidential-initial-comment-on-the-google-deepmind-independent-reviewers-report/

medConfidential initial comment on the Google DeepMind Independent Reviewers’ report

UPDATE 2pm: responding to Google’s claims that doctors use secure messaging to send photos, Phil Booth said: “Had Google managed to buy Snapchat, they wouldn’t have said anything about it. The report blames doctors for hygiene, and the hospital for it’s IT systems. Now they’re blaming doctors for their choice of secure messaging apps to care for patients with whom they have a direct care relationship.”

Doctors care for their patients, and it’s up to them which safe and lawful tool to use. The only reason DeepMind care is they have an tool to sell; and they’re still in denial that they way they built it was unlawful.


The report answers none of the obvious questions that a supposedly independent Review of unlawful data copying should have answered.  

The ICO confirmed on Monday that DeepMind Health’s deal with the Royal Free had broken the Data Protection Act in at least 4 ways [1], and they have been given weeks to fix it. There is now a formal undertaking in place for correction of their project’s ongoing breaches of the Data Protection Act [2]. As of this week, DeepMind remains in clear breach of UK privacy laws. (page 7)

The National Data Guardian’s letter, referred to by the Review, shows clearly that DeepMind were aware of the unlawful nature of their processing last December[3] and the Review suggests they chose to do nothing about it.

In addressing “law, regulation and data governance”, the Reviewers say “We believe that there must be a mechanism that allows effective testing without compromising confidential patient information” (page 9, right column). So many people agree that there are already such processes – DeepMind just didn’t use any of them. It is unclear why the “Independent Reviewers” feel this is anyone but Google’s problem. (Here’s the sandbox for Cerner – which the Royal Free uses.)

If, as Prof John Naughton analogises, the Royal Free’s response to the ICO decision was “like a burglar claiming credit for cooperating with the cops and expressing gratitude for their advice on how to break-and-enter legally”, this report is DeepMind saying “It wasn’t me! Ask my mum…” thinking that’s an alibi.

DeepMind accepts no reponsibility [4], and its Reviewers seem happy with that.  Which, given DeepMind’s broad AI ambitions, should frankly be terrifying…

Responding to the Review, medConfidential Coordinator Phil Booth said:

“If Page 7 (right column) is accurate in its description of record handling at the Royal Free, then CQC must conduct an urgent inspection of data hygiene at the hospital; or was this just “independent” hyperbole to make Google look good?”

“The Reviewer’s way to not criticise DeepMind is to avoid looking at all the things where DeepMind did anything wrong. The Reviewers may think “this is fine”, but anyone outside the Google bunker can see that something has gone catastrophically wrong with this project.”

“Google DeepMind continues to receive excessive amounts of data in breach of four principles of the Data Protection Act, and the Independent Reviewers didn’t think this worth a mention. DeepMind did something solely because they thought it might be a good idea, ignorant of the law, and are now incapable of admitting that this project has unresolvable flaws. The ICO has forced both parties to fix them within weeks having ignored them for approaching 2 years.

“DeepMind Health needs real senior management with a experience of caring for patients, i.e. a Regulated Medical Professional, as a Chief Medical Officer. The second paragraph on the inside front cover (which isn’t even a numbered page in the printed document, but page 2 in the PDF) shows how badly they have failed from the start.”

For further information or for immediate or future interview, please contact Phil Booth, coordinator of medConfidential, on 07974 230 839 or coordinator@medconfidential.org

 

Notes to editors:

  1. Information Commissioner’s Office summary of their finding https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/2017/07/royal-free-google-deepmind-trial-failed-to-comply-with-data-protection-law/
  2. The ICO requires that the Royal Free and DeepMind take actions within a month of the undertaking issuance – page 7. https://ico.org.uk/media/action-weve-taken/undertakings/2014352/royal-free-undertaking-03072017.pdfMany of these issues were highlighted to DeepMind by MedConfidential last year, and which they have repeatedly and systemically ignored.
  3. Sky News reported in May that the unlawful nature of the DeepMind data processing was first formally brought to the Royal Free & DeepMind’s attention in December 2016 by the National Data Guardian. http://news.sky.com/story/google-received-16-million-nhs-patients-data-on-an-inappropriate-legal-basis-10879142 Paragraph 4 of the letter from the National Data Guardian to the Hospital clearly shows that they were first formally of their legal failings in December.
  4. Details of medConfidential’s complaint are available here:
  5. This complaint has now been vindicated by the investigation, despite an extremely strong PR response from Google. Contemporary quotes from project advocates, which now ring hollow, include: [all emphasis added]a) Mustafa Suleyman, Co-Founder at DeepMind, has said:

    i) “As Googlers, we have the very best privacy and secure infrastructure for managing the most sensitive data in the world. That’s something we’re able to draw upon as we’re such a core part of Google.” [Guardian, 6/5/16]
    ii) “We have, and will always, hold ourselves to the highest possible standards of patient data protection.” [Daily Mail, 4/5/16]
    iii) How this came about all started with Dr Chris Laing, of the Royal Free Hospital: “We went for coffee and ended up chatting for four hours.” [BBC News Online, 19/7/16]
    iv) More recently, in an interview with Mr Suleyman published on 20/3/17: “When pushed on how the public would be assured that its sensitive data was safe, Suleyman replied, “first there is the law”.” [Digital Health, 20/3/17]

    b) George Freeman MP, at the time a Minister in the Department of Health: “NHS patients need to know their data will be secure and not be sold or used inappropriately, which is why we have introduced tough new measures to ensure patient confidentiality.” [Daily Mail, 4/5/16]

    c) Professor Hugh Montgomery, (consultant for Google’s DeepMind project) said, on Radio 4’s PM programme on 4 May 2016:

    i) “So this is standard business as usual. In this case, it was a standard information data sharing agreement with another supplier, which meets all of those levels of governance. In fact, the agreement there, or the standards of management of those data, meets the very very highest levels. It meets something called HSCIC level 3, which most hospitals trusts don’t even reach.” [Recording of audio available, see link below]
    ii) “So firstly, this isn’t research. Research is governed by an entirely separate process that would require anonymisation of data and all sorts. This is data processing.”
    iii) “It’s fair to say again that not only is this data at the very highest standards, and beats every standard, and more in the United Kingdom. But the data is encrypted end-to-end, and they have to, like everyone else in the health service, stick to the law.”
    iv) Recording of audio available at: https://www.dropbox.com/s/cfimojgec24rlrj/
    20160504­deepmind­radio4­pm.mp3?dl=1
    20160504­deepmind­radio4­pm.mp3?dl=1

    d) Will Cavendish, now Strategy Lead for DeepMind Applied, formerly Informatics Accountable Officer at the Department of Health, said (when IAO):

    …“The vital importance of trust, security, and cyber security.” … “To be honest, it used to be that not a week goes by, now it’s not a day goes by, without stories of hacking, data leaks, inadvertent data sharing. This absolutely erodes the trust that underpins the work that we do.” https://www.youtube.com/watch?v=5Ej3PRF1jUw&t=2h15m5s

    e) Dr Julian Huppert, Chair and “on behalf of the Panel of Independent Reviewers for Google DeepMind Health” said in an e-mail to medConfidential on 6/7/16:

    i) “one of our roles is to look in detail at how DeepMind Health uses patient data, and to confirm that it complies with the highest ethical and regulatory standards.”
    ii) “We believe from what we have seen so far that DeepMind has a clear commitment to the Caldicott Principles, and that they have to date been honest in their public and private comments. We also believe they are willing to work constructively with regulators, and remain within the law.

     

  6. DeepMind’s response to the ICO finding has been to blame everyone but themselves. As they begin to regularly refresh part of their Review board, perhaps Shaun Spicer will be available to help.

 

-ends-

[PRESS RELEASE] Google DeepMind deal with the Royal Free Hospital broke the law

The Information Commissioner’s Office has today ruled that the deals which gave Google DeepMind copies of 1.6 million patients’ hospital records are unlawful:

https://ico.org.uk/action-weve-taken/enforcement/royal-free-london-nhs-foundation-trust/

The ICO’s ruling determines that the deals breached four of the Data Protection principles:

https://ico.org.uk/media/action-weve-taken/undertakings/2014353/royal-free-undertaking-cover-letter-03072017.pdf

medConfidential first complained to the National Data Guardian and ICO in June 2016. [1]

In February 2017, the National Data Guardian said that copying of patients’ data to develop the Streams app was on an “inappropriate legal basis”:

http://news.sky.com/story/google-received-16-million-nhs-patients-data-on-an-inappropriate-legal-basis-10879142

Google DeepMind – the AI company developing the app – has given various contradictory quotes about its intent over time, repeatedly asserting that what it was doing was lawful. [2]

Apparently entirely coincidentally, the “Independent Reviewers” of Google DeepMind Health have a report due out, via the Science Media Centre at 00:01 this Wednesday. The timing may be a coincidence – just as it was apparently a complete coincidence that the Royal Free released a press release about how wonderful the project was, without mentioning the word Google once, 72 hours after receiving the letter from the National Data Guardian saying the data use was unlawful. [3]

On seeing the ICO’s ruling, Phil Booth, coordinator of medConfidential said:

“We look forward to Google DeepMind’s Independent Reviewers’ report on Wednesday.”

For further information or for immediate or future interview, please contact Phil Booth, coordinator of medConfidential, on 07974 230 839 or coordinator@medconfidential.org

Notes to editors

1) Details of medConfidential’s complaint are available here:

a) Timeline of events, as of 31/5/16: https://medconfidential.org/wp-content/uploads/
2016/06/medconfidential-deepmind-timeline.pdf

b) Complaint to Regulators: https://medconfidential.org/wp-content/uploads/2016/06/
medconfidential-to-regulators.pdf

c) Shortly after submission, the MHRA found that the project should have been registered with them (and wasn’t): https://techcrunch.com/2016/07/20/
deepminds-first-nhs-health-app-faces-more-regulatory-bumps/

2) This complaint has now been vindicated by the investigation, despite an extremely strong PR response from Google. Contemporary quotes from project advocates, which now ring hollow, include: [all emphasis added]

a) Mustafa Suleyman, Co-Founder at DeepMind, has said:

i) “As Googlers, we have the very best privacy and secure infrastructure for managing the most sensitive data in the world. That’s something we’re able to draw upon as we’re such a core part of Google.” [Guardian, 6/5/16]

ii) “We have, and will always, hold ourselves to the highest possible standards of patient data protection.” [Daily Mail, 4/5/16]

iii) How this came about all started with Dr Chris Laing, of the Royal Free Hospital: “We went for coffee and ended up chatting for four hours.” [BBC News Online, 19/7/16]

iv) More recently, in an interview with Mr Suleyman published on 20/3/17: “When pushed on how the public would be assured that its sensitive data was safe, Suleyman replied, “first there is the law”.” [Digital Health, 20/3/17]

b) George Freeman MP, at the time a Minister in the Department of Health: “NHS patients need to know their data will be secure and not be sold or used inappropriately, which is why we have introduced tough new measures to ensure patient confidentiality.” [Daily Mail, 4/5/16]

c) Professor Hugh Montgomery, (consultant for Google’s DeepMind project) said, on Radio 4’s PM programme on 4 May 2016:

i) “So this is standard business as usual. In this case, it was a standard information data sharing agreement with another supplier, which meets all of those levels of governance. In fact, the agreement there, or the standards of management of those data, meets the very very highest levels. It meets something called HSCIC level 3, which most hospitals trusts don’t even reach.” [Recording of audio available, see link below]

ii) “So firstly, this isn’t research. Research is governed by an entirely separate process that would require anonymisation of data and all sorts. This is data processing.”

iii) “It’s fair to say again that not only is this data at the very highest standards, and beats every standard, and more in the United Kingdom. But the data is encrypted end-to-end, and they have to, like everyone else in the health service, stick to the law.”

iv) Recording of audio available at: https://www.dropbox.com/s/cfimojgec24rlrj/
20160504­deepmind­radio4­pm.mp3?dl=1

d) Will Cavendish, now Strategy Lead for DeepMind Applied, formerly Informatics Accountable Officer at the Department of Health, said (when IAO):

…“The vital importance of trust, security, and cyber security.” … “To be honest, it used to be that not a week goes by, now it’s not a day goes by, without stories of hacking, data leaks, inadvertent data sharing. This absolutely erodes the trust that underpins the work that we do.” https://www.youtube.com/watch?v=5Ej3PRF1jUw&t=2h15m5s

e) Dr Julian Huppert, Chair and “on behalf of the Panel of Independent Reviewers for Google DeepMind Health” said in an e-mail to medConfidential on 6/7/16:

i) “one of our roles is to look in detail at how DeepMind Health uses patient data, and to confirm that it complies with the highest ethical and regulatory standards.”

ii) “We believe from what we have seen so far that DeepMind has a clear commitment to the Caldicott Principles, and that they have to date been honest in their public and private comments. We also believe they are willing to work constructively with regulators, and remain within the law.

3) https://www.royalfree.nhs.uk/news-media/news/new-app-helping-to-improve-patient-care/

 

medConfidential Bulletin, 30th June 2017

So, we have a new Government (after a fashion). And, whatever else, there’s some continuity at the Department of Health…

Given this continuity, the completely unambiguous Conservative Manifesto commitment, and cross-party support for the National Data Guardian, it was a bit disappointing that a statutory footing for NDG was absent from the Queen’s Speech.

We can’t help but note – with a Data Protection Bill on its way, arbitrary data-sharing powers available in the Digital Economy Act, and Theresa May threatening to roll back human rights – that it is protections such as these that underpin the privacy of all our medical records.


What just happened?

The election put a lot on hold, but you may remember a dodgy deal with the Royal Free Hospital that got Google DeepMind into a spot of trouble with the ICO and National Data Guardian when we complained about it.

The NDG’s formal view came out during the election period, and we await the ICO’s ruling – due any day now. We are therefore entirely unsurprised that DeepMind’s “Independent” Reviewers’ report is also delayed. One might question “independence” when a whitewash coincidentally comes out a day after the regulator’s critique…

What’s happening next?

We don’t comment on every future project press release from Google DeepMind – their PR flacks cost many times our annual budget. But last week’s announcement that its next project will be to provide a hospital IT system for Taunton is worthy of some attention; the relevant detail is at the bottom of page 2 of this document.

It’s understood that companies will provide the NHS with IT systems – GPs and hospitals buy in systems all the time. But accepting ‘gift horses’ from aggressively data-seeking US info corps already known for not playing by the rules may not necessarily be wise. For one thing, as many have learned, if you’re not a paying customer you tend to end up being the product.

If, however, the decision is that the people of Taunton are most in need of better infrastructure – NHS England certainly felt they were, this area being one of the ‘pathfinders’ for the cancelled care.data scheme (more on its successor below) – then starting in Somerset is as good a place as any.

But this doesn’t mean you can ignore the regulatory implications. Or future cost.

As recently as January, DeepMind assured Regulators that its tools were not used for clinical decision making, yet in June it has signed contracts to run a hospital using it. To be used in direct care, the central IT system of a hospital is a closely regulated system – these are, after all, the systems that run Intensive Care – although Google, chasing the profits rather than patients, probably won’t choose to help those in most acute need.

Has Google started the Regulatory  process to run that system, or is it trying ‘deployment via press release’? Does it want DeepMind to mark its own homework too?

The only way for patients to know if their data was used in such a programme is for everyone to know where, when and why their medical records have been accessed. Google says it won’t use patients’ data for other purposes; our concern is that minds change. After all, the company said it wouldn’t start building this system for 3 years – that was 7 months ago.

For as long as DeepMind Health is led by an entrepreneur – and has no Chief Medical Officer who is bound by the Hippocratic Oath – its position can change, purely for business reasons. Its corporate officers may stand on stage and say they won’t, but they say many things which they change their minds about. One can be an AI visionary, or run a health infrastructure service – but people have every right to be nervous when you try to do both, especially if you claim you aren’t doing so.

It is inevitable that the future model for this service will be ‘AI assistants’ offering hints and references to doctors via the Streams app; the principle of A&E triage, applied hospital-wide.

This being the case, if these AI systems are modular and compartmentalised for the delivery of care, then they can each be regulated separately. If, however, the individual systems are not interoperable and transparent, then the entire infrastructure must be regulated tightly. (Research, i.e. the development of such systems – including the justification, with evidence, of what data they actually need – is already regulated, by MHRA and other bodies.)

Until the situation is clear, questions as to whether DeepMind’s approach to Regulators is the same as Uber’s (they do, after all, share investors) will remain.

We should point out, as DeepMind buried it in the small print, that no money is changing hands here – and neither party is obligated to do anything. This may yet be just another Silicon Valley startup (the TV show, that is – not the place) that puts out a stream of press releases, delivering for investors over patients.

 

What’s happening where you live? And what can you do?

Wherever you live, in England, there are changes coming to your local NHS.

The ever-so-subtly again renamed STPs (now “Sustainability and Transformation Partnerships”, not just Plans) and their further regional reorganisation – over “several years” – into Kaiser Permanente-style Accountable Care Organisations represent the Government’s and NHS England’s view of the future.

Bearing in mind the massive democratic deficit in the NHS, will accountability be to patients or to the analogue administrators?

Given that – most of the time at least – care records follow patients, one of the best ways to see how the NHS works is to look at the data trail that you leave behind you.

So if you have a login for your GP practice’s website, we encourage you to look at the letters that have been scanned into your record, and to simply count the logos. (If you don’t already have a login for online access, here’s how to get one.) Then, as your NHS changes over the next few years, keep count; over time do you see more commercial logos, or fewer?

While you’re at it, you might also want to check who’s accessed your GP record. Both EMIS and TPP have now switched on basic access to your GP record’s ‘audit trail’ – and as more and more people use it, this vital transparency feature should improve over time.

Things are clearly going to stay busy for a good while yet. Four years in, medConfidential exists entirely through your donations and the generosity of the Joseph Rowntree Reform Trust, to whom we are applying for a further grant. We appreciate all donations – and your support helps with other funding.

 

medConfidential rapid responses to DeepMind’s statements about their “legally inappropriate” data copying

We shall update this page as more information becomes available (newer items at the top).


Tuesday 11am:

Yet more questions raised about the usage of the Streams app

The Sky News footage shows that the Streams app is still in use, displaying information from patients – their name, their date of birth, gender, and other demographic information.

Where does that information come from? How does the ‘calendar gap’ affect patient care?

There are 3 choices:

  1. It comes via the first contract that has been found to be unlawful (with the calendar gap)
  2. The second contract is being breached (which also contains the calendar gap)
  3. There is a third secret contract hidden from scrutiny

Or Google’s AI team has come up with something else legally dubious to continue to unlawfully copy 1.6 million patient records… this suggests an uber-esque approach to the law, and to safety.

What is the ‘calendar gap’?

The data Google DeepMind unlawfully copy is up until “last month”. It is currently the 16th May 2017, and at best, the data they copy will run up until 30 April 2017. On the 29 May, they will only have data until the end of April. When there’s a new month, they get an updated dataset covering the new “last month”. (It possibly takes a few days to actually happen, but you get the idea.)

Streams will help you if you were in the RFH last month. If you were there last week, however, the effect of the contract is that Streams could cause you harm – as Google’s app may prioritise longer-term patients it knows more about, over newer ones it knows less about.

Such problems are why MHRA approvals and a proper testing regimen are so important. To be absolutely clear, this failure is not endemic to Streams – the DeepMind deal with Imperial does not contain it, for example – but it appears as a dangerous symptom of the deal from DeepMind, that has been found to be unlawful.

We’ll ask the National Data Guardian for clarity later today.


Tuesday 10am:

We’ve seen this piece being discussed: the article is correct about patients who were receiving direct care – but out of the 1.6 million patients’ data it copied, DeepMind in February 2017 said it had assisted in the direct care of just “more than 26”.

So while 27 records may have had a lawful basis, 1,599,973 didn’t.

It is the 1,599,973 records that are of concern here. Similarly, while there is not necessarily any problem with testing an app, testing an app isn’t the same as providing direct care. It is a separate process that DeepMind didn’t go through, as their interviews at the time made very clear (Note 6).


Tuesday 10am:

If Google DeepMind didn’t receive the letter containing the NDG’s finding, as they have said to medConfidential (after the date on the letter), they should have a chat to the gmail team about such a convenient problem that no one else sees…

Even if that excuse was valid in the past, there are now lots of copies of the letter on the internet, evidencing their unlawful behaviour. Although Dodgy Donald from DeepMind might be in denial about even that.


Monday night:


Under the heading, ‘What we’ve learned so far’, a newly updated page on DeepMind’s website states:

There’s very low public awareness of NHS technology, and the way patient data is routinely used in the provision of care. For example, many people assume that patient records are normally stored on NHS-owned computers, when they are in fact routinely processed by third party IT providers. This confusion is compounded by the potential future use of AI technologies in the NHS.

medConfidential comment:

This response by Google shows that DeepMind has learnt nothing. There may well be lawful reasons for third party IT providers to process data for direct care for 1.6 million patients – unfortunately for Google’s AI division, developing an app is not one of them.

Google told the public as little as they thought they could get away with – and being duplicitous, they still are. And, in so doing, they are trying to force the NHS into taking the blame for their mistakes.


Regarding the investigation by Sky News into the sharing of patients’ records, which begins:

Google’s artificial intelligence arm received the personally identifying medical records of 1.6 million patients on an “inappropriate legal basis”, according to the most senior data protection adviser to the NHS.

medConfidential comment:

Google’s lawyers are expensive, but “inappropriate legal basis” is still a euphemism for unlawful.

Buried in the interview footage is a statement from a nurse that the app is still in use with patients today. Also:

“The testing for the Streams app has now concluded and it is being used at the Royal Free Hospital, Prof Powis told Sky News, under a second agreement which is not being investigated.” (Sky News article)

Unfortunately for Google, their own press release from last November states that the same data is shared under both agreements.


 

[PRESS RELEASE] Google DeepMind unlawfully copied the medical records of 1.6 million NHS patients

“A core part of Google” has been told it has no lawful basis to process 5 years’ of patient data from the Royal Free Hospital in London. [1] With no legal basis, the data must be deleted.

In May 2016, the New Scientist reported [2] that Google DeepMind had access to a huge haul of patient data, seemingly without appropriate approvals. In July 2016, the MHRA confirmed [3] that DeepMind had not received any approvals for a trial involving patients, using patient data. In November 2016, DeepMind signed a replacement contract covering exactly the same data. [5d]

The National Data Guardian has provided a view on this matter (all emphasis added): [1]

The Royal Free “…confirmed to us [NDG] that 1.6 million identifiable patient records were transferred to Google DeepMind and that implied consent for direct care was the legal basis for the data processing.

“…Streams was going through testing and therefore could not be relied upon for patient care, any role the application might have played in supporting the provision of direct care would have been limited and secondary to the purpose of the data transfer. My considered opinion therefore remains that it would not have been within the reasonable expectation of patients that their records would have been shared for this purpose.

It is unclear whether Google DeepMind has complied with the finding that it had no legal basis for processing this data; nor is it clear what it was that first attracted DeepMind executives to unlawfully copy 1.6 million people’s medical records, repeatedly insisting on direct care as the sole legal basis. [8]

medConfidential agrees with the Information Commissioner, when she said in a speech to technology companies: “I do not believe data protection law is standing in the way of your success.” She reminded her audience: “It’s not privacy or innovation – it’s privacy and innovation.” [4]

In this case, this DeepMind project turned out to be neither of those things. [9]

The National Data Guardian’s investigation has made clear – despite their claims to the contrary – that DeepMind had no legal basis for their actions in this project.

medConfidential coordinator, Phil Booth, said:

“This letter shows that Google DeepMind must know it had to delete the 1.6 million patient medical records it should never have had in the first place. There were legitimate ways for DeepMind to develop the app they wanted to sell. Instead they broke the law, and then lied to the public about it.

“Every flow of patient data in and around the NHS must be safe, consensual and transparent. Patients should know how their data is used, including for possible improvements to care using new digital tools. Such gross disregard of medical ethics by commercial interests – whose vision of ‘patient care’ reaches little further than their business plan – must never be repeated.

“While the NHS sent doctors to a meeting, DeepMind sent lawyers and trained negotiators. What this boils down to is whether Google’s AI division followed the law and told the truth; it now appears they may have done neither.

“As events this weekend have shown, it’s the number of copies of patient data that matter – encryption locks won’t reassure anyone, if the wrong people have been given the keys.”

medConfidential campaigns for confidentiality and consent in health and social care, seeking to ensure that every flow of data into, across and out of the NHS and care system is consensual, safe and transparent. Founded in January 2013, medConfidential is an independent, non-partisan organisation working with patients and medics, service users and care professionals.

For further information or for immediate or future interview, please contact Phil Booth, coordinator of medConfidential, on coordinator@medconfidential.org

Notes to editors

1) “The NDG has provided a view on this matter to assist the ICO’s investigation” was the National Data Guardian’s comment on the publication of the University of Cambridge paper, ‘Google DeepMind and healthcare in an age of algorithms’: https://link.springer.com/article/10.1007%2Fs12553-017-0179-1 and http://www.cam.ac.uk/research/news/
deepmind-royal-free-deal-is-cautionary-tale-for-healthcare-in-the-algorithmic-age

Sky News published a copy of the letter from the National Data Guardian on 15 May 2017: http://news.sky.com/story/google-received-16-million-nhs-patients-data-on-an-inappropriate-legal-basis-10879142

2) medConfidential raised a complaint [4] to the ICO following reports in the New Scientist, and follow-ups elsewhere, about secretive data use by Google DeepMind:

a) New Scientist, 29/4/16: https://www.newscientist.com/article/2086454-
revealed-google-ai-has-access-to-huge-haul-of-nhs-patient-data/

b) New Scientist, 13/5/16: https://www.newscientist.com/article/2088056-did-
googles-nhs-patient-data-deal-need-ethical-approval/

c) Daily Mail, 4/5/16: http://www.dailymail.co.uk/news/article-3573286/NHS-
trust-handed-private-patient-details-Google-says-implied-permission-emerges-hospital-talks-internet-giant.html

d) BBC, 19/7/16: http://www.bbc.co.uk/news/technology-36783521

e) Guardian, 6/5/16 (note 9 May & 25 July updates at the bottom of the article): https://www.theguardian.com/technology/2016/may/06/deepmind-best-privacy-infrastructure-handling-nhs-data-says-co-founder

3) “DeepMind is currently working with the MHRA to ensure that the device complies with all relevant medical device legislation before it is placed on the market” – TechCrunch, 20/7/17: https://techcrunch.com/2016/07/20/deepminds-first-nhs-health-app-faces-more-regulatory-bumps/

4) Information Commissioner’s speech, ‘Transparency, trust and progressive data protection’, 29 September 2016: https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/
2016/09/transparency-trust-and-progressive-data-protection/

5) medConfidential’s complaint is available here:

a) Timeline of events, as of 31/5/16: https://medconfidential.org/wp-content/uploads/
2016/06/medconfidential-deepmind-timeline.pdf

b) Complaint to Regulators: https://medconfidential.org/wp-content/uploads/2016/06/
medconfidential-to-regulators.pdf

c) Shortly after submission, the MHRA found that the project should have been registered with them (and wasn’t): https://techcrunch.com/2016/07/20/deepminds-first-nhs-health-app-faces-more-regulatory-bumps/

d) The end of the first ‘Note to editors’ in a press release from the Royal Free Hospital on 22 November 2016 clearly states: “The new agreement does not change the number of patients whose data will be processed by Streams”: https://www.royalfree.nhs.uk/news-media/news/nhs-and-technology-leaders-agree-groundbreaking-partnership-to-improve-safe/

6) Claims by the New Scientist have been vindicated by the investigation, despite an extremely strong PR response from Google. Contemporary quotes from project advocates, which now ring hollow, include: [all emphasis added]

a) Mustafa Suleyman, Co-Founder at DeepMind, has said:

i) “As Googlers, we have the very best privacy and secure infrastructure for managing the most sensitive data in the world. That’s something we’re able to draw upon as we’re such a core part of Google.” [Guardian, 6/5/16]

ii) “We have, and will always, hold ourselves to the highest possible standards of patient data protection.” [Daily Mail, 4/5/16]

iii) How this came about all started with Dr Chris Laing, of the Royal Free Hospital: “We went for coffee and ended up chatting for four hours.” [BBC News Online, 19/7/16]

iv) More recently, in an interview with Mr Suleyman published on 20/3/17: “When pushed on how the public would be assured that its sensitive data was safe, Suleyman replied, “first there is the law”.” [Digital Health, 20/3/17]

b) George Freeman MP, at the time a Minister in the Department of Health: “NHS patients need to know their data will be secure and not be sold or used inappropriately, which is why we have introduced tough new measures to ensure patient confidentiality.” [Daily Mail, 4/5/16]

c) Professor Hugh Montgomery, (consultant for Google’s DeepMind project) said, on Radio 4’s PM programme on 4 May 2016:

i) “So this is standard business as usual. In this case, it was a standard information data sharing agreement with another supplier, which meets all of those levels of governance. In fact, the agreement there, or the standards of management of those data, meets the very very highest levels. It meets something called HSCIC level 3, which most hospitals trusts don’t even reach.” [Recording of audio available, see link below]

ii) “So firstly, this isn’t research. Research is governed by an entirely separate process that would require anonymisation of data and all sorts. This is data processing.”

iii) “It’s fair to say again that not only is this data at the very highest standards, and beats every standard, and more in the United Kingdom. But the data is encrypted end-to-end, and they have to, like everyone else in the health service, stick to the law.”

iv) Recording of audio available at: https://www.dropbox.com/s/cfimojgec24rlrj/
20160504­deepmind­radio4­pm.mp3?dl=1

d) Will Cavendish, now Strategy Lead for DeepMind Applied, formerly Informatics Accountable Officer at the Department of Health, said (when IAO):

i) …“The vital importance of trust, security, and cyber security.” … “To be honest, it used to be that not a week goes by, now it’s not a day goes by, without stories of hacking, data leaks, inadvertent data sharing. This absolutely erodes the trust that underpins the work that we do.” https://www.youtube.com/watch?v=5Ej3PRF1jUw&t=2h15m5s

e) Dr Julian Huppert, Chair and “on behalf of the Panel of Independent Reviewers for Google DeepMind Health” said in an e-mail to medConfidential on 6/7/16:

i) “one of our roles is to look in detail at how DeepMind Health uses patient data, and to confirm that it complies with the highest ethical and regulatory standards.”

ii) “We believe from what we have seen so far that DeepMind has a clear commitment to the Caldicott Principles, and that they have to date been honest in their public and private comments. We also believe they are willing to work constructively with regulators, and remain within the law.

7) The claim to reach “HSCIC level 3” was a self-assessment by DeepMind, which was revoked upon examination. [See the 25 July update to this Guardian article].

8) In a controversial press release by the hospital on 24 February 2017, the word “Google” did not appear once, despite point 6 (a)(i) above: https://www.royalfree.nhs.uk/news-media/
news/new-app-helping-to-improve-patient-care/
and a subsequent Guardian article on 9 March 2017, from a press release by Google DeepMind, which explicitly attributes actions to Google DeepMind: https://www.theguardian.com/technology/2017/mar/09/google-deepmind-health-records-tracking-blockchain-nhs-hospitals

9) “ “With health data, and government acquired health data, we need to be sure we aren’t, in effect, giving oxygen away for free to a private company that will start to sell it back to us,” says Azeem Azhar, who writes the popular Exponential View newsletter…” – Quartz, 17/3/17: https://qz.com/934137/googles-goog-deepmind-got-too-much-health-data-from-
britains-nhs-research-paper-says/

– ends –

medConfidential comment on Google DeepMind briefing on an academic paper

We read many academic papers about data projects. It is rare they result in anything at all, let alone anonymous briefings against academic inquiry.

We were therefore intrigued by two points in this Wired article, written with access to Google DeepMind executives:

  1. It reuses a quote from medConfidential that is 9 months old, as if nothing has changed in the last 9 months. If that was true, why did Wired write about it again?
  2. That the quote from the Google DeepMind executive suggests the academic paper to which the article refers has errors.

If, as DeepMind says, “It makes a series of significant factual and analytical errors”, we look forward to DeepMind publishing evidence of any errors as a scientifically rigorous organisation would, rather than hiding behind anonymous briefings from their press office and a hospital. Google claims “ “we’re completely at the mercy and direction” of the Royal Free”, but from the last 2 paragraphs of the same article, that’s obviously not completely true…

medConfidential has confidence in the scientific inquiry process – and we are aware DeepMind also do, given their own authorship of academic articles about their work.

While it is highly unusual, it is not a factual or analytical error to write an academic paper that is readable by all.

We expect that DeepMind was aware of the substance of the paper prior to publication, and didn’t say anything about any of those problems then. This behaviour is entirely consistent with DeepMind’s duplicity regarding our timeline of public facts about their original deal – they claim errors in public, but will say nothing about them when asked.

Colleagues at the Wellcome Trust are right – mistakes were made.

This is how AI will go wrong; good people with good intentions making a mistake and being institutionally incapable of admitting that most human of characteristics, imperfection.

medConfidential response to “technology company DeepMind” Press Release

For immediate release – Tuesday 28 February 2017

One year after first telling the public that “technology company DeepMind” [1] was going to help the NHS, it is still unclear whether Google’s duplicitous offer still includes forcing the NHS to hand over the medical history of every patient who has visited the hospital. [2]

It is no surprise that digital tools help patients, but is Google still forcing the NHS to pay with its patients’ most private data?

As the NHS reorganises itself again with the Secret Transformation Plans, [3] NHS England plans a ‘National Data Lake’ for all patient data. [4] Of which this is one. In defending giving data on all its patients to Google, Royal Free’s Chief Executive, David Sloman, said “it is quite normal to have data lying in storage”. [5]

Tomorrow the Government announces the UK’s new digital strategy, [6] including new money for the Artificial Intelligence in which DeepMind specialises. Is copying of data on a whim what the future holds?

Clause 31 of the Digital Economy Bill suggests precisely that [7] – data can be ‘shared’ (copied) to anyone associated with a public or NHS body [8] who can justify it as “quite normal to have data lying in storage”.

As Downing Street takes the Trump approach to health data, [9] does Google now say the ends justify the means?

Phil Booth, coordinator of medConfidential said:

“So toxic is the project, the latest press release doesn’t even use the word “Google”.

“It is good that 11 patients a day get faster care due to this tool; but Google will still not say why they wanted data on thousands of patients who visit the hospital daily.

“Until patients can see where their medical records have gone, companies will continue to predate upon the NHS to extract its most important resources.”

Notes to Editors

1) This is how Google’s wholly-owned subsidiary, DeepMind – based in the Google offices in London – was misleadingly described in this press release published by the Royal Free: https://www.royalfree.nhs.uk/news-media/news/new-app-helping-to-improve-patient-care/

2) ‘Google handed patients’ files without permission: Up to 1.6 million records – including names and medical history – passed on in NHS deal with web giant’, Daily Mail, 3/5/16: http://www.dailymail.co.uk/news/article-3571433/Google-s-artificial-intelligence-access-private-medical-records-1-6million-NHS-patients-five-years-agreed-data-sharing-deal.html

3) Hospital cuts planned in most of England: http://www.bbc.co.uk/news/health-39031546

4) medConfidential comments on NHS England’s National Data Lake: https://medconfidential.org/2017/fishing-in-the-national-data-lake/

5) The Government confirms that the bulk data copied by DeepMind, i.e. SUS, “are maintained for secondary uses” and not direct care: http://www.parliament.uk/business/publications/written-questions-answers-statements/written-question/Lords/2016-12-07/HL3943

6) Due to launch on Wednesday, being now pre-briefed by the Minister: https://twitter.com/MattHancockMP/status/835835027611127809

7) Clause 31 of the Digital Economy Bill as currently drafted would allow any provider of a service to a public body (such as Google to the NHS) to share data with (i.e. provide a copy to) any other provider.

8) While the Draft Regulations for Clause 31 state that Department of Health bodies are excluded from the Clause, medConfidential has received confirmation that such bodies will be included in the final regulations after Parliament has considered the Clause without health included.

9) The NHS is being forced to release the names and addresses of vulnerable patients to the Home Office: http://buzzfeed.com/jamesball/trumping-donald-trump

Questions that remain unanswered from May 2016 include:

  • What was the basis for Google to get 5 years of secondary uses data on every patient who visits the hospital? Google is getting thousands of people’s data per day, yet the hospital admits it is helping only a small fraction of them.
  • Why did the app not simply access the data it could clinically justify, when it needed to display it? That would have provided all the benefits of the app to patients and clinicians, and not given Google the medical records of patients which it had no justification for receiving. Did Google even talk to the hospital’s IT provider about access to only the data it needed before demanding all the data the hospital held?

medConfidential made a complaint to the ICO and National Data Guardian about the project in June 2016. Google and the Royal Free Hospital have failed to yet provide satisfactory answers and we understand the investigation remains ongoing.

-ends-

Deepmind try again – November 2016

DeepMind this morning reannounced their partnership with the Royal Free Hospital. Updates are at the bottom – details are in the 9:50 and 10:10 updates.

There’s apparently a new legal agreement to copy exactly the same data that caused so much controversy over the summer. We have not yet seen the new legal agreement, so can’t comment on what it permits or disallows.

Responding to the press release, Phil Booth, Coordinator of medConfidential said:

“Our concern is that Google gets data on every patient who has attended the hospital in the last 5 years and they’re getting a monthly report of data on every patient who was in the hospital, but may now have left, never to return.

“What your Doctor needs to be able to see is the up to date medical history of the patient currently in front of them.

“The Deepmind gap, because the patient history is up to a month old, makes the entire process unreliable and makes the fog of unhelpful data potentially even worse.

As Deepmind publish the legal agreements and PIA, we will read them and update comments here.


8:50am update. The Deepmind legal agreement was expected to be published at midnight. As far as we can tell, it wasn’t. Updated below.

TechCrunch have published a news article, and helpfully included the DeepMind talking points in a list. The two that are of interest (emphasis added):

  • An intention to develop what they describe as “an unprecedented new infrastructure that will enable ongoing audit by the Royal Free, allowing administrators to easily and continually verify exactly when, where, by whom and for what purpose patient information is accessed.” This is being built by Ben Laurie, co-founder of the OpenSSL project.
  • A commitment that the infrastructure that powers Streams is being built on “state-of-the-art open and interoperable standards,” which they specify will enable the Royal Free to have other developers build new services that integrate more easily with their systems. “This will dramatically reduce the barrier to entry for developers who want to build for the NHS, opening up a wave of innovation — including the potential for the first artificial intelligence-enabled tools, whether developed by DeepMind or others,” they add.

Public statements about streams (an iPhone app for doctors) don’t seem to explain what that is. What is it?


9:30 update: The Deepmind website has now been updated. We’re reading.

The contracts are no longer part of the FAQ, they’re now linked from the last paragraph of text. (mirrored here)


9:40 update: MedConfidential is greatly helped in its work by donations from people like you.


9:50 update: Interesting what is covered by what…

screen-shot-2016-11-22-at-09-54-17screen-shot-2016-11-22-at-09-45-28

screen-shot-2016-11-22-at-09-47-34


10:10 update: What data does the DeepMind FIHR API cover? What is the Governance of that API? Is it contractually, legally, and operationally independent of the Streams app?

(it’s clearly none of those things, as the above screenshots say).

Deepmind have made great play of their agreement being safe, but consent is determined in a google meeting room, and the arrangements for the “FIHR API” are secretive and far from transparent.

There is likely to only be one more update today around 1pm. Unless Google make an announcement that undermines their contractual agreements.


1pm update: The original information sharing agreement was missing Schedule 1, and has been updated.


3:30 update: DeepMind have given some additional press briefings to Wired (emphasis added):

“Suleyman said the company was holding itself to “an unprecedented level of oversight”. The government of Google’s home nation is conducting a similar experiment…

““Approval wasn’t actually needed previously because we were really only in testing and development, we didn’t actually ship a product,” which is what they said last time, and MHRA told them otherwise.

Apparently “negative headlines surrounding his company’s data-sharing deal with the NHS are being “driven by a group with a particular view to pedal”.”. The headlines are being driven by the massive PR push they have done since 2:30pm on Monday when they put out a press release which talked only about the app, and mentioned data as an aside only in the last sentence of the first note to editors. – Beware of the leopard.

As to our view, MedConfidential is an independent non-partisan organisation campaigning for confidentiality and consent in health and social care, which seeks to ensure that every flow of data into, across and out of the NHS and care system is consensual, safe and transparent. Does Google Inc disagree with that goal?