The U.Okay.’s medical gadget regulator has admitted it has considerations about VC-backed AI chatbot maker Babylon Health. It made the admission in a letter despatched to a clinician who’s been elevating the alarm about Babylon’s strategy towards affected person security and company governance since 2017.
The HSJ reported on the MHRA’s letter to Dr. David Watkins yesterday. TechCrunch has reviewed the letter (see beneath), which is dated December 4, 2020. We’ve additionally seen extra context about what was mentioned in a gathering referenced within the letter, in addition to reviewing different correspondence between Watkins and the regulator during which he particulars quite a lot of wide-ranging considerations.
In an interview he emphasised that the considerations the regulator shares are “far broader” than the (vital however) single difficulty of chatbot security.
“The issues relate to the corporate governance of the company — how they approach safety concerns. How they approach people who raise safety concerns,” Watkins advised TechCrunch. “That’s the priority. And among the ethics around the mispromoting of medical units.
“The overall story is they did promote something that was dangerously flawed. They made misleading claims with regards to how [the chatbot] should be used — its intended use — with [Babylon CEO] Ali Parsa promoting it as a ‘diagnostic’ system — which was never the case. The chatbot was never approved for ‘diagnosis.’”
“In my opinion, in 2018 the MHRA should have taken a much firmer stance with Babylon and made it clear to the public that the claims that were being made were false — and that the technology was not approved for use in the way that Babylon were promoting it,” he went on. “That should have happened and it didn’t happen because the regulations at that time were not fit for purpose.”
“In reality there is no regulatory ‘approval’ process for these technologies and the legislation doesn’t require a company to act ethically,” Watkins additionally advised us. “We’re reliant on the health tech sector behaving responsibly.”
The marketing consultant oncologist started elevating pink flags about Babylon with U.Okay. healthcare regulators (CQC/MHRA) as early as February 2017 — initially over the “apparent absence of any robust clinical testing or validation,” as he places it in correspondence to regulators. However with Babylon opting to disclaim issues and go on the assault towards critics his considerations mounted.
An admission by the medical units regulator that every one Watkins’ considerations are “valid” and are “ones that we share” blows Babylon’s deflective PR techniques out of the water.
“Babylon cannot say that they have always adhered to the regulatory requirements — at times they have not adhered to the regulatory requirements. At different points throughout the development of their system,” Watkins additionally advised us, including: “Babylon never took the safety concerns as seriously as they should have. Hence this issue has dragged on over a more than three-year period.”
During this time the corporate has been steaming forward inking wide-ranging “digitization” offers with healthcare suppliers around the world — together with a 10-year deal agreed with the U.Okay. metropolis of Wolverhampton last year to offer an built-in app that’s supposed to have a attain of 300,000 folks.
It additionally has a 10-year settlement with the federal government of Rwanda to help digitization of its well being system, together with by way of digitally enabled triage. Other markets it’s rolled into embody the U.S., Canada and Saudi Arabia.
Babylon says it now covers greater than 20 million sufferers and has finished 8 million consultations and “AI interactions” globally. But is it working to the excessive requirements folks would count on of a medical gadget firm?
Safety, moral and governance considerations
In a written abstract, dated October 22, of a video name which occurred between Watkins and the U.Okay. medical units regulator on September 24 final 12 months, he summarizes what was mentioned within the following approach: “I talked through and expanded on each of the points outlined in the document, specifically; the misleading claims, the dangerous flaws and Babylon’s attempts to deny/suppress the safety issues.”
In his account of this assembly, Watkins goes on to report: “There appeared to be general agreement that Babylon’s corporate behavior and governance fell below the standards expected of a medical device/healthcare provider.”
“I was informed that Babylon Health would not be shown leniency (given their relationship with [U.K. health secretary] Matt Hancock),” he additionally notes within the abstract — a reference to Hancock being a publicly enthusiastic consumer of Babylon’s “GP at hand” app (for which he was accused in 2018 of breaking the ministerial code).
In a separate doc, which Watkins compiled and despatched to the regulator final 12 months, he particulars 14 areas of concern — masking points together with the protection of the Babylon chatbot’s triage; “misleading and conflicting” T&Cs — which he says contradict promotional claims it has made to hype the product; in addition to what he describes as a “multitude of ethical and governance concerns” — together with its aggressive response to anybody who raises considerations about the protection and efficacy of its know-how.
This has included a public assault marketing campaign towards Watkins himself, which we reported on last year; in addition to what he lists within the doc as “legal threats to avoid scrutiny and adverse media coverage.”
Here he notes that Babylon’s response to security considerations he had raised again in 2018 — which had been reported on by the HSJ — was additionally to go on the assault, with the corporate claiming then that “vested interest” have been spreading “false allegations” in an try to “see us fail.”
“The allegations were not false and it is clear that Babylon chose to mislead the HSJ readership, opting to place patients at risk of harm, in order to protect their own reputation,” writes Watkins in related commentary to the regulator.
He goes on to level out that, in May 2018, the MHRA had itself independently notified Babylon Health of two incidents associated to the protection of its chatbot (one involving missed signs of a coronary heart assault, one other missed signs of DVT) — but the corporate nonetheless went on to publicly garbage the HSJ’s report the next month (which was entitled: “Safety regulators investigating concerns about Babylon’s ‘chatbot’”).
Wider governance and operational considerations Watkins raises within the doc embody Babylon’s use of employees NDAs — which he argues results in a tradition inside the corporate the place employees really feel unable to talk out about any security considerations they could have; and what he calls “inadequate medical device vigilance” (whereby he says the Babylon bot doesn’t routinely request suggestions on the affected person consequence publish triage, arguing that: “The absence of any robust feedback system significant impairs the ability to identify adverse outcomes”).
Re: unvarnished employees opinions, it’s fascinating to notice that Babylon’s Glassdoor rating on the time of writing is simply 2.9 stars — with solely a minority of reviewers saying they’d suggest the corporate to a pal and the place Parsa’s approval ranking as CEO can be solely 45% on mixture. (“The technology is outdated and flawed,” writes one Glassdoor reviewer who’s listed as a present Babylon Health worker working as a scientific ops affiliate in Vancouver, Canada — the place privateness regulators have an open investigation into its app. Among the listed cons within the one-star evaluate is the declare that: “The well-being of patients is not seen as a priority. A real joke to healthcare. Best to avoid.”)
Per Watkins’ report of his online assembly with the MHRA, he says the regulator agreed NDAs are “problematic” and impression on the power of workers to talk up on questions of safety.
He additionally writes that it was acknowledged that Babylon workers could concern talking up due to legal threats. His minutes additional document that: “Comment was made that the MHRA are able to look into concerns that are raised anonymously.”
In the abstract of his considerations about Babylon, Watkins additionally flags an occasion in 2018 which the corporate held in London to advertise its chatbot — throughout which he writes that it made quite a lot of “misleading claims,” akin to that its AI generates well being recommendation that’s “on-par with top-rated practicing clinicians.”
The flashy claims led to a blitz of hyperbolic headlines about the bot’s capabilities — serving to Babylon to generate hype at a time when it was prone to have been pitching traders to boost extra funding.
The London-based startup was valued at $2 billion+ in 2019 when it raised an enormous $550 million Series C spherical, from traders together with Saudi Arabia’s Public Investment Fund and a big (unnamed) U.S.-based medical health insurance firm, in addition to insurance coverage large Munich Re’s ERGO Fund — trumpeting the elevate on the time as the biggest ever in Europe or U.S. for digital well being supply.
“It should be noted that Babylon Health have never withdrawn or attempted to correct the misleading claims made at the AI Test Event [which generated press coverage it’s still using as a promotional tool on its website in certain jurisdictions],” Watkins writes to the regulator. “Hence, there remains an ongoing risk that the public will put undue faith in Babylon’s unvalidated medical device.”
In his abstract he additionally contains a number of items of nameless correspondence from quite a lot of folks claiming to work (or have labored) at Babylon — which make quite a lot of extra claims. “There is huge pressure from investors to demonstrate a return,” writes one in every of these. “Anything that slows that down is seen [a]s avoidable.”
“The allegations made against Babylon Health are not false and were raised in good faith in the interests of patient safety,” Watkins goes on to claim in his abstract to the regulator. “Babylon’s ‘repeated’ attempts to actively discredit me as an individual raises serious questions regarding their corporate culture and trustworthiness as a healthcare provider.”
In its letter to Watkins (screengrabbed beneath), the MHRA tells him: “Your concerns are all valid and ones that we share.”
It goes on to thank him for personally and publicly elevating points “at considerable risk to yourself.”
Babylon has been contacted for a response to the MHRA’s validation of Watkins’ considerations. At the time of writing it had not responded to our request for remark.
The startup advised the HSJ that it meets all of the native necessities of regulatory our bodies for the nations it operates in, including: “Babylon is committed to upholding the highest of standards when it comes to patient safety.”
In one aforementioned aggressive incident last year, Babylon put out a press launch attacking Watkins as a “troll” and searching for to discredit the work he was doing to spotlight questions of safety with the triage carried out by its chatbot.
It additionally claimed its know-how had been “NHS validated” as a “safe service 10 times.”
It’s not clear what validation course of Babylon was referring to there — and Watkins additionally flags and queries that declare in his correspondence with the MHRA, writing: “As far as I am aware, the Babylon chatbot has not been validated — in which case, their press release is misleading.”
The MHRA’s letter, in the meantime, makes it clear that the present regulatory regime within the U.Okay. for software-based medical gadget merchandise doesn’t adequately cowl software-powered “health tech” units, akin to Babylon’s chatbot.
Per Watkins there is no such thing as a approval course of, presently. Such units are merely registered with the MHRA — however there’s no legal requirement that the regulator assess them and even obtain documentation associated to their improvement. He says they exist independently — with the MHRA holding a register.
“You have raised a complex set of issues and there are several aspects that fall outside of our existing remit,” the regulator concedes within the letter. “This highlights some issues which we are exploring further, and which may be important as we develop a new regulatory framework for medical devices in the U.K.”
An replace to pan-EU medical units regulation — which can herald new necessities for software-based medical units and had been initially supposed to be applied within the U.Okay. in May final 12 months — will now not happen, given the nation has left the bloc.
The U.Okay. is as an alternative within the technique of formulating its personal regulatory replace for medical gadget guidelines. This means there’s nonetheless a gap around software-based “health tech” — which isn’t anticipated to be totally plugged for a number of years. (Although Watkins notes there have been some tweaks to the regime, akin to a partial lifting of confidentiality necessities final 12 months.)
In a speech final 12 months, well being secretary Hancock told parliament that with the federal government aimed to formulate a regulatory system for medical units that’s “nimble enough” to maintain up with tech-fueled developments akin to well being wearables and AI whereas “maintaining and enhancing patient safety.” It will embody giving the MHRA “a new power to disclose to members of the public any safety concerns about a device,” he mentioned then.
In the in the meantime the present (outdated) regulatory regime seems to be persevering with to tie the regulator’s arms — a minimum of vis-a-vis what they’ll say in public about security considerations. It has taken Watkins making its letter to him public to do this.
In the letter the MHRA writes that “confidentiality unfortunately binds us from saying more on any specific investigation,” though it additionally tells him: “Please be assured that your concerns are being taken seriously and if there is action to be taken, then we will.”
“Based on the wording of the letter, I think it was clear that they wanted to provide me with a message that we do hear you, that we understand what you’re saying, we acknowledge the concerns which you’ve raised, but we are limited by what we can do,” Watkins advised us.
He additionally mentioned he believes the regulator has engaged with Babylon over considerations he’s raised these previous three years — noting the corporate has made quite a lot of adjustments after he had raised particular queries (akin to to its T&Cs, which had initially mentioned it’s not a medical gadget however have been subsequently withdrawn and modified to acknowledge it is; or claims it had made that the chatbot is “100% safe” which have been withdrawn — after an intervention by the Advertising Standards Authority in that case).
The chatbot itself has additionally been tweaked to place much less emphasis on the analysis as an consequence and extra emphasis on the triage consequence, per Watkins.
“They’ve taken a piecemeal approach [to addressing safety issues with chatbot triage]. So I would flag an issue [publicly via Twitter] and they would only look at that very specific issue. Patients of that age, undertaking that exact triage assessment — ‘okay, we’ll fix that, we’ll fix that’ — and they would put in place a [specific fix]. But sadly, they never spent time addressing the broader fundamental issues within the system. Hence, safety issues would repeatedly crop up,” he mentioned, citing examples of a number of points with cardiac triages that he additionally raised with the regulator.
“When I spoke to the people who work at Babylon they used to have to do these hard fixes … All they’d have to do is just kind of ‘dumb it down’ a bit. So, for example, for anyone with chest pain it would immediately say go to A&E. They would take away any thought process to it,” he added. (It additionally after all dangers losing healthcare assets — as he additionally factors out in remarks to the regulators.)
“That’s how they over time got around these issues. But it highlights the challenges and difficulties in developing these tools. It’s not easy. And if you try and do it quickly and don’t give it enough attention then you just end up with something that is useless.”
Watkins additionally suspects the MHRA has been concerned in getting Babylon to take away sure items of hyperbolic promotional materials associated to the 2018 AI occasion from its web site.
In one curious episode, additionally associated to the 2018 occasion, Babylon’s CEO demoed an AI-powered interface that appeared to indicate real-time transcription of a affected person’s phrases mixed with an “emotion-scanning” AI — which he mentioned scanned facial expressions in actual time to generate an evaluation of how the individual was feeling — with Parsa occurring to inform the viewers: “That’s what we’ve done. That’s what we’ve built. None of this is for show. All of this will be either in the market or already in the market.”
However neither function has really been delivered to market by Babylon as but. Asked about this final month, the startup advised TechCrunch: “The emotion detection functionality, seen in old versions of our clinical portal demo, was developed and built by Babylon‘s AI team. Babylon conducts extensive user testing, which is why our technology is continually evolving to meet the needs of our patients and clinicians. After undergoing pre-market user testing with our clinicians, we prioritized other AI-driven features in our clinical portal over the emotion recognition function, with a focus on improving the operational aspects of our service.”
“I certainly found [the MHRA’s letter] very reassuring and I strongly suspect that the MHRA have been engaging with Babylon to address concerns that have been identified over the past three-year period,” Watkins additionally advised us right now. “The MHRA don’t appear to have been ignoring the issues but Babylon simply deny any problems and can sit behind the confidentiality clauses.”
In an announcement on the present regulatory state of affairs for software-based medical units within the U.Okay., the MHRA advised us:
The MHRA ensures that producers of medical units adjust to the Medical Devices Regulations 2002 (as amended). Please consult with existing guidance.
The Medicines and Medical Devices Act 2021 gives the muse for a brand new improved regulatory framework that’s presently being developed. It will contemplate all points of medical gadget regulation, together with the danger classification guidelines that apply to Software as a Medical Device (SaMD).
The U.Okay. will proceed to acknowledge CE marked units till 1 July 2023. After this time, necessities for the UKCA Mark should be met. This will embody the revised necessities of the brand new framework that’s presently being developed.
The Medicines and Medical Devices Act 2021 permits the MHRA to undertake its regulatory actions with a higher degree of transparency and share information the place that’s within the pursuits of affected person security.
The regulator declined to be interviewed or reply to questions about the considerations it says within the letter to Watkins that it shares about Babylon — telling us: “The MHRA investigates all concerns but does not comment on individual cases.”
“Patient safety is paramount and we will always investigate where there are concerns about safety, including discussing those concerns with individuals that report them,” it added.
Watkins raised another salient level on the difficulty of affected person security for “cutting edge” tech instruments — asking the place is the “real-life clinical data”? So far, he says the research sufferers should go on are restricted assessments — usually made by the chatbot makers themselves.
“One quite telling thing about this sector is the fact that there’s very little real-life data out there,” he mentioned. “These chatbots have been around for a good few years now … And there’s been enough time to get real-life clinical data and yet it hasn’t appeared and you just wonder if, is that because in the real-life setting they are actually not quite as useful as we think they are?”