When the contraceptive pill first became available in the US, women weren’t warned of possible side effects, such as heart attacks and blood clots. It took around a decade before anything was done. In her 1969 book The Doctor’s Case Against The Pill, journalist and activist Barbara Seaman who had long experienced these symptoms. Her work led to US Senate hearings on the safety of the drug, which prompted a move to lower doses and inclusion of mandatory information on side effects.

This wouldn’t be the last time voices of women who have poorly understood conditions would long go unheard, even in connection with the pill. Despite strong anecdotal evidence from users, it took until the 2010s for .

It is a similar story for those with chronic fatigue syndrome, of which . Regardless of the strength of testimonies on its debilitating impact, research into this condition was scant until a similar one – long covid – .

When thousands are expressing similar sentiments about their health, we must pay attention

This week’s cover story on page 36 gives the latest example. For decades, women with endometriosis, an often painful gynaecological condition, have also complained of autoimmune illness. Only in the past handful of years has a possible link been . And it was only this year that a full analysis identified a shared genetic pathway between the two, leading to potential new treatments.

Why did it take so long? Lack of funding for female health conditions and squeamishness around female anatomy play a role. But it is vital to be aware of how much less attention seems to be paid – and how much less legitimacy is given – to the complaints of women. When thousands are expressing similar sentiments about their health, we must take heed. Robust data is always needed for concrete medical advice and safe treatment. We might well get there faster, though, if we listen promptly to people, especially women, when they state the realities of their experience.

An artificial intelligence model trained on the medical data of 57 million people who have used the National Health Service in England could one day assist doctors in predicting disease or forecast hospitalisation rates, its creators have claimed. However, other researchers say there are still significant privacy and data protection concerns around such large-scale use of health data, while even the AI’s architects say they can’t guarantee that it won’t inadvertently reveal sensitive patient data.

The model, called Foresight, was first developed in 2023. That initial version used OpenAI’s GPT-3, the large language model (LLM) behind the first version of ChatGPT, and trained on 1.5 million real patient records from two London hospitals.

Now, at University College London and his colleagues have scaled up Foresight to create what they say is the world’s first “national-scale generative AI model of health data” and the largest of its kind.

Foresight uses eight different datasets of medical information routinely collected by the NHS in England between November 2018 to December 2023 and is based on Meta’s open-source LLM Llama 2. These datasets include outpatient appointments, hospital visits, vaccination data and records, comprising a total of 10 billion different health events for 57 million people – essentially everyone in England.

Tomlinson says his team isn’t releasing information about how well Foresight performs because the model is still being tested, but he claims it could one day be used to do everything from making individual diagnoses to predicting broad future health trends, such as hospitalisations or heart attacks. “The real potential of Foresight is to predict disease complications before they happen, giving us a valuable window to intervene early, and enabling a shift towards more preventative healthcare at scale,” he told a press conference on 6 May.

While the potential benefits are yet to be supported, there are already concerns about people’s medical data being fed to an AI at such a large scale. The researchers insist all records were “de-identified” before being used to train the AI, but the risks of someone being able to use patterns in the data to re-identify the records are well-recorded, particularly when it comes to large datasets.

Free newsletter

Sign up to The Daily

Sign up to newsletter

“Building powerful generative AI models that protect patient privacy is an open, unsolved scientific problem,” says at the University of Oxford. “The very richness of data that makes it valuable for AI also makes it incredibly hard to anonymise. These models should remain under strict NHS control where they can be safely used.”

“The data that goes into the model is de-identified, so the direct identifiers are removed,” said at NHS Digital, speaking at the press conference. But Chapman, who oversees the data used to train Foresight, admitted that there is always a risk of re-identification: “It’s then very hard with rich health data to give 100 per cent certainty that somebody couldn’t be spotted in that dataset.”

To mitigate this risk, Chapman said the AI is operating within a custom-built “secure” NHS data environment to ensure that information isn’t leaked out of the model and is accessible only to approved researchers. Amazon Web Services and data company Databricks have also supplied “computational infrastructure”, but can’t access the data, said Tomlinson.

at Imperial College London says one way to check whether models can reveal sensitive information is to verify whether they can memorise data seen during training. When asked by 91av whether the Foresight team had conducted these tests, Tomlinson said it hadn’t, but that it was looking at doing so in the future.

Using such a vast dataset without communicating to people how the data has been used can also weaken public trust, says at the University of Oxford. “Even if it is being anonymised, it’s something that people feel very strongly about from an ethical point of view, because people usually want to keep control over their data and they want to know where it’s going.”

But existing controls give people little chance to opt out of their data being used by Foresight. All of the data used to train the model comes from nationally collected NHS datasets, and because it has been “de-identified”, , says an NHS England spokesperson, though people who have chosen not to share data from their family doctor won’t have this fed into the model.

Under the General Data Protection Regulation (GDPR), people must have the option to withdraw consent for the use of their personal data, but because of the way LLMs like Foresight are trained, it isn’t possible to remove a single record from an AI tool. The NHS England spokesperson says that “as the data used to train the model is anonymised, it is not using personal data and GDPR would therefore not apply”.

Exactly how the GDPR should address the impossibility of removing data from an LLM is an , but the UK Information Commissioner’s Office’s website states that “de-identified” data should not be used as a synonym for anonymous data. “This is because UK data protection law doesn’t define the term, so using it can lead to confusion,” it states.

The legal position is further complicated because Foresight is currently being used only for research related to covid-19, says Tomlinson. That means exceptions to data protection laws enacted during the pandemic still apply, says Sam Smith at , a UK data privacy organisation. “This covid-only AI almost certainly has patient data embedded in it, which cannot be let out of the lab,” he says. “Patients should have control over how their data is used.”

Ultimately, the competing rights and responsibilities around using medical data for AI leave Foresight in an uncertain position. “There is a bit of a problem when it comes to AI development, where the ethics and people are a second thought, rather than the starting point,” says Green. “But what we need is the humans and the ethics need to be the starting point, and then comes the technology.”

Doctors, as a whole, are a pretty clever bunch, but they can be resistant to change. The most famous example is probably the 19th-century surgeons who refused to wash their hands when moving from mortuary to labour ward, spreading as-yet-undiscovered microbes and leading to infant deaths. Hungarian physician Ignaz Semmelweis, who collected statistics to make the case that soap and water could save lives, was ridiculed and ostracised.

Today, we live in more enlightened times, and medical practice is generally backed by evidence – but are we always getting the right evidence to bring about change? For example, there are signs that bringing artificial intelligence into clinical use could also save lives. As we report in “AI helps radiologists spot breast cancer in real-world tests”, radiologists who chose to use an image-classifying AI to help spot breast cancer picked up an extra case per 1000 people screened. Across healthcare systems, the effect could be big.

Does that mean we should encourage doctors to hang up their scrubs and let the machines take over? Far from it. While large language model AI systems like ChatGPT can ace multiple-choice medical tests, they do less well on conversational diagnoses (see “AI chatbots fail to diagnose patients by talking with them”). A medic with a good bedside manner and listening ear is still vital, for now.

We should be bolder in testing medical AI systems in real-world settings

Instead, there are two conclusions we can draw from these studies. The first is that we should be careful about using the generic term “artificial intelligence”. Although the two systems we report on share an underlying neural network technology, image classification is a very different task to text generation, and the latter has a much higher risk of the AI spitting out plausible but incorrect results. In other words, not all AIs are made equal.

The second conclusion is that we should be bolder in testing medical AI systems in real-world settings, rather than just in the lab or simulations. The breast cancer study, by giving radiologists control over when to use AI, shows it can be a useful tool. With a push to get more evidence like this, lives could be saved, just as after Semmelweis, who is now considered a medical hero.