AI is no better at predicting whether someone has covid-19 than a simple questionnaire asking people for self-reported symptoms.
Many earlier studies had suggested that AI systems could detect certain sounds in people’s coughs and voices that indicate a covid-19 infection, with reported accuracies as high as 99 per cent. But by testing AIs on a comprehensive covid-19 dataset from the UK National Health Service, researchers have shown how supposedly impressive AI performance mostly relies on detecting demographic factors associated with likelihood of covid-19 infection.
“We show that the meat of the AI classification performances was due to confounding factors,” says at Imperial College London.
Advertisement
“We collected the largest data set to date of people with and without covid-19 providing four [sounds],” he says. The study tested three AI systems – including one of the latest audio-classifying AIs – on audio recordings from more than 67,000 PCR-tested volunteers from the UK, including 23,514 people who tested positive for covid-19.
Those audio recordings included four different types of sounds, says Coppock: “Single cough, three coughs, a sentence of them saying ‘I love nothing more than an afternoon cream tea’ – very British of us – and the fourth one was exhalation.”
Anonymised health data from the volunteers’ National Health Service records allowed researchers to check whether the AIs are simply predicting covid-19 infection based on 16 confounding factors – such as gender, ethnicity and location. By matching the profiles of people who had tested positive with similar profiles of people who had tested negative, the researchers aimed to remove any AI predictive power based on the shared demographic factors.
The AI saw a “massive drop-off in performance” from 90 per cent to 60 per cent when accounting for the confounding factors, says Coppock. Based on additional experiments, the researchers suspect that the remaining 10 per cent of AI performance beyond flip-of-the-coin accuracy is also due to undetected biases.
The bottom line is that the AI was outperformed by a simple symptom checker questionnaire that would be both cheaper and easier to deploy, says Coppock.
“This work is important because if we don’t understand how an audio-based AI screening of covid-19 might work in real-life conditions, then we will overestimate the value of a purely technical solution in deployment,” says at the Massachusetts Institute of Technology.
Reference:
