Driverless cars may be worse at detecting children and people with darker skin, tests on artificial intelligence systems suggest. The researchers who carried out the work say that tighter government regulation is needed and that car-makers must be transparent about the development and testing of these vehicles.
at King’s College London and her colleagues assessed eight AI-based pedestrian detectors used in driverless car research. The overall detection accuracy for adults was almost 20 per cent higher than it was for children, and was 7.5 per cent higher for pedestrians with light skin compared with those with dark skin.
Car-makers don’t release details of their own software, but Zhang says it is likely to have the same problems. “It’s their confidential information and they won’t allow other people to know what models they use, but what we know is that they’re usually built upon the existing open-source models,” she says. “We can be certain that their models must also have similar issues.”
Advertisement
Zhang says part of the problem is that the main open-source collections of pedestrian images, which are used to train AI models, feature more people with light skin than dark skin.
Manufacturers don’t reveal the data they use to train their models, but they may use the same open-source collections used in this experiment, proprietary collections or a combination. Zhang says that more transparency is needed on how commercial models are trained and how they perform, so their safety can be objectively measured.
“We need much better regulations,” says Zhang. “I actually checked the current UK regulations. They just briefly mention the importance of fairness, but it’s very vague.”
Racial bias in AI has been revealed before, including some AIs being less accurate at identifying the faces of Black people than those of white people. But while these problems can block people from accessing vital services or even skew the severity of sentences handed out in court, the repercussions of such bias in driverless car software could be fatal.
91av contacted several leading companies developing autonomous cars, but Tesla, Ford, Toyota and Volkswagen did not respond. Mercedes declined an interview and to provide information about what its own software is based on, but said in a statement that it attaches “great importance to the greatest possible safety for all traffic participants”.
at the University of Oxford says the problems identified in the study need to be fixed before AI systems are used in cars on real roads. However, engineers must make sure their solutions don’t have unintended consequences that harm overall safety, she adds.
“Hypothetically, [efforts to improve safety] could make the car more unsafe because it stops every time a small object is in its view in case it’s a child, and that, of course, can create more accidents in theory,” says Véliz. “One of the lessons of AI ethics is that there’s no easy solution. What’s important is first to identify what the problems are, then to make sure that the company is auditing those algorithms to be as fair as possible, and that the trade-offs are made in a conscious way and in a way that can be defensible and explained.”
arXiv
