91av

People seem to be answering research survey questions randomly on Amazon’s crowdsourcing website. The findings could mean that many academic studies are wrong.

Michael Chmielewski at Southern Methodist University in Texas and Sarah Kucker at Oklahoma State University recently revisited data they had collected on Amazon’s Mechanical Turk (MTurk) platform, a virtual labour marketplace where people are paid to perform short tasks. MTurk is often used to gather survey responses for social science research.

Since 2015, Chmielewski and Kucker had used MTurk to collect data on how a child’s language skills developed depending on their parents’ personalities. When 91av published an article in 2018 claiming automated bots were targeting the site and ruining academic studies, the pair revisited their data and found inconsistencies. But rather than bots ruining their data, it seems humans racing through possible survey answers and not reading the questions were causing the problems.

By performing a statistical analysis on their results, the team found that the responses just weren’t right. “The conclusions were just massively wrong,” says Chmielewski. “Well-established links between neuroticism and depression weren’t there. We were seeing links in the wrong directions. Things that should have been negatively related were now positively related.”

Read more: Genetic privacy attack could reveal DNA secrets from genealogy sites

Chmielewski believes that the low quality of results is down to participants rushing through the answers and not reading them. “You need to be much more careful using MTurk now than people were before,” he says. After screening for inconsistent answers, the pair removed 60 per cent of survey responses in one portion of their study and 38 per cent in another.

“As a regular MTurk user I also had the perception that the data quality was deteriorating in recent times,” says Florian Kunze at the University of Konstanz, Germany. Amazon didn’t respond to a request for comment for this story.

Kristy Milland who does work offered on MTurk wasn’t surprised by the results. “MTurk is a labour platform, not a participant pool,” she says. “We signed up to make money, not to help science.”