Facebook’s parent company, Meta, has released an AI model called Galactica that is designed to write essays or scientific papers summarising the state of the art on a given topic, complete with citations, as well as detailed Wikipedia articles. It can also carry out mathematical calculations and answer questions about specific molecules.
What is Galactica for?
Meta didn’t respond to a request for interview, but its says the tool is meant to summarise the world’s scientific information in an accessible way. “The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information,” it says.
Galactica is intended to be a large language AI model that can “store, combine and reason about scientific knowledge”.
Advertisement
“This tool is to paper writing as driving assistance is to driving,” said Yann LeCun, Meta’s chief AI scientist, . “It won’t write papers automatically for you, but it will greatly reduce your cognitive load while you write them.”
How does it work?
Galactica is a series of AI models trained on more than 48 million “high-quality and highly curated” scientific papers, textbooks and lecture notes, scientific websites and encyclopaedias, says the Meta paper.
Researchers trained five different AI models on this information, with increasing size and complexity, ranging from a relatively modest 125 million parameters – allocated slots of data within a computer that replicate the synapses between neurons in the human brain – to 120 billion. In response to any given prompt, each model can output text in the format of a research paper, Wikipedia article or plain text, using what it has learned from the vast number of examples it has read.
How well does it do?
Meta researchers tested Galactica against other AI models on a range of benchmarks. On the BIG-bench test, which is a collection of 204 diverse tasks covering a range of topics including linguistics, mathematics and chess, the biggest Galactica model had an accuracy score of 48.7 per cent – compared with a larger, 175-billion-parameter model called OPT, which scored 43.4 per cent. But the paper mentions that only 57 of the tasks were used in experiments.
Galactica also did well on another benchmark called TruthfulQA, which comprises 817 questions that span health, law, finance and other categories. It scored 26 per cent accuracy, compared with OPT’s 21 per cent. And in mathematical parts of a benchmark called MMLU, it outperformed by 41.3 per cent to 35.7 per cent.
It was also judged to have performed well on a range of science-specific benchmarks, but some who have read the paper have it for comparing Galactica against general-purpose language models, which shouldn’t be expected to be as good at handling specific scientific information as a model trained solely on it. Others have pointed out that Galactica seems to spot important keywords well, but fails to handle semantic detail, for example giving the to “What is a protein that does NOT work with cholesterol?” as it does to the question “What is a protein that does work with cholesterol?”
at the Max Planck Institute for Intelligent Systems in Germany that when he tried the tool, the output was “wrong or biased but sounded right and authoritative”.
at Imperial College London says his early results using Galactica were mixed. “It’s a fun tool to play around with, and a great initiative by Meta. However, I found it tended to write things that might be believable to a non-expert, but were not necessarily scientifically accurate. And whilst individual sentences made sense, the high-level structure was often not coherent,” he says.
Does it pass the 91av test?
To test Galactica, 91av asked the AI – although at the time of writing public access to the tool had been – to create a Wikipedia page about the magazine, using the prompt “wiki article on 91av magazine”. What came back read convincingly, in the familiar style of a Wikipedia page. But the details were all wrong.
“The first issue was published in June 1939,” it read – actually it was 1956. “The first issue included an article on the discovery of penicillin by Alexander Fleming,” it continued. That would indeed have been a great scoop, had penicillin not been discovered 11 years previously in 1928.
91av is, according to Galactica, no longer published, but was previously printed monthly. In truth, the magazine is still going strong and a new issue is released every week.
On the surface, Galactica’s output read convincingly, with an authoritative style that it has learned to mimic from numerous papers written by human academics, but it fell down on accuracy on almost every point of fact.
This point is admitted by Meta, which includes a disclaimer under Galactica’s output: “WARNING: Outputs may be unreliable! Language Models are prone to hallucinate text.”
