Google DeepMind recently introduced SAFE (Search-Augmented Factuality Evaluator). This system was developed by DeepMind to improve fact-checking in large language models (LLM). This is a method that breaks down answers into individual facts and then checks each fact separately using Google Search.
Even more powerful AI for DeepMind
Created by Google, DeepMind is positioned as one of the leading companies in the AI market, with varied applications in many areas such as health, energy, and transport.
With the introduction of the SAFE system, it gains precision and reliability to process the information it receives even more efficiently.
SAFE: How does it work?
SAFE employs a distinct methodology that involves breaking down extensive textual responses into singular facts.
Each of these facts is then subject to rigorous verification via queries carried out on Google Search.
This approach allows for autonomous and accurate assessment of information, thereby expanding the horizons of factuality in AI-generated responses.
“Superhuman” results?
In comparative experiments, SAFE demonstrated notable agreement with human assessments, occurring 72% of the time.
Additionally, in a series of 100 discrepancies between human and SAFE assessments, the system was correct 76% of the time.
These results indicate not only the effectiveness of SAFE as a fact-checking system but also its potential for cost-effectiveness, given that it is 20 times less expensive than traditional human methods.
However, the attribution of the term “superhuman” to SAFE has provoked academic debate.
Researchers such as Garcy Marcus have expressed reservations, suggesting that this terminology could lead to an overestimation of the system's true capabilities.
According to Marcus, to earn this designation, SAFE should be evaluated against a broader range of professional human fact-checkers rather than crowdsourced contributors.
Sharing SAFE code on GitHub
Google DeepMind has made the source code for SAFE available on GitHub. This initiative would allow the scientific community to access, use, and contribute to the improvement of SAFE.
The GitHub repository includes various essentials like LongFact, a set of 2,280 prompts requiring long responses, as well as the automated SAFE evaluator itself.
For more details, the SAFE code is available on GitHub.