Beware of Botshit: How Researchers Hope to Fix AI’s BS Issue

Generative AI’s problems handling the truth just won’t go away. Error rates are stuck where they were a year ago, and as more people use the technology, more hallucinations pop up. That’s a worry for CIOs and CDOs charged with leading their company’s AI push. MVPs look less viable, while promising use cases sit in limbo.

The issue has attracted the attention of the US military establishment, and there’s a growing body of academic research devoted to tackling machine-generated ‘bullshit.’

We look at the efforts being made to understand AI’s epistemic risks – and fix them.

Table of Contents Table of Contents

1. Key Takeaways
2. Deep-Rooted Drivel Generation
3. Bullshit as a Technical Concept
4. A Problem of Verification
5. The Hallucinations Hall of Shame
6. From Dream to Nightmare?
7. How to Fix It
8. The Military Weighs In
9. The Bottom Line
10. FAQs
11. References

Show Full Guide

Table of Contents

1. Key Takeaways
2. Deep-Rooted Drivel Generation
3. Bullshit as a Technical Concept
Show Full Guide
4. A Problem of Verification
5. The Hallucinations Hall of Shame
6. From Dream to Nightmare?
7. How to Fix It
8. The Military Weighs In
9. The Bottom Line
10. FAQs
11. References

Deep-Rooted Drivel Generation

Are hallucinations really such a big deal, or have a few high-profile glitches warped perceptions?

GenAI applications are churning out an astounding amount of content. In September 2023, Amazon felt compelled to limit the number of books an author could post to three in a single day. That came less than a week after a new requirement compelling authors to tell the company when their works are AI-generated.

Both measures were prompted by the removal of books attributed to a well-known author, which turned out to be AI fakes. A month earlier, Amazon was forced to delete AI-written titles about mushroom foraging that may have offered dangerous advice.

The web’s becoming an anaerobic botshit lagoonand the quantum of human “content” in any core sample is dwindling to homeopathic levels. Even sources considered high-quality, from Cnet articles to legal briefs, are contaminated with botshit:https://t.co/2P1sZfVJeV

16/

— AFK Cory Doctorow NONCONSENSUAL BLUE TICK (@doctorow) March 14, 2024

The applications used to create that content have variable rates of accuracy. Vectara’s hallucination leaderboard on GitHub currently ranks GPT 4 Turbo on top with a 2.5% hallucination rate. The worst performer at the time of writing was Apple’s OpenELM-3B-Instruct, at 22.4%. Most AI models on the list generate made-up facts at rates of between 4.5 and 10%.

With estimates of the number of daily API requests processed by ChatGPT alone reaching into the trillions, even a 2.5% rate could result in an astounding amount of misinformation.

Bullshit as a Technical Concept

While AI’s penchant for making stuff up may be well known, its roots haven’t been broadly understood. That’s starting to change.

One of the GenAI revolution’s unexpected outcomes is a growing body of academic literature devoted to ‘bullshit.’ Not the sweary kind, but rather the technical concept developed by Princeton University professor Harry Frankfurt in 2005 ‘for comprehending, recognizing, acting on, and stopping forms of communication that have no basis in fact.’

A team of researchers from Simon Fraser University, The University of Alberta, and the City University of London applied Frankfurt’s BS model to GenAI and found something new.

Their paper, Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots, says, “Chatbots can produce coherent sounding but inaccurate or fabricated content.” When humans use this untruthful content for tasks, it becomes ‘botshit.’

Bullshit vs. Botshit

Source: McCarthy et al.

Ian McCarthy, Professor of Technology and Operations Management at Simon Fraser University’s Beedie School of Business, told Techopedia that LLMs shouldn’t be seen as machines designed to lie.

Instead, they are “prediction machines that sometimes produce contaminated responses. The risk and harms that come with LLMs is when their nonsensical content is trusted and used by humans and organizations.”

“Chatbots do not ‘know’ the meaning of their responses, so when they generate hallucinations, they are also not lying,” the paper says. “In contrast, the human approach to generating truthful knowledge relies on reflexivity and judgment.”

A Problem of Verification

Another paper by Glasgow University researchers is more emphatic, calling bullshit since it’s “a more useful and more accurate way of predicting and discussing the behavior of (GenAI) systems.” It also captures a more fundamental truth that “these models are in an important way indifferent to the truth of their outputs.”

Professor Xiaowei Huang, Director of the Trustworthy Autonomous CPS Lab at Liverpool University, told Techopedia that the underlying problem with GenAI is that its built-in neural network-based classifiers “are designed to generate new content. Whenever new content is created, its truthfulness must be verified. In this sense, hallucinations are inherent to GenAI.”

Huang led a team of researchers studying hallucinations and noted the increasing difficulty of developing rigorous engineering methods for GenAI similar to those used in safety-critical systems like cars and airplanes.

“While the community has attempted to adapt these techniques for neural network-based classifiers with some success, GenAI introduces new challenges that complicate the process,” Huang says. “These include the unavailability of training data, open-source issues, and nondeterministic outputs – meaning the same input may produce different outputs across multiple executions of the GenAI.”

The Hallucinations Hall of Shame

What can happen when chatbots go awry? The flaws and complications that cause GenAI to make dodgy inferences have harmed reputations and threatened revenues.

• In February 2024, Air Canada lost a tribunal case and was forced to honor a steeply discounted fare that its customer-support chatbot had offered a passenger in error.
• In May 2024, Google was compelled to repair its recently launched text summary feature at the top of search results when it told some users that they could safely eat rocks.
• In June 2023, two US lawyers were fined $5,000 after one of them admitted to using ChatGPT to write a court filing. The AI had even appended fake citations to the submission for cases that never existed.
• ChatGPT invented a sexual harassment case and named a real law professor as the accused, citing a made-up Washington Post article as evidence for its assertions.
• Students at Texas A&M University were failed en masse by their professor after ChatGPT claimed it had written their papers (something that hadn’t actually happened).
• KPMG complained to the Australian government when a committee revealed it had not verified the accuracy of a Google Bard document, which falsely accused companies of non-existent scandals and named people who had never worked for those companies.
• In January 2023, researchers at Northwestern University found that fake research papers and scientific abstracts created with ChatGPT tricked scientists into believing they were real close to a third of the time.
• In February 2023, Google Bard made a factual error in its first live public demonstration, claiming that the James Webb Space Telescope took the first pictures of a planet outside the solar system. This actually happened in 2004, well before the telescope’s launch.
• In September 2023, an article published in Nature claimed that Google Deepmind had “helped discover 2.2 million new crystals.”Academics quickly rubbished the claim’s “scant evidence.”
• It’s more than just words. AI image generation tools like Google Gemini have created glaringly obvious historical inaccuracies, including representations of people of color dressed in Nazi-era uniforms.

From Dream to Nightmare?

Left unchecked, GenAI’s reliability issues could intensify. ChatGPT, for example, has already subsumed the entire internet into its training data set. Now, it’s starting to train on AI-generated content.

I coined a term on @machinekillspod that I feel like needs its own essay: Habsburg AI – a system that is so heavily trained on the outputs of other generative AI's that it becomes an inbred mutant, likely with exaggerated, grotesque features. It joins the lineage of Potemkin AI.

— Jathan Sadowski (@jathansadowski) February 13, 2023

A study by British and Canadian researchers warns that this could turn out to be generative AI’s ‘recursive curse,’ one with the potential to collapse AI models entirely.

Imagine a skyscraper-sized photocopier that churns out tons of photocopies, then scans those photocopies to create progressively more diluted photocopies, continuously on repeat, ad infinitum.

Generative AI would become a massive ouroboros, progressively collapsing inward as it consumes itself.

How to Fix It

As academics delve deeper under GenAI’s bonnet to understand its flaws, they are also working out solutions.

Simon Fraser’s McCarthy says he and his fellow researchers have developed a risk-management framework that orients generative AI chatbot use based on two dimensions:

1. Response Veracity Importance – how important it is that a chatbot’s response to a task is correct.
2. Response Veracity Verifiability – how easy it is to verify the truthfulness of a chatbot response.

“Work involving chatbots will vary according to these dimensions,” he says. “For companies to address the risk of botshit, they must recognize which of four modes of chatbot work – authenticated, autonomous, automated, and augmented – they are dealing with.”

They must also assess botshit-related risks like ignorance, miscalibration, routinization, and black boxing that come with each mode of chatbot work, he adds.

The different modes require different fact-checking modules within the LLM and different levels of critical thinking for their use.

Liverpool’s Prof Huang recommends that companies develop the ability for GenAI to be aware of its own limitations.

“For example, when GenAI operates in specialized areas such as pharmaceutical or biomedical, where it lacks dedicated knowledge, it should refrain from offering recommendations. And when it is provided with domain-specific knowledge or databases, it should also be able to quantify any improvement of its confidence in the domain.”

For now, he says, tightening up processes of verification and validation remains the most effective way to firm up the safety and trustworthiness of GenAI.

The Military Weighs In

Outside of academia, a new initiative from the US government’s Defense Advanced Research Projects (DARPA) indicates how seriously AI’s reliability issues are being taken.

Under its Artificial Intelligence Quantified (AIQ) program, the agency is sponsoring a series of research projects looking at how AI can be engineered to guarantee trustworthy outcomes. Darpa says it wants to “explore the hypothesis that mathematical foundations, combined with advances in measurement and modeling, will guarantee an AI system’s capabilities, when they will or will not manifest, and why.”

The program, which will run in 18-month phases, will consider AI capabilities at three levels:

Dr. Patrick Shafto, DARPA’s Artificial Intelligence Quantified (AIQ) program manager, said:

“AI has achieved near human-level performance in text generation, game playing, and such, which raises the prospect of widespread integration with human partners in the military and society. And at the most general level, we’re interested in determining how to ensure AI systems will have the properties needed to solve various problems.”

The Bottom Line

Given the gargantuan sums being invested in the Generative AI space, the technology’s persistent fib problem continues to worry investors, customers, CIOs, and cybersecurity experts.

Happily, it’s also generating fascinating research and live experiments that could lead to a better understanding of how AI’s algorithms and inference engines can be improved.

FAQs