Beware of Botshit: How Researchers Hope to Fix AI’s BS Issue

Why Trust Techopedia

Generative AI’s problems handling the truth just won’t go away. Error rates are stuck where they were a year ago, and as more people use the technology, more hallucinations pop up. That’s a worry for CIOs and CDOs charged with leading their company’s AI push. MVPs look less viable, while promising use cases sit in limbo.

The issue has attracted the attention of the US military establishment, and there’s a growing body of academic research devoted to tackling machine-generated ‘bullshit.’

We look at the efforts being made to understand AI’s epistemic risks – and fix them.

Key Takeaways

  • AI hallucinations just keep coming, and some are starting to question if they’re a feature or a bug.
  • That puts a lot of money and resources devoted to new AI projects in jeopardy.
  • A spate of new research examines the issue and recommends solutions. Meanwhile, DARPA is accepting submissions for a new program designed to instill trust in AI and ensure its outputs are legit.
  • Researchers have proposed building in a ‘limitation awareness’ functionality, which would stop AI applications from making recommendations when they do not have sufficient data.

Deep-Rooted Drivel Generation

Are hallucinations really such a big deal, or have a few high-profile glitches warped perceptions?

GenAI applications are churning out an astounding amount of content. In September 2023, Amazon felt compelled to limit the number of books an author could post to three in a single day. That came less than a week after a new requirement compelling authors to tell the company when their works are AI-generated.

Both measures were prompted by the removal of books attributed to a well-known author, which turned out to be AI fakes. A month earlier, Amazon was forced to delete AI-written titles about mushroom foraging that may have offered dangerous advice.


The applications used to create that content have variable rates of accuracy. Vectara’s hallucination leaderboard on GitHub currently ranks GPT 4 Turbo on top with a 2.5% hallucination rate. The worst performer at the time of writing was Apple’s OpenELM-3B-Instruct, at 22.4%. Most AI models on the list generate made-up facts at rates of between 4.5 and 10%.

With estimates of the number of daily API requests processed by ChatGPT alone reaching into the trillions, even a 2.5% rate could result in an astounding amount of misinformation.

Bullshit as a Technical Concept

While AI’s penchant for making stuff up may be well known, its roots haven’t been broadly understood. That’s starting to change.

One of the GenAI revolution’s unexpected outcomes is a growing body of academic literature devoted to ‘bullshit.’ Not the sweary kind, but rather the technical concept developed by Princeton University professor Harry Frankfurt in 2005 ‘for comprehending, recognizing, acting on, and stopping forms of communication that have no basis in fact.’

A team of researchers from Simon Fraser University, The University of Alberta, and the City University of London applied Frankfurt’s BS model to GenAI and found something new.

Their paper, Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots, says, “Chatbots can produce coherent sounding but inaccurate or fabricated content.” When humans use this untruthful content for tasks, it becomes ‘botshit.’

Bullshit vs. Botshit

Bullshit Botshit
Definition Human-generated content that has no regard for the truth, which a human then applies to communication and decision-making tasks Chatbot-generated content that is not grounded in truth and is then uncritically used by a human for communication and decision-making tasks.
Types Pseudo-profound bullshit: statements that seem deep and meaningful

Persuasive bullshit: statements that aim to impress or persuade

Evasive bullshit: statements that strategically circumvent the truth

Social bullshit: statements that tease, exaggerate, joke, or troll

Intrinsic botshit: the human application of a chatbot response that contradicts the chatbot’s training data

Extrinsic botshit: the human application of a chatbot response that cannot be verified as true or false by the chatbot’s training data

Insights Humans are more likely to generate and use bullshit:

  • The more unintelligent, dishonest, and insincere they are
  • The expectations for them to have an opinion are high, and they expect to get away with it
  • If their bosses frequently spout bullshit
Chatbots are more likely to generate hallucinations for humans to use and transform into botshit when there are:

  • Data collection, preprocessing and tokenization problems that limit factual knowledge alignment between the training data and the desired response
  • Ambiguous prompts that misdirect the chatbot
  • Problems with the training and modeling choices of the LLM transformer
  • Issues with fine-tuning efforts based on uncertainty around grounded truth

Source: McCarthy et al.

Ian McCarthy, Professor of Technology and Operations Management at Simon Fraser University’s Beedie School of Business, told Techopedia that LLMs shouldn’t be seen as machines designed to lie.

Instead, they are “prediction machines that sometimes produce contaminated responses. The risk and harms that come with LLMs is when their nonsensical content is trusted and used by humans and organizations.”

“Chatbots do not ‘know’ the meaning of their responses, so when they generate hallucinations, they are also not lying,” the paper says. “In contrast, the human approach to generating truthful knowledge relies on reflexivity and judgment.”

A Problem of Verification

Another paper by Glasgow University researchers is more emphatic, calling bullshit since it’s “a more useful and more accurate way of predicting and discussing the behavior of (GenAI) systems.” It also captures a more fundamental truth that “these models are in an important way indifferent to the truth of their outputs.”

Professor Xiaowei Huang, Director of the Trustworthy Autonomous CPS Lab at Liverpool University, told Techopedia that the underlying problem with GenAI is that its built-in neural network-based classifiers “are designed to generate new content. Whenever new content is created, its truthfulness must be verified. In this sense, hallucinations are inherent to GenAI.”

Verification and Validation Techniques for Evaluating AI Outputs

Huang led a team of researchers studying hallucinations and noted the increasing difficulty of developing rigorous engineering methods for GenAI similar to those used in safety-critical systems like cars and airplanes.

“While the community has attempted to adapt these techniques for neural network-based classifiers with some success, GenAI introduces new challenges that complicate the process,” Huang says. “These include the unavailability of training data, open-source issues, and nondeterministic outputs – meaning the same input may produce different outputs across multiple executions of the GenAI.”

The Hallucinations Hall of Shame

What can happen when chatbots go awry? The flaws and complications that cause GenAI to make dodgy inferences have harmed reputations and threatened revenues.

  • In February 2024, Air Canada lost a tribunal case and was forced to honor a steeply discounted fare that its customer-support chatbot had offered a passenger in error.
  • In May 2024, Google was compelled to repair its recently launched text summary feature at the top of search results when it told some users that they could safely eat rocks.
  • In June 2023, two US lawyers were fined $5,000 after one of them admitted to using ChatGPT to write a court filing. The AI had even appended fake citations to the submission for cases that never existed.
  • ChatGPT invented a sexual harassment case and named a real law professor as the accused, citing a made-up Washington Post article as evidence for its assertions.
  • Students at Texas A&M University were failed en masse by their professor after ChatGPT claimed it had written their papers (something that hadn’t actually happened).
  • KPMG complained to the Australian government when a committee revealed it had not verified the accuracy of a Google Bard document, which falsely accused companies of non-existent scandals and named people who had never worked for those companies.
  • In January 2023, researchers at Northwestern University found that fake research papers and scientific abstracts created with ChatGPT tricked scientists into believing they were real close to a third of the time.
  • In February 2023, Google Bard made a factual error in its first live public demonstration, claiming that the James Webb Space Telescope took the first pictures of a planet outside the solar system. This actually happened in 2004, well before the telescope’s launch.
  • In September 2023, an article published in Nature claimed that Google Deepmind had “helped discover 2.2 million new crystals.”Academics quickly rubbished the claim’s “scant evidence.”
  • It’s more than just words. AI image generation tools like Google Gemini have created glaringly obvious historical inaccuracies, including representations of people of color dressed in Nazi-era uniforms.

From Dream to Nightmare?

Left unchecked, GenAI’s reliability issues could intensify. ChatGPT, for example, has already subsumed the entire internet into its training data set. Now, it’s starting to train on AI-generated content.

Some predict this could lead to ‘Habsburg AI’, where GenAI applications are trained so extensively on the outputs of other AI tools that they become inbred and febrile, exhibiting strange new behaviors that undermine faith in their operational capability.

A study by British and Canadian researchers warns that this could turn out to be generative AI’s ‘recursive curse,’ one with the potential to collapse AI models entirely.

Imagine a skyscraper-sized photocopier that churns out tons of photocopies, then scans those photocopies to create progressively more diluted photocopies, continuously on repeat, ad infinitum.

Generative AI would become a massive ouroboros, progressively collapsing inward as it consumes itself.

How to Fix It

As academics delve deeper under GenAI’s bonnet to understand its flaws, they are also working out solutions.

Simon Fraser’s McCarthy says he and his fellow researchers have developed a risk-management framework that orients generative AI chatbot use based on two dimensions:

  1. Response Veracity Importance – how important it is that a chatbot’s response to a task is correct.
  2. Response Veracity Verifiability – how easy it is to verify the truthfulness of a chatbot response.

“Work involving chatbots will vary according to these dimensions,” he says. “For companies to address the risk of botshit, they must recognize which of four modes of chatbot work – authenticated, autonomous, automated, and augmented – they are dealing with.”

They must also assess botshit-related risks like ignorance, miscalibration, routinization, and black boxing that come with each mode of chatbot work, he adds.

The different modes require different fact-checking modules within the LLM and different levels of critical thinking for their use.

Liverpool’s Prof Huang recommends that companies develop the ability for GenAI to be aware of its own limitations.

“For example, when GenAI operates in specialized areas such as pharmaceutical or biomedical, where it lacks dedicated knowledge, it should refrain from offering recommendations. And when it is provided with domain-specific knowledge or databases, it should also be able to quantify any improvement of its confidence in the domain.”

For now, he says, tightening up processes of verification and validation remains the most effective way to firm up the safety and trustworthiness of GenAI.

The Military Weighs In

Outside of academia, a new initiative from the US government’s Defense Advanced Research Projects (DARPA) indicates how seriously AI’s reliability issues are being taken.

Under its Artificial Intelligence Quantified (AIQ) program, the agency is sponsoring a series of research projects looking at how AI can be engineered to guarantee trustworthy outcomes. Darpa says it wants to “explore the hypothesis that mathematical foundations, combined with advances in measurement and modeling, will guarantee an AI system’s capabilities, when they will or will not manifest, and why.”

The program, which will run in 18-month phases, will consider AI capabilities at three levels:

  1. Specific Problem Level – mapping individual inputs and outputs.
  2. Classes of Problem Level – looking at collections of inputs and associated outputs.
  3. Natural Class Level – looking at which inputs behave as expected with respect to outputs.

Dr. Patrick Shafto, DARPA’s Artificial Intelligence Quantified (AIQ) program manager, said:

“AI has achieved near human-level performance in text generation, game playing, and such, which raises the prospect of widespread integration with human partners in the military and society. And at the most general level, we’re interested in determining how to ensure AI systems will have the properties needed to solve various problems.”

The Bottom Line

Given the gargantuan sums being invested in the Generative AI space, the technology’s persistent fib problem continues to worry investors, customers, CIOs, and cybersecurity experts.

Happily, it’s also generating fascinating research and live experiments that could lead to a better understanding of how AI’s algorithms and inference engines can be improved.


What is ‘botshit’?

How to fix AI hallucinations?

What is DARPA doing with AI?


Related Reading

Related Terms

Mark De Wolf
Technology Journalist
Mark De Wolf
Technology Journalist

Mark is a freelance tech journalist covering software, cybersecurity, and SaaS. His work has appeared in Dow Jones, The Telegraph, SC Magazine, Strategy, InfoWorld, Redshift, and The Startup. He graduated from the Ryerson University School of Journalism with honors where he studied under senior reporters from The New York Times, BBC, and Toronto Star, and paid his way through uni as a jobbing advertising copywriter. In addition, Mark has been an external communications advisor for tech startups and scale-ups, supporting them from launch to successful exit. Success stories include SignRequest (acquired by Box), Zeigo (acquired by Schneider Electric), Prevero (acquired…