Not All Hallucinations Are Bad: The Constraints and Benefits of Generative AI

Recent Examples of Hallucinations

If we look back at the past year, we can find plentiful examples that highlight the risks of blindly trusting outputs from large language models (LLMs) like ChatGPT. In one case, lawyers used ChatGPT to fabricate legal citations for a court filing, inventing three fictitious case titles that sounded plausible but had no basis in reality. When the opposing counsel could not find these cases in legal databases, the judge fined the lawyers $5,000 for submitting bogus research.^ⅰ

Another incident involved a radio host being defamed when ChatGPT falsely claimed he had embezzled funds, making up details from thin air when prompted about a case summary. The host has now sued OpenAI, the first defamation lawsuit against the company related to its AI model's hallucinated outputs.^ⅱ

While amazing at synthesizing fluent text, LLMs can also confidently generate content that is completely fabricated or factually inaccurate. This is because, despite their fluency, LLMs lack true understanding and rely on pattern recognition to generate plausible-sounding outputs. These examples underscore why AI-generated content cannot be blindly trusted, especially for critical (and/or high-risk) applications. Human oversight and fact-checking are still essential when using these powerful but flawed models.

From fabrication to defamation, we've seen the risks of AI hallucinations. To avoid these landmines, businesses must exercise caution and implement safeguards against blindly accepting AI-generated content as factual without verification.

GenAI: Foundations and Technology

GenAI empowers machines with humanlike creative capabilities. At its core, this artificial imagination is driven by advanced machine learning techniques. A key approach is generative adversarial networks (GANs), which pit a generator AI against a discriminator AI.

In GANs, the generator creates synthetic content that aims to be realistic, while the discriminator tries to distinguish it from real data. This adversarial training improves the generator's ability to produce outputs indistinguishable from reality—but it also allows the machine to veer into surreal hallucinations that go past normal boundaries.

Other foundational techniques include variational autoencoders (VAEs) that iteratively refine synthetic content and autoregressive models that construct narratives step-by-step like human storytellers. The transformer architecture with self-attention mechanisms has been particularly transformative for generative language models, launching the authors of the 2017 paper 'Attention Is All You Need' to near-celebrity status in the technology community.^ⅲ

At a higher level, GenAI leverages deep neural networks that learn complex patterns from vast datasets, essentially modeling the creative process in silica. This involves extensive training reinforced by human feedback to steer the AI towards intended goals.

If we want to demystify GenAI's capability to hallucinate, we must understand this technological foundation. The combination of cutting-edge machine learning and scalable training pipelines empowers machines to venture into realms of imagination previously thought uniquely human - for better or worse.

Understanding the Mystery of Data

In the realm of AI, data serves a similar role to human experiences—it provides the raw material and molds the AI's capabilities. For GenAI models, data acts as both the canvas on which the student paints new content and as the tutor who guides what the student creates.

This leads to a phenomenon called "generative anthropomorphism" - the attribution of human-like qualities to AI systems as they learn to mimic and extend human creativity from the data. One of the most famous examples was Google engineer Blake Lemoine, who claimed that its chatbot LaMDA was a self-aware person.^ⅳ

The biggest risk arises when an AI's output matches user expectations superficially while lacking grounding in verified facts. In such cases, users may readily accept fabricated content as truthful, inadvertently amplifying misinformation. Distinguishing between plausible but unfounded outputs versus substantiated results will become a critical capability.

In some cases, generative anthropomorphism could manifest as AI-generated images depicting mythical scenes, text crafted into narratives that defy logic and reality, or music by composers who never existed. This tendency to generalize and stretch boundaries based on detected patterns in training data is a key driver of hallucination.

On one hand, this generalization makes it possible for GenAI models to create innovative and unpredictable outputs that can spark creativity and lead to new offerings. However, it also risks disseminating misinformation, biases, and harmful stereotypes present in the training data that gets amplified during the generative process.

As we explore how training data influences this phenomenon, we must strive to capitalize on its creative potential while instituting guardrails for ethical and responsible development.

Types of Hallucination

Hallucinations in GenAI can manifest in various forms depending on the model architecture and intended use case. Some common types include:

• Visual Hallucinations - Image generation models may create visuals depicting non-existent objects, scenes or patterns ranging from abstract art to completely fabricated creatures.
• Textual Hallucinations - Language models can generate sentences or paragraphs containing fictional information, invented events, or factual inaccuracies that have no basis in reality.
• Content Expansion - Generative models sometimes diverge by producing extraneous content beyond what is present in the input data, such as adding peripheral details to images or extending narratives far beyond the original context.
• Inferential Errors - When tasked with language understanding, LLMs may draw flawed inferences or conclusions not substantiated by the provided information, resulting in responses that misrepresent the actual context.
• Bias Amplification - Models can hallucinate outputs that reflect and amplify societal biases inherent in their training data, perpetuating stereotypes, discrimination, and unethical perspectives.
• Context Hallucinations - Text generated by language models may seem superficially relevant but contain factual inaccuracies that deviate from the true context.

The severity and impact of these hallucination types can vary across different GenAI architectures. Mitigating and controlling hallucinated outputs is crucial to ensuring the technology's safety, reliability, and alignment with real-world objectives as research in this space progresses.

Hallucination in Action - In Different Sectors

The phenomenon of hallucination with GenAI is not constrained to any single domain - though in detail-oriented areas such as law, its impact can be felt more keenly. However, just as human creativity transcends boundaries, the artificial imagination of generative models can manifest across diverse industry sectors in unique and transformative ways.

Imagine GenAI being applied in telecommunications, where hallucinated language could spark new approaches to customer communication. Or in business process outsourcing, where hallucinated process flows could optimize operations. Realizing this added value from AI hallucinations, however, requires us to carefully evaluate these outputs against technical feasibility, industry standards, and operational objectives.

A key risk is that GenAI models trained primarily on open web data may hallucinate outputs incompatible with an organization's proprietary data, processes, and policies when deployed in that context. Failing to root the AI's responses in relevant domain knowledge increases the likelihood of detached hallucinations.

GenAI hallucinations have the potential to spark creative solutions and uncover novel possibilities across sectors, from manufacturing to healthcare to finance. However, using this powerful capability responsibly means striking a balance between entrepreneurial innovation and pragmatism that varies from industry to industry.

As we explore GenAI deployment in specialized domains, we must carefully evaluate each hallucinated idea's business utility while mitigating risks around caveats or blindspots emerging from the model's disconnect from real-world constraints. Simultaneously, we must remain open to paradigm shifts that seemingly implausible hallucinations may provoke.

Leveraging Hallucination: Transforming Drawbacks into Advantages

While hallucinations are often viewed as an undesirable quirk of GenAI, understanding their root causes can instead turn them into beneficial applications. One promising use is idea generation—providing seed prompts and allowing language models to hallucinate freely can spark novel concepts for products, advertisements, stories, or other creative works that humans may take longer to conceive of by themselves.

Hallucination can also aid synthetic data generation. AI-generated synthetic data could augment real-world datasets for training machine learning models, testing systems, or simulating complex environments. Models can hallucinate corner cases that are theoretically possible, but highly unlikely, to stress-test AI robustness.

Generative art represents another frontier, with AI hallucinations enabling new forms of digital media, music, films, and interactive experiences unconstrained by human creators' assumptions.

However, until verified, it's critical to view AI-generated content with healthy skepticism. Hallucinations can propagate inaccuracies, so human oversight is essential before incorporating speculative outputs into real-world applications. As models continue advancing, we'll likely see hallucinations leveraged in increasingly innovative yet responsible ways.

Mitigating Hallucination in GenAI

Despite their creative potential, GenAI models' tendency to hallucinate poses risks when outputs drastically deviate from the intended domain. Let's examine mitigation strategies that organisations can use to keep the AI's creative process grounded in reality and aligned with practical applications.

Common mitigation techniques include carefully crafting prompts to steer models toward realistic outputs, training on high-quality datasets to reduce hallucinations, employing model architectures like GANs designed for realism, incorporating contextual information anchors like Wikipedia (RAG), filtering outputs for potential hallucinations, and maintaining human oversight to validate AI-generated content aligns with real-world facts and objectives.

Promising research advances have been made in automatically detecting hallucinations in model outputs and in algorithms to correct such instances. However, no single silver bullet exists—the most effective approach combines multiple mitigation strategies. It's also crucial to be cognizant of GenAI's current limitations and employ the technology with judicious care while this field progresses.

Ethical and Societal Implications of Hallucination in AI

A key issue ethical implication of hallucinations is the blurred line between fact and fiction when models confidently generate fabricated content detached from reality. This threatens to undermine truth, sow misinformation that impedes informed decision-making, and erode public trust.

There are also concerns about AI amplifying historical biases and negative societal stereotypes present in its training data through hallucinated outputs. Perpetuating discrimination and unethical viewpoints represents a harmful pitfall. Privacy rights are another angle—hallucinating personal details or identifiable characteristics without consent raises issues around data usage and potential harm.

In the societal sphere, AI hallucinations could fuel the spread of misinformation and disinformation, degrading civic discourse. They may also enable nefarious exploitation, such as defamatory deepfakes and criminal fraud vectors, as a cybersecurity risk.

As GenAI's capabilities grow, AI developers and enterprises have a responsibility to uphold transparency, accountability, ethical guidelines, and responsible practices aligned with their core values. Proactive governance over generative models will be critical in harnessing their innovative benefits while minimizing the negative societal ramifications.

Charting the Future with GenAI - NTT DATA's Path Ahead

Our exploration of GenAI hallucinations has uncovered the immense potential that firms can unlock from this technology's trajectory and the profound responsibility they share in shaping it. The creative capacities exemplified through hallucinations could open doors across sectors, reimagining customer communications, optimizing operations, and driving future breakthroughs we can scarcely conceive today.

However, the path will require organizations to carefully navigate the challenges before them. Hallucination threatens to disseminate misinformation and undermine universal recognition of the truth. If left unchecked, it also risks perpetuating biases and discrimination. Privacy rights and consent emerge as ethical minefields as AI systems hallucinate personal data, and societal impacts - such as eroded civic discourse and abuse for fraud - loom as existential risks without responsible guardrails.

As an industry leader, NTT DATA is committed to upholding transparency, ethics, and accountability in developing GenAI capabilities centered on our core values and customer needs. To ensure hallucinations remain a productive creative force rather than a runaway liability, we are exploring mitigation strategies, including prompt engineering, data curation, model innovations, output filtering, and human oversight.

The future promises remarkable possibilities - but to unlock GenAI's full potential, the journey ahead will require innovative practices, multi-stakeholder collaboration, and an unwavering commitment to deploying the technology responsibly and ethically as co-creators of a new era where the bounds of machine imagination are unbounded.

If you'd like more information on harnessing the power of GenAI hallucinations in a responsible framework, you can discover more in our whitepaper.