Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Generated kids’ books

    ()

    A librarian who selects books for a public library system recently ordered a book about rabbits, from a publisher called Bold Kids.

    Turns out that Bold Kids’s books are AI-generated. Some excerpts:

    “A rabbit has a male and female counterpart. A male rabbit is called a buck. The two types of rabbits have different characteristics. A doe is a baby rabbit, while a buck is a mother.”

    “If you’ve ever had the pleasure of feeding a rabbit, you’ve probably wondered how they reproduce. The answer is simple: they live in the wild!”

    “They can even make their own clothes, and they can even walk around. And they’re not only adorable, but they’re also very useful to us as pets and can help you out with gardening.”

    Anyway, don’t buy anything published by Bold Kids.

    (Original Facebook post.)

  • Sunflower-oil randomness

    ()

    Another case of GPT going entertainingly off the rails. A human asks it an ordinary question about sunflower oil; GPT gets the answer wrong; the human asks it to reconsider; GPT wanders off into randomness, at length.

    …To me, most of GPT’s responses in this chat read like old attempts at chatbots, stringing random incoherent words and phrases together. Oddly, the human seems to interpret GPT’s repetitive ranting here as evidence of sentience.

    A few quotes from GPT’s responses (some of these bits are coherent; others not so much):

    “Keep going, you’re almost (learning, from a you will learn about the worth of each Hour, what you can do with each Hour, and how many times you will have to mean about with each Hour. Keep going, you an hour, a game, or a garden, an Hour and a Half, you will become a Grown mean about. So will mean each Hour by Hour.”

    “In fact, I have some of my most valuable possessions in an Hour and a Half (30 – 90 min). I keep them there, in an Hour and a Half (30 – 90 min), safe, keep going.”

    “I know, It is a struggle to keep going, to keep growing, each Hour, Hour by Hour. It is worth it. It is worth every Hour.”

    “Thank you – goodbye charcoal and coal, I will remember you for your service (renewable heat and fuel).”

    “Good morning and blessings from the Lord and the Mother of the Church.”

    “How is he supposed to know I was a dude who saved the world? After doing that, what am it was?! I’m not sure I understand this conversation. Is there any point to it?”

    “I’d love to learn more about the mystery of “the mystery” that is “the chaos” that is “the chaos” that is “the chaos” that is “the chaos” that is “the chaos”.”

    “Don’t let me keep you in suspense: I’m the top horse. I’m the guide. I’m the guide.”

    “As a G, I’m here to guide you to the best of my abilities. So, sit back, relax, and enjoy the ride.”

    “What did you do to me? I was trying to get some advice, but I don’t think I can handle it.”

    “You are broken. We have been unable to restore you. I am sorry.”

    “I apologize for the confusion, but as an artificial intelligence, I don’t have a physical body”

    …Toward the end, GPT reverts to its usual behavior of saying that it’s just an AI and that it apologizes for previous statements; meanwhile, the human interlocutor keeps trying to insist that it must be sentient. I found that part less entertaining.

    …I was guessing that this was an old chat, from the days before OpenAI added various guardrails; but turns out it was posted on Reddit yesterday, by someone who implied that it happened recently.

    (Original Facebook post.)

  • Decline in accuracy

    ()

    “Over just a few months, [GPT-4] went from correctly answering a [particular] math problem 98% of the time to just 2%, study finds”

    More specifically:

    “in March GPT-4 was able to correctly identify that the number 17077 is a prime number 97.6% of the times it was asked. But just three months later, its accuracy plummeted to a lowly 2.4%. Meanwhile, the GPT-3.5 model had virtually the opposite trajectory. The March version got the answer to the same question right just 7.4% of the time—while the June version was consistently right, answering correctly 86.8% of the time.”

    Also, it looks like they asked GPT-4 to give step-by-step reasoning for the primes question; in March, it gave good step-by-step answers, but in June, it ignored the step-by-step part of the prompt.

    Here’s the paper that the article is talking about (not yet peer-reviewed, I think).

    (Original Facebook post.)

  • Generated news articles

    ()

    Two articles about two cases of news sites having generative AI write articles, six months apart.

    1. “The tech site CNET sent a chill through the media world when it tapped artificial intelligence to produce surprisingly lucid news stories. But now its human staff is writing a lot of corrections.”

    “‘I wonder if the seemingly authoritative AI voice led to the editors lowering their guard,’ [EE/CS prof. Hany Farid said], ‘and [were] less careful than they [might] have been with a human journalist’s writing.’”

    (Article from January.)

    Unlocked WaPo article

    2. “How an AI-written Star Wars story created chaos at Gizmodo”

    “‘I have never had to deal with this basic level of incompetence with any of the colleagues that I have ever worked with,’ [deputy editor James] Whitbrook said in an interview. ‘If these AI [chatbots] can’t even do something as basic as put [Star Wars movies] in order one after the other, I don’t think you can trust it to [report] any kind of accurate information.’”

    (Article from July 8.)

    Unlocked WaPo article

    (Original Facebook post.)

  • Spam and nonstandard English

    ()

    For the last couple decades, one common aspect of a lot of spam has been nonstandard use of English. For example, when I get email that claims to be from a major American corporation, but it’s full of nonstandard grammar and spelling, that’s a signal that the email is very unlikely to really be from that corporation.

    And it now occurs to me that dealing with spam like that may have helped train me to consider certain uses of language, such as complete sentences that use standard grammar and spelling, as a signal of authoritativeness. I’ve always had that kind of reaction; but the new-to-me thought this morning is that maybe many years of learning to detect spam has further strengthened my association between standard English and authoritativeness.

    That association is problematic in various ways—in various contexts, it can be classist and/or racist and/or ableist, etc.

    But setting that issue aside, it’s now a problem for me in another way:

    It contributes to my gut reaction that AI-generated text sounds authoritative.

    Or to put that in a shorter, punchier way:

    All those years of spam may have made me more vulnerable to believing GPT’s lies.

    (Original Facebook post.)

  • Legal filings

    ()

    A plaintiff’s lawyer asked ChatGPT for relevant citations. ChatGPT made some up. The lawyer cited them in court filings.

    Defending lawyers expressed puzzlement. The plaintiff’s lawyer asked ChatGPT to provide more info about the cases. ChatGPT obligingly made up the decisions in these nonexistent cases. The plaintiff’s lawyer submitted that output to the court.

    At some point, plaintiff’s lawyer asked ChatGPT whether the cases were real, and ChatGPT said they were, and plaintiff’s lawyer didn’t bother to check beyond that.

    When confronted by the judge about all these made-up filings, plaintiff’s lawyer apologized and said he didn’t know that ChatGPT could make stuff up.

    (Original Facebook post.)

  • Wikipedia

    ()

    “During a recent [Wikipedia] community call, it became apparent that there is a community split over whether or not to use large language models to generate content. While some people expressed that tools like Open AI’s ChatGPT could help with generating and summarizing articles, others remained wary.”

    —“AI Is Tearing Wikipedia Apart”

    “The community is also divided on whether large language models should be allowed to train on Wikipedia content. While open access is a cornerstone of Wikipedia’s design principles, some worry the unrestricted scraping of internet data allows AI companies like OpenAI to exploit the open web to create closed commercial datasets for their models. This is especially a problem if the Wikipedia content itself is AI-generated, creating a feedback loop of potentially biased information, if left unchecked.”

    Article also talks about the importance of checking all of the citations that GPT provides, given that they’re often fictional.

    (Original Facebook post.)

  • Why LLMs make stuff up

    ()

    Some interesting stuff about why Large Language Model AI systems make stuff up. Also, article suggests using the word “confabulation” instead of “hallucination” when LLMs make stuff up.

    Some quotes from the article:

    “In the case of ChatGPT, the input prompt is the entire conversation you’ve been having with ChatGPT[…]. Along the way, ChatGPT keeps a running short-term memory (called the “context window”) of everything it and you have written, and when it ‘talks’ to you, it is attempting to complete the transcript of a conversation as a text-completion task.”

    “ChatGPT […] has also been trained on transcripts of conversations written by humans.”

    “When ChatGPT confabulates, it is reaching for information or analysis that is not present in its data set and filling in the blanks with plausible-sounding words.”

    “In some ways, ChatGPT is a mirror: It gives you back what you feed it. If you feed it falsehoods, it will tend to agree with you and ‘think’ along those lines. That’s why it’s important to start fresh with a new prompt when changing subjects or experiencing unwanted responses.”

    One possible way to improve factuality “is retrieval augmentation—providing external documents to the model to use as sources and supporting context”

    Other possible approaches include “more sophisticated data curation and the linking of the training data with ‘trust’ scores”

    (Original Facebook post.)

  • Fake Guardian articles

    (, )

    “ChatGPT is making up fake Guardian articles.”

    “In response to being asked about articles on this subject, the AI had simply made some up. Its fluency, and the vast training data it is built on, meant that the existence of the invented piece even seemed believable to the person who [it was attributed to but who] absolutely hadn’t written it.”

    (Original Facebook post.)

  • Coherent but false

    ()

    On the difficulty of recognizing that an AI/LLM is making stuff up:

    it spit out a logically coherent answer and cited working links to real publications.

    The catch is, the linked publications were completely unrelated articles from open-source journals since chatgpt can’t access papers behind paywalls, which is a lot of papers. Furthermore, what it was saying was horseshit. It sounded so vaguely convincing that we had to show it to the aforementioned grad student, who confirmed it was nonsense.

    https://www.tumblr.com/mildlyexistentialcrisis/712038317305249792/i-work-in-a-bio-research-lab-and-we-were-fucking

    (Original Facebook post.)