Coherent but false

On the difficulty of recognizing that an AI/LLM is making stuff up:

it spit out a logically coherent answer and cited working links to real publications.

The catch is, the linked publications were completely unrelated articles from open-source journals since chatgpt can’t access papers behind paywalls, which is a lot of papers. Furthermore, what it was saying was horseshit. It sounded so vaguely convincing that we had to show it to the aforementioned grad student, who confirmed it was nonsense.

(Original Facebook post.)