Long but excellent article about generative AI (LLM) chatbots: “Who are we talking to when we talk to these bots?”, by Colin Fraser.
(Article from early 2023; I’m sure some details of LLM behavior have changed since then, but I think the core ideas here are still valid and relevant.)
Fraser’s central points are that LLMs are designed to create sequences of words, and that a chatbot persona is a fictional character overlaid (by means of training) on that word-sequence-creation software. And that our experience of having a conversation with an LLM chatbot is partly shaped by the user interface and by our expectations about how conversations go.
A few quotes:
“At least half of the reason that interacting with the bot feels like a conversation to the user is that the user actively participates as though it is one.”
“the first-person text generated by the computer is complete fiction. It’s the voice of a fictional ChatGPT character, one of the two protagonists in the fictional conversation that you are co-authoring with the LLM.”
“A strong demonstration that the LLM and the ChatGPT character are distinct is that it’s quite easy to switch roles with the LLM. All the LLM wants to do is produce a conversation transcript between a User character and the ChatGPT character, it will produce whatever text it needs to to get there.”
“my interlocutor is not really the ChatGPT character, but rather, an unfeeling robot hell-bent on generating dialogues.”
“The real story isn’t that the chat bot is bad at summarizing blog posts—it’s that the chat bot is completely lying to you”
“[AI companies] actually have very little control what text the LLM can and can’t generate—including text that describes its own capabilities or limitations.”
“The LLM [is never truthfully refusing to answer a question]. It’s always just trying to generate text similar to the text in its training data”
Leave a Reply