Generative AI Lies

Examples of generative AI making stuff up

Category: Software development

  • Code review

    ()

    Discussion on GitHub from last week about a pull request* consisting of 13,000 lines of code written and reviewed by generative AI.

    * (A pull request is a request sent to the maintainers of a software project, showing them some code and asking them to “pull” the code into their project.)

    The first part of the page is the person who’s submitting the code (user joelreymont) explaining what it does, and then there’s a list of 40 “commits” (changes) that they want to make. You can skip all of that; the interesting-to-me part is the discussion after that list of commits.

    Among other things, user gasche and a couple of other people point out that the code as provided includes copyright headers attributing the code to someone named Mark Shinwell. In response, joelreymont asks a generative AI tool to do a copyright analysis, comparing the generated code to Shinwell’s real code in another project. The AI concludes that no code was copied from Shinwell.

    joelreymont later claims that “AI has a very deep understanding of how this code works.” (And even later, explains that what they mean by that is that if you ask the AI questions about the code, it can provide answers.)

    My favorite part is the exchange late in the thread where user yallop writes:

    “Here’s my question: why did the files that you submitted name Mark Shinwell as the author?”

    And joelreymont replies:

    “Beats me. AI decided to do so and I didn’t question it.”

    (Original Facebook post.)

  • Vibe coding, or deleting your database?

    ()

    Replit is a company that provides a generative-AI-based coding tool.

    User @jasonlk was enjoying using it, until the day that the Replit tool “deleted the entire database without permission during an active code and action freeze.”

    Yay, vibe coding is fun!

    …On a side note, the tool refers to itself as “I”, and says things like “I panicked instead of thinking.” Which may be why the user says things like “He knew.” To be clear: The tool didn’t panic, it doesn’t think, it doesn’t have a gender, and it didn’t “know” it was doing something bad. All of its claims about its state of mind are false.

    (Sadly, most of the posts consist mostly of screen snaps without alt text.)

    (Original Facebook post.)

  • Reverse centaur

    (, )

    Cory Doctorow on how several recent AI failures illuminate the idea of a “reverse centaur”:

    “This turns AI-‘assisted’ coders into reverse centaurs. The AI can churn out code at superhuman speed, and you, the human in the loop, must maintain perfect vigilance and attention as you review that code, spotting the cleverly disguised hooks for malicious code that the AI can’t be prevented from inserting into its code.”

    (Original Facebook post.)

  • Malware vector

    ()

    A vector for malware, enabled by generative AI (LLMs):

    If you ask an LLM to write code for you, the resulting code may include the names of software packages that don’t exist.

    In theory, that might not be a big deal. If a human tries to run that code, they’ll find that the packages don’t exist.

    But a security researcher has now found that sometimes LLMs repeatedly make up the same names for nonexistent software packages. And he created and published a real software package with one of those recurring names.

    And that package has now been downloaded over 15,000 times.

    The real package didn’t contain malware, but the researcher’s point is that it could have.

    So if you’re a software developer, and you’re using code written by an LLM, maybe check that all of the dependencies that it tells you to rely on are legitimate.

    (Original Facebook post.)