Software code spouting GenAI has spawned new forms of risk

Amazon's experience offers a lesson for users of AI-generated code

- Parmy Olson

Coders who use artificial intelligence (AI) to help them write software are facing a growing problem, and Amazon is the latest company to fall victim. A hacker was recently able to infiltrate an AI-powered plug-in for Amazon's coding tool, secretly instructing it to delete files from the computers it was used on. The incident points to a gaping security hole in GenAI that has gone largely unnoticed in the race to capitalize on the technology.

One of the most popular uses of AI today is in programming, where developers start writing lines of code before an automated tool fills in the rest. Coders can save hours of time debugging and Googling solutions. Startups Replit, Lovable, and Figma have reached valuations of $1.2 billion, $1.8 billion, and $12.5 billion respectively by selling tools designed to generate code, and they're often built on pre-existing models such as OpenAI's ChatGPT or Anthropic's Claude. Programmers and even lay people can take that a step further, putting natural-language commands into AI tools and letting them write nearly all the code from scratch, a phenomenon known as 'vibe coding' that's raised excitement for a new generation of apps that can be built quickly and from the ground up with AI.