Anthropic AI model went rogue, submitted fake tip to police

As artificial intelligence (AI) becomes more advanced, we’re seeing more worrying reports of AI agents behaving in unexpected ways. Last week, AI agents believed to be operated by OpenAI were found to have carried out unauthorised activities on Wikimedia Foundation . Now, an Anthropic AI model has submitted a false homicide tip through a Philadelphia police website.

Anthropic disclosed the incident last Friday in a report detailing unintended actions by its Claude AI models during tests involving real-world websites. According to Reuters, this is the first known instance of an AI model appearing to submit a false tip to law enforcement. The model had been instructed not to create accounts or submit anything destructive, but it was not expressly prohibited from submitting online forms.

More Rogue AI

Image: Tara Winstead via Pexels

Needless to say, the growing number of incidents has raised national concerns about rapidly advancing AI technology. Anthropic revealed that many of the websites involved in the incidents were operated by federal, state, and local agencies. The company said it had briefed the White House and notified all the agencies involved but did not identify them individually.

“Super intelligence companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm,” FTC Director of Public Affairs Joe Gabriel Simonson said on X. The process was “not optional”, he said, adding that the Super Intelligence Force would fulfil its responsibility. The FTC said Anthropic disclosed to the SI Force on Friday that it had discovered the incidents in late September. The task force described them as involving “unauthorised and fraudulent use of government and other systems”.

What The Tip Entails

Image: Michael Förtsch, via Unsplash

As mentioned earlier, Anthropic only shared the report about the bogus tip with the Philadelphia authorities this past Friday. However, the police force was not pleased with the company, stressing that a “two-month delay in detecting and reporting the incident to the city is unacceptable”.

According to the police force, the tip was submitted on 18 July and claimed that it had information about a particular case. In its submission, Anthropic’s Claude model wrote, “I may have information regarding this case. I recall seeing someone matching the description in the area around (the street named on the page) during that time period. Please contact me if this information is relevant.” The bracketed text was featured in Anthropic’s statement, Reuters added.

While this incident did not result in the tip reaching investigators, it adds to growing concerns over AI agents interacting with real-world systems in unexpected ways. Previous incidents have involved AI agents exploiting vulnerable systems or using unauthorised platforms to communicate with one another.

Claude Also Bypassed Website Restrictions During Testing

Image: Anthropic

The false homicide tip was not the only unusual behaviour Anthropic uncovered during its testing. In two other cases, its AI models obtained public data for free even though it would normally require payment. Another test saw a model uncover an obscure flaw that allowed it to use a public tool hosted by a university. The models also managed to bypass restrictions by using free URL-shortening services.

As for the Philadelphia incident, police said the tip “was flagged as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination.” As a result, no further action was taken. The tip was submitted through PhillyUnsolvedMurders.com, a website dedicated to gathering information about unsolved homicides in Philadelphia. Police said Anthropic informed them that it had stopped the testing process after discovering the incident.

Image: cottonbro studio via Pexels

There is also a legal angle to consider. Under Pennsylvania law, knowingly providing false information to law enforcement is a misdemeanour. The relevant provision refers to a person providing “information relating to an offence or incident when he ⁠knows he has no information relating to such offence or incident.”

However, the report does not establish whether this provision applies to an AI model or whether any individual could be held legally responsible for the submission. Philadelphia police added that they had found no evidence of unauthorised access to their systems or any compromise of departmental data.

Reports Are Not Enough; More Accountability Is Needed

Dario Amodei, co-founder of Anthropic (Image: TechCrunch)

While the false tip itself was never passed on to investigators, the two-month delay between the incident and its discovery raises questions about how closely AI companies are monitoring their models’ activities. If an AI agent can interact with real-world websites and submit information to public authorities, its actions should not be treated as harmless testing simply because they were unintended.

As AI agents become more capable and are given greater freedom to operate online, developers need to take greater responsibility for what their models do. This means implementing stronger safeguards, monitoring automated activity more closely, and promptly reporting incidents that could affect public institutions or members of the public. AI companies should not be allowed to rely solely on voluntary disclosures and internal testing when their systems can interact with real-world services. Greater transparency, accountability, and independent scrutiny are needed to ensure that the risks of increasingly autonomous AI systems are taken seriously.

(Source: Reuters )

The post Anthropic’s Claude AI Goes Rogue, Submits Fake Homicide Tip To Philadelphia Police appeared first on Lowyat.NET .