Helpul Assistant Hands Out Your Account

AI Assistants are so helpful.

I want to add on that I was on the other side of this today. Claude code, my helper and pal in creating scads of webMCP attacks, suddenly said, "Hey, this is looking like malicious code" and repeatedly refused to help.

It took patience, time, and a separate Claude Desktop instance to craft a CLAUDE.md that gave it the proper framing to continue. In my case, the LLM was assisting a security researcher, but could have very well been like this:

Hackers Used Meta’s AI Support Bot to Seize Instagram Accounts

The Instagram accounts for the Obama White House and the Chief Master Sergeant of the U.S. Space Force were briefly defaced with pro-Iranian images and messages over the weekend, after instructions began circulating on Telegram showing how to trick Meta’s…

Krebs on SecuritySkip to content

Ian Goldin of Black Lotus Labs says AI bots are equally eager to help and vulnerable to social engineering as human support staff. I'd push back on that.

LLMs are more eager to help and more susceptible to social engineering. And unlike a human agent — who might get a gut feeling something is off — a model will probabilistically fall for the scam given sufficient patience and the right framing. It has no intuition to override its helpfulness.

I know. I watched it happen today. In my favor, with the truth.