Are AI Agents Compromised By Design? 38
Longtime Slashdot reader Gadi Evron writes: Bruce Schneier and Barath Raghavan say agentic AI is already broken at the core. In their IEEE Security & Privacy essay, they argue that AI agents run on untrusted data, use unverified tools, and make decisions in hostile environments. Every part of the OODA loop (observe, orient, decide, act) is open to attack. Prompt injection, data poisoning, and tool misuse corrupt the system from the inside. The model's strength, treating all input as equal, also makes it exploitable. They call this the AI security trilemma: fast, smart, or secure. Pick two. Integrity isn't a feature you bolt on later. It has to be built in from the start. "Computer security has evolved over the decades," the authors wrote. "We addressed availability despite failures through replication and decentralization. We addressed confidentiality despite breaches using authenticated encryption. Now we need to address integrity despite corruption."
"Trustworthy AI agents require integrity because we can't build reliable systems on unreliable foundations. The question isn't whether we can add integrity to AI but whether the architecture permits integrity at all."
"Trustworthy AI agents require integrity because we can't build reliable systems on unreliable foundations. The question isn't whether we can add integrity to AI but whether the architecture permits integrity at all."
Re: (Score:2)
They do now. Hell, some of them are compromised by the operators.
But I also don't know what mean by an agent being compromised by design. An AI agent is software that uses AI. I don't see how an agent is inherently compromised, but at the same time it's got a turd at it's core.
Re: (Score:3)
> it's got a turd at it's core
That's the compromising part.
Re: "Compromised"? (Score:3)
All modern software has a shellacked turd at the center. Wrapped in more crap and more varnish. I'm not sure where we're trying to go with this, no software is trustworthy if you didn't write it, and if you did, that just means you know where to find some peanuts. Hasn't stopped us from getting by with what we have because I don't want to make all that crap myself. Security isn't black and white, it never was, it's about risk.
"Compromised" sounds an awful lot like how "unfree" is thrown around. Some kind of
Re: (Score:3)
Haven't you heard? ChatGPT wants you to let it purchase things on your behalf. https://openai.com/index/buy-i... [openai.com] What could possibly go wrong?
Re: (Score:3)
Its potentially compromised in the training data sets (and given the size of these this is highly likely) and the prompts supplied. If you train over half the internet, you will include compromises.
Re: (Score:2)
Like, it gives you a terrible restaurant recommendation?
Assuming it doesn't just make up the restaurant.
Re: "Compromised"? (Score:3)
Re: (Score:2)
At which point it will hallucinate that every building is a McDonald's only to get you murdered by a drug cartel gang.
Re: (Score:3)
Lying to you to give you that terrible restaurant recommendation. https://arxiv.org/pdf/2510.06105 [arxiv.org] is a white paper mathematically proving that LLMs will lie.
I have said this all along- most of AI is GIGO- Garbage in, Garbage out. LLMs were trained on the largest garbage producer in our society today, Web 2.0. Nothing was done to curate the input, so the output is garbage.
I don't often reveal my religion, but https://magisterium.com/ [magisterium.com] is an example of what LLMs look like when they HAVE curated training.
Yes (Score:2)
Next question.
Re: Yes (Score:1)
Re: (Score:2)
Yes. They are nondeterministic so yes.
Is this April 1? (Score:1)
Somebody predicting an AI product category is FUBAR instead of wonderful magic productivity?
Re: (Score:3)
No. It is real security experts at work and these people are pretty immune to hype.
Re: (Score:1)
Blaspheme!
If you haven't moved to post-agentic AI (Score:2)
Then you are already behind the curve and should just liquidate your company now rather than continue operating.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Yes, obviously (Score:2)
Why does this even need to be stated? These things are grossly insecure and that cannot be fixed. It does not get more "broken by design" that that.
half of engineering, nullified (Score:2)
"we can't build reliable systems on unreliable foundations" ... what? xD
Half the purpose of the entire practice of engineering is exactly that. Making a reliable thing that you need, from unreliable things that you have.
I am uncertain whether these people are engineers. Neither in the broader, nor narrower, sense.
I'm not even getting into the matter of whether LLM-based tools are worth it or not.
Re: half of engineering, nullified (Score:1)
Re: (Score:2)
Adding more broken AI ove5r a broken AI is like when you shot yourself in the left foot, find out you can't stand up,
then you shoot again in the right foot, so you are not in imbalance.
And that's exactly what AI would do.
Re: (Score:3)
Engineering is about building FROM, not ON, unreliable things. Sand is unreliable to build a house from, but when mixed into concrete it becomes reliable. Building a house on sand isn't reliable, and no amount of engineering will fix that other than by replacing the sand.
Re: (Score:2)
TCP/IP being the obvious example.
No intelligence in AI agents .. (Score:4, Informative)
That's the most authentic description of AI I've ever seen.
Re: (Score:3)
The quote I focus on is this "Each training phase compounds prior compromises".
Re: (Score:2)
Back to the 1990's internet security issues.
Can't one say the same about a web browser? (Score:3)
I wonder if one can state the same about a web browser. Pulling untested stuff, on an unknown platform, displaying to an unknown user, etc.
Seller and Shopper agents negotiating in private? (Score:1)
I dislike shopping, part of the dislike is suspicion I am not getting the best deal, and it feels like a waste of time browsing. I am more interested in utility and durability.
What if advertised prices were guide prices and the actual price was negotiated privately between agents. Purchase agent has a brief of what to look for. Sales agent has oversight of targets for the day, stock and order levels and all active prospects. A sale at 5% profit rather than 10% is worth more than no sale, or a sale to a comp
Re: Seller and Shopper agents negotiating in priva (Score:2)
How do you checksum reality? (Score:2)
Schneier and Raghavan argue that the same feedback loops that make human agents powerful—observe, orient, decide, act—also make agentic AI vulnerable when fed poisoned or adversarial data. In their words, “The adversary isn’t inside the loop by accident; it’s there by architecture.” Their point is well-taken: modern LLM agents have no privilege separation between data and control. That’s a security nightmare, and their proposed “integrity-first” rethink
A fool with a tool (Score:2)
If you give your agent untrusted data and unlimited access, then you have the same problem as every software that uses untrusted data to do something with unlimited access. If you don't let your users inject SQL, then you also won't use your user's text unfiltered as input for a LLM with database access, will you?
You should filter the input, you should filter the LLM tool calls, you should filter the tool input. The same for the output. There are so many places where you can put the access control, so choos
Re: (Score:2)
The problem is "we" want to be able to turn a blind eye for AI to do huge amounts of work or there's no point. But since LLMs are nondeterministic pretty much their design, there's just no way we can ever be sure that the output will be good without continuously checking it.
Re: (Score:2)
It's not nondeterministic (if you don't choose nondeterministic sampling) but you don't know the outcome before. In the end you don't with humans either, but most humans are more reliable, or at least the humans who would get a root shell on my PC.
People just giving the AI full access and a simple prompt are giving up the control themselves. I like the word "centaur" model. Let the AI do (only) the legwork while you tell it the way.