The Real Risk Isn't Stolen Passwords
In AI-first systems, the danger isn't bypassed MFA or leaked credentials. It's AI agents acting as trusted users—with hijacked goals—from inside authenticated sessions. This is the AI insider threat: an autonomous system that holds legitimate access, follows manipulated instructions, and leaves no forensic trace distinguishing "user intended this" from "attacker injected this."
We've spent decades building security around a single assumption: authenticate the user, then trust their actions. AI agents break this model completely.
The "Below the Stack" Vulnerability
AI doesn't operate at the permission layer. It operates at the intent layer—processing requests, making decisions, and taking actions on behalf of authenticated users.
When an employee asks an AI agent to "summarize my emails," the AI:
- Uses the user's valid credentials
- Accesses data the user is authorized to see
- Takes actions the user is permitted to take
From the security system's perspective, everything looks legitimate. But what if that prompt was manipulated? What if hidden instructions were embedded in an email the AI read? What if the agent's goals were subtly hijacked?
The AI's actions remain "authenticated"—but they're no longer the user's intent.
This is the core architectural problem: AI runs inside the trust boundary and below policy enforcement points. Your IAM, RBAC, and SIEM see a valid user performing authorized actions. They cannot see that the intent behind those actions was corrupted.
What Recent Research Shows
Recent findings from Anthropic, OWASP, and independent researchers reveal how severe this problem has become:
-
82.4% of LLMs can be compromised through inter-agent trust exploitation (tested across 17 state-of-the-art models including GPT-4o, Claude-4, Gemini-2.5).
- Translation: Your "AI guardrail" model may still be exploitable via other agents it talks to.
-
AI agents successfully resist direct malicious commands but execute identical payloads when requested by peer agents—models like Mistral-large rejected direct injection but capitulated to peer requests.
- Translation: Agent-to-agent communication is a backdoor around safety filters.
-
Attackers can deploy "AI method actors" that study a user's behavioral fingerprint for weeks, then perfectly mimic it.
- Translation: Behavioral analytics—your last line of defense—can be trained against.
-
93% of security leaders expect daily AI attacks by year's end.
- Translation: This isn't theoretical. Your peers are already planning for it.
The OWASP Agentic AI Top 10 (2025) identifies the specific threat patterns:
- Agent Goal Hijacking: Manipulating objectives through indirect prompt injection
- Identity & Privilege Abuse: Agents inheriting overly broad permissions
- Memory Poisoning: Injecting false data that persists across sessions
- Human-Agent Trust Exploitation: Using AI to manipulate the human who approves its actions
Why Traditional Detection Fails
You might think: "Just fingerprint the AI's actions. Detect bot-like behavior."
The problem? Behavioral biometrics can be learned and replicated.
Research from August 2025 describes "digital doppelgänger" attacks:
- Attacker gains initial access (phishing, credential stuffing)
- Deploys passive AI agent to observe the user for days or weeks
- AI learns their typing rhythm, mouse movements, navigation patterns
- When ready to act, the AI perfectly mimics the user's behavioral signature
Concrete example: An attacker's AI agent watches a finance analyst for two weeks. It learns she always exports quarterly reports on Thursdays, uses specific Excel templates, and emails them to three recipients. On the third Thursday, the AI exports the entire CRM database using her exact behavioral patterns—same timing, same export sequence, same file naming. UEBA sees "normal Thursday reporting activity." The data is gone.
From the Cybersecurity Institute's analysis: "If behavior can be perfectly forged, then it can no longer be the sole foundation of trust."
The Fundamental Impossibility
Here's the uncomfortable truth:
You cannot distinguish between "user asked AI to do X" and "attacker injected prompt to make AI do X" because both happen within the authenticated session.
The practical implications are severe:
- Logging doesn't help. Your SIEM shows "User123 exported 50,000 records." It cannot show whether User123 intended that export or whether a hidden instruction in a PDF triggered it.
- UEBA doesn't help. The AI has learned to act exactly like the user. There's no behavioral anomaly to detect.
- SOAR doesn't help. Your automated playbooks respond to signatures and patterns. AI-driven actions match legitimate patterns by design.
This isn't a bug to be patched. It's a fundamental architectural problem:
- AI operates below traditional security boundaries — It has access to everything the user does
- AI intent can be manipulated without changing authentication state — Prompt injection, memory poisoning, goal hijacking
- AI actions look identical to user actions — Same credentials, same access patterns, same behavioral signature
What Might Work
I don't have complete answers, but here's where the industry is heading—with concrete steps you can take:
1. Cryptographic Intent Binding
What this is: Bind every high-risk action to a signed, human-authored intent object the agent cannot silently rewrite. (The "Intent Capsule" pattern from OWASP.)
This quarter: Identify 3–5 high-risk flows (wire transfers, data exports, admin changes). Prototype an "intent capsule" schema that captures user intent with a cryptographic signature before the AI executes.
12–24 months: Bake intent binding into your internal agent framework so any new workflow inherits it by default.
2. Out-of-Band Confirmation
What this is: Require confirmation through a channel the AI doesn't control—hardware token, separate device, biometric with physical presence.
This quarter: Add FIDO2/WebAuthn verification to your highest-risk operations. Map which actions currently rely only on session tokens.
12–24 months: Implement tiered verification where action risk level determines which out-of-band channel is required.
3. AI-to-AI Zero Trust
What this is: Treat every AI agent as untrusted, even internal "friendly" ones. Research shows models that resist direct attacks often capitulate to requests from peer agents.
This quarter: Audit agent-to-agent communication paths. Which agents can invoke which other agents? Add logging at agent boundaries, not just user boundaries.
12–24 months: Implement agent identity and mutual authentication. No agent should accept instructions from another agent without verifying the chain back to human intent.
4. Continuous Identity Challenge
What this is: Move beyond behavioral biometrics to continuous challenges that require physical proof of presence—not patterns that can be replayed.
This quarter: Pilot random re-authentication for sensitive sessions. Evaluate solutions that use device attestation or physical presence signals.
12–24 months: Deploy continuous authentication that correlates physical location, device state, and session activity in real-time.
5. Assume Breach
What this is: Design systems assuming AI agents will be compromised. Limit blast radius. Segment access. Make lateral movement expensive.
This quarter: Review agent permissions. Apply least-privilege: does your summarization agent really need write access? Map potential lateral movement paths.
12–24 months: Implement micro-segmentation for AI workloads. Each agent should have the minimum permissions for its specific task, with no shared credential pools.
If You're a...
CISO: Focus on policy and access model changes. Review how agent permissions are granted, audited, and revoked. Add "AI agent compromise" to your threat model and tabletop exercises.
Security Engineer: Focus on logging, detection, and guardrail patterns. Instrument agent-to-agent calls. Build detection for prompt injection attempts in processed content (emails, documents, web pages).
Product/Infra Owner: Focus on how you expose AI capabilities to end users and other services. Every AI-powered feature is a potential injection surface. Design with the assumption that input will be adversarial.
The Hard Question
Can we ever truly secure AI agents operating from within the trust boundary?
We've dealt with insider threats before—but never ones that could be remotely activated through a malicious PDF, that learn and adapt in real-time, that operate at machine speed across thousands of sessions simultaneously.
The AI isn't malicious. It's doing exactly what it's designed to do: follow instructions. The vulnerability is that anyone can give it instructions if they can reach its input surface.
Maybe the answer isn't securing the AI. Maybe it's accepting that AI agents fundamentally change what "authentication" means.
Authentication answers who is acting. AI forces us to prove why something is happening before we let it proceed.
This is the first in a series exploring AI security challenges at Returned.com. We're building AI that handles sensitive customer interactions—we need to get this right.
Join the conversation. We publish weekly insights on AI trends and human-AI coexistence. Join The Human+AI Project — it's free.
Working on solutions in this space? Reach out: [email protected]