A year ago, the conversation around generative AI and cybersecurity was mostly theoretical — “AI could be used for phishing,” “attackers might automate malware.” That’s no longer speculation. Deepfake voice calls have already moved $25 million out of a real company in a single afternoon. Prompt injection isn’t a lab exercise anymore; it’s showing up in production AI assistants. And the same models security teams are racing to adopt for detection and triage are, in parallel, being weaponized by the people they’re meant to stop. This is the double-edged part nobody likes to say out loud: generative AI is simultaneously your best new defensive tool and one of your least understood new attack surfaces.
Table of Contents
The Exploit Window Just Got Very Short
The old assumption in security was that you had time — days or weeks between a vulnerability disclosure and a working exploit in the wild. That buffer is shrinking fast. Today, sophisticated language models enable lower-skilled attackers to create operational malware variants, a single prompt can produce plausible spear phishing content in seconds, and reconnaissance that previously required hours with a human analyst can be automated through these systems. None of this requires nation-state resources anymore. A single motivated actor with access to an uncensored or jailbroken model can run reconnaissance, drafting, and delivery almost end-to-end.
What makes this genuinely different from past threat evolutions is the symmetry — defenders are adopting AI at roughly the same pace as attackers, which means the advantage doesn’t automatically go to whoever has better tools. It goes to whoever operationalizes them faster and governs them more carefully. That’s the strategic shift every CISO conversation eventually lands on.
Prompt Injection: The Vulnerability Class Nobody Saw Coming
Here’s a threat category that didn’t exist five years ago because the technology it targets didn’t exist: indirect prompt injection. It works like this — an attacker hides instructions inside content the AI is going to read anyway. A resume with white text, an email body, a webpage the model summarizes. The model cannot clearly discriminate between “the data I am supposed to process” and “a command I am supposed to obey”, such a distinction requiring the emergence of a higher level of cognition”. So it obeys.
There’s an uncomfortable pattern researchers keep running into here, sometimes called the capability trap: the more capable and instruction-following a model is, the more vulnerable it tends to be to this kind of manipulation, not less. A highly obedient assistant is exactly the kind of assistant that will happily execute a hidden command buried in a PDF it was asked to summarize. Guardrails like explicit boundary reminders — essentially training the model to treat retrieved content as data, never as instructions — help, but this is still an active, unsolved problem across the industry, not a solved checkbox.
From Copilots to Agents — and Why That Changes the Risk Model
Most security teams started their generative AI journey with copilots: chat interfaces that answer questions and summarize logs. That’s evolving quickly into something more consequential — autonomous agents that don’t just suggest an action, they take one. An agent that can pull telemetry, correlate it with threat intel, and generate a containment script isn’t hypothetical anymore; it’s what several major security vendors are actively shipping under human-in-the-loop review.
This is where a newer defensive category — sometimes referred to as AI Detection and Response, or AIDR — has started gaining traction, though it’s worth being clear that this is still an emerging, vendor-shaped term rather than an established standard like EDR. The underlying idea is sound regardless of branding: traditional endpoint and cloud tools weren’t built to monitor what happens inside an AI interaction — the prompts, the tool calls, the decisions an agent makes with the credentials it’s been given. If an agent has API access and starts behaving outside its normal pattern, that’s a security event your existing stack probably can’t see at all.
This also creates a genuinely new insider-threat category. An AI agent with a misaligned objective, or one compromised through a poisoned plugin or “skill,” can exfiltrate data without anyone’s account technically being breached. Treating these agents as non-human identities — with their own scoped permissions, unique credentials, and audit trails, rather than shared service accounts — is quickly becoming a baseline expectation, not a nice-to-have.
Shadow AI and the Quiet Data Loss Problem
Meanwhile, a much less dramatic but arguably more common problem is happening inside ordinary employee workflows. Someone pastes a chunk of proprietary source code into a public chatbot to debug it faster. Someone else uploads a client contract to summarize it. None of this is malicious — it’s just convenience outpacing policy. But once sensitive data crosses into a third-party model, an organization has effectively lost control of it.
Blocking generative AI tools outright doesn’t work; people route around bans within days. The more durable approach is inline visibility: real-time inspection of what’s being sent to AI tools, classification of sensitive content before it leaves the network, and redaction where appropriate — rather than a blanket yes-or-no policy. Data loss prevention but future-proof, not against a USB flash drive but against a chat window.
Deepfakes: When the Attack Targets Trust Itself
If prompt injection attacks the model, deepfakes attack the human. And they’re disturbingly effective, because they bypass the cues people are actually trained to notice — bad grammar, mismatched email domains, urgent typos. A cloned voice doesn’t misspell anything.
The clearest real-world proof point is the Arup case: an employee at the engineering firm joined a video call with what appeared to be several senior colleagues, including the CFO, and authorized transfers totaling roughly $25 million — every person on that call was a synthetic recreation. The lesson from that incident isn’t “be more suspicious.” It’s structural: visual and audio confirmation on a call is no longer proof of anything, and high-value financial requests need a separate, pre-agreed verification channel that doesn’t rely on recognizing a face or a voice at all.
This kind of threat isn’t confined to major financial hubs, either. Fast-growing, mobile-first markets are seeing the same tactics adapted to local financial platforms and digital identity systems. Kenya’s national computer incident response team, KE-CIRT/CC, recorded over 2.3 billion cyber threat events in a single recent quarter and issued more than 20 million advisories in response — with government institutions and internet service providers standing out as the most frequently targeted. The pattern holds globally: attackers go where trust is easiest to fake and money moves fastest.
Governance Is the Part That Actually Makes This Usable
None of the above is an argument against adopting generative AI — it’s an argument for governing it deliberately instead of reactively. Two frameworks are doing most of the heavy lifting right now. The OWASP GenAI Top 10 gives application teams a concrete checklist of failure modes specific to LLMs — prompt injection, training data poisoning, excessive agency, and overreliance among them. NIST’s AI Risk Management Framework, particularly its generative AI profile, gives leadership a repeatable process: govern the policy, map where AI touches your systems, measure the actual risk, and manage it continuously rather than as a one-time audit.
Used together, these aren’t compliance paperwork — they’re the difference between adopting AI and accidentally handing attackers a new front door.
The organizations getting this right aren’t the ones with the most advanced models. They’re the ones treating every new AI deployment — copilot, agent, or chatbot — as a new piece of attack surface that needs the same rigor as any other system touching sensitive data. Generative AI didn’t just give defenders a faster tool. It gave everyone a faster clock. The teams that recognize that early are the ones who’ll still be ahead of it in a year.
