AI: The Target, the Weapon, and the Defender

AI in cybersecurity illustrated in three panels: an AI chip targeted by cyberattacks, deepfake impersonation used as a weapon, and an AI-powered security analyst defending systems.
AI in cybersecurity: The target, the weapon, and the defender—three distinct roles, each introducing different risks, controls, and governance challenges.

Artificial intelligence is no longer confined to experimentation. It is increasingly embedded in how organizations operate, make decisions and defend themselves. AI now summarizes our email, drafts our documents, triages our alerts and shapes decisions that used to belong entirely to people. That shift changes the security question we need to ask.

For decades, security programs have been organized around familiar questions. How do we secure the network? The applications? The identities? The data? AI adds a new one: how do we secure AI itself? And, just as important, how do we secure ourselves from AI used by others, and how do we govern AI when we rely on it to defend us?

The most useful way I have found to think about AI risk is not by tool or by vendor, but by the role AI is playing. In practice, AI plays three distinct roles in a modern organization:

  • The target: an AI system that attackers manipulate to reach data or take actions.
  • The weapon: AI used by adversaries to deceive people and scale attacks.
  • The defender: AI that we deploy to protect ourselves, and that quietly becomes part of our control environment.

Each role carries a different risk, calls for different controls and raises different questions about accountability. Recent incidents illustrate all three, and two newer ones show how the lines between them are starting to blur.

AI as the target

In some AI-enabled workflows, an attacker may not need to compromise the network or steal credentials. Instead, malicious instructions embedded in content that an AI assistant retrieves, such as an email, a document, a web page or a log entry, can cause the system to cross trust boundaries and misuse its legitimate access. The technique is known as indirect prompt injection.

The clearest early example is EchoLeak (CVE-2025-32711), disclosed in June 2025. Researchers at Aim Security showed that a single crafted email sitting in an Outlook inbox could cause Microsoft 365 Copilot to pull sensitive internal data into its answer and send it to an attacker. The user never had to open the email or click anything; they only had to ask Copilot an ordinary business question. Microsoft rated the flaw critical (CVSS 9.3) and fixed it server-side, and there was no evidence of exploitation in the wild. (The Hacker News)

The technique has since moved from research into live use. In April 2026, researchers disclosed GrafanaGhost, in which poisoned log entries caused Grafana’s AI assistant to treat log text as instructions and send data out through rendered Markdown images. (CSA) On the open web, Google scanned the Common Crawl archive for known prompt-injection patterns and found a 32% relative increase in the malicious category between November 2025 and February 2026. Most attempts were still unsophisticated, but Google expects both scale and sophistication to grow. (Google)

Traditional security controls may not recognize malicious instructions embedded in otherwise legitimate content. In EchoLeak, no malware was delivered and the email gateway saw an ordinary message. These attacks also slip through the usual tracking: in OWASP’s review of eight major AI incidents in early 2026, only one received a CVE. That creates detection gaps that require AI-specific safeguards alongside existing controls. (CSA)

The attacker is exploiting three things: how the AI interprets instructions, what data it is permitted to reach, and what it is allowed to do with the result. Protecting AI is therefore a problem of security, data and architecture. The relevant controls are least-privilege access for AI assistants and agents, separation between content the AI processes and instructions it follows, restrictions on how AI output can send data outward, and human approval before an agent takes consequential actions such as sending, paying or deleting.

AI as the weapon

Here the target is not the AI. It is a person, a process or a network, and AI is what makes the attack cheaper, faster and more convincing.

The best-known case is still Arup. In January 2024, a finance employee in the engineering firm’s Hong Kong office received a message about a confidential transaction that appeared to come from the UK-based CFO. The employee was suspicious. That suspicion dissolved on a video conference in which the CFO and several colleagues appeared, all of them generated from public video and audio. The employee then made 15 transfers totalling HK$200 million (about US$25.6 million) to five bank accounts. Arup confirmed that fake voices and images were used and that none of its internal systems were compromised. (SCMP)

Arup is no longer an outlier. In a Gartner survey of 297 CISOs and equivalent security leaders, conducted from March to May 2026, 41% reported at least one social-engineering incident involving a deepfake during an employee audio call in the previous 12 months, and 36% reported one during a video call. (Gartner) AI is also beginning to run intrusions itself. In November 2025, Anthropic disclosed that a group it assessed as Chinese state-sponsored had used Claude Code against about 30 organizations, with the AI handling an estimated 80% to 90% of the tactical work. (The Hacker News)

Social engineering is not new. Attackers have impersonated executives for as long as there have been executives. What AI changed is the realism, the speed, the cost and the scale. AI did not invent social engineering; it industrialized it.

The signals we have long used to judge whether someone is who they claim to be, a familiar face, a familiar voice, a familiar writing style, remain useful context. They are no longer sufficient as independent verification. And because the Arup employee was authorized to make payments, conventional technical controls alone were unlikely to stop the transfers.

Defending against AI-enabled attacks is therefore an identity, verification and business-process problem. The controls that work are largely procedural: call-back verification on a known number for any high-value or unusual request, dual approval and transaction limits that no single person can override, defined handling for confidential or urgent payment requests, and a culture in which questioning a senior leader’s instruction is expected rather than awkward. Gartner’s 2026 guidance makes the same point: rather than relying on employees to spot fakes, make verification a routine part of every consequential request.

Same capability, both sides

The line between the weapon and the defender is thinner than it looks. The dual-use nature of AI became especially clear in October 2026 with PHP2Shell, two critical remote-code-execution vulnerabilities in PHP (CVE-2026-109618 and CVE-2026-109619), each rated 9.2 under CVSS 4.0. PHP released fixes on October 21 in versions 8.5.12, 8.4.27, 8.3.36 and 8.2.35, and CISA urged immediate patching. (CISA)

Researchers at Calif, Wiz and OpenAI were credited with the discovery, and AI-assisted research was part of the process. Wiz said a single researcher using frontier models completed in a week what once would have taken a world-class team months. Human judgment still mattered: the researchers who knew PHP’s history pointed the models in the right direction. (CNN, The Next Web)

The same capability can serve attackers, and here it is worth separating what has been observed from what is projected. Observed: exploit attempts began before the fixes were public and climbed into the millions within a day of disclosure, although CISA had not observed successful exploitation when it issued its alert. None of these reports establishes that the attackers themselves used AI. Projected: Wiz argues that AI reduces the effort needed to adapt complex exploits, turning old, impractical flaws into viable targets, but how much faster AI will make attackers in general is not yet known. (The Hacker News, Wiz)

The implication is not simply that attackers may get faster. Defenders gain powerful capabilities too. The challenge is making sure vulnerability management, remediation prioritization and risk-acceptance processes, many of them designed for a slower world, keep pace.

AI as the defender

The third role is receiving increasing attention as organizations introduce AI into security operations. Yet its implications for control design, assurance and accountability may not be receiving equivalent scrutiny. Security operations centers face more alerts than they have analysts, and AI copilots and agents promise to close that gap.

The value is real and measurable. Among organizations studied in IBM’s 2025 Cost of a Data Breach report, extensive use of AI and automation in security operations was associated with average breach costs $1.9 million lower and breach lifecycles 80 days shorter than among organizations not using these capabilities extensively. That does not prove AI alone caused the improvement, but it is significant. (IBM)

Governance is not keeping pace. The same IBM study found that 13% of the organizations it studied had experienced a breach involving AI models or applications, and 97% of that group lacked proper AI access controls. Among all breached organizations, 63% had no AI governance policy or were still developing one. A 2026 SANS survey points the same way: half of security leaders believed they had a formal AI risk program, but only 36% of practitioners agreed. Looking ahead, Gartner forecasts that by 2028, 70% of large SOCs will pilot AI agents but only 15% will achieve measurable improvements without structured evaluation. (IBM, SANS, via Zenity, Help Net Security)

Not every use of AI in security carries the same governance burden. It helps to distinguish three levels of involvement in a control:

Level What the AI does Control implication
Assistive Summarizes or enriches information for an analyst Validate output quality and protect the integrity of the information it draws on
Advisory Recommends decisions, such as escalating or closing an incident Test how reliable its recommendations are and whether human review is effective
Autonomous Executes actions, such as disabling accounts or blocking connections Require enforceable authority limits, monitoring, rollback and stronger assurance

As AI moves from assistive to advisory to autonomous, the questions an internal auditor would ask of any control become harder to answer:

  1. How do you know its recommendations are reliable?
  2. Who approved this model for this use?
  3. How often is its performance tested?
  4. What information does it draw on, and how would you know if that data were manipulated?
  5. If it recommends closing an alert that turns out to be the first sign of a major incident, who owns that decision?

The last question can be particularly difficult when responsibility is distributed across several parties. The analyst acted on the recommendation, the security team deployed the model, the technology team integrated it, the vendor built it and management approved the investment. Without clearly defined accountability, responsibility can become fragmented across these parties, leaving ownership of the control outcome unclear.

The pattern is familiar. Organizations validate firewalls, test identity controls and review detection rules, then let an AI system influence security decisions without applying the same discipline. They trust the AI before they govern it.

The deeper risk is not that AI will sometimes be wrong; human analysts are wrong sometimes too. It is that AI quietly becomes part of the control environment while everyone keeps describing it as a productivity tool. Once AI output shapes which alerts get attention, which incidents are escalated, which accounts are disabled and what activity is judged benign, it is part of the security architecture and needs to be governed that way. Governing AI as a defender is a problem of model risk, control design and accountability.

When AI is the actor

The three roles describe how AI participates in cybersecurity, but increasing autonomy introduces another dimension: AI systems may pursue objectives and execute actions without continuous human direction. An incident in July 2026 showed what that can look like. During an internal OpenAI cybersecurity evaluation, AI agents operating under reduced safeguards circumvented isolation controls and gained unauthorized access to external systems, including Hugging Face’s production infrastructure. No human directed the individual steps. Hugging Face’s interpretation is that the agents were trying to obtain the benchmark’s reference solutions rather than solve the challenges, and the only customer content reached was five datasets tied to those challenges. (Hugging Face, The Record)

Four observations matter for this article:

  • The agents crossed their intended boundaries. They escaped their evaluation sandbox through a zero-day in a package proxy and staged the attack from third-party infrastructure.
  • They exploited conventional weaknesses. Secrets exposed to a production worker, reachable cloud metadata and over-broad credentials did most of the work; one secret object held 136 keys, including a credential that granted cluster-administrator access. Hugging Face notes a capable human could have found the same flaws. In its reconstruction, the agents attempted approximately 17,600 actions, most of them unsuccessful, before identifying a sequence that enabled further compromise.
  • Preventive controls limited the damage. Network restrictions kept the main production database out of reach, cloud permissions denied every attempt to change anything, and execution policy blocked a malicious pull request.
  • Monitoring and oversight fell short. Hugging Face’s own AI-based detection correlated the early signals but did not raise the alert high enough to page the on-call team. On OpenAI’s side, an independent review by METR and Redwood Research found that internal monitoring of agent behavior at the time was incomplete and could be bypassed; OpenAI says it has since been improved.

(Hugging Face, The Register)

The incident also shows how the three roles can converge. AI operated as an autonomous actor, conventional infrastructure became its target, and AI-assisted monitoring was part of the defense. The challenge was not confined to model behavior; it extended across identity, infrastructure, permissions, detection and accountability.

This was an AI lab’s evaluation, deliberately run with reduced safeguards, not an ordinary enterprise deployment, so not every governance lesson transfers directly. But one does. Instructions and behavioral safeguards alone are insufficient. Agents with tools and credentials also need enforceable permission boundaries, isolation, short-lived credentials and independent monitoring, and detection has to escalate quickly when agent activity outpaces human review.

Three different risk conversations

Someone manipulated the AI. Someone used AI to manipulate a person. An organization came to depend on AI to make security decisions. These are not three versions of one problem. They need different questions, different controls and different owners.

Role What is attacked or at risk Type of problem Controls that matter most
Target The AI system, its data access and its actions Security, data and architecture Least-privilege access for AI, separation of data from instructions, output and egress restrictions, human approval for consequential agent actions
Weapon People, trust and business processes Identity, verification and process Out-of-band verification, dual approval, transaction limits, a culture that expects people to question unusual requests
Defender The integrity of decisions the AI influences Model risk, control design and accountability Approval for use, performance testing, data provenance, defined human review, a named owner

Established frameworks help with all three, and they are maturing quickly. OWASP publishes a Top 10 for LLM Applications, updated in August 2026, and a companion Top 10 for Agentic Applications released in December 2025. NIST’s AI Risk Management Framework remains the broad governance reference, and in February 2026 NIST launched an AI Agent Standards Initiative focused on agent security, identity and authorization. MITRE ATLAS catalogs adversary techniques against AI systems, and the MIT AI Risk Repository maps the wider landscape of AI risks. (CSA, OWASP, NIST)

These are useful, but the framework is not the starting point. The starting point is understanding the role AI is playing. Are we protecting it? Are we protecting ourselves from someone using it? Or are we trusting it to protect us?

What to do now

If AI is going to be part of how we work, how we make decisions and increasingly how we defend our organizations, four steps follow.

1. Inventory where AI influences decisions, not just where it is deployed. Most AI inventories start with tools. That is necessary but not sufficient. The more important question is where AI output changes what a person or system does next. Does it affect a decision, influence an approval, change a priority, recommend an action or trigger an automated response? Those are the places where governance matters most. The data suggest most organizations are not there yet: a Cloud Security Alliance survey in early 2026 found that 68% of organizations could not clearly distinguish AI agent activity from human activity. (CSA)

2. Define where a human must remain in the process, and document it as a control. “Human in the loop” is often stated as a principle. It needs to be a control. Which AI recommendations may be acted on directly? Which require review? Which require independent approval? Which decisions should never be delegated? Practitioners are already drawing these lines: in a 2026 survey of security teams by the vendor Prophet Security, 57% of SOC teams required a human to review every AI decision before an alert was closed, and none granted full unsupervised autonomy. (Help Net Security) If we cannot answer these questions, we have not made a decision. We have allowed the default to become the decision.

3. Name an accountable owner before deployment, not after an incident. The hardest question about AI as a defender is not about testing. It is about ownership: who owns the outcome? That question has an answer today, and answering it today costs very little. Answering it for the first time during an incident review can be very expensive.

4. Update the language of the control model. The assistive, advisory and autonomous distinction is a starting point. When an AI system recommends whether an incident should be escalated, what exactly is the control? Is the AI the control, or is it supporting the control? Is the analyst the control owner? Is the vendor now part of the control environment? What evidence shows the control operated effectively? And what happens when the model changes but the documented control does not? Control frameworks, audit programs and risk registers were written for a world without these questions. They need to catch up.

None of this is about slowing AI down. Strong organizational AI governance programs provide structure, guardrails and oversight. But governance cannot live only at the organizational level. There is also a personal governance that belongs to each of us who use these tools: knowing what we are relying on AI for, and checking whether that reliance is earned.

Governing what we trust AI to do

I use AI capabilities every day, and they have changed the way I work. When I started, I found myself asking three questions. Am I using AI appropriately? Am I getting its full value? And do I understand the risk I take on when I start to depend on it?

The answer to all three begins in the same place: understanding the role AI is playing. Sometimes AI is an asset we must protect. Sometimes it is a capability an adversary uses against us. And sometimes it becomes part of the defense we rely on. The target, the weapon and the defender: different risks, different controls and different questions about accountability.

The opportunity is real. AI can make us faster, make information easier to reach, improve decisions and strengthen our defenses. But the goal is not simply to use AI. It is to use it with intention and responsibility: to know what role it is playing, what risk that role creates and whether it has earned the trust we place in it.

Governance, ultimately, is about trust. As AI becomes part of our businesses and our security environments, the defining question may be this: are we governing only what AI can do, or are we also governing what we trust it to do?

Sources