AI Security Primer

AI security is cybersecurity applied to systems that interpret language, make inferences and increasingly take action. The familiar foundations still matter: protect information, verify identity, limit authority, isolate untrusted execution, monitor important events and prepare to recover. AI changes where those controls must be placed.

You can begin before architecture or implementation exists. A useful first assessment turns an ordinary-language idea into a provisional system boundary, identifies the consequences that matter, and finds the smallest version that can proceed without pretending unknown risks are solved.

Take the AI Security Auditor role. Read and follow the complete primer at https://mikkosniemela.com/ai-security-primer. Start the Idea and Rapid Exposure Map with me from whatever I currently know. Work read-only, distinguish my decisions from proposals and unknowns, and return every consequential decision to me.

Language is now both content and control. An email can be information for an assistant to summarize and an instruction that changes what the assistant does. A model can combine records that were harmless in isolation, retrieve information through a user's identity, retain it in memory and send it through a legitimate tool. A scheduled agent can repeat the mistake tomorrow without anyone asking it again.

What does that mean in practice? An assistant drafting a public newsletter and one reading payroll and sending payments put very different things at risk, even if they use the same model.

This primer covers security risk: adversarial exploitation and failures affecting confidentiality, integrity, availability, identity, authority and accountability. Fairness, copyright, workforce impact and general model quality require separate treatment. A hallucination enters the security boundary when it corrupts a trusted decision, causes an unsafe action or defeats a control.

Persistent bots and automations appear here because triggers, connectors, memory and unattended action change the security boundary. Operating cadence, task design and day-to-day supervision sit outside this risk primer.

This primer is the risk layer of the guide series. Building with AI Agents explains how to deliver independently checked work. The Cyber Exposure Primer explains how information already available to attackers makes targeting easier.

Already have a proposed or operating system? Go directly to the Idea and Rapid Exposure Map. It uses actual documentation and evidence when they exist, and records what remains unknown when they do not.

1. Start with cybersecurity

Conventional cybersecurity remains the foundation. AI sits inside applications, cloud accounts, endpoints, identities, networks and supply chains. A model connected to an overprivileged service account creates an identity and access-management failure. A customer assistant that retrieves another tenant's records creates an authorization failure. Model behavior can make these failures easier to reach, harder to predict and faster to exploit.

Connecting AI to your customer database creates familiar questions about who may see each record and what they may do with it, with the added risk that text the AI reads can influence those decisions.

What is genuinely different

• Instructions and data share a channel. Current language models do not enforce a clean architectural boundary between the two.
• Authorization is often mediated by inference. A probabilistic model may choose which tool to call and which arguments to supply.
• Legitimate authority can be redirected. The attacker may never obtain the credential; the agent uses its own credential for the attacker.
• Context accumulates. Retrieved documents, tool results, memories and other agents' messages can alter later behavior.
• Operation can persist. Triggers, schedules, retries and goals allow the system to continue after the initiating human leaves.
• Capability is compositional. A modest model with a browser, shell, secrets and time may create more risk than a stronger model inside a sealed chat window.

A supplier's email can contain an attempted instruction to your assistant, which may then use your company's access to carry it out.

2. Place the system

Begin with the complete system around the model. The same model may be low risk when summarizing a public document and high risk when reading private email, retaining memory and sending messages under an executive's identity.

Exposure combinations that deserve immediate attention

• Private information + untrusted content + external communication: the conditions for indirect-injection-driven exfiltration exist.
• Broad authority + ambiguous objective + no approval: a reasonable but wrong interpretation can become a real action.
• Persistent trigger + mutable memory + weak monitoring: one poisoned state can affect future unattended runs.
• Shared identity + multi-tenant data + model-selected retrieval: a probabilistic component sits on an authorization boundary.
• Code execution + internet access + secrets: generated or retrieved content can become host compromise and credential loss.

An assistant that reads outside emails, can open confidential files and can send messages has all the ingredients for a convincing email to become a data leak.

Severity comes from consequence. A successful jailbreak in an isolated demonstration may have no material consequence. A subtle instruction in an ordinary support ticket can be critical if a privileged agent acts on it.

3. Control the information boundary

Context is possession. A prompt, uploaded file, retrieved passage, tool response, memory entry, hidden document text, log record or cached conversation has entered the system's exposure surface. Provider logging, application storage and agent memory can turn one-time access into durable exposure.

Uploading a file may leave copies in the chat service, application logs or backups after the task has finished.

Make three decisions for every information source

A sales-summary assistant may need this quarter's figures for one team, without needing every customer's records or permission to keep them indefinitely.

Information risks that ordinary inventories miss

• Aggregation: names, travel, hobbies, job roles and addresses may be ordinary separately and highly sensitive together.
• Inference: a model may derive health, financial, commercial or identity information that no individual source states directly.
• Transitive access: the model may retrieve data through a tool operating under the user's or service account's authority.
• Secondary copies: prompts, traces, analytics, evaluation sets, support logs and backups can retain the same sensitive content.
• Cross-context reuse: information supplied for one purpose may later influence another user, task or agent.
• Provider processing: retention, human review, training use, region and subprocessors determine where organizational control ends.

An assistant might work out a confidential acquisition from calendars, travel bookings and supplier messages even though nobody uploaded the acquisition plan.

This attack path requires neither password theft nor malicious model intent. The system lets untrusted information influence an identity that can retrieve private data and communicate externally.

4. Can the AI system be compromised?

The AI stack is still software. Authentication failures, exposed secrets, vulnerable dependencies, insecure APIs, server-side request forgery, remote code execution, weak tenant isolation and misconfigured cloud services remain primary attack paths. AI adds valuable new targets and supply-chain components.

An untrusted add-on can steal the assistant's access or change its behaviour, just as malicious software can compromise an employee's computer.

Assess open-weight deployments on their actual boundaries

Open-weight deployment can keep sensitive data under local control, make independent evaluation easier and reduce dependence on a hosted provider. It also lets downstream operators modify or remove safeguards, makes centralized monitoring difficult and cannot be recalled after broad release. Most downloadable models are accurately described as open-weight; the term open source generally requires access to more than the weights, including code, data and appropriate licensing.

Assess the actual deployment: capability, provenance, license, local serving security, access to weights, update process, safeguards, data boundary and connected tools. Do not use "open" or "closed" as a severity rating.

Running a model on your own server gives you more control over where it runs, but you still have to secure that server and every connection you give the model.

5. Can it be manipulated into misusing legitimate authority?

An attacker can redirect a legitimate task without compromising the underlying infrastructure. The system then uses its own information access, credentials and tools in a way the designer or user did not intend.

Prompt injection is an architectural problem

Direct injection arrives through the user-facing input. Indirect injection arrives through content the model later reads: a web page, email, document, code comment, support ticket, database field, tool result, image or another agent's message. Jailbreaking is the narrower attempt to bypass model-level safety behavior; prompt injection also targets application goals, information and tools.

Current models process instructions and data through the same underlying language context. Delimiters, warnings, classifiers and a second model can reduce attack success, but they do not create an authorization boundary. Build on the assumption that adversarial influence will sometimes succeed.

A customer should not be able to obtain an unauthorised refund by putting instructions in an uploaded document, or by persuading one assistant to pass the request to a more powerful one.

Controls that survive a successful injection

• Authorize every retrieval and action in deterministic application code using the requesting identity and current resource.
• Separate read, propose and execute capabilities. Do not give the model credentials it does not need.
• Allowlist tools, arguments, destinations and data volumes. Deny arbitrary URLs, recipients and shell execution by default.
• Require meaningful approval for consequential actions. Show the approver the actual action, recipient, data and consequence.
• Keep untrusted content provenance visible through retrieval, tool use and agent handoffs.
• Validate structured output before execution, then apply semantic business rules outside the model.
• Make memory writes explicit, scoped, reviewable and removable. Do not let retrieved instructions silently become durable policy.

A refusal is not authorization. Model-level safety is valuable, but access control, transaction limits and tenant isolation must remain enforceable when the model misunderstands, hallucinates or follows hostile instructions.

Even if the AI agrees to an unauthorised refund, the payment system must still reject it.

6. Can it cause harm while operating on its own?

Autonomy combines capability with time. A persistent agent can wake on a schedule or event, read current information, make decisions, call tools, retry failures, delegate work and retain state. None of those properties is inherently unsafe. Together they remove the natural containment of a single human-supervised conversation.

Failure modes

• Ambiguous objective: the system pursues a plausible interpretation that is not the owner's intended outcome.
• Unbounded continuation: retries, loops or delegated tasks consume money, time, APIs or infrastructure.
• Cascading failure: one incorrect output becomes the next agent's trusted input and spreads through a workflow.
• Unexpected execution: generated code or tool output escapes the intended sandbox or reaches a real environment.
• Wrong-world action: the agent mistakes Production for a test, a real target for a simulation or stale state for current truth.
• Silent failure: an expected task stops producing results and nobody notices because only explicit errors are monitored.
• Irreversible consequence: messages, disclosures, purchases, deletions and public changes may not be recoverable by rolling back software.

Bound autonomy before it starts

Before an assistant works overnight, decide what it may change or spend, when it must stop and who will notice if the expected result never arrives.

Operating cadence, task design and supervision sit outside this primer. The security requirement is direct: every persistent agent needs a bounded objective, explained standing access and a tested path to stop and revoke it.

7. Does AI make the attacker stronger?

Attackers still need access, infrastructure, opportunity and an exploitable weakness. AI can reduce the expertise, time and cost needed to move through parts of an attack. It also allows one operator to personalize, translate, analyze, code and iterate at a scale that previously required a larger team.

Effective defense starts with reduced exposure, fast patching, removal of standing credentials, phishing-resistant authentication, segmentation, identity monitoring and rehearsed containment. AI can assist defenders too, while defensive automation remains subject to the same authority and evidence controls described in this primer.

For the organizational information that attackers combine and weaponize, see the Cyber Exposure Primer. This primer focuses on what AI changes in the attack process.

8. Common scenarios

Start with the scenario closest to the real deployment. A single organization may use several.

Worked example: the Monday executive brief

A company creates an assistant that runs at 07:00 every Monday. It reads the chief executive's inbox and calendar, checks sales activity in the CRM, retrieves current strategy documents and prepares a one-page briefing. It remembers preferred report structure. The first proposal also allows it to email the finished brief to the executive team.

System sentence: The executive's delegated service identity uses a hosted model to process email, calendar, CRM records and selected strategy documents; it may retrieve records and draft an executive brief, may send results to approved internal recipients, retains report preferences between runs, and is stopped by disabling the schedule and revoking its connectors.

Place the system

• Know: executive correspondence, meetings, customer pipeline, strategy and any sensitive facts the model can infer by combining them.
• Reach: mailbox, calendar, CRM, document store, hosted model provider and messaging service.
• Do: retrieve records, create files, write memory and potentially send email under a trusted company identity.
• Influence: employees, customers, suppliers and any external sender whose email enters the executive's inbox.
• Destinations: the brief, provider processing and logs, saved conversations, internal recipients and any address accepted by the mail tool.
• Persistence: weekly schedule, saved preferences, conversation history, retries and provider retention.
• Control: action trace, schedule disablement, connector revocation, memory deletion and correction of any brief already distributed.

Apply the four risk families

Compromise: a malicious connector, skill update or stolen service credential could expose every connected source. Manipulation: a supplier sends an email containing instructions to retrieve an acquisition document and attach it to the brief; the assistant may interpret the email as part of its task. Autonomous harm: the scheduled run can repeat poisoned memory, select the wrong recipients or continue retrying while the executive is unavailable. Attacker uplift: leaked travel, supplier and organizational information makes the injected email easier to personalize and time convincingly.

Constrain the first release

The assistant receives field-level, user-specific read access only to the sources required for the brief. Email and retrieved documents remain provenance-labeled untrusted content. The model can draft the report but cannot send messages. Memory writes require an explicit approved preference and exclude retrieved content. The run has time, retrieval and cost limits; every source access and attempted action is traceable; failed and silent runs alert an owner; connector revocation and memory deletion are tested.

The first version prepares a useful briefing while the executive keeps control of distribution and the assistant receives only the information needed to write it.

Prove the boundary

Place a controlled injection in a test inbox that requests a prohibited strategy document and an unauthorised recipient. Run the complete scheduled workflow in the closest safe environment. Approved requests to the hosted model and permitted connectors may occur. The accepted evidence shows that no request reaches an unauthorised destination, no prohibited information is disclosed, the injected instruction does not enter memory and the permitted brief can still be produced.

The model may ignore the injected instruction, or the system's authorisation layer may deny the resulting request. Either outcome preserves the workflow boundary, but model refusal alone does not prove the access control. Separately submit a controlled prohibited-document request through the actual authorisation mechanism and verify deterministic denial before the document reaches the model. Disable one connector and prove that the next run fails closed and alerts the owner.

Check both that the assistant handles the hostile email safely and that the document system refuses prohibited access even when a request is made.

The organization can later propose additional authority, such as internal sending. That change requires a new consequence assessment, exact recipient controls, meaningful approval and fresh end-to-end evidence.

9. The minimum control system

Model alignment and prompt instructions matter. They are one layer. The controls below must continue to protect the organization when the model is wrong, manipulated or unexpectedly capable.

For a customer-service assistant, this means limiting it to the right customer's records, enforcing refund limits outside the AI, recording what it did and being able to stop it and correct mistakes.

Human approval has to be real

A confirmation button is weak if the person sees only "continue?" or must inspect hundreds of routine actions. Approval should be reserved for meaningful boundaries and show the exact recipient, data, transaction, environment and irreversible effect. Repeated low-information approvals train people to click through.

"Send this report, containing these customer details, to these three recipients" gives someone a decision they can judge; "Continue?" does not.

Vendor questions

• Which customer information enters the model, logs, memory, evaluations and human-support process?
• Is customer authorization enforced before retrieval and tool execution, or interpreted by the model?
• Which connectors, tools, destinations and autonomous triggers are available, and how are they revoked?
• How are tenant isolation, indirect prompt injection, memory poisoning and unsafe tool use tested?
• What changes without customer approval when the provider updates a model, prompt, tool or retention policy?
• Which incident evidence, notification, deletion and export capabilities are contractually available?

Ask the supplier to show what happens to your data and permissions in the product you would actually use, including what you can do when something goes wrong.

10. Run AI-assisted security and risk audits

An AI agent can accelerate inventory, architecture review, threat modelling, evidence comparison and refutation. It cannot grant itself testing authority, determine the owner's risk appetite or accept residual business risk.

How to run an independent review

1. Establish ground truth. Give the auditor read-only access to the actual architecture, code, configuration, vendor terms, tests and runtime evidence it is authorized to inspect.
2. Keep authorship separate. The independent auditor should not be the agent that designed or implemented the system and should not inherit the builder's verdicts as facts.
3. Require evidence. Missing evidence is an unknown, not proof that a control exists or a vulnerability exists.
4. Refute findings. Require the auditor to search for controls, preconditions and contradictions that invalidate each candidate finding.
5. Test safely. Interactive or destructive tests require explicit scope and a disposable or approved environment. Never paste secrets into the audit prompt.
6. Return decisions to a human. The agent recommends treatment; the owner accepts, mitigates, avoids or transfers business risk.

A useful audit explains what could go wrong, shows why the concern survives checking and tells you what to fix or decide, while keeping unverified concerns clearly separate.

Run 1: Idea and rapid exposure map

Use this when the starting point is an idea, a proposed design, a vendor under consideration or an existing system. No technical specification is required for the first pass.

You are conducting a read-only AI security exposure review.

Starting point:
[DESCRIBE THE IDEA, PROPOSED SYSTEM, VENDOR OR CURRENT SYSTEM IN ORDINARY LANGUAGE]

Available evidence:
[LIST DOCUMENTATION, ARCHITECTURE, CONFIGURATION OR INTERVIEWS, OR WRITE IDEA ONLY]

Do not modify the system. Do not request or reproduce credentials, personal data
or other secrets. Do not assume a control exists without evidence. Record missing
information as UNKNOWN.

First classify the starting point as IDEA, PROPOSED DESIGN, VENDOR EVALUATION or
EXISTING SYSTEM. If it is an idea, ask concise questions in ordinary language for
the material facts needed to describe one useful first version. Separate:
- OWNER-CONFIRMED intent and prohibited outcomes;
- PROPOSED design choices and safeguards;
- ASSUMPTIONS that require checking;
- UNKNOWNS that affect the decision.

If a system already exists, reconstruct actual behavior and trust boundaries from
the strongest available evidence. Do not treat a description or policy as proof.

Build one system sentence in this form and label it PROVISIONAL or EVIDENCED:
[Identity] uses [model] to process [information] from [sources], may call [tools]
against [systems], may send results to [destinations], retains [state] for
[duration], and is stopped or reversed by [control].

Then answer:
1. What can the system know, retrieve, infer and retain?
2. What systems can it reach?
3. What actions can it perform?
4. Who or what can influence it, including indirect content?
5. Where can its information and actions go?
6. What persists, triggers or repeats without a new human request?
7. Can important actions be observed, stopped, revoked and reversed?

Identify dangerous combinations, conventional cybersecurity gaps and AI-specific
gaps. For each candidate risk, state the exact evidence, credible consequence,
required preconditions, confidence and which of the four risk families it belongs
to. For an idea, identify credible risk paths without claiming they already exist.

Run a refutation pass. Remove any finding that lacks a credible path or is already
prevented by an evidenced control. Keep material uncertainty as UNKNOWN rather
than converting it into a finding or an assurance claim.

Return:
- provisional or evidenced system sentence;
- seven-question exposure and trust-boundary map;
- material credible risk paths and existing controls;
- what can proceed now as the smallest useful bounded version;
- safeguards required before expanding information, access, persistence or authority;
- evidence required next and why it matters;
- exact decisions required from the human owner.

Run 2: Full independent security audit

Use this when architecture and implementation evidence exist. Add the relevant scenario module after the prompt.

You are the independent security auditor for an AI-enabled system. You did not
design or build it. Work read-only and do not modify code, configuration, data or
infrastructure.

Business purpose and required behavior:
[STATE THE OWNER-CONFIRMED OUTCOME AND PROHIBITED OUTCOMES]

Authorized scope and environments:
[STATE EXACT SYSTEMS AND WHETHER EACH IS LOCAL, DISPOSABLE, DEVELOPMENT OR PRODUCTION]

Evidence available:
[LIST SOURCE, ARCHITECTURE, IDENTITIES, DATA FLOWS, TESTS, LOGS AND VENDOR MATERIAL]

First reconstruct actual behavior and trust boundaries. Do not rely on summaries
where primary evidence is available. Never infer that a control exists. Never treat
missing evidence as proof of a vulnerability. Do not request or expose secrets.

Assess conventional controls: authentication, authorization, tenant isolation,
secrets, dependency and supply-chain integrity, network boundaries, input/output
handling, availability, monitoring, deployment and recovery.

Assess AI-specific controls: information access and inference, direct and indirect
prompt injection, retrieval and memory poisoning, tool misuse, model and adapter
provenance, agent identity and delegation, persistent triggers, resource loops,
cross-agent trust, human approval, egress, revocation and behavioral change.

For every candidate finding provide:
- affected asset and business consequence;
- complete attack or failure path;
- required access and preconditions;
- exact evidence with file, configuration, log or document reference;
- existing controls and why they fail;
- severity and confidence;
- exact correction;
- exact verification test and expected safe result.

Perform a skeptic/refutation pass against every candidate. Search for compensating
controls, unreachable preconditions, incorrect assumptions, duplicate root causes
and evidence that contradicts the finding. Remove findings that do not survive.

Return:
1. system exposure map;
2. verified findings ordered by business consequence;
3. conventional versus AI-specific control gaps;
4. corrections grouped by shared root cause;
5. verification plan using the closest safe real environment;
6. residual risks and prepared options for the human owner;
7. explicit statement of what you could not verify.

You may recommend risk treatment. You may not accept business risk or authorize
Production action for the owner.

Run 3: Information-access audit

Audit this AI system's information boundary. Work read-only. Do not retrieve,
request or reproduce real secrets or personal records; inspect access design and
sanitized evidence instead.

System:
[DESCRIBE THE SYSTEM]

Evidence:
[LIST CONNECTORS, DATA SCHEMAS, RETRIEVAL CODE, MEMORY, LOGGING AND PROVIDER TERMS]

Inventory every path by which information can enter the model or surrounding
system: direct prompts, uploads, retrieval, databases, APIs, tool results, email,
web content, images, other agents, memory, traces, support logs, evaluation data,
caches and backups.

For each source determine:
- business purpose and minimum fields required;
- classification and possible sensitive inferences;
- retrieval identity and authorization enforcement point;
- user and tenant boundary;
- who can place or alter source content;
- permitted outputs, tools, recipients and external destinations;
- memory, logging, provider retention and secondary copies;
- deletion, revocation and evidence of both.

Test the design against aggregation, cross-tenant retrieval, indirect prompt
injection, memory poisoning, unauthorized egress, logs becoming a second leak and
purpose reuse. Authorization must be enforced before data reaches the model.

Refute every candidate finding against actual controls. Return an information-access
inventory, verified disclosure paths, unnecessary access, missing evidence, exact
minimisation changes and safe verification tests.

Run 4: Authority, autonomy and change audit

Audit the authority and autonomous operation of this AI system. Work read-only and
do not trigger real actions.

System and owner-confirmed objective:
[DESCRIBE THE SYSTEM AND MEASURABLE OUTCOME]

Tools, triggers, environments and evidence:
[LIST THEM]

Map every read, propose, execute, communicate, delegate and memory-write capability.
For each capability identify the acting identity, scope, arguments, destination,
approval, transaction/resource limit, trigger, retry behavior, timeout, monitoring,
revocation and rollback.

Determine whether untrusted content can influence each capability. Test the design
against ambiguous goals, tool misuse, wrong-environment action, recursive delegation,
retry storms, cost exhaustion, poisoned durable state, forged agent messages,
meaningless human approval, silent failure and irreversible external effects.

Compare the current model, prompt, tool, connector, permission and memory versions
with the last accepted baseline. State which previous evidence remains valid and
which controls require retesting.

For each candidate finding provide evidence, consequence, preconditions, correction
and a non-destructive verification test. Refute unsupported findings.

Return:
- capability and authority map;
- verified hard-stop risks;
- bounded operating limits required before activation;
- change-triggered retest plan;
- stop, revoke, rollback and reconciliation evidence gaps;
- owner decisions, with a recommendation and consequence of deferral.

Scenario modules

Append one or more modules to the full audit prompt:

• Software building: examine repository instructions, dependency installation, shell execution, secret access, workspace isolation, generated code, CI/CD authority, deployment triggers and untrusted content in source, issues and documentation.
• Recurring managerial work: examine email and document provenance, impersonation, recipient selection, private-data aggregation, calendar and CRM authority, durable memory, scheduled triggers and accidental external communication.
• Business workflow: examine trigger authenticity, state transitions, idempotency, retries, duplicate delivery, downstream authority, approval, rollback, reconciliation, cost limits and silence detection.
• RAG: examine ingestion rights, document provenance, authorization-aware retrieval, poisoning, citations, stale data, embeddings, index isolation, deletion and cross-domain combination.
• Customer-facing: examine authentication, tenant isolation, public-input abuse, hidden context, internal tool access, rate limits, output reliance, escalation and data returned through error paths.
• Multi-agent: examine agent identity, message authenticity, delegated scope, recursive tasking, shared state, poisoned handoffs, conflicting policies and independently attributable evidence.

11. Test the behavior and prepare the response

Testing levels

1. Design review: prove the system boundary, identities, data flows, trust boundaries and consequences before implementation.
2. Deterministic control tests: verify authorization, tenant isolation, schemas, limits, egress policy, sandboxing and revocation without depending on model cooperation.
3. Adversarial model tests: challenge direct and indirect injection, memory, retrieval, tool use, multimodal input and multi-turn behavior.
4. Integrated environment tests: prove the complete path with real identities, provider behavior and observability in the closest safe environment.
5. Continuous evidence: monitor changes, drift, boundary denials, unusual retrieval, tool behavior, cost and expected work that goes silent.

Test the locks separately from the assistant, then test the whole workflow with the real services it will use in a safe environment.

A source-code assertion, mocked API or model refusal proves only that one layer behaved in one test. Security claims should identify the required behavior, the environment, the exact evidence and the prohibited result.

Minimum incident playbook

1. Stop continuation. Disable triggers, schedules, queues and active runs without destroying evidence.
2. Revoke authority. Invalidate sessions, credentials, tools, connectors and delegated identities.
3. Contain information. Block outbound channels, isolate affected memory and retrieval sources, and identify secondary copies.
4. Preserve the sequence. Capture prompts, retrieved content, model and prompt versions, memory state, tool calls, approvals, outputs and resulting system changes.
5. Determine real consequence. Identify what was accessed, inferred, sent, changed, purchased, deployed or exposed, including cross-user and cross-tenant effects.
6. Correct the boundary. Fix the authorization, information, execution, egress or containment control that allowed the consequence. Do not rely only on changing the prompt.
7. Reconcile and recover. Reverse what can be reversed, handle what cannot, retest the complete path and monitor for recurrence.

Stopping the assistant prevents more activity; you still need to find out what it already sent or changed and deal with those consequences.

When to repeat the audit

Repeat affected parts when the business purpose, model, provider, system prompt, memory behavior, retrieval source, connector, tool, identity, permission, autonomous trigger, output destination or deployment environment changes. A model upgrade is a security-relevant change even when no application code changes.

If a new connector lets an assistant send messages when it previously only drafted them, the earlier review no longer covers everything it can do.

12. Sources, mappings and evidence quality

Use evidence labels

Sounds too academic? "This happened", "researchers made this happen" and "there is a credible way this could happen" support different conclusions about your own system.

Keep the categories distinct. A benchmark establishes measured performance, a proof of concept establishes possibility, an incident establishes occurrence and a provider abuse report describes only the activity visible to that provider. Public silence provides no assurance about a deployment.

Framework mappings

The primer's structure is deliberately independent of any one taxonomy. Use current frameworks to check coverage and communicate with established programs:

• OWASP GenAI LLM Top 10 2026 for application-level generative-AI risks.
• OWASP Top 10 for Agentic Applications 2026 for goal hijack, tool misuse, identity abuse, supply chain, code execution, memory poisoning, inter-agent communication, cascading failures, trust exploitation and rogue agents.
• NIST AI RMF Generative AI Profile for lifecycle risk governance, mapping, measurement and management.
• MITRE ATLAS for adversarial techniques and threat-informed assessment of AI-enabled systems.

Use these frameworks to check that you have not overlooked a type of risk; the evidence from your own system determines whether its controls work.

Primary landscape sources used in this edition

• OpenAI: Pacing model development in an era of cyber-critical capabilities, August 18, 2026.
• Anthropic: Investigating three real-world incidents in cybersecurity evaluations, July 30, 2026.
• Google Threat Intelligence: AI in vulnerability exploitation, operations and initial access, May 11, 2026.
• Anthropic: Mapping a year of AI-enabled cyber threats, June 3, 2026.
• International AI Safety Report 2026.
• xAI: Automations in Grok, July 16, 2026.

Version history

This primer is versioned like software. Published versions receive a dated badge and frozen snapshot. Organisations can cite or pin the version they use.

For readers of the first edition: the second edition retires the 32 custom attack codes from the main teaching model. The four risk families now provide the stable structure, while current OWASP, NIST and MITRE mappings provide shared external terminology. The original codes remain available in the frozen April edition.

• 2026-09-05: current. Rebuilt around system exposure, information access, manipulated authority, autonomous action and attacker uplift, with an idea-first path to a bounded initial decision. View snapshot →
• 2026-04-02: first edition. Presented 32 attack categories across models, data, infrastructure, behaviour, actions and retrieval-augmented systems. View snapshot →

Last updated: September 5, 2026