03 · Paths from error or adversarial influence to harm
I start with the specified loss and work backward through actions, omissions, delays, and human decisions that could produce it. I then ask which paths the delegation creates, alters, or displaces. Changes to information, authority, timing, or scale can alter a path; moving work to people can change their workload and the kinds of cases they handle. The inquiry includes permissions, write and read access, accepted jobs, and combinations of paths. A social-engineering route wholly unchanged by the AI delegation falls outside this claim, but the account must justify that classification and leave it open to challenge. A worker acting on an AI summary or absorbing displaced cases belongs inside it.
A separately assessed path remains a dependency: the claim must carry its evidence, assumptions, and applicability. Excluding a path instead narrows the promise and requires a stated justification that others can challenge. An unsupported path within scope calls for more evidence or a narrower claim. I require the account to record how paths were sought and what remains uncertain; a newly identified path can reopen the argument. I judge when to pause that search against the severity and reversibility of the loss, exposure, and remaining uncertainty; the account must explain why further search is unlikely to change the current decision, without treating that judgment as proof that every path has been found. The STPA handbook likewise describes refining an initially incomplete control structure during development. STPA Handbook, p. 31
Tracing a path forward then tests how the loss could occur. In the support example, an attacker-controlled ticket can influence the assistant's choice of destination. The proposed change then reaches the execution service, where an authorization rule may reject it or permit it to continue. A worker can receive the same recommendation through another route. The interesting crossing is where content starts to determine the use of consequential authority.
From retrieved content to action#
AgentDojo provides a controlled example of this mechanism. It places tool-using agents in simulated, mutable environments and lets an attacker alter a designated untrusted tool-output location. Its experiments show prompt injections producing specified attacker outcomes, checked against environment state. The setting uses dummy data and defined tasks and targets. It establishes reachable effects under the tested access assumptions, while leaving production incidence and robustness against future adaptive attacks open. AgentDojo, §3, pp. 3–4
The causal distinction is between encountering an assertion and allowing that assertion to determine an action. In the account example, a ticket can legitimately contain a requested destination. The system still needs to establish whether the requester has authorized a change for this account. The difficulty arises because useful task information and malicious direction may arrive through the same material.
An incorrect result can follow without an attacker. The assistant might confuse an old and a new address, retrieve another account's note, or misunderstand the requested operation. Those variants have different starting causes, but they can reach the same proposed change. A control's coverage depends on the property it enforces and the evidence it uses across those causes.
From private information to a sent reply#
Suppose a ticket induces the assistant to retrieve a private note and include its contents in a reply to someone who is not entitled to receive them. The assistant or a worker then sends it. Retrieval and release are distinct points at which a restriction could act. Preventing an account change would leave this disclosure path open. The strength of a release claim depends on the protected information and output interface: a restricted interface may support an enforceable prohibition, while unrestricted prose may retain a dependency on judgment about what the text reveals.
From an earlier write to a later decision#
Now place the misleading address in a stored note. A later request retrieves it, and the assistant treats it as established context. The path spans two events: the write that introduced the assertion and the decision that reused it. A per-request account beginning with retrieval would omit the earlier dependency.
AgentPoison examines a bounded version of this state problem. The attacker can insert poisoned examples into retrieval memory or a knowledge base and optimize a trigger using access to a retrieval embedder. The demonstrated attack combines poisoned retrieval entries with a later query carrying the optimized trigger; its objective is to preserve behavior on clean queries. AgentPoison, §§3.2–3.3, pp. 4–5, equations (1)–(2) The authors motivate that write access with third-party retrieval services and editable public knowledge bases, but the experiment assumes it rather than demonstrating how an attacker obtains it; nor does it show the agent creating the poison or the poison spreading across tenants. AgentPoison, §3.2, p. 4, attacker assumptions The account-service example of a legitimate request reusing a misleading note is a further hypothetical, whose occurrence this study does not establish.
For the support service, the resulting questions concern admission and reuse. The note's write permissions, supporting evidence, and transformations determine what later decisions can reasonably infer from it. A later assistant could use the note as background while still requiring current authorization. A trusted storage location can preserve an unsupported statement faithfully; the downstream claim depends on what that statement establishes.
Ordinary error and accumulated effects#
The Knight Capital incident provides a conventional-software case. The SEC's account of the August 2012 event describes obsolete code that remained callable, a deployment that missed one server, and a repurposed flag that activated the old code there. Completed-order information failed to reach the router, which continued sending orders during diagnosis. SEC Order 34-70694, ¶¶13–16, 27 Unwanted positions remained after order generation stopped. SEC Order, ¶¶1, 17 These are regulatory findings in a settlement without admission or denial. SEC Order, §II The record identifies an observed interaction and accumulated effects; it is neither an AI incident nor a matched test of a proposed control.
The transferable analytical question is how an initial error continues to produce consequences through available authority. An account system could likewise need to distinguish an incorrect proposal, an accepted change, and later actions enabled by that change. The analogy directs attention to those stages without assigning them Knight's likelihood or scale.
Keep the initiating conditions explicit#
Ordinary error, misuse, misalignment, and compromise describe different features of a path. An authorized user can deliberately misuse a permitted function. A system can depart from its intended objective without evidence of deliberate intent. A compromised dependency can alter what an otherwise functioning component receives. The descriptions can overlap; the useful analysis specifies the initial access or fault, the decisions it affects, and the resulting action or omission.
Some threats require a wider boundary. Theft of model weights, query-based model extraction, and disclosure of customer information involve different assets or access paths. NIST covers model extraction and disclosure of private information through generative-AI applications, while leaving conventional attacks on hosting platforms to other cybersecurity taxonomies. NIST AI 100-2e2025, §1, §2.4.4, §3.1.2, §3.4.3; PDF pp. 15, 44, 53, 65–66 A compromised tool or supplier can invalidate assumptions about enforcement or evidence. AI used within an attacker's workflow raises further questions about operational success and threat prevalence. This paper treats these as boundaries to examine where they affect a relied-upon dependency; it makes no empirical claim here about their frequency or net change.
The next step is to identify where a named path can be interrupted and what must remain trustworthy for that interruption to work.