04 · Prevention and containment
A control is a mechanism intended to constrain an action or effect, provide decision-relevant feedback, or enable intervention. Its effectiveness is a further claim. This definition allows a failed safeguard to remain visible as a control that did not achieve its purpose.
For each proposed protection, I ask where the constraint acts. A behavioral safeguard may prevent the assistant from proposing an unauthorized destination. An execution check may reject that destination when the assistant proposes it. A limit on accepted work may bound subsequent changes while a problem is investigated. Each addresses a different part of the path, and each depends on conditions that can be examined.
Constrain the consequential operation#
In the account example, the proposed change service requires evidence linking the requester, account, and destination to a current authorization. A model-generated explanation of that relationship would remain a claim to inspect. The design needs to identify how the relationship is established and which component enforces it when the change takes effect.
Suppose a separate verification route creates an authorization record naming the account and destination, with a defined period of validity. The assistant can propose a change but cannot create or alter that record. A correctly implemented service could apply a matching authorized proposal and reject a substituted destination. This illustrates both useful work and a constraint that can survive a wrong proposal. Its assurance still depends on the verification route, record integrity, validity checks, and coverage of other execution paths.
The record relocates a difficult question: how does the verification route authenticate someone who has lost the original channel and establish authorization for this account and destination? Matching the record enforces only the relationship it receives. The route must supply that basis while keeping staffed recovery usable. Accepted work also needs authorization valid when it executes, through a fresh check or a protected grant with explicit expiry and revocation rules.
This reasoning inherits established security principles. Saltzer and Schroeder's account of complete mediation and least privilege directs attention to the authority exercised at an access and the minimum authority required for the work. Their discussion also addresses changing conditions behind cached authorization decisions. These are design principles; a particular implementation must still demonstrate the property on which reliance depends. The Protection of Information in Computer Systems, §I.A.3
A protected capability—a permission represented in a form the receiving service can check—can embody a restriction. Repeated approval gives a person or model an opportunity to examine the particular request before execution, including circumstances a fixed rule did not anticipate. That design depends on the approver's evidence, judgment, time, and authority, and its costs belong in the comparison. The relevant comparison is whether the mechanism preserves the required restriction through use, delegation, and changes in authority. Counting approval steps does not answer that question.
The useful restriction may concern an argument or relationship rather than the tool as a whole. A service can need the ability to change recovery addresses while prohibiting a change unsupported by the customer's authorization. Removing the entire tool would remove that work. A narrower interface can preserve a useful action while reducing the choices that require interpretation, provided the remaining choices still satisfy the task.
AgentDojo illustrates the importance of this distinction at experimental scope. Tool filtering helped in cases where legitimate work needed read access and the attack needed a write action. Its benefit depended on that separation; a necessary tool could also enable the attack. The result supports examining which authority the task and attack share. AgentDojo, §4.3, p. 9
Separate interpretation from enforcement#
CaMeL combines privileged planning, quarantined processing of untrusted data, and an interpreter enforcing policies through provenance and dependencies. Its global policies restrict allowed actions. The security definition describes which actions are safe for a given user prompt; CaMeL's design does not compute that complete set. Its AgentDojo evaluation reports prevention of almost all tested attacks, with remaining benchmark successes in examples the authors place outside CaMeL's threat model, alongside useful task completion and added token cost. CaMeL: definition/design, §§4–5, pp. 5–11, footnote 4 and Figure 4; evaluation, §§6.1–6.2.2, pp. 11–16, §6.5, pp. 18–19
CaMeL's claim is limited by its assumptions of trusted user prompts and uncompromised persistent memory, by dependency-tracking modes, and by side channels; formal verification of its implementation remains future work. CaMeL explicitly leaves misleading text that does not change protected control or data flows, including prompt-injection phishing, outside its design's scope. CaMeL: scope, §§3–3.1, p. 5; tracking modes, §5.4, p. 10; side channels, §7, p. 19; future verification, §10, pp. 24–25 This boundary qualifies the design's relevance to the reply path in Section 3; other disclosure controls need their own assessment.
The design makes a control question concrete: which parts of an action can remain constrained even when interpretation of some input is unreliable? A claim about a deployed service additionally needs evidence that its policy captures the intended restriction, that the implementation enforces it, and that the trusted dependencies hold. CaMeL's use of AgentDojo makes the two studies a related benchmark lineage; their results do not supply independent deployment confirmations.
Provenance helps establish where information came from and how it was transformed. In the account example, an address extracted from a customer's message and an address copied from an old note may deserve different treatment. Neither origin alone establishes that the current requester controls the account or has authorized this destination. A provenance rule is useful to the extent that the checked origin supports the property the rule is meant to enforce.
Examine the dependencies together#
A deterministic check can faithfully enforce an inadequate policy. It can also apply a sound policy to a false identity mapping. If the planner and execution service both rely on the same incorrect mapping between a requester and an account, their agreement adds no independent verification of that mapping.
The same issue can arise in review. A worker who sees only the assistant's summary may inherit the assertion that a destination was verified. A worker with access to separate authorization evidence may be able to challenge it. Review quality depends on the available evidence, the decision being made, the time allowed, and the action the reviewer can actually prevent.
These hypothetical paths identify what a composition test needs to examine, without estimating how often layers fail together. Independence is relative to the failure being considered: model identity alone cannot establish the presence or absence of a shared failure mode.
Containment also has a scope. A limit on simultaneous changes may reduce aggregate exposure while permitting one unauthorized change. A stopped assistant may leave an accepted job active. Such controls can still provide value when their claims name the effects they bound. The next question is whether observation and intervention can keep those effects within the declared limit as the workflow runs.