AI Safety & Security
Delegated Authority and Warranted Reliance
Introduction — Consequential delegation#
Consider a subscription service that uses an AI assistant to help customers recover access to their accounts. The assistant reads tickets, retrieves prior case notes, drafts replies, and proposes changes to recovery addresses. A separate service can apply those changes, and a support worker has an administrative console. This is a hypothetical workflow; all its details and variants are invented for analysis.
Suppose an attacker puts an address into a ticket and persuades the assistant to recommend it as the customer's new recovery destination. The assistant has proposed a change the customer did not authorize. Whether that proposal becomes an unauthorized account change depends on what happens next: the evidence required by the change service, the permissions used, any accepted job, and the worker's separate decision. A misleading note can also preserve the assertion for a later session.
The service makes two commitments: protect accounts against changes without the customer's authorization, and provide a staffed recovery route when the customer has lost the original contact channel. Both matter. A system that closes every difficult case may avoid some unauthorized changes while failing the people it was meant to help.
My central claim is:
A claim that a delegated AI workflow controls a specified loss must be supported across the paths to that loss that the delegation creates, alters, or displaces, including paths through execution, stored state, delegated work, and human decisions. Any exclusion must be named, justified, and open to challenge.
I offer this as a standard for warranted reliance: reliance justified by evidence about the work entrusted to a system and the conditions and limits under which it can meet that commitment. Its practical question is what prevents a mistaken or manipulated decision from producing the loss we say the system controls. Answering it requires a causal account: which mechanism acts on which path, what that mechanism depends on, and how much of the claim the evidence supports.
The stronger proposition—that enforceable limits and timely intervention reduce harm while preserving useful operation—is an empirical hypothesis. Particular controls can support it under particular conditions; their presence alone cannot establish it, and a simpler design may meet the same requirements more effectively. The comparison must include the work people need completed and the costs of controlling it.
This paper focuses on consequential digital workflows. Its unit of analysis includes models, tools, data, retained state, identities, services, and the people whose decisions produce or bear the effects. It draws on systems safety, security engineering, bounded AI experiments, and a conventional software incident. These sources support different inferences without collectively establishing an overall safety rate for deployed AI.
One objection is that competent security and systems-safety engineering already supplies this account. In the workflows examined here, models interpret untrusted material, select actions and arguments at runtime, reuse context, and supply explanations that people may act on. These mechanisms give the analysis its AI focus; none needs to be exclusive to AI. Where conventional analysis adequately explains the paths and supports the decision, I would use it. The contribution I claim is a synthesis that makes the connection between delegation, controls, and evidence easier to examine.
AI Vision & Future examines the conditional path from capability to dependable value and wider tractable work. AI Operators Handbook develops practical operating decisions and records. Here I examine the mechanisms and evidence behind a claim of control: how delegation creates exposure, how prevention and intervention can constrain it, and what warrants continued reliance when conditions change.
Section 6 proposes a comparison that could support or weaken the harm-reduction hypothesis, including a simpler design that could satisfy the commitment more effectively. This paper specifies that test; it does not report having performed it.