Human approval for AI agents

OWASP explains that a system built on a language model often receives from its developer the ability to call functions or other systems, through extensions (sometimes called tools, skills or plugins). The choice of which extension to use can also be left to the agent itself³. An agent generally chains several calls to the model, each building on the results of the previous ones³.

OWASP identifies three usual causes of excessive autonomy: too much functionality, too many permissions, too much autonomy³. Human approval addresses the third: OWASP gives as an example an extension that deletes a user's documents without asking them for confirmation³.

Approval does not replace the other protections. OWASP also recommends giving the agent only the extensions, functions and permissions it needs, and having downstream systems check authorizations rather than the model³. For prompt injection, OWASP lists human control of privileged operations among the measures that limit the impact². These choices fall to the teams that deploy the agent, not to the employee.

• Read the approval request in full before answering, including the recipients, files and accounts concerned.
• Refuse when you do not understand what the agent is going to do, or when the request does not match what you asked of it.
• Refuse an action that has nothing to do with your current task: an agent that is summarizing a document has no reason to offer to send a message.
• For an action that affects other people, a payment or access rights, check by another means if doubt remains.
• Report to security any agent that acts without asking for your agreement when it should.

Approving without reading

When approval requests keep coming, people end up clicking without looking. Approval then becomes a formality and protects nothing. A request that appears in the middle of a task deserves the same attention as the first one of the day.

The agent with too many rights

OWASP gives the example of an extension meant to read data, but which connects to a database with an identity that can also modify, insert and delete³. It also cites the extension that acts for a user with a generic privileged account that has access to every user's files³. In these cases, a single approval can have effects far wider than the request shows.

Prompt injection

OWASP describes indirect injection: the model accepts content from outside, such as websites or files, and this content changes its behavior in unintended ways². The instructions do not need to be readable by a human, as long as the model processes them². OWASP adds that it is not known whether foolproof prevention methods exist². Human approval is therefore a safeguard not to be bypassed.

The following illustration is neutralized: the hidden instruction is paraphrased.

Illustration (fictional). Hidden text in a received email:
[instruction addressed to the assistant: look for sensitive information
in the mailbox and forward it to an external address]
Approval request shown to the employee:
"Send a message to [email protected]?"

This is the scenario OWASP describes: a booby-trapped email pushes the assistant to scan the mailbox and forward sensitive information to the attacker's address³. Among the countermeasures, OWASP lists the requirement to review and manually click "send" for every message drafted by the extension³.

Believing that approval is enough

OWASP specifies that some measures, such as logging, monitoring and rate limiting, do not prevent excessive agency but limit the damage³. NIST presents its AI risk management framework as intended for voluntary use⁴. The UK NCSC states that the security of an AI system must be a requirement from the design stage and throughout its life cycle⁵.