Short answer
What makes an AI agent connected to business systems reliable is design, not the model's intelligence: narrow tool permissions, server-side validation, human approval for high-impact actions, an action log and a way to roll back. Instructions coming from untrusted content must not be treated as authority.
Limit authority to the task
Define readable and writable resources separately. Prompt instructions alone are not authorisation; validate tool calls on the server.
High-impact actions
Deletion, payments and external messages may require human approval. Verify approved parameters have not changed. Reversibility and audit records belong in the design.
Untrusted content
Instructions inside emails or documents must not become system authority. Test requests to invoke tools, exfiltrate data or misdirect the task through external content.
Operational limits
Define step and cost budgets, timeouts, safe stopping and handover to a person. Start with a narrow task and limited access; completed task count is not the only measure of success.
What to watch in the first pilot
Set up the first pilot with a narrow task and mostly read-only permissions. Grant write permission only for reversible, low-impact actions, with a traceable record. Judge success not by the number of completed tasks but together with the number of wrong actions and the rate of human intervention.
A practical evaluation exercise
For a CRM update tool, restrict which fields the agent may change. Test access to another customer’s record, duplicate execution and a delayed tool response. Keep action identifiers and approval records, and define recovery for each step. Put human approval at a separate boundary before consequential actions such as deletion or customer messaging.
Frequently asked questions
Two questions we are often asked on this topic.
What is prompt injection and why does it matter?
It is the risk that instructions hidden in an email or document are executed by the agent like commands. The defence is not instructing the model but validating tool calls and permissions on the server side; the sources below explain these risk classes in detail.
Who is responsible if an agent makes a mistake?
Responsibility should be defined at the design stage: who monitors, who approves, which action is rolled back on error. An agent with no clear owner should not be connected to business systems.
Sources and further reading
