We build AI agents and automation for a living. So this may read as an odd article for us to publish. We want to talk about where AI should not act on its own, and which decisions still need a person.
We publish it because the question comes up in almost every project. A client asks the assistant to draft a follow-up, and it produces something excellent. They then ask a fair question. If it can do that, why can it not just send the email? Or update the invoice? Or push the change to production?
Sometimes it can. Often it should not. This article explains how to tell the difference.
Trust is a dial, not a switch
Anthropic’s own guidance on this makes a useful point. Trust in AI is not a yes-or-no question. It works more like a dial that you turn up or down depending on what is at stake.
They use a medical analogy. You might take casual advice about eating more vegetables without checking it. You would want a second opinion before surgery. The same logic applies to your business systems. A summary of yesterday’s enquiries deserves less scrutiny than a change to a customer’s payment record.
So the right question is never “do we trust AI?” It is “how much does this particular action cost us if the AI gets it wrong?”
The two ways AI gets things wrong
Two failure modes matter most in business use.
Hallucination. The model produces information that sounds plausible but is not true. It might invent an order number, a policy detail or a contact name. In a chat window you notice. Inside an automated workflow, that invented value can travel straight into your database.
Sycophancy. The model tries to please you rather than correct you. Ask “this discount is approved, right?” and a sycophantic answer agrees. Ask the same question neutrally and you may get a different response. Leading questions produce agreeable answers.
There is a third problem that catches experienced people out. Confident presentation hides weak content. A well-structured answer with headings, bullet points and citations feels more reliable than a hesitant one. That feeling is not evidence. Format tells you nothing about accuracy, and neat output can make a wrong answer harder to spot.
Reading, writing and deploying are three different risks
When we design AI into a client’s systems, we separate what the AI does into three levels. This one distinction prevents most problems.
Level 1: the AI reads and suggests. It summarises tickets, drafts replies, scores leads, answers questions about your data. A person decides what happens next. Mistakes cost seconds.
Level 2: the AI acts, and a person approves. It prepares the record update, queues the email, or drafts the invoice. Nothing takes effect until someone clicks approve. Mistakes get caught at the gate.
Level 3: the AI acts on its own. It writes to live systems without review. This is appropriate for narrow, reversible, well-tested actions. It is dangerous everywhere else.
Most organisations should run at Level 1 and Level 2 for the first several months. Move individual actions to Level 3 only after you have watched them behave correctly, and only where a mistake is cheap to undo.
Where a human must stay in the loop
These are the areas where we insist on human approval in client systems, whatever the AI’s accuracy looks like in testing.
Money leaving the business. Payment approvals, refunds, supplier payments and payroll. AI can prepare the batch and flag anomalies. A person authorises it.
Client funds and regulated accounts. Trust accounts, escrow and any account governed by professional rules. The compliance risk sits with the firm, not the software.
Accounting entries and financial reporting. AI can match, suggest and reconcile. Someone qualified signs off the ledger and anything filed with a tax authority.
Contracts, quotes and pricing. An AI-generated discount or contract clause can create a binding commitment. Draft with AI. Approve with people.
HR decisions. Hiring, performance, discipline and termination. These carry legal exposure and affect people’s livelihoods. AI may summarise. It must not decide.
Legal, medical and safety advice. Anything a professional would need to sign. AI drafts a starting point at best.
Anything a customer receives that carries a commitment. Delivery promises, service levels, apology compensation, legal notices. A wrong number here becomes a dispute.
Irreversible data operations. Deleting records, merging duplicates in bulk, mass field updates. If undo is not possible, approval is mandatory.
Anything that touches personal data at scale. Exports, cross-border transfers and bulk processing all carry data-protection obligations under GDPR and similar laws. Those obligations do not transfer to a model.
Where AI should never hold deployment or write access
Some access should not be granted at all, regardless of approval steps. In our client systems, these stay closed.
- Production deployment. No AI agent pushes code to production, changes infrastructure, or runs migrations unsupervised. Use a pipeline with review, staging and rollback, exactly as you would for any developer.
- Direct write access to the production database. Agents work through an API with defined operations. They do not run arbitrary SQL against live data.
- Credentials, permissions and user roles. An agent that can widen its own access has no meaningful limits. Keep identity management with humans.
- Payment execution. Preparing a payment is fine. Releasing funds is not.
- Bulk outbound communication. No agent sends to a full customer list without a person reviewing the content and the audience. A bad send cannot be recalled.
- Deleting or overwriting records without a recovery path. If you cannot restore it within minutes, the agent should not be able to destroy it.
- Security and compliance controls. Firewall rules, audit settings, retention policies and consent records stay under human control.
- Anything with no audit trail. If you cannot see what the agent did and why, you cannot govern it.
Where AI genuinely earns its place
None of this argues against AI. It argues for putting it where the value is high and the risk is low. These uses work well with light oversight.
- Summarising calls, tickets, emails and long documents
- Drafting replies, proposals and internal updates for a person to review
- Classifying and routing enquiries
- Scoring and prioritising leads
- Answering questions about your own data in plain language
- Flagging anomalies for a human to inspect
- Enriching and tidying records inside a defined, reversible scope
- Generating internal reports and first-draft analysis
These uses save real hours. They also fail safely, because a person sees the output before it matters.
How we build the guardrails
Good AI implementation is mostly boring engineering. Here is what we put around an agent before it touches a client’s live systems.
Scoped permissions. The agent gets its own identity with the narrowest access that lets it do its job. It does not inherit an administrator’s rights.
An explicit action list. We define exactly which operations the agent may perform. Anything outside that list fails rather than improvising.
Approval gates with thresholds. Small, routine actions proceed. Anything above an agreed value, volume or risk level waits for a person.
Dry runs first. The agent runs in shadow mode against real data and writes nothing. We compare its proposed actions with what your team actually did.
Full audit logging. Every action records what happened, which data informed it and who approved it. This matters for compliance and for debugging.
Rate and volume limits. An agent that misbehaves should affect ten records, not ten thousand.
Rollback and recovery. We test the undo path before go-live, not after an incident.
Ongoing evaluation. Accuracy drifts as your data and processes change. We review agent decisions on a schedule rather than assuming last quarter’s performance holds.
Five questions to ask before you give an AI agent write access
- What is the worst realistic outcome if this action is wrong? If the answer involves money, regulators or reputation, add an approval step.
- Can we undo it, and how quickly? Reversible actions can run with less supervision.
- Would we let a new employee do this unsupervised in their first week? If not, the agent should not either.
- Can we see what it did? No audit trail, no autonomy.
- Who is accountable when it goes wrong? The answer must be a named person. It cannot be the model.
Our position at ClubCode Technology
Building AI agents, CRM automation and business-process automation across Zoho, Salesforce and custom platforms is what we are expert at. We take a deliberate position on this. We would rather deliver a system your team trusts than one that impresses in a demo and worries your finance director.
In practice that means we design the approval gates with you at the start of the project, not after an incident. We tell you which parts of your process should stay manual. And when a client asks for full autonomy in a place we think is unsafe, we say so and explain why.
Adopting AI well is not about handing over control. It is about deciding, deliberately, where the dial should sit for each decision your business makes.
Want help mapping which parts of your process suit automation and which need a person? Email info@clubcodetechnology.com or use the contact form on this site.