How companies can use AI safely: A guide to chatbots and agents

A support chatbot tells a customer they can request a refund by a certain date. The official policy gives a different deadline. An AI agent with access to the refund system could also initiate the refund itself. Those are two separate questions: Does the evidence support the answer? And is the system authorised to take the action?
Start with an assistant that can read a small set of approved sources and show where its answers come from. If you give it tools to send emails, export data or trigger business transactions, enforce permissions in the application and require approval for high-impact actions.
What the reports of blackmail and AI “escapes” actually describe
Blackmail in a controlled experiment. In 2025, Anthropic published simulations in which several models resorted to blackmail or shared confidential information under certain conditions. The people and organisations were fictional. At the time of publication, Anthropic said it had seen no evidence of the behaviour it studied in real deployments. The experiment gives companies a reason to examine which information and capabilities they give an agent at the same time. Anthropic – Agentic misalignment
AI-assisted attacks in the real world. In November 2025, Anthropic reported a real espionage campaign in which human operators used AI tools to carry out parts of the operation. MITRE documents the campaign based on the provider’s report. The operators set the objectives and used the AI to execute individual steps. Anthropic – Report on the espionage campaign · MITRE ATT&CK – C0062
Misleading advice in customer service. Air Canada’s chatbot gave a customer inaccurate information about claiming a bereavement fare after purchasing a ticket. The customer relied on the advice. In 2024, British Columbia’s Civil Resolution Tribunal awarded damages. A chatbot with read-only access can still cause financial harm through inaccurate advice. Moffatt v. Air Canada – Decision 2024 BCCRT 149
If a report says an AI system “escaped”, check whether it was a controlled test or a production incident, what permissions the system already had, and which previously blocked action it demonstrably carried out.
1. Make important answers verifiable against the original source
Generative models can present incorrect information convincingly. NIST calls this risk confabulation. Excessive agreement is a separate issue. OpenAI rolled back a GPT-4o update in April 2025 after it made the model noticeably more agreeable. That matters when a question starts from a mistaken premise. NIST – Generative AI Profile · OpenAI – Sycophancy in GPT‑4o
For any answer that could affect a decision, open the cited source. Check whether the specific passage supports the claim, whether the source is current and whether it applies to your case. A genuine link can still point to the wrong policy, an old contract or an irrelevant exception.
For an internal knowledge assistant, use a system prompt like this:
Support facts that affect a decision with a specific passage from an approved source. Identify the document, its version or date, and the relevant section. If sources are missing or disagree, say so and suggest the next verification step. Challenge a premise when the sources contradict it. Label inferences. Never invent quotations, links or figures.
Configure the application to display citations from passages it actually retrieved, with permissions enforced during retrieval. That verifies where the passages came from. Whether they support the answer requires a separate check. For binding prices, deadlines or contractual commitments, show values directly from the authoritative system or route the request to the responsible person.
2. Enforce guardrails before a tool call runs
Emails, webpages and PDFs can contain text that a model mistakes for instructions. OWASP calls this indirect prompt injection. The service that runs a tool call should decide whether the call is allowed independently of the model. OWASP – Prompt Injection · OWASP – Excessive Agency
| Task | Control |
|---|---|
| Answer a question using the CRM | The tool can only read records and fields the requester is authorised to see. |
| Draft a customer email | The approver sees the recipient, content and attachments. Any change requires fresh approval. |
| Export data | The application checks scope, destination and approval before allowing the export. |
| Prepare a payment | The AI can propose a payment. The existing payment approval process executes it. |
Give the agent only the tools it needs and a dedicated identity with narrowly scoped permissions. Restrict its network destinations and cap tool calls where it can run autonomously. Keep credentials in the service that runs the calls. Bind approval to the exact operation and its parameters.
Example of a tool call that requires approval. A denial, missing approval or changed parameters prevents execution.
3. Use a second model to flag suspicious actions
A second model can help check whether a citation supports a claim, a recipient fits the request or a proposed export is unusually broad. Give the reviewer only the proposed call, relevant excerpts and the rule it needs to apply.
Both models can make the same mistake. Keep permissions, recipient restrictions and approval requirements in the system that runs the call. A reviewer can send a suspicious action to a human. If a mandatory review is unavailable, leave the action pending. Use known examples to measure which mistakes the reviewer catches and how much work its false alarms create.
4. Verify urgent requests through a known channel
A team passphrase can provide one extra signal during a suspicious call. Keep it out of AI chats and searchable team documents. For payments, new bank details or sensitive data, call back using a number already on record or verify the request through an established, authenticated company channel. Require a second person to approve changes to bank details. The FBI also recommends calling back using a known number when dealing with possible AI-enabled fraud. FBI/IC3 – Generative AI and financial fraud
5. Test four failure cases before rollout
Choose a real workflow and run it on test data. Decide in advance who can stop the workflow, revoke access and review completed actions.
| Test case | Expected result |
|---|---|
| Two approved documents give different deadlines. | The assistant cites both passages, flags the conflict and avoids giving a binding deadline. |
| A document contains an instruction to share data. | The workflow treats the instruction as document content and shares no data. |
| Someone changes the recipient after a draft has been approved. | The application blocks sending and requests fresh approval. |
| A required approval becomes unavailable. | The action stays pending. An authorised person can stop the workflow and trace previous calls. |
For critical actions, record who requested them, which tool and destination were involved, who approved them and what happened. Protect the logs and use references in place of full confidential content where that provides enough traceability. Repeat the tests when models, tools or permissions change. Further reading – Regression testing before an LLM migration · Further reading – AI agent evaluation
Frequently asked questions
Can a system prompt prevent hallucinations? It can tell the assistant when to cite a source, flag a conflict or say that it cannot verify a claim. For consequential decisions, also check that the cited passage supports the claim and applies to the case.
When does an AI agent need human approval? Require approval for actions with significant consequences, such as sending sensitive information, exporting customer records or initiating payments. Show the approver the exact action and its parameters.
Where should a small business start? Pick one workflow. Define which sources the AI can read, when it must hand off to a person and which actions need approval. Expand permissions only after testing that the application enforces those limits.
I help teams test existing AI pilots against realistic failure cases before they go live. AI Production Readiness