AI Agents Are Getting More Powerful. Who’s Checking Their Work?

Why continuous oversight matters for AI agents, custom GPTs, plugins, and the AI tools your business already uses.

Artificial intelligence is evolving faster than most businesses can keep up with. New models, integrations, and autonomous agents arrive at a remarkable pace, and businesses are embracing them to save time, reduce costs, and improve productivity.

But there’s an important question that isn’t getting enough attention: Who is making sure these AI tools continue to work as intended after they’ve been approved and deployed?

This applies to nearly every form of AI businesses use today, from autonomous agents to custom GPTs, plugins, chatbots, and AI features embedded in everyday software.

A business might carefully evaluate an AI solution before implementing it. The tool produces accurate results and follows established rules. Employees are trained, and management approves its use. Six months later, is anyone checking whether those things are still true?

The AI tool you approved in January may not be the same tool you’re using in October.

AI Doesn’t Stand Still After Deployment

Traditional software changes through updates and new releases, which is why organizations rely on change management, testing, and monitoring. AI adds another layer of complexity: it can change even when the business using it hasn’t modified anything.

Consider what might happen over time:

The underlying model changes. A provider releases an updated model with different capabilities, limitations, or behavior.
Integrations evolve. A plugin or connected application gains new functionality or requests broader permissions.
Instructions behave differently. A custom GPT that consistently followed specific guidance may respond differently on a new model.
Automation expands. A tool designed to recommend actions may gain the ability to execute them.
Business use changes. Employees begin relying on an assistant for decisions beyond its originally approved purpose.

Many updates improve reliability, security, and performance. But improvement in one area doesn’t guarantee that every existing safeguard or expected behavior remains unchanged. That’s why initial testing alone isn’t enough.

The Growing Gap Between AI Adoption and Oversight

This concern isn’t hypothetical.

In a survey published by Ernst & Young in September 2026, 98% of senior AI executives reported having formal AI governance policies. Yet 47% said their organizations had previously not applied their governance process to urgent AI deployments. And 26% of respondents whose organizations used agentic AI said they couldn’t detect unauthorized AI agents operating internally.

The survey covered 202 senior AI executives at publicly traded U.S. companies with at least $1 billion in annual revenue, so its findings shouldn’t automatically be generalized to small businesses. But they show how difficult AI oversight can be, even for organizations with substantial resources.

Meanwhile, the National Institute of Standards and Technology (NIST) published research in March 2026 on the challenges of monitoring AI systems after deployment, emphasizing that testing before deployment cannot fully capture how AI systems will behave under real-world conditions.

The message is clear: having an AI governance policy is not the same as verifying that AI systems remain appropriately controlled.

A Simple Business Example

Imagine a small accounting firm that introduces an AI assistant to help employees summarize client correspondence.

Before implementation, the firm tests the assistant using sample documents. It produces accurate summaries, follows instructions, and has no permission to modify client records. Management approves the tool.

Over the next several months, the underlying model is updated. The platform adds new integrations. Employees discover the assistant can also draft client responses and start using it for more complex financial correspondence.

Nothing necessarily goes wrong. But important questions arise:

Does the assistant still produce accurate summaries?
Does it distinguish between verified facts and assumptions?
Can it now access additional client information?
Are employees relying on it for work that was never evaluated?
And most importantly, who is responsible for answering these questions?

The firm may believe it’s using the same approved tool. It may even have the same name and interface. But its capabilities, behavior, and business use may have changed significantly.

An approval made six months ago doesn’t automatically provide assurance about the system operating today.

It’s Not Just About AI Agents

Autonomous agents deserve attention because they can take actions: managing customer inquiries, updating records, scheduling appointments. The more authority an agent has, the greater the consequences when something goes wrong.

But risk doesn’t require autonomy. A custom GPT can produce misleading recommendations. A plugin can gain access to sensitive information. An assistant embedded in accounting software can influence financial decisions without anyone noticing its model has changed. Even a basic chatbot creates problems if employees treat its answers as authoritative.

The right level of oversight depends on how the AI is used, what information it can access, and the consequences of an error. The principle is the same: if your business relies on AI, someone needs to remain accountable for how that AI performs.

From One-Time Approval to Continuous AI Assurance

Businesses don’t need elaborate governance programs to address this. They need a repeatable process that ensures AI tools are periodically evaluated and meaningful changes don’t go unnoticed.

I think of this as continuous AI assurance. Rather than asking whether a tool passed its initial evaluation, it asks whether the tool remains appropriate, reliable, and controlled throughout its use.

For small businesses, that can begin with five practical steps.

1. Know What AI Your Business Is Using

Create a simple inventory of AI tools: standalone chatbots, custom GPTs, plugins, connected applications, automated workflows, and AI functionality embedded in other software.

For each tool, document its purpose, who uses it, what information it can access, and who is responsible for overseeing it. You can’t monitor something you don’t know exists.

2. Establish a Baseline

Define what acceptable performance looks like. This doesn’t require sophisticated testing software.

For example, a business using AI to draft customer emails might create a set of sample requests and expected results. The tool should use accurate information, maintain an appropriate tone, protect confidential details, and avoid making unauthorized commitments. Save those scenarios and results; they become the baseline for comparing future performance.

Ideally, you set a baseline before adopting a tool. But if you’re already using AI, it isn’t too late. Start today with the tools you rely on most.

3. Monitor Changes and Reevaluate Risk

AI providers regularly introduce new models, features, and integrations. Review release announcements, significant functionality changes, and permission requests.

Not every update requires a full reassessment. A minor interface change carries different risk than granting an assistant permission to modify financial records. Periodic reviews may be enough for lower-risk tools. AI involved in sensitive data, financial transactions, regulatory obligations, or customer-facing decisions warrants more frequent monitoring and stronger controls.

Meaningful changes should trigger a reassessment rather than waiting for the next scheduled review.

4. Retest What Matters

Rerun your test scenarios periodically. For important business functions, consider testing:

Accuracy: Are outputs still reliable and appropriate?
Instructions: Does the AI still follow established restrictions?
Permissions: Can it access, share, or modify information beyond its intended scope?
Actions: Does it require approval before executing consequential tasks?
Security and privacy: Are sensitive information and connected systems protected?
Human oversight: Are employees reviewing outputs and decisions where necessary?

Because AI responses can vary, a single successful test doesn’t prove a system will always behave correctly. Repeated testing, review of actual outputs, and tracking of errors give a more useful picture.

The objective isn’t to guarantee AI will never make a mistake. It’s to detect meaningful changes and failures before they create unacceptable consequences.

5. Assign Accountability

Every business AI tool should have an owner. That person doesn’t need to be an IT professional or AI expert, but they do need to understand the tool’s purpose, its limitations, and when an issue should be escalated.

The owner should know when the tool was last reviewed, whether its capabilities have changed, what problems employees have encountered, and who can restrict or discontinue its use. For tools that can take actions, establish clear boundaries around what the AI can do independently and what requires human approval.

Without ownership, even well-intentioned monitoring tends to become inconsistent.

AI Vendors Have Responsibilities, Too

Businesses shouldn’t shoulder this alone. AI providers are better positioned to understand changes to their models, infrastructure, and capabilities, and they should communicate meaningful information about updates, security controls, known limitations, and changes that could affect customers.

But vendor oversight and business oversight serve different purposes. A provider can test whether a new model performs well against its own criteria. It can’t determine whether that model still meets every customer’s specific requirements.

A vendor may tell you its newest model is more capable. Only your business can determine whether it’s still appropriate for your workflow.

Governance Should Enable AI Adoption, Not Prevent It

The purpose of oversight isn’t to make AI adoption complicated. It’s to give businesses enough confidence to use these technologies responsibly and to expand their use where it makes sense.

Effective governance also means proportionality. An assistant helping brainstorm social media ideas doesn’t warrant the same oversight as an agent authorized to issue customer refunds. Small businesses can start with simple documentation, a handful of recurring tests, and periodic reviews, and grow their oversight as their AI use grows.

The Question Every Business Should Be Asking

AI agents will become more capable. Models will keep evolving. Plugins and integrations will offer increasingly powerful functionality. None of that eliminates the need for human accountability. The more businesses depend on AI, the more important that accountability becomes.

The question isn’t whether your AI worked correctly when you first adopted it.

The question is whether anyone is checking that it still does.

A Practical Next Step

If your business already uses AI, take 30 minutes this week and answer five questions for each tool you rely on:

What do we use it for?
What information can it access, and what can it change?
Who owns it?
When was it last reviewed?
Has anything about it, or how we use it, changed since then?

That simple inventory is the foundation for everything else. When you’re ready to go a step further, download my free Before You Trust the Output: AI Risk Checklist (PDF). You don’t need to stop innovating. You need to make sure your oversight keeps pace.

At juliecella.com, I explore practical ways small businesses can adopt AI, improve operations, and manage emerging technology risks without unnecessary complexity.

Sources

Ernst & Young (September 2026). EY Survey Finds That Autonomous AI Implementation Outpaces Oversight.
National Institute of Standards and Technology (March 2026). Challenges to the Monitoring of Deployed AI Systems (NIST AI 800-4).