blog·4 min read

Built with AI agents: the checklist from a real audit

Eight things we found while auditing a platform built with AI agents. Not one is a mistake by the model; all eight sit in the environment around it.

Go through my website for the obvious security mistakes: who can get in,
are the permissions right, and which packages are out of date?

A platform built with AI agents often looks better than a platform built by hand. Tidy code, consistent names, comments everywhere. And then you audit it and find eight things that have nothing to do with code.

That is exactly what stood out in an audit of such a platform this month. We name no names and no domains, but the list itself is worth holding up against your own project. Not one of the eight points is something a model got wrong. All eight are about the environment the agents worked in.

1. Two-factor authentication on the owner accounts

The first account on a project is almost always the account with the most rights, and almost always the account nobody looks at again. On a project that was built fast, it usually also holds the hosting login, the address for password resets and the link to the repository.

Switch on two-factor authentication for every account that owns something, not just the accounts you work with daily. It is ten minutes of work and it is the only point on this list with no trade-off at all.

2. Out of date packages nobody looks at

An agent picks a package, installs it, and that package stays on the version of that day. Work on the project for six months and you are left with a list of dependencies that contain known problems.

This is a check you can run in a single request and then repeat monthly. What matters is not the list but the habit: without a fixed moment nobody looks at it, because it is never urgent.

3. The permissions on your settings file

The file with your passwords and API keys should be readable by exactly one user. In practice it regularly sits on permissions that let any user on the server open it. On your own server that is not so bad. On shared hosting it is the difference between "my data" and "everybody's data".

An agent does not pay attention to this, because the file works fine with wider permissions. It is something you set correctly once and then verify.

4. Mockups left in the open

Building fast produces design samples, test pages and screens with example data. They are made to show and then forgotten. They are still online, without a login, findable by anyone who knows the folder or by a search engine that picks them up.

Go through your own project for folders with names like demo, test, mockup and old. Anything that is not part of the site does not belong in the web root.

5. The repository on the same server as the site

This is the most common one, and the least visible. The agents work on a server, the version control repository sits on that same server, and that repository holds the only history of the project. If the server fails you have lost not just the site but also the way back.

Put your repository somewhere else. It does not have to be an outside service if you would rather not; a second place is already enough.

6. No check that runs by itself

No test that runs on every change, no alert when the site goes down. On a project people work on, somebody notices within a day. On a project agents work on, there is no somebody.

Start small. A monitor that messages you when the site stops responding, and one check that runs before anything goes live. That alone is the difference between "I had no idea" and "I knew within two minutes".

7. Two agents in the same working folder

Two requests at once in the same folder is asking for trouble. Agent A reads a file, agent B writes it, agent A writes its own version back. The result is not an error message but work that quietly disappears, and you find out days later.

Let one request finish before you start the next, or give every request its own copy to work in.

8. Agents with full access to personal data

The heaviest of the eight. An agent that has to build a feature around user data usually does not need production data. Yet it often works in it, because that was the environment that happened to be ready.

Separate those. Building happens on data that does not belong to real people. If it concerns health, finances or children, this stops being a matter of good practice and becomes an obligation.

Common mistakes

Thinking this is about the model. Not one of the eight points changes if you pick a different model. They are about permissions, separation and checks. Those are choices made by whoever sets the project up.

Ticking the list off once. Points 2, 4 and 6 drift away as soon as you start building again. Give them a fixed moment, for example the first Monday of the month.

Trying to pin it all on the agent. A line in your instructions telling it to be careful is not protection. Permissions, separated environments and a place for your history are.

Waiting until the project is finished. The eight points together cost less than a day at the start. Afterwards, point 8 alone costs a rebuild.

If you want to go through this on your own hosting, you can ask the assistant for the first four points literally; the request above this article is a good start. What it can check by itself is covered in Which checks the assistant can run itself.

MK
Maarten Keizer

Founder of Invoker. Over twenty years in hosting, system administration and web development; builds the Claude hosting himself and tests everything on our own servers first.

about maarten