Keeping an AI agent out of production: the AI writes code; a person merges, runs production and types secrets

The setting

Together with Claude Code, we added a new kind of product (about 70,000 items) to a service: add database tables, fix the daily import, mix the items into the site's search, and make 70,000 vectors (numbers for searching by meaning). Three repositories were involved; production was PostgreSQL on a VM and a site on Cloudflare Workers.

This article records how the work was split and what actually happened. How to write the settings is covered in the permissions article and the .env experiment.

The split: how much to give the AI

The line is simple. What stays local and can be undone goes to the AI. What goes outside, is hard to undo, or involves secrets goes to a person.

A person runs things on the production server

The deploy script logs into the VM over SSH, copies files, applies database changes, runs once, then enables the timers. The AI does not run it.

When the AI once issued an SSH command meaning only to read the production database, auto mode's check (another model that reviews each action) stopped it. Entering the production server counts as a person's job, even just to read.

Instead, the AI moves on by reading the output the person pastes.

• Person: runs the deploy script and pastes the output as is
• AI: reads it and confirms, for example, that “the new table was created this time (no ‘already exists’ notice)” and “the imports of all three categories completed”
• AI: what it cannot check, it hands over as a command for the person, including what output means success

Scripts are written so that running them again gives the same result (CREATE TABLE IF NOT EXISTS, an UPDATE that changes nothing the second time). If a person can stop halfway and just run it again, it is easy to ask them to.

Secrets stay out of the conversation

Writing 70,000 vectors into the production database needed its password. The AI never gets it. The person types it in their own terminal, without showing it.

read -s "PGPASSWORD?DB password: " && export PGPASSWORD && echo
PG_URL='postgres://app@127.0.0.1:55440/app' node scripts/embed-products.mjs --other
unset PGPASSWORD

• read -s does not echo the input (zsh syntax; in bash, read -rs -p "..." PGPASSWORD)
• The connection string has no password; it comes from PGPASSWORD, which Node's pg and psql both read. A password with symbols needs no URL escaping
• unset it afterwards. It stays in neither the shell history nor the AI's conversation

The AI checks whether a value exists, not the value. The deploy script starts by checking, without printing anything, that both required variables are in the settings file.

[ "$(grep -c -E '^(API_ID|AFFILIATE_ID)=.' "$ENV_FILE")" = 2 ] || { echo "API_ID and AFFILIATE_ID not both in $ENV_FILE"; exit 1; }

Tokens work the same way: the person types them locally and pipes them over SSH into a file only root can read. They are kept in a password manager (1Password).

What actually went wrong

1. A push before permission

Once, the AI pushed a commit that added CI settings without waiting for permission. The person stopped it (“What are you doing?”), and the AI explained what happened and apologized. No harm was done, but a push leaves the machine.

Since then, “do not push or deploy until clearly told to” is written in the AI's memory (the project notes) and read in every conversation. You need a way to keep a rule said once in force in the next conversation too.

2. A script the AI wrote failed first in production

The AI added an --other option to the vector script so that it targets the new table. When the person ran it against production, it failed at once.

70568 titles to embed
error: syntax error at or near "0.0059967041015625"

Inside the loop, a variable named vectors holding the computed vectors hid the variable of the same name that held the table's name. INSERT INTO ${vectors} got a list of numbers instead of a table name.

It did no harm because:

• It failed on the first write, so nothing had been written
• The script only targets rows with no vector yet, so after the fix it could simply run again (70,000 items in about 10 minutes)
• A person ran it and pasted the output as is, so the AI found the cause right away

The lesson: after adding an option, run a little of it on the local database (--limit 100, say) before handing it to production. The search SQL had local tests; this script did not.

Checking from outside can go to the AI

Checking after the deployment only reads published pages, so it went to the AI.

• Whether the sitemap lists the new products (36,407 for 2026)
• Whether a new product's page opens with 200 and shows the “similar titles” block
• Whether existing pages load as fast as before
• robots.txt, and whether pages carry noindex

Actions whose requests directly cost money or skew numbers (following ad links, calling a paid AI) are avoided even when checking. The chat search costs money on every call, so the AI did not try it and reported “not tried”.

Tell the AI about the environment's limits too

The AI does not know limits nobody told it. In this project, the notes said:

• The internal disk is full, so temporary files go on the external SSD (run dev servers and tests with TMPDIR=/Volumes/.../.tmp)
• Do not stop processes you did not start (other work's dev servers run on the same machine)
• A person makes changes on the production server
• Never open ad links yourself (it skews the click numbers)

Each is a small rule, but unless it is written down, the AI trips at the same place every time.

Checklist

• Push, merge and production runs wait for a person's explicit instruction
• Secrets enter neither the AI's conversation nor shell history (read -s, environment variables, a password manager)
• The AI checks only whether a secret exists, never its value
• Scripts for production give the same result when run again, and can resume after stopping
• New options and scripts run in small amounts locally before production
• Rules decided once are written where the next conversation reads them (notes, CLAUDE.md)
• The AI reports what it did not check as not checked