AI coding / Field notes

Before you add more AI agents, give one a way to prove its work

By · · 6 min read

The part of AI coding that slows me down is deciding whether the work is actually finished. An agent can produce a convincing explanation and a clean diff while leaving me to launch the app, repeat the user journey, and discover what it missed.

Before running more agents, I need a reliable way to check the output of one.

In this article
  1. Start with a user journey the agent can verify
  2. Turn repeated review findings into checks
  3. Add a lint ratchet to an existing codebase
  4. Make the correct path easy to find
  5. Audit rules before deleting them
  6. Save the heavier tools for a real problem
  7. A daily loop I can actually use

These are the changes I chose for my projects to make that check easier.

Start with a user journey the agent can verify

"Run the tests" is useful, but it is not a complete description of what a user does. A test suite can pass while a button leads nowhere or an offline transition loses the state the user expected to keep.

A feature map closes part of that gap. For each feature, write down:

For an import feature, that might mean starting with an empty library, selecting a synthetic file, checking the imported rows, and confirming that a rejected file produces a useful error. The map should also explain how to clean up.

I added a map for six existing journeys in intent: trade. The first verified browser path was the signed-out app load. That is evidence for that path, not proof that billing, signed-in sync, or every other journey works. Naming that limit matters as much as recording the pass.

Keep the map next to the code. When a change alters a journey, update its entry in the same change. A stale recipe can be worse than no recipe because it gives the agent confidence in the wrong steps.

Turn repeated review findings into checks

Written instructions are useful for judgment and context. They are weaker at preventing the same mechanical mistake every week.

When a review finding repeats, ask what would make the next occurrence fail automatically. An undefined variable belongs in lint. A forbidden import belongs in an import guard. A concurrency bug belongs in a regression test that reproduces the failure.

The check should sit at the narrowest layer that can enforce the rule. Adding another paragraph to an instruction file does not make a compiler or CI job reject the mistake.

This also changes what review is for. Reviewers can spend more time on the decisions a check cannot settle: whether the behavior fits the product, whether the boundary is sensible, and whether the implementation creates unnecessary work for the next person.

Add a lint ratchet to an existing codebase

Enabling lint in a mature project can produce a wall of existing errors. Making all of them a prerequisite for one small improvement turns the change into a cleanup project.

A baseline offers a practical starting point. Record the existing findings, then fail CI when a file's count for a rule increases. As old findings are fixed, prune the baseline.

That is the approach I added to intent: trade, alongside an existing import guard that had not been running in CI. The existing tests stayed in place.

There is a limit worth understanding: a count-based baseline can miss a new violation if an old violation of the same rule disappears in the same file. It limits growth in the recorded counts; it does not prove that every changed line is clean. Review and the rest of the test suite still have work to do.

Make the correct path easy to find

Agents learn from the patterns already in a repository. If the same job has four competing implementations, a longer instruction file will not reliably tell the next agent which one to copy.

Before creating a new helper or skill, look for the existing route. If there is already a verification command, improve its inputs instead of adding another command with nearly the same name. If a feature has one clear home, put its code and documentation there.

The goal is a codebase where the obvious implementation follows the intended boundaries. This helps people as well as agents.

Audit rules before deleting them

One idea I tested was shortening written rules wherever a check already enforced them. The audit changed my mind about doing a broad cleanup.

Much of the text expressed judgment or operating constraints that no existing check enforced. The proposed cuts also lost useful detail. I kept the rules rather than treating a smaller instruction file as an improvement by itself.

An audit can still produce a useful backlog. When a supposedly enforceable rule is broken repeatedly, that is a concrete reason to build its check. Until then, removing the instruction would remove the only protection in place.

Save the heavier tools for a real problem

I also considered skill evaluations and formal models for state machines. Both are useful tools, but neither needs to become a requirement for every small change.

A recurring skill failure is a reason to build a behavior evaluation using synthetic fixtures. A defect in a concurrent or stateful subsystem is a reason to consider a model. If a model finds a counterexample, replay it against the real implementation before calling it a product bug. A model checks the assumptions and bounds you gave it.

I also left automatic merging out of this work. More throughput is only useful when the verification and review process can support it.

A daily loop I can actually use

  1. Find the feature and its verification path.
  2. Make one bounded change.
  3. Run the relevant checks and drive the affected journey when available.
  4. Record the result and its limits.
  5. Review the remaining judgment calls.
  6. Turn repeated findings into checks.

That is where I would start before adding more agents. Give one agent a clear path, an observable finish line, and a check that catches yesterday's mistake.