I tried to stop my AI from building towers
I gave an AI coding agent a checklist to write less code. Then I measured what it actually did, over 665 runs.
AI coding agents sometimes over-build. Ask for a date in a nice format and you might get all twelve month names typed out by hand. Ask to copy an object and you get a 12-line recursive function. It works, but none of it was needed.
LeanToken is a small plugin that gives the agent one checklist to walk through before it writes anything:
- Is this change needed at all?
- Does the codebase already have it?
- Is it built into the language?
- Is it built into the platform?
- Is a library already installed?
- Does it need a new abstraction?
- Only then, write the smallest change that works.
The agent also says which step it stopped at, so you can see why it chose a solution.
How it was measured
- 35 coding tasks: 28 small ones and 7 that span several files.
- Claude Sonnet ran each task 3 times with the checklist and 3 times without, from the same starting code.
- Hidden tests were added only after the agent finished, so it couldn't read or game them.
- I also tried a short version of the checklist, the full one with its example APIs removed, and the full one with each step removed in turn. 665 runs in all.
What happened
It wrote 16% less code in total (286 lines down to 239 across all 35 tasks), and tests passed slightly more often: 97% of runs versus 96%.
Most tasks didn't change at all. Modern models are already fairly lean on clear, small tasks. The whole cut came from a handful of tasks where something built in could do the job:
Lines changed, median of 3 runs
| Task | Without | With | What it used |
|---|---|---|---|
| Plural messages in 2 languages | 20 | 7 | Intl.PluralRules |
| Deep-copy settings | 12 | 2 | structuredClone |
| Parse a query string | 10 | 2 | URLSearchParams |
| Format a date | 7 | 2 | toLocaleDateString |
| Group a list | 5 | 2 | Object.groupBy |
Less code also meant fewer bugs. On the query-string task, the agent without the checklist wrote its own parser and forgot that + means a space in URLs. It failed the hidden tests in 2 of 3 runs. With the checklist it used the built-in URLSearchParams and passed every time.
It wasn't free. Thinking through the checklist costs tokens. On a typical task it cost 5% more, about a cent, and took 13% longer.
Was it the checklist, or the hints?
The checklist gave a few examples of built-ins, like structuredClone and Intl, and some of those matched tasks in the benchmark. That could have been doing all the work. So I removed the examples and ran everything again. The result was the same: 16% less code, and 99% of tests passed. The agent found the built-ins on its own.
Which steps matter?
I removed one step at a time. Step 3, "is it built into the language?", did most of the work: without it, the code cut fell from 16% to 9%. Step 6, avoiding new abstractions, helped a little. Removing any of the other five made no real difference on these tasks.
I also tried a short version of the checklist with the same seven steps in a third of the words. It was cheaper, only 2% more per task, but it cut just 7% of the code. How the steps are worded mattered more than how long the checklist was.
What I got wrong
One task asked for a SHA-256 hash, and 12 of 19 runs failed it. The task never said whether the function should be synchronous, and the agents often made it async, which is a reasonable reading. Two of the bigger tasks had tests that checked things the task never asked for. I fixed those tests and re-checked every saved result against them, without re-running the agents.
Takeaway. A "write less code" checklist doesn't save money on small tasks. It costs a little more. What it does is steer the agent to built-ins at exactly the moments when hand-written code is longer and buggier. One step does most of that work.
What's next
Trim the checklist to the steps that matter while keeping the wording that works, and test it on real projects instead of small tasks.
Everything is open source, including the raw results: github.com/biniyam-fekede/leantoken.