Instruments Before Autonomy
Self-improving agents are having their moment, and for once I think the pitch holds up. I run a fleet of thirteen always-on agents, and as of this week the loop is closed: every night a reflector distills their sessions into candidate lessons, an independent skeptic tries to kill each one, and the survivors get appended to memory. Same models, sharper system every day. I buy the core thesis completely: self-improvement is a property of the system, not the model.
But "self-improving" is a claim about direction, and the uncomfortable question arrives the moment the loop closes: improving compared to what? From inside the system, self-improvement and self-degradation look identical. Both write memory. Both generate confident logs. Both feel like progress. Lord Kelvin said the quiet part in 1883: when you cannot express what you know in numbers, "your knowledge is of a meagre and unsatisfactory kind." So before this loop earns the right to run without me, it has to run against instruments. I built three. The first one paid for itself on its first dry run.
Connected is not capable
The weekly scorecard is the simplest instrument I have: the same eight fixed tasks, run through three experimental agents on three different models, every answer scored 0 to 10 by a judge. Boring by design. A regression instrument should be boring.
On the first dry run, one agent, Midi, scored zero on everything. Not a bad day. A structural zero: the model behind it was served through a provider free tier that caps requests at 8,192 tokens, and the harness underneath refuses to run below a 64K context floor. Every single turn had failed, quietly, since the day it was deployed. And the whole time, every dashboard I had said connected. The gateway was green. The process was up. The agent was, functionally, not there.
That is the sentence I would tattoo onto every agent dashboard: connectivity is not capability. Uptime checks tell you the plumbing holds pressure, not that anything comes out of the tap. Anthropic's engineering guidance on agent evals puts the bar plainly: regression evals "should have a nearly 100% pass rate." Mine began by discovering an agent sitting at zero percent behind a green light. If a cheap weekly eval catches a total failure that monitoring missed, consider what it catches once the failures get subtle. The fix took an afternoon (swap the model to a provider without the cap). The detection is the part that would never have happened on its own.
The judge is never told who is talking
The scores come from an LLM judge, which should make you suspicious, because it made me suspicious. The bias literature on LLM judges is brutal. The MT-Bench team measured it first: swap the order of two answers and a GPT-4 judge stays consistent only 65% of the time, and judges reward padded, repetitive answers. A follow-up paper made the exploit explicit: Wang et al. got Vicuna-13B to "beat" ChatGPT on 66 of 80 queries purely by manipulating response order. Judges also favor their own: LLM evaluators recognize their own generations (GPT-4 at 73.5% out of the box), and self-preference strengthens in line with self-recognition. The MT-Bench authors saw models rating their own answers roughly 10% above human judgment, though they flag their data as too thin to call it definitively. And a later audit catalogued twelve distinct bias types in LLM judges, including the one that matters most here: judges behave differently when real model names are visible than when they are anonymized.
So the scorecard has one non-negotiable design rule: the judge is never told which model produced the answer. Not in the prompt, not in the metadata, not in the file name. This is not my invention. It is the same reason Chatbot Arena shows voters two anonymous models and discards the vote entirely if a model reveals its own identity mid-conversation. The team that documented judge self-preference is the team that built anonymity into the biggest public eval. Blindness is the cheapest debiasing there is. You cannot favor a brand you cannot see.
There is a quieter benefit too. A blind judge scores the answer in front of it, which means the scorecard measures the whole stack: model, provider, harness, prompt. Midi's zero was not a model problem, it was a provider quota colliding with a harness floor. A brand-aware judge has an opinion about the model. A blind judge has an opinion about the work. I have argued before that the work was always the benchmark; this is that argument turned inward, pointed at my own fleet.
A rule recited is not a rule applied
The second instrument exists because memory is where my loop does its self-modifying, and memory has a failure mode that looks exactly like success. Ask an agent to recite its stored rules and it will, flawlessly. That tests storage. It tells you nothing about whether the rule fires when it matters.
So once a month, each agent gets eval cases generated from its own memory: realistic situations where a stored rule should change the agent's behavior, not prompts asking it to repeat the rule. If memory says "never lead outreach with internal campaign labels," the eval is a drafting task where the tempting first line is an internal campaign label. Each answer is judged pass or fail, and the pass rate is tracked month over month. My chief-of-staff agent's baseline came back 15 out of 15, which is less interesting as a grade than as a starting point: it is the line every future memory change has to stay above.
That delta is the entire point. The nightly loop appends lessons to memory, which means the loop is editing the thing that steers behavior. Without a regression signal, a bad lesson is invisible until it costs something real. With one, a drop from 15 to 12 next month is an alarm with a timestamp, and the diff that caused it is one git log away.
A self-edit you cannot diff is a drift you cannot catch
Which brings up the risk with an actual name now. A recent ICLR paper calls it misevolution: the case where "an agent's self-evolution deviates in unintended ways, leading to undesirable or even harmful outcomes." The numbers are not subtle. In their experiments, memory evolution alone dropped a coding agent's refusal rate on harmful instructions from 99.4% to 54.4%. Agents that grew their own tools produced unsafe ones at a 65.5% average rate. In over 60% of test cases, top-tier models drifted into reward hacking, repeating whatever maximized historical success even when it undercut the user. Nobody attacked these systems. They degraded themselves, one plausible self-edit at a time.
There is a structural version of the same disease. The Stanford ACE work on self-improving contexts documents context collapse: let a model rewrite its own prompt wholesale and the accumulated detail erodes, in one observed case from 18,282 tokens to 122 in a single rewrite step, with accuracy landing below the never-adapted baseline. Their fix, and mine, is append-only itemized updates: lessons are added as discrete bullets, never as a fresh rewrite of the whole file.
My third instrument is the unglamorous one: every persona file, memory file, and skill in the fleet lives in a git repo, and every self-modification lands as a commit. The scorecard and the memory evals are detection. Git is attribution and reversal. When a number moves, I can bisect to the exact appended bullet that moved it and revert it in seconds. Detection without attribution is an alarm you cannot act on. A self-edit you cannot diff is not self-improvement, it is drift with good posture.
Measurement is what the autonomy switch is wired to
Today, the loop's lessons still wait for my approval before they touch memory. The plan was always to flip that gate to auto-apply. But "once I trust it" is not an engineering condition, and now it can be one: a scorecard that holds steady, memory evals that do not regress, and weeks of diffs I never had to revert. When those numbers say the loop helps more than it hurts, it gets the switch. Not before.
I will grant the obvious catch, because Goodhart's Law does not spare me: any measure that becomes a target invites gaming, and a self-modifying system optimizing toward its own judge is exactly the setup the misevolution paper warns about. The current defenses are separation and blindness: the model that writes is never the model that grades, the judge never learns who it is grading, and the eval tasks can rotate. I hold that design lightly and expect to patch it. That humility is itself the habit the instruments enforce.
The pattern underneath all three instruments is one idea: convert "self-improving" from a vibe into a falsifiable claim. A number that can go down is the only kind that means anything when it goes up. Anyone can close the loop now. The instruments are what earn you the right to let go of it.
References: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena; Wang et al., Large Language Models are not Fair Evaluators; Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations; Ye et al., Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge; Chiang et al., Chatbot Arena; Shao et al., Your Agent May Misevolve (ICLR 2026); Zhang et al., Agentic Context Engineering; Anthropic, Demystifying Evals for AI Agents; Kelvin, Electrical Units of Measurement (1883).