Measuring Whether the Agent Is Actually Making You Faster

You finish a feature in what feels like half the usual time, then spend Thursday afternoon working out why the retry logic behaves oddly under load, in code you accepted on Tuesday without fully reading. The ten minutes of typing you skipped are vivid. The forty minutes spent reviewing a diff you did not entirely understand are not, because reviewing feels like progress in a way that staring at a blank file does not.

That asymmetry is the whole measurement problem. Agents reliably make the visible part of the work shorter and the invisible part longer, so your gut reading of the speedup is biased before you start.

Why lines of code lie here, specifically

Diff size and lines per hour were always weak proxies. With an agent they become actively misleading, because the agent solves the same problem with more code than you would have written. It adds the defensive branch you would have skipped, the docstring, the three extra test cases, the error class. Your LOC per hour goes up sharply. Whether value per hour went up is a completely separate question that the number cannot answer.

The same applies to commit counts, files touched, and pull requests opened. All of them measure output volume, and output volume is the input the agent inflates. If you track those, you will conclude you got much faster, and you will be measuring the inflation.

What to measure instead

Four things, in rough order of how much trouble they save you.

Rework rate. Of the work the agent produced, what fraction did you revert or substantially rewrite within a week? Pick a definition and hold to it. A workable one: you changed more than half of it, or you threw it away entirely.

Cycle time. Wall clock from "task defined and understood" to "merged and working in the place it needs to work". Not typing time. The whole span, review included, because review is where the agent moves the cost.

Review time versus writing time. A simple split of where your hours actually went. If writing drops from four hours to one and review climbs from one to three, you saved an hour, not three.

Escaped defects. Bugs found after merge, attributed to the change that introduced them. Noisy, slow to accumulate, and worth the trouble anyway, because it is the only one that notices you shipped faster and worse.

The two-week log

The simplest honest experiment a solo developer can run needs no tooling. Open a plain text file. One line per task, appended as you go, never reconstructed from memory at the end of the day:

2026-08-03 | auth rate limiter        | agent | 09:15-11:40 | est 3h  | rework: no
2026-08-03 | vitest suite for parser  | agent | 13:00-13:50 | est 1.5h| rework: no
2026-08-04 | fix session race         | hand  | 09:30-12:15 | est 2h  | rework: -
2026-08-05 | auth rate limiter (redo) | hand  | 10:00-11:00 | est -   | rework: yes -> 08-03

Task, approach, start and end, your honest pre-task estimate of doing it by hand, and one line on whether it needed rework. Two weeks. Backfill the rework column on Fridays, since rework is only visible in hindsight.

The pre-task estimate is the weakest field and you should say so out loud. It is self-reported, it is biased, and estimating a task you are about to start with help is genuinely hard. It is still better than nothing, and writing it down before you begin at least stops you from adjusting it afterwards to match how you felt.

Comparing like with like when no two tasks are alike

You cannot pair tasks. Nobody writes the same feature twice under controlled conditions, and if you try you will spend the fortnight designing an experiment instead of doing the work.

Bucket instead. Four or five categories, defined before you start, assigned when you log the task rather than when you analyse it. Something like: boilerplate and scaffolding, tests, migrations and mechanical refactors, unfamiliar API surface, core domain logic. Buckets are coarse enough to be assigned in two seconds and specific enough that the variation inside one is smaller than the variation between them. That is all you need.

A worked example

Assumptions, stated so you can disagree with them. One developer, ten working days, roughly six hours of loggable task work per day. Estimates are pre-task and self-reported. Rework means more than half the output was rewritten or discarded within seven days, and rework hours are tracked separately from the original task.

Bucket	Tasks	Hours	Rework	Rework hrs	Hand estimate
Boilerplate, scaffolding	11	9.5	1	1	19
Tests	8	7	1	1	13
Migrations, refactors	5	6	0	0	10
Unfamiliar API surface	4	8	1	1	12
Core domain logic	6	21	4	8	24

Thirty-four tasks, 51.5 hours logged, 11 hours of rework, so 62.5 hours of actual time against 78 hours of estimated hand-written time. Net saving 15.5 hours over two weeks, a shade under 20 per cent. Overall rework rate is 7 of 34, about 21 per cent.

The headline number is the least interesting thing in the table. Split it.

The four non-domain buckets took 33.5 hours against 54 hours estimated, a saving of 20.5 hours, roughly 38 per cent. Core domain logic took 21 hours plus 8 hours of rework, 29 hours in total, against a 24 hour estimate. That bucket was about 20 per cent slower with the agent than without, and it burned the majority of the fortnight's rework.

Play it forward. Hand-write the domain work and use the agent for everything else, and the fortnight costs 33.5 plus 24, which is 57.5 hours instead of 62.5. Five hours better, from doing less with the agent rather than more.

Now look at rework rate by bucket. Boilerplate: 1 of 11, about 9 per cent. Core domain: 4 of 6, about 67 per cent. That spread was visible in week one, well before the hours added up to anything conclusive. Rework rate moves early, it needs no estimates, and it points directly at the buckets to stop delegating.

Where the gain is real, and where it goes negative

The reliable wins are boilerplate, test scaffolding, mechanical migrations, and unfamiliar API surface, where the agent's advantage is that it has read the documentation more recently than you have.

The reliable losses cluster around three things. Subtle concurrency and ordering bugs, where plausible-looking code is the most expensive possible output. Anything depending on domain knowledge that exists only in your head or in a conversation from six weeks ago. And code where you are the reviewer of last resort, because nobody else will catch it, so the review cost lands entirely on you and lands in full.

The trap you are walking into

Any metric you track starts steering you. Track cycle time alone and you will merge faster by reviewing less, which improves the number and degrades the software. Escaped defects are the counterweight, but they arrive weeks late.

Rework rate is the one to put on the wall. It is cheap to collect, it needs no estimate, and it catches the failure mode nearly everyone actually has, which is not slowness but confidently accepting output that has to be redone. A developer gaming rework rate downward has to either understand the diff properly before merging or delegate less of the work that keeps coming back. Both of those are the behaviour you wanted anyway.

Two weeks of logging is enough to find your own boundary between the buckets where the agent pays and the ones where it does not. That boundary is personal, it moves as the tools change, and it is worth re-checking about twice a year.