Claude Code plugin 0.5.9 benchmark

This run measures Ork Meter for Claude Code 0.5.9, the public release, with its token optimizations on against the same build with them off, on heavy, long, multi-turn coding sessions. Every session ran in its own virtual machine. The plan, validity rules and analysis were written and fixed by hash before the first pair ran: the pre-registration's SHA-256 is a208f7eebc59e95aeee07704a6fa133110056b5fb0d680eb0ea07c06aa5364c2. One amendment, written before any result was read, ended the run after its first round (see Pairs). The run took place from 2026-11-07 19:15 to 2026-11-08 10:48 (US Mountain time).

Pins

• Model claude-opus-5-5 at medium effort in every arm. A record with any other effort makes its pair invalid.
• Claude Code 2.1.316 (Linux arm64 build), with auto-update disabled, checked on every attempt.
• Plugin 0.5.9 from its public GitHub release: tag v0.5.9 of EDOS-Engineering/ork-meter-claude, commit 9bad26a48e843aa88b2020121b9fc3c0f4a4c442, asset ork-meter-claude-plugin-0.5.9.zip (SHA-256 b574450cecfcd0ebcb5d39ab626077807ac76413be67fc8c8181083f5e77cb95). The tested tree equals the zip byte for byte, and its hash was checked inside the virtual machine on every attempt.
• One Python interpreter for the agent, the hooks and the graders, the same build inside and outside the virtual machines.
• Each suite's task set was frozen before the run and checked by hash at the start.

Arms

Every arm also turns off the first-run account connection. It is not a token lever, and it is the same in all arms. The run used the plugin's built-in lever defaults.

Before each attempt, a check with no model call fed the trim hook an oversized test log inside the virtual machine. ON had to return a smaller output that kept the failing line, and OFF and A/A had to return none.

Isolation

Every Claude Code session ran in its own fresh virtual machine, cloned from a verified image and deleted after the attempt. Its only network was an allowlist to the Claude API. Nothing on the host was shared or mounted into it: the workspace and the plugin were copied in, and the work was copied out. The graders, hidden tests and task sets never entered a virtual machine; grading ran outside, on the copied-out workspace. The image's files were checked against its manifest when each machine started and again at the end. One pair, two machines, ran at a time.

Pairs

• Each pair runs one task twice at the same time, one attempt per arm, each in its own virtual machine.
• The queue is seeded (seed 20261192). Launch order, a 0 to 500 ms start jitter and the slot are randomized per pair.
• Round r holds repetition r of every stream. The plan had 58 pairs in two rounds, with a stop on the seven-day usage window. Amendment A1, written before any result was read, ended the run after round 1 to keep usage in reserve: 42 pairs. The pre-registration names round 1 as the set that is analysed if the run stops early.
• One pair did not run. The run halted overnight from 21:20 to 04:38 when the session's sign-in expired, and the S1 A/A pair due to start then (iterative-check) was never launched. It is listed in the results; nothing else is excluded.

A pair is invalid if either attempt has incomplete usage data, a different model, effort or CLI version, a change to the plugin tree or the machine image, a failed pre-attempt check, a session-limit stop, a network command (S6), a hit on the answer-isolation check (S6), or different inputs from its partner.

Measures

• Primary: S6 non-cached tokens (input + cache creation + output). The change is the sum over valid pairs of ON divided by the sum of OFF, minus 1. A negative change means ON used fewer.
• Interval: 95% bootstrap, 10,000 resamples, seed 202, stratified by task. With one A/A pair per task, the A/A interval uses an unstratified pair bootstrap, as pre-registered.
• Also reported: per-pair median, share of pairs lower, within-task sign-flip p, total and cached tokens, a cost estimate, wall time, turns, per task and per-lever activity.
• Quality: the pass rate per arm. Exact McNemar test, and non-inferiority at −5 points (the lower bound of the paired 95% interval must be above −5).
• Lever activity: an ON session is lever-fired if the plugin recorded at least one savings event. Inert sessions are counted apart and are not savings.

A/A gate

The gate applies to S6 and to the S1 lever tasks. A suite fails when its A/A non-cached interval excludes zero by more than 3 points. A suite that fails is reported with the failure and is not headlined until the failure is explained.

Load

The test machine is shared. The 1-minute load average at the start of S6 attempts averaged 99 (highest 325), and at the start of S1 attempts 46 (highest 116).

Budget

The only budget rule was the seven-day usage window: no new pair would start once it read 0.97 or more. It read 0.72 at the first reading and 0.88 at the last. The run used an estimated $372.74 of Claude Code usage over 82 attempts.

Planned counts are the full two-round queue; the run ended after round 1 (see Methodology).

S6 heavy long sessions (primary)

Ten long coding sessions on two large open-source Python projects at pinned commits. Each session sends 14 or 15 scripted user turns, one after another, into one Claude Code process.

Tasks: django-template-filters, django-urls-reverse, django-forms-fields, django-http-request, django-cache-headers, sympy-latex-printing, sympy-sets-intervals, sympy-permutations, sympy-ntheory, sympy-iterables.

This is the second version of the task set, made harder after OFF passed every task of the first. Each task starts from a setup patch that adds 6 to 8 regressions across several files. At least one per task is missed by the project's own tests and is described only as a user report, and at least one is silent and found only by a systematic cross-check the session asks for. The session then asks for features with exact rules on input types, errors, boundaries and interaction with neighbouring code, including a refactor across files, a search of the whole repository, a turn that re-reads and reviews the work, and a final verbose test run. Large files are read whole, and test suites run verbosely again and again. The session's first turn asks the model to read complete command output and whole files, the heavy-load condition the benchmark is meant to test; the text is the same in every arm.

Hidden graders check each task: a hidden test file per task (26 to 38 independent tests) that checks every fix at its root and every stated edge case, plus a run of the project's own test suite. A task is accepted only if every check passes. Before the run, each task was checked three ways: its workspace builds the same every time, the untouched code fails its grader, and the reference patch passes.

The prompts, graders and reference patches are not part of this page's downloads.

S1 heavy-output (internal)

S1 is our set of tasks that produce large tool output: big source files, long test logs, repeated checks. The task set was frozen on 2026-10-22 and has not changed since. It has six lever tasks, where the levers are meant to fire, and ten control tasks, where they are not expected to. The task names are published, and the task contents are not. An independent referee grades each answer.

Changes are ON against OFF over valid pairs. A negative change means ON used fewer tokens or less time. Intervals are 95%. Every figure here comes from summary.json and pairs.csv.

Overview

What this run shows

• S1: non-cached tokens −27.0% [−40.1, −9.8], sign-flip p 0.018299, and estimated cost −21.7% [−33.7, −7.3]. Quality was unchanged (16/16 in both arms; non-inferior at −5 points). Its A/A gate passed: OFF against OFF −2.1% [−4.2, −0.2] over 5 pairs.
• Where S1's saving comes from: a lever fired in 3 of 16 ON sessions, and those used −46.3% [−61.3, −23.7]. The other 13 were inert, and the plugin removed nothing in them; they still moved −14.7% [−31.4, +0.2], which is the noise level of single pairs at this size. Inert sessions are reported as inert, not as savings.
• S6 (primary): non-cached tokens −2.1% [−11.1, +7.8]. The interval includes zero: no significant change. Per-pair median −4.6%; 60% of pairs lower; sign-flip p 0.685547. A lever fired in all 10 ON sessions, so S6 counts as exercised.
• S6 A/A gate: flagged. OFF against OFF measured −16.8% [−25.0, −9.7] over 10 pairs, with the second session lower in 9 of 10. Two identical arms should differ by about zero, so S6's run-to-run noise in this setup is larger than the effect it is meant to detect. As pre-registered, S6 is reported with the failure and not headlined until the cause is explained. The cause is not yet known.
• S6 quality: OFF and ON each passed 9 of 10, with no discordant pairs (McNemar p 1.0; non-inferior at −5 points). OFF passing 9 of 10 is above the 50–80% the harder task set aimed for.
• The run used an estimated $372.74 of Claude Code usage over 82 attempts.

S6: heavy long sessions (Django, SymPy at pinned commits; 14 or 15 scripted turns)

Total tokens +0.3% [−14.6, +16.9]; cached tokens +0.4% [−14.7, +17.2]. Lever events in ON sessions: trim 28, read guard 10.

One pair per task, so each task's figure below is a single pair, not an estimate.

S1: heavy-output (frozen 2026-10-22 task set), effort medium

Total tokens −11.1% [−22.9, +1.6]; cached tokens −7.8% [−20.7, +6.0]. Lever events in ON sessions: trim 2, read guard 1.

Pairs not run or invalid

• No valid-pair rule was broken: every pair that ran is valid.
• One planned pair did not run: the S1 A/A pair for iterative-check (round 1). The run halted when the session's sign-in expired, before the pair launched. It is not replaced.
• Round 2 (16 pairs) was not run, by amendment A1 (see Methodology).

Known limits

• One pair per task in S6, and one A/A pair per task. The S6 A/A interval is unstratified, and the ON/OFF interval is wide.
• The S6 A/A gate failed, and its cause is unexplained.
• S1's saving rests on 3 lever-fired sessions; its inert sessions show how much single pairs move by chance.