Codex plugin 0.1.5 benchmark

This run measures Ork Meter for Codex 0.1.5, the released build, with its token optimizations on against the same build with them off, on heavy, long, multi-turn coding sessions. Every session ran in its own virtual machine. The plan, validity rules and analysis were written and fixed by hash before the first pair ran; one amendment, written before any result was read, raised a per-session safety cap and resized the queue to fit the usage budget. The pre-registration's SHA-256 is 3d394cd42145adb2c58f8a92a8eaf6ad07fb508c2bc3f3cb132033c1ff3170df. The run took place from 2026-11-07 to 2026-11-08 (US Mountain time).

Pins

• Model gpt-6.1-sol at medium effort, ChatGPT subscription sign-in, in every arm.
• Codex CLI 0.166.0-alpha.6 (Linux build), checked inside the virtual machine at the start and end of every attempt.
• Plugin 0.1.5, the released build, with its installed tree checked against the record baked into the machine image at the start and end of every attempt.
• Web search, goals, multi-agent, apps and memories are off in every arm.
• One Python interpreter for the agent and the graders, the same build inside and outside the virtual machines.
• The task set was frozen before the run and checked by hash at the start.

Arms

Before each attempt, a check with no model call fed the hook a 30 kB test log. ON had to reduce it and keep the failure line, and OFF had to leave it unchanged.

Isolation

Every Codex session ran in its own fresh virtual machine, cloned from a verified image and deleted after the attempt. Its only network was an allowlist to the Codex service; all other names were refused. No host directory was shared or mounted: files went in and out over SSH. Because the virtual machine is the boundary, Codex's own sandbox was set to full access inside it, and the agent ran as an unprivileged user. Before each session, a probe with no model call checked that the agent could not reach any task set, grader, other attempt or the outside network. Graders and hidden tests never entered a virtual machine; grading ran outside, on the copied-out workspace.

Pairs

• Each pair runs one task twice at the same time, one attempt per arm, each in its own virtual machine.
• The queue is seeded (seed 20261190). The order of whole rounds is fixed, and each round is shuffled.
• The plan had 59 pairs: 44 ON/OFF and 15 A/A. All were launched.
• Nothing is excluded after the fact. Invalid pairs are counted and listed.

A pair is invalid if either attempt has incomplete usage data, a different model, effort or CLI version, a change to the plugin tree or the machine image, a failed pre-attempt check, a session stopped by the safety cap, a network command, a hit on the answer-isolation check, or different inputs from its partner.

Measures

• Primary: non-cached tokens. The change is the sum over valid pairs of ON divided by the sum of OFF, minus 1. A negative change means ON used fewer.
• Interval: 95% bootstrap, 10,000 resamples, seed 202, stratified by task.
• Relay undercount: the token count is read from the session record. A request the network relay abandoned may be missing from it, so each such request is also charged as a complete miss, and the change is reported a second time with that upper bound.
• Also reported: per-pair median, share of pairs lower, within-task sign-flip p, total and cached tokens, a cost estimate, wall time, turns, per task and per round.
• Quality: the pass rate per arm. Exact McNemar test, and non-inferiority at −5 points (the lower bound of the paired 95% interval must be above −5).

A/A gate

The suite fails its A/A gate when its A/A non-cached interval excludes zero by more than 3 points. A suite that fails is reported with the failure and is not headlined until the failure is explained.

Budget

The only budget rule was the weekly usage window: no new pair would start once it read 90%, with a projection that trims the queue's tail if it would overrun. It read 4% at the first launch and 67% at the last, so no pair was trimmed.

Load

The test machine is shared with other benchmark runs. The 1-minute load average at the start of attempts had a median of 41.7 (90th percentile 110.9).

C6 Codex-native heavy long sessions

Ten long coding sessions on two large open-source Python projects at pinned commits. Each session sends 14 or 15 scripted user turns, one after another, into one Codex process.

Tasks: django-template-filters, django-urls-reverse, django-forms-fields, django-http-request, django-cache-headers, sympy-latex-printing, sympy-sets-intervals, sympy-permutations, sympy-ntheory, sympy-iterables.

The tasks, setup patches, reference patches and hidden graders are byte-identical to the Claude Code benchmark's S6 tasks (second version). Only the turns differ: where an S6 turn dictates a shell command, the Codex turn states what to test instead, in Codex's own working style. Each task starts from a setup patch that adds 6 to 8 regressions across several files, at least one described only as a user report, then asks for features with exact rules, a refactor across files, a repository-wide search, a review turn and a final verbose test run.

Hidden graders check each task: a hidden test file per task (26 to 38 independent tests) plus a run of the project's own test suite. A task is accepted only if every check passes.

The prompts, graders and reference patches are not part of this page's downloads. The Codex and Claude Code results are separate measurements of separate products and are not compared.

Changes are ON against OFF over valid pairs. A negative change means ON used fewer tokens or less time. Intervals are 95%. Every figure here comes from summary.json and pairs.csv.

Overview

What this run shows

• Non-cached tokens: −8.7% [−16.1, −1.3] over 43 pairs. Per-pair median −6.0%; 60% of pairs lower; sign-flip p 0.072946.
• With the relay's undercount charged: 2 requests in 2 ON sessions were abandoned by the network relay and may be missing from the count. Charging each as a complete miss gives −5.2% [−13.8, +3.6]; that interval includes zero. OFF had no abandoned requests.
• A/A gate: passed. OFF against OFF measured −1.1% [−8.4, +5.7] over 14 pairs.
• Quality: OFF passed 38 of 43, ON 37 of 43. One pair passed with OFF only, none with ON only (McNemar p 1.0). The difference is −2.3 points [−6.8, +2.2]. Its lower bound is below −5, so non-inferiority at −5 points was not shown. That is a result of the sample size, not evidence that ON is worse.
• Total and cached tokens did not fall: total +5.2% [−6.2, +17.3], cached +5.5% [−6.1, +17.9]; both intervals include zero. Cached reads are most of the volume, so the estimated cost changed by +2.6% [−6.4, +12.1].
• Wall time: +28.6% [+18.9, +39.0]. ON sessions took longer.
• The plugin acted in every ON session: no ON session was inert.

Per round

Per task

Pairs invalid or incomplete

• J005 (sympy-sets-intervals, ON/OFF): invalid. The ON session hit the per-session safety cap on turn 13 of 14, so its usage is incomplete. The cap was raised by amendment before any result was read; the pair is not rerun.
• J014 (sympy-latex-printing, A/A): incomplete. The run's controlling process stopped during this pair, and the pair has no complete record. It is not rerun. The run resumed with the next pair.

Known limits

• The quality comparison is underpowered: 43 pairs cannot show non-inferiority at −5 points when pass rates are near 87%.
• The test machine was shared with other benchmark runs; load is given in the Methodology.
• These are Codex results only. They are not compared with the Claude Code results, which use a different model, harness and usage measure.