Agent documentation benchmark

Same agent. Same task. Different documentation.

AI coding agents spend much of their effort looking things up. We measured what happens when an off-the-shelf agent gets its platform knowledge from GetPlane instead of the open web: the same knowledge, reshaped for machines and served from the GetPlane AI control plane.

GetPlane, Inc.
getplane.ai
Task D4, retrain window orchestrator
Claude Code 2.1.280 with Claude Sonnet 5
What we tested

One agent, one engineering task, two ways to find the answers.

Before a coding agent can write working code against an unfamiliar platform, it has to learn that platform's tools, field names and types. Today it does that by searching the web and reading pages written for people. We ran the same agent on the same task and changed only where its documentation came from.

Out of the box

The standard setup

Claude Code 2.1.280 with Claude Sonnet 5, default configuration, web search and fetch enabled. It pulls the platform's public documentation pages as it needs them, the way coding agents work today.

Agent→Web search→Human docs pages
vs
With GetPlane

The same agent, the same knowledge, with no internet

The platform documentation is reshaped into Knowledge Content Objects (NCO, patent pending) and served from the GetPlane AI control plane through its MCP reader.

Agent→GetPlane control plane→NCO corpus

The difference is shape, not content. Web documentation is organised as pages for people to browse. An NCO corpus is organised for a machine to look things up: one fixed vocabulary, an index of every exact name, and a catalogue of every tool.

Held constant

Agent and model (Claude Code 2.1.280, Claude Sonnet 5), prompt, starter code, simulated backend, hidden acceptance tests, and call and time limits. A third lane with no documentation at all confirms the task can't be solved from memory.

How a run is scored

The agent works until it declares the task done. Hidden tests it never sees then decide pass or fail. We record wall-clock time, tokens and list-price cost for every run.

GetPlane MCP reader with NCO corpus v4, compared with the standard setup

Half the time.Half the cost.

In the standard setup, the median run took 13.2 minutes and $2.75. With GetPlane serving the NCO corpus and no web access, the median run took 6.1 minutes and $1.38. Both passed every hidden test.

GetPlane vs standard setupmedians, 3 runs each
54%
less time to a tested solution6.1 min vs 13.2 min
54%
fewer tokens3.03M vs 6.52M
52%
lower cost per accepted solution$1.33 vs $2.77
3/3
runs passed every hidden testthe standard setup also passed 3/3
The race

Every GetPlane run finished before the fastest standard-setup run.

Each dot is one complete run: the agent reads the documentation, writes and tests its code, and stops; hidden acceptance tests then judge the result. The slowest GetPlane run took 6.7 minutes; the fastest standard-setup run took 10.9.

GetPlane MCP reader, NCO v4 NCO v4, plain file reader Standard setup
← swipe to see the full chart →
Output tokens (the agent's own writing)
−57%24.9k vs 57.3k
Cheapest single run
$1.15standard best $2.47
Spread between runs
2.4 minstandard 5.8 min
Unit economics

What each accepted solution costs.

Cost per accepted solution is the total spend of all runs divided by the runs that passed. Prices are the model provider's list prices.

Cost per accepted solution
$2.77→$1.33
−52%
Tokens per solution
6.52M→3.03M
−54%
Output tokens per solution
57.3k→24.9k
−57%
Minutes per solution
13.2→6.1
−54%
Accepted solutions per $100
Standard setup36
GetPlane75
Accepted solutions per agent-hour
Standard setup4.5
GetPlane9.8

Derived from the medians above: $100 / $2.77 and $100 / $1.33; 60 / 13.2 and 60 / 6.1 minutes. The agent runs one task at a time.

How we got here

Each version was measured on the same task and tests.

Median tokens per run. Each change to the corpus or the reader was pre-registered, then run with the same agent, task and hidden tests.

← swipe to see the full chart →
Baseline

Standard setup, web access

$2.77
cost per accepted solution
13.2 min, 6.52M tokens
NCO v2

Machine corpus, file reader

$2.30
cost per accepted solution
−17%, 7.8 min
NCO v4

GPX corpus format

$1.45
cost per accepted solution
−48%, 6.3 min
NCO v4 + GetPlane MCP

The purpose-built reader

$1.33
cost per accepted solution
−52%, 6.1 min
What the agent did differently

It read less and rewrote less.

Web documentation is organised as pages for people. The GPX corpus format uses one fixed vocabulary, an exact-name index, and a catalogue of every tool. The GetPlane MCP reader returns one tool at a time. The run logs show the agents working in four steps.

1

Read the catalogue

A list of every tool and what it does. Every agent read it first or near the start (nco_open).

2

Look up exact names

Field and type names come from an index instead of guesswork (nco_find).

3

Fetch one tool

Its input, output and the types it uses, in one call. No read was truncated in any of the six v4 runs (nco_get).

4

Write the code

The agent produced 57% fewer output tokens than the standard setup: less drafting and rework.

The evidence

Supporting results

This task needs documentation

The task cannot be solved from memory.

With no documentation at all, the same agent passed once in three runs. The two failures used their full 300-call budget trying request shapes against the API.

1/3
No documentation
passed
3/3
GetPlane, NCO v4
passed
Smaller, and nothing lost

The corpus is less than half its previous size.

Corpus v4 carries the whole reference in 2.25 MB against 5.05 MB. Every field reference resolves, every exact name is indexed, no string is lost, and a rebuild comes out byte-identical.

−55%
Corpus size
5.05 → 2.25 MB
100%
Build checks
674 of 674 files validate
Faster than the plain file

The reader adds to the format's gain.

With the same v4 corpus, reading through the GetPlane MCP reader instead of opening the files directly used 13% fewer tokens and cost 8% less per accepted solution.

3.49M
v4, file reader
$1.45 per accepted
3.03M
v4, GetPlane MCP
$1.33 per accepted
Network access

It works without internet access.

Every GetPlane run reached one network host, the model API. The documentation stayed on the machine, which matters for teams that cannot let agents browse the internet.

1
GetPlane
host: the model API
2–4
Standard setup
hosts incl. Google's docs and live API
Confirmed on a second task

On a second task, GetPlane again halves the time and cost.

The NCO corpus format was developed on the retrain task above. To test it independently, we drew a new task at random from tools the format was not shaped around: an audit of a managed prompt library against policy documents. We ran it seven times in each setup: the standard setup with web access, the same agent with no documentation, and GetPlane’s NCO corpus through the MCP reader and as plain files, 42 runs in all.

Minutes to a tested solution
13.0→6.3
−51%
Tokens per solution
6.18M→3.49M
−43%
Cost per accepted solution
$3.16→$1.56
−50%
Runs that passed every hidden test
7 of 7→7 of 7
Equal
No documentation at all
3 of 7passed, $6.17 per pass
NCO as plain files vs web
−41%cost, $1.87 vs $3.16
Corpus format, NCO v4 vs v2
−54%tokens, MCP reader

Cards compare the standard setup with web access against NCO v4 through the GetPlane MCP reader, 7 runs each, medians. The GetPlane runs were made one day before the web and no-documentation runs, with the same agent, model, task and hidden tests. The web agent read Google’s public documentation; GetPlane runs reached only the model API. A gain the same size as on the retrain task shows the earlier result was not an artefact of developing the format on that task.

Summary

Same result.
Half the time.
Half the cost.

On a task that requires documentation, an AI coding agent working from GetPlane's NCO corpus and reader produced the same accepted result as the same agent in its standard setup with web access, in 54% less time, with 54% fewer tokens and at 52% lower cost per accepted solution.

MethodPre-registered plan committed before any scored run. Seeded run order. Every run recorded, pass or fail.
Held constantClaude Code 2.1.280, Claude Sonnet 5, identical prompts, starter code, simulated backend, hidden tests and limits.
These figuresThe retrain window task, 3 runs per setup, medians, for everything above the second-task section. The prompt audit task, 7 runs per setup, medians, for that section. Cost is the list-price estimate.
Full recordFinal Consolidated Results, Specification (As Run), RESULTS-4 and every run's data are retained in the benchmark repository.