Rerun it yourself

git clone https://github.com/dfrostar/neuralmind && cd neuralmind
pip install -e . tiktoken                            # source checkout ships the harness
NEURALMIND_ORT_THREADS=1 python -m evals.public.run  # clones the pinned repos, prints the table

The harness lives in the source tree, not the PyPI wheel, so run it from a clone. The repos are cloned at fixed commit SHAs and nothing in the run is random, so recall and found-rate should match the tables below exactly. Token counts can shift slightly between machines — CI runners without AVX-512 land up to 0.8% off on a per-repo mean; NEURALMIND_ORT_THREADS=1 matches CI’s setup (what moves, and the evidence). Raw per-query data is committed in bench/public/results.json; the query set is in evals/public/manifest.json.

What it comes to

Two kinds of result, side by side: retrieval, where NeuralMind gets the right code into the window in far fewer tokens, and task outcomes, where a pre-registered pilot found it made no difference.

Retrieval: what reaches the window

Gold file found

40 of 40

Keyword ripgrep, top five files whole: 32 of 40.

Reproducible

Fewer tokens vs ripgrep

31–67×

The baseline closest to how an agent works.

Reproducible

Vs pasting every file

51–242×

670–961 mean tokens per question against 41,729–232,483.

Reproducible

Top-5 hit rate

93.3%

150 questions on 5 repos; 10 misses, listed.

Reproducible

Task outcomes: does the agent do better

Tasks solved

18 vs 18

Of 20, without and with NeuralMind.

Pilot

Cost per task

+0.0%

Median change with NeuralMind; 95% CI −5.2% to +8.6%.

Pilot

Runs solved

53 vs 55

Of 60 each, without and with.

Pilot

NeuralMind tool calls

0

In 60 runs with its 28 MCP tools available.

Pilot

What the pilots found, and why

Task outcomes: three pilots, no significant difference

Pilot

Fewer tokens in the window isn’t the same as a cheaper or better agent, so we pre-registered a task-outcome benchmark and ran its pilot on 2026-10-20: 20 real bug-fix tasks, Claude Code with and without NeuralMind, 3 seeds each, 120 runs. The claim it tested, that NeuralMind lowers cost per task without solving fewer, is not supported.

Task-outcome pilot, baseline against NeuralMind
Per arm	Without	With NeuralMind
Tasks solved	18 of 20	18 of 20
Runs solved	53 of 60	55 of 60
Cost per solved task	$0.227	$0.216
Median turns per task	10.3	10.8
Median explore calls before the first edit	4	4

Analyzer report All 120 runs Pre-registered spec

The same tasks on a local model: 10 vs 8, not significant

Pilot

A second, separate pilot ran the same 20 tasks with a different agent: Hermes-Agent v0.22.5 driving gpt-oss:20b-64k on one local GPU, 3 seeds per arm, 120 runs, finished 2026-10-30. NeuralMind reached it only through its Hermes plugin (a context block on every prompt, the session recap, edit learning); neither arm had an MCP server. A local model costs nothing per call, so the primary endpoint is tasks solved. The claim it tested, that the model solves more of these tasks with NeuralMind, is not supported.

Hermes pilot, baseline against NeuralMind
Per arm	Without	With NeuralMind
Tasks solved (most of 3 seeds)	8 of 20	10 of 20
Runs solved	26 of 60	29 of 60
Median minutes per task	18.4	17.9
Median turns per task	91.3	88.3
Runs that hit the timeout	3	0

Analyzer report All 120 runs Pre-registered spec

Carrying what NeuralMind learned from task to task: 9 vs 7, not significant

Pilot

A third pilot asked whether what NeuralMind learns during one task helps with the next. The same 20 tasks ran in a fixed order on Hermes-Agent v0.22.5 and gpt-oss:20b-64k, along 3 chains that each carried NeuralMind’s learned state (its synapse store and session recaps) from one task to the next, beside 3 fresh runs per task that started with nothing learned: 120 runs, side by side on the same days, finished 2026-11-01. The claim it tested, that the model solves more of these tasks with the carried state, is not supported.

Memory track, fresh against carried
Per condition	Fresh	Carried
Tasks solved (most of 3 runs)	7 of 20	9 of 20
Runs solved	21 of 60	25 of 60
Tasks where most context blocks led with a file the fix touched	11 of 20	11 of 20
Median minutes per task	14.7	15.8
Median turns per task	86.8	100.2

Analyzer report All 120 runs Pre-registered spec

What share of a bill is exploration

Field report

NeuralMind’s per-question ratio applies inside one slice of a session: the code an agent reads and searches before it changes anything. The size of that slice bounds what any retrieval tool can save, so we pre-registered the question and measured it on one developer’s own sessions, 2026-10-02 to 2026-11-01. On the Mac, about a fifth of what the Claude Code sessions were billed for was code being read or searched and re-sent, and about three in ten of the context the agent could see. On the second machine, running Hermes mostly for operations and chat, it was 10.5% of the billed prompt.

Exploration share of the token bill, interactive sessions, one developer’s two machines
Interactive sessions	Sessions	Billed prompt tokens	Share of recorded context	Share of billed prompt	Median per session
Claude Code, on the Mac	90	3,900,183,630	29.6%	19.0%	20.4%
Hermes, on the second machine	67	860,795,351	10.5%	10.5%	5.4%

Not usage, and never pooled into either row: the task-outcome harness’s 261 Hermes sessions spent 50.6% of their context on exploration. The harness drives a small model through a fixed turn budget, which reads and searches far more than a developer does.

Read it as a field report

One developer, two machines, 30 days. It is not a sample of developers or projects, and the Hermes sessions are mostly operations and chat, so they say little about what a coding agent reads.
Both figures sit within a few points of a threshold the spec fixed before any share was computed: under 10% is arithmetic a bill doesn’t feel, 10% to 30% is a claim that stands with the share beside it, 30% or more is a material part of the bill. Claude Code’s 29.6% of the recorded context reads “10% to 30%”, 0.4 points under the top line, and its 19.0% of the billed prompt is in the same band. Hermes’s 10.5% reads “10% to 30%”, 0.5 points over the bottom line.
Both are within noise. Across the passes of the computation, each corrected after code review and kept beside this one, the Claude Code headline has read 27.4%, 30.6%, 30.7%, 29.6%, on both sides of the line. The classifier fixes between two of those passes moved the pooled shares by only 0.03 and 0.02 points on identical sessions; what moves the headline is how one resumed conversation, three transcript files under one session id, is counted: about a point. The last pass counts each file as a session, as the spec defines one. So neither is a finding about the line. A nightly job adds Hermes sessions and the Mac is re-run by hand, so the figures will move.
Claude Code’s transcripts account for 64.2% of what was billed. The rest (tool schemas, system-reminders, thinking blocks) holds no tool results, so the share of the billed prompt is the recorded share times that ratio, not a bound. Hermes bills per session, so its estimate matches billing by construction and its two shares coincide.
It is a slice, not a saving. What NeuralMind does to a bill is the question the task-outcome pilot asked, and it found no difference.

The field report Pre-registered spec Claude Code data Hermes data

How it is built to be hard to dismiss

Six design decisions, each aimed at a specific way a vendor benchmark usually fails.

Real, recognizable repos: requests, click, flask and rich, pinned to release SHAs. Anyone can check out the commit and audit. No vendor fixture, no repos picked for being easy.
Objective gold, no LLM judge: Each query's gold file is the definition site of a named symbol, verifiable with one ripgrep command. A backend answers if and only if the gold file lands in its assembled context. Nothing to rig.
Cost and correctness together: The headline is always recall at N× fewer tokens, never a lone ratio. A token-reduction number with no correctness number attached is meaningless.
Strong baselines, disclosed: Not just whole-file dumps. A keyword baseline (ripgrep) and a vector baseline that returns the top 8 entries of NeuralMind’s own vector index — in effect, NeuralMind’s retrieval core in isolation. It sends a symbol’s name, module and docstring, not its code, so its token count is a floor: a chunk RAG that sends code bodies would pay more.
Pre-registered queries: Every query is committed to the manifest before tuning, and every one is reported. Losses are shown, not dropped.
Deterministic per machine: Synapse injection is off, the repos are pinned and nothing is sampled, so a re-run on the same machine is byte-identical. Across machines, recall and found-rate have come back identical. On the GitHub-hosted CI runners without AVX-512, token counts shift slightly, by up to 0.8% on a per-repo mean in this run, and so does the vector-RAG baseline’s MRR on click; an Apple M3, also without AVX-512, matched the committed run exactly.

Results, per repo

Tokenizer: tiktoken o200k_base. Token counts are shown for every backend rather than a ratio, so the columns compare directly, including where NeuralMind is the more expensive one.

requests

@ 0e322af87714 queries51.2× fewer tokens

requests at commit 0e322af877 — 14 pre-registered queries
Backend	Recall	Found	Tokens/query	MRR
full-file	1.00	100%	41,729	1.00
ripgrep	0.79	71%	26,543	0.60
embedding-rag	1.00	100%	229	0.92
neuralmind	1.00	100%	816	0.93

click

@ 874ca2bc1c7 queries117.1× fewer tokens

echo-util, missed from the v4.6.0 run until v4.12.0, is found again.

click at commit 874ca2bc1c — 7 pre-registered queries
Backend	Recall	Found	Tokens/query	MRR
full-file	1.00	100%	78,514	1.00
ripgrep	0.79	71%	45,059	0.60
embedding-rag	1.00	100%	220	0.86
neuralmind	1.00	100%	670	0.86

flask

@ c12a5d874c10 queries68.5× fewer tokens

xfile-dispatch-context, missed in the v4.10.0 run, is found at v4.12.0. The plain vector baseline misses two of these ten queries.

flask at commit c12a5d874c — 10 pre-registered queries
Backend	Recall	Found	Tokens/query	MRR
full-file	1.00	100%	59,013	1.00
ripgrep	0.85	80%	26,891	0.65
embedding-rag	0.85	80%	272	0.74
neuralmind	1.00	100%	861	0.72

rich

@ 7f580bdcf09 queries242.0× fewer tokens

rich at commit 7f580bdcf0 — 9 pre-registered queries
Backend	Recall	Found	Tokens/query	MRR
full-file	1.00	100%	232,483	1.00
ripgrep	1.00	100%	43,437	0.75
embedding-rag	1.00	100%	236	0.94
neuralmind	1.00	100%	961	0.83

Where NeuralMind loses

Two loss modes, reported plainly. A benchmark that never reports a loss is not evidence, it is advertising.

What this does not measure

Every number the site quotes

Rendered from site/claims.json, the file CI holds every page to: each figure with the kind of evidence it is, where it was measured, and the command that reproduces it.

Every ratio in site/claims.json, with its evidence level, source and reproduction command
Figure	What it is	Evidence	Source and reproduce command
51–242×	Fewer tokens than pasting every source file, across the 40 public-benchmark queries (per repo in the tables above).	Reproducible	docs/benchmarks/public.md`python -m evals.public.run`
31–67×	Fewer tokens than ripgrep opening its top five files whole, same queries. An agent that reads only the matching lines pays less than this baseline; that is not measured.	Reproducible	docs/benchmarks/public.md`python -m evals.public.run`
78.6×	psf/requests v2.32.3, `neuralmind benchmark .` at v4.5.0: the measured code the index covers over the tokens each generic question cost.	Reproducible	docs/use-cases/benchmark-your-repo.md`git clone --depth 1 --branch v2.32.3 https://github.com/psf/requests && cd requests && neuralmind build . && neuralmind benchmark .`
41.8×	The same requests run against the fixed 50K-token estimate: the legacy line the CLI prints beside the measured one.	Reproducible	docs/use-cases/benchmark-your-repo.md`git clone --depth 1 --branch v2.32.3 https://github.com/psf/requests && cd requests && neuralmind build . && neuralmind benchmark . --naive-50k`
12–50×	Field reports measured by `neuralmind benchmark` before v4.5.0, against its fixed 50K-token estimate. A new run is not on this scale.	Field report	docs/wiki/Benchmarks.md`neuralmind benchmark <your repo> --naive-50k`
48.8×	One field report: a ~9,300-node private TypeScript codebase through a major rebuild, against the fixed 50K-token estimate the CLI used before v4.5.0.	Field report	docs/use-cases/measure-memory-across-a-refactor.md`neuralmind benchmark <your repo> --naive-50k`
65.6×	Community submission: a 241-node JavaScript project, against the fixed 50K-token estimate.	Community	docs/community-benchmarks.json`neuralmind benchmark . --naive-50k --json`
46×	Community submission: a 1,626-node Python project, against the fixed 50K-token estimate.	Community	docs/community-benchmarks.json`neuralmind benchmark . --naive-50k --json`
5.1×	The ~500-line CI fixture, measured at v4.3.4. It moves with retrieval changes; CI gates only the floor below.	Reproducible	docs/wiki/Benchmarks.md`python -m tests.benchmark.run`
4.0×	The CI regression floor on that fixture: the build fails below it.	CI-gated	docs/wiki/Benchmarks.md`python -m tests.benchmark.run`
5.5×	Average reduction scripts/demo.sh prints across its three bundled-fixture queries.	Reproducible	scripts/demo.sh`bash scripts/demo.sh`
5.7×	One query in the same demo.	Reproducible	scripts/demo.sh`bash scripts/demo.sh`
8–16×	TurboVec’s 4-bit index vectors against float32: size, not tokens.	CI-gated	docs/wiki/Benchmarks.md`python -m evals.parity.run`
—	ripgrep (query keywords, top five files opened whole) found the gold file on 32 of 40 public-benchmark queries; NeuralMind on 40 of 40	Reproducible	docs/benchmarks/public.md`NEURALMIND_ORT_THREADS=1 python -m evals.public.run`
—	A gold file in the top five for 93.3% of the 150-question retrieval eval (30 pre-registered questions on each of requests, click, flask, rich and NeuralMind's own repo); 10 misses, listed per question	Reproducible	bench/retrieval/on-v4.12/report.md`python -m evals.retrieval.run`
—	NeuralMind's assembled context costs 670-961 mean tokens per question (per repo) on the public benchmark; pasting every source file costs 41,729-232,483	Reproducible	docs/benchmarks/public.md`NEURALMIND_ORT_THREADS=1 python -m evals.public.run`
—	Synapse recall never lowers top-k hit rate, at a neutral token budget	CI-gated	docs/wiki/Benchmarks.md`python -m tests.benchmark.run`
—	Faithfulness vs naive truncation at a matched token budget: CI fails if the mean delta falls below -0.10 (built-in backend, mean of 3 runs on the reference fixture). At v4.12.0 it measured +0.265 (0.839 vs 0.574 expected-fact recall) with CI's command on one machine, where v4.11.1 measured +0.027; at v4.3.4 CI measured -0.054, a loss the site published.	CI-gated	docs/wiki/Benchmarks.md`python -m evals.faithfulness.runner --run`
—	TurboVec fact recall 0.800 vs chroma float32 0.744, identical top-k hit@4	CI-gated	docs/wiki/Benchmarks.md`python -m evals.parity.run`
—	Task outcomes, pilot: on 20 pre-registered bug-fix tasks, Claude Code with and without NeuralMind solved the same number of tasks at no measurable difference in cost; the agent called none of NeuralMind's MCP tools	Pilot	bench/task_outcome/pilot/report.txt`python -m evals.task_outcome.run --tasks evals/task_outcome/tasks.json --out bench/task_outcome/pilot --model claude-sonnet-5-5 --seeds 3 --tool-venv <venv with neuralmind==4.15.1> --test-python <venv>/bin/python --work-dir <scratch> && python -m evals.task_outcome.analyze bench/task_outcome/pilot`
—	Task outcomes, Hermes pilot: on the same 20 bug-fix tasks, Hermes-Agent on a local gpt-oss:20b model solved 10 with NeuralMind's plugin and 8 without, too few tasks apart to be significant (one-sided McNemar p = 0.25), with no measurable difference in tokens per task	Pilot	bench/task_outcome/hermes-pilot/report.txtpython -m evals.task_outcome.run --agent hermes --tasks evals/task_outcome/tasks.json --out bench/task_outcome/hermes-pilot --model gpt-oss:20b-64k --seeds 1 --max-turns 120 --timeout 3600 --tool-venv <venv with neuralmind==4.23.1> --test-python <venv>/bin/python --work-dir <scratch>, then the same with --seeds 3 (stage 2 skips the recorded runs), then python -m evals.task_outcome.hermes_sessions bench/task_outcome/hermes-pilot && python -m evals.task_outcome.analyze bench/task_outcome/hermes-pilot
—	Task outcomes, memory track: on the same 20 tasks in a fixed order, Hermes-Agent on a local gpt-oss:20b model solved 9 with NeuralMind's learned state carried from task to task and 7 starting fresh, too few tasks apart to be significant (one-sided McNemar p = 0.3125)	Pilot	bench/task_outcome/memory-track/report.txt`python -m evals.task_outcome.run --agent hermes --track memory --chains 3 --fresh 3 --tasks evals/task_outcome/tasks.json --out bench/task_outcome/memory-track --model gpt-oss:20b-64k --max-turns 120 --timeout 3600 --tool-venv <venv with neuralmind==4.23.1> --test-python <venv>/bin/python --work-dir <scratch>, then python -m evals.task_outcome.hermes_sessions bench/task_outcome/memory-track && python -m evals.task_outcome.analyze bench/task_outcome/memory-track`
—	Exploration share, field report: reading and searching code was 19.0% of the prompt tokens billed for one developer's 90 interactive Claude Code sessions (29.6% of the context their transcripts record) and 10.5% for 67 interactive Hermes sessions, 2026-10-02 to 2026-11-01	Field report	bench/exploration-share/2026-11-01/README.md`python -m evals.exploration_share claude-code --root ~/.claude/projects --since 2026-10-02T00:00:00Z --until 2026-11-02T00:00:00Z --out rows.jsonl && python -m evals.exploration_share hermes --db ~/.hermes/state.db --since 2026-10-02T00:00:00Z --until 2026-11-02T00:00:00Z --out rows.jsonl && python -m evals.exploration_share report --rows rows.jsonl --out-json summary.json --out-md summary.md`

The public benchmark

Rerun it yourself

What it comes to

Task outcomes: three pilots, no significant difference

The same tasks on a local model: 10 vs 8, not significant

Carrying what NeuralMind learned from task to task: 9 vs 7, not significant

How it is built to be hard to dismiss

Results, per repo

requests

click

flask

rich

Where NeuralMind loses

What this does not measure

Every number the site quotes

Run it on your own repo