Measure it on your own repo
The number that decides anything for you is the one from your codebase. Three commands get it, a fourth checks the learning later — and we want you to share it, good or bad.
We do not have testimonials yet. What we have is a benchmark that runs locally, against your code, in about fifteen minutes. Run it and read your own number. If it is good, tell us — we will publish it. If it is not, tell us that too: we will explain why, or fix it.
Step by step
Install
Measuring your own repo needs only the PyPI package — no clone. (A source checkout is needed only to rerun the public four-repo benchmark.)
$ pip install neuralmind
Build your index
tree-sitter parses the structure and embeddings are computed on your machine. The first build downloads a public embedding model once; after that nothing else is fetched.
$ cd /path/to/your-repo
$ neuralmind build .
Measure token reduction
Runs five generic code questions (or the ones in your .neuralmind.eval.yaml) and reports wake-up tokens, average tokens per question, and the reduction: your code — every code file the index covers, measured at the same ~4 characters per token as the context — over the tokens a question costs. It scales with your repo. The legacy line repeats the run against the fixed 50,000-token estimate releases before v4.5.0 used, so older numbers stay comparable.
$ neuralmind benchmark .
# machine-readable:
$ neuralmind benchmark . --json
Project: requests
Baseline: measured: 94,069 tokens in 36 indexed code files
Questions: generic
Wake-up tokens: 510
Avg query tokens: 1200.6
Avg reduction: 78.6x
Legacy reduction: 41.8x (vs the fixed 50K-token estimate used before v4.5.0)
Summary: 78.6x average token reduction vs measured: 94,069 tokens in 36 indexed code files
# psf/requests v2.32.3 — yours will differ
Check retrieval on your own code
Samples symbols from your index, asks for each one by its own description, and reports how often the right file comes back — plus the blind spots it could not find, by name. No labels or setup needed.
$ neuralmind probe .
Sampled 50 of 801 indexed symbols, retrieval depth k=10
answerability : 84% (file found in top-10)
MRR : 0.620
recall@1/3/5 : 0.460 / 0.780 / 0.820
blind spots : 8
# psf/requests v2.32.3 — yours will differ
Check whether the learning helps (after a few weeks of use)
Write down questions your team really asks, each with the file that answers it — past incidents work well, phrased as the symptom you saw, not the fix you found. Then run them twice, with the synapse layer’s recall on and off (v4.14.0+). Both runs are read-only. Recall also follows the call and import edges every build seeds, so a fresh build can already differ, and no difference is a normal result too. Rerun it as usage teaches the layer more: eval --report shows whether the gap grows with its usage-learned edges and transitions.
$ neuralmind eval . --suggest --write # drafts .neuralmind.eval.yaml; edit it
$ neuralmind eval . --synapse-ab
$ neuralmind eval . --report
Share your results
Emit a schema-ready submission and send it to us, or open a pull request against the community benchmarks. Nothing is uploaded — you choose what to paste, and whether it is attributed or anonymized.
$ neuralmind benchmark . --contribute
What you get
- Tokens per question
- The mean context NeuralMind hands your agent for a code question, from benchmark. The hardest number on this page — it does not depend on any baseline.
- Reduction
- Your code over NeuralMind’s context: the measured size of every code file the index covers, divided by the tokens each question costs and averaged — on psf/requests, 94,069 tokens of code at ~1,200 a question, 78.6×. Docs and changelogs are left out, so prose can’t pad it. (The public benchmark’s 51.2× on the same repo, at v4.12.0, is a different measurement — 14 pre-registered questions against non-test source only — so the two don’t compare directly.) Before v4.5.0 the CLI divided a fixed 50K-token estimate instead (41.8× on the same run); the community submissions so far (46× to 65.6×) and the 12–50× field-report range were measured that way.
- Answerability & recall@k
- From probe: how often the right file is in the top k when you ask for one of your own symbols.
- Synapse recall, on vs off
- From eval --synapse-ab: how many of your questions the synapse layer moved up or down, hit@5 won and lost, and an exact McNemar p-value — with the size of the layer it read, seeded and usage-learned, so a zero is explained rather than hidden. Nothing we publish shows this improving over time on a real repo yet; this is how you would find out on yours.
- MRR and blind spots
- Mean reciprocal rank (1.0 means the right file always comes first), and the symbols the index could not retrieve — the list to fix first.
Want correctness scored as well as cost? The public benchmark measures against every source file and scores gold-file recall on pre-registered queries — it needs a source checkout because the harness ships in the repo.
Already run it?
Send your output to hello@neuralmind.uk and we will publish it — attributed or anonymized, your choice. The only requirement: it comes from your own codebase, under your own run. We are building social proof the honest way, one real number at a time.