perf(python): one-pass mirror of the parsed tree; map lookups in linkProject

[swapnilpaliwal-sd]

What

The Python extraction stages each re-walked the tree-sitter tree through the JS<->C++ binding, so every node's properties were re-marshalled once per stage. Profiling the parse stage on a 526k-LOC subject put 54.7% of samples inside the binding's property getters (get type alone 16.1%); Parser.parse itself was 1%. Two lookups in linkProject additionally ran as linear scans inside loops (modules by hash per import record, bindings by hash per alias).

• materializePyTree mirrors the freshly parsed tree into plain JS in one cursor pass; every stage then reads JS properties. Tree-sitter still parses every file — nothing about parsing changes.
• linkProject prebuilds both maps with the same first-match semantics.
• Parser version 0.1.2 -> 0.1.3.

Measurements (two large corpus subjects, warm, same machine)

subject	python stage before	after	speedup
subject A (526k LOC)	211.7s	66.9s	3.2x
subject B (638k LOC)	160.9s	62.9s	2.6x

Controls

• IR is byte-identical on both subjects (full diff -rq of the per-language output trees against the pristine base commit's parser).
• Node-level A/B harness asserts every mirrored property (type, spans, counts, text, flags, fields) against the real tree: 2.3M nodes, 0 mismatches. It caught the one real subtlety: an ERROR node absorbed during recovery is isExtra, so ERROR nodes read that flag from the real node (only reachable in files that failed to parse).
• npx tsx src/test/python-tests.ts: 23/23 gates pass.
• Known accepted divergence (unused by any stage): tree-sitter's childForFieldName('alternative') on match_statement pierces into the match block; the stages iterate case_clauses by type instead.

[swapnilpaliwal-sd]

Extended to Java and C# (same branch): the mirror is now shared (parser/src/parsers/mirror-tree.ts), parameterized by each grammar's extras; JS/TS are untouched (their extractors run on the compiler AST, not tree-sitter).

subject	stage before	after	speedup	IR vs pristine
Python, 526k LOC	211.7s	66.9s	3.2x	byte-identical
Python, 638k LOC	160.9s	62.9s	2.6x	byte-identical
Java, 318k LOC	66.2s	13.3s	5.0x	identical except the one column below
C#, two subjects	73.2s	17.6s	4.2x	byte-identical

Validation: node-level A/B against real trees (django 400 files, jackson-scale subject all 1,386 files = 2.2M nodes, both C# subjects all 1,946 files — 0 consumer-visible mismatches), and all four language gate suites green: python 23/23, java 103 checks/0 failed, csharp 62/62 (torture manifest included), javascript 49/49.

One deliberate behaviour change, Java only: nested-annotation argument linking used a Map keyed by node OBJECT, and tree-sitter hands out a fresh wrapper per access, so the join hit only when a wrapper instance happened to be reused — 159/267 nested-annotation args linked on the large Java subject. With the mirror's stable identities it links 260/267. Every other Java relation is byte-identical.

Documented, consumer-unused divergences: childForFieldName('alternative') piercing on Python match_statement, and named-sibling resolution around ERROR/hidden boundaries in C# (no C# stage reads named siblings).

[swapnilpaliwal-sd]

Third commit: the TS/JS-side and pipeline fixes.

TypeScript parse (its extractors run on the compiler AST, so the mirror does not apply there — the costs were elsewhere):

• relation CSVs were verified by re-reading and re-decoding the finished file; the same width + line-break rules now run per row at append time, allocation-free, and publish() checks written-bytes against file size (the torn-write defect the read-back existed for). The streaming verifier stays exported for its gate.
• the root-program closure walk read + cheap-parsed every file to follow imports, then extraction read everything again; the text is now handed over (consumed per file), and resolveModuleName gets a ModuleResolutionCache instead of re-probing node_modules per specifier.
• warm A/B on the 1.1M-LOC TS package: stage 57.2s → 50.9s, IR byte-identical; the removed double read was ~12% of a cold parse. TypeScript suite 49/49.

Pipeline: all solved languages strictly in sequence. The largest language still solves alone first (publish-first behaviour unchanged), the rest now run in waves (AXIOMCODE_SOLVE_JOBS, default 2; 1 restores the old line), each with its own intermediate because run-souffle writes fixed file names there. Serial vs concurrent produce row-identical relations across all 255 tables on a three-language tree.

End to end on this five-language repository: axiomcode index wall time 207.5s → 129.9s (−37%).