Skip to content

D241 — The link graph is a stat-validated in-memory cache

Decision. The link graph is still derived from the concept files (D108), but the per-file parse results are cached in memory and every graph read validates them against the files instead of re-reading all of them. A file is re-parsed when its size, modification time or mode changed, when its modification time is within 2 s of when it was observed (git's "racily clean" rule), when an extensionless link's asset lookup now answers differently, or when it is a symlink. Readers get an immutable view, built only when something changed. Links, the content hash and four frontmatter facets are cached, never bodies, and nothing is persisted. The policy's per-concept type lookup reads the same cache for one file at a time. graph_neighbors gains a missing flag on link targets that are not concepts.

Why. Every graph access walked and regex-parsed the whole KB, and several callers did it once per concept: lint with scope_neighbors (one walk per concept in scope), the Atlas concept panel (two walks per click) and every Visible check for a principal with a type selector (one read and parse per concept). Validation keeps D108's guarantee without depending on every write path to notify: the existing notification was incomplete by construction (OnSyncIn only with SQLite, supersede touching no index, operators editing files directly). Measured on the synthetic 1,000-concept, ~15 MB KB in internal/kb/graphcache_bench_test.go (Apple M-series, -benchtime 3x):

before after
GraphNeighbors, first call 32 ms 45 ms
GraphNeighbors, warm 31 ms 2.6 ms
GraphSnapshot, warm 32 ms 5.1 ms
lint, 111-concept scope, scope_neighbors 3.75 s 0.36 s

The cost is a slower first read (hashes and asset probes are recorded) and a cache whose correctness rests on the validation rules, so each rule has a test that fails when it is removed.

Alternatives rejected. - Invalidation hooks on the write paths: incomplete by construction, and blind to git pulls and editors. - Persisting a link index: a second source of truth to reconcile on disk. - An opt-out setting: a second code path; the equivalence test is the safety net instead. - Size and mtime only: a chmod changes neither, yet decides whether the uncached walk can read the file. The equivalence test found this.

Consequences. Every existing graph API returns exactly what it returned before: graphcache_oracle_test.go keeps verbatim copies of the uncached readers and graphcache_test.go compares them after each step of a mutation script. The policy change is proved the same way in policy_facets_test.go. ConceptFacets resolves a file as ReadConcept does (the direct form wins over an expanded index.md) and never runs a full validation. A file rewritten with its size, mode and an old modification time restored is not detected until one of them changes; git has the same limit. The maps Links and IncomingLinks return are shared and must never be mutated.