Skip to content

Data-plane decisions

Knowledge-base layout, OKF parsing and hashing, concepts, Maps, imports and model migrations. Current behavior: ../data-plane.md.

These records explain why choices were made and may describe superseded behavior. For the supported interface, follow the current-state page linked above.

D5 — Normalized content-hash

ContentHash = sha256 hex of the normalized content: CRLF→LF, whitespace trim, removal of trailing empty lines, removal of timestamp: (auto-generated). Canonical ordering of YAML keys + per-section hashes (SectionHashes) since M2. Avoids spurious stale_write on non-substantive variations.

D8 — Frontmatter: stdlib-only YAML parser

Hand-rolled parser in internal/okf (ParseFrontmatter/Serialize/CanonicalString): covers the OKF YAML subset (scalars, flow and block lists, empty values). *Frontmatter interface separated from serialization → gopkg.in/yaml.v3 pluggable without rewrites.

D9 — Canonical ordering + SectionHashes

CanonicalString() sorts YAML keys alphabetically (without comments). SectionHashes computes per-section hashes (h1/h2) + _full. Two files with keys in different order produce the same hash.

ExtractLinks parses only [text](path.md). Absolute links, anchor-only (#), and paths outside the KB are ignored. (Superseded by D72: [[id]] wiki-links are first-class in ExtractLinks, so graph and lint see them.)

D15 — Secret scrubbing: blocking regex, 6 patterns ✅ Removed

The internal/scrub package (6 regexes: private_key, aws_key, github_token, generic_secret, bearer_token, connection_string) was removed together with source_ingest and the event-driven exporter (June 2026). PII prevention is now the responsibility of the LLM-as-judge quality-gate at the ingest checkpoint.

D19 — Agent contract: AGENTS.md generated by Init

kb.Init creates AGENTS.md with soft guidance on style, naming, workflow, when to lint. Added to reservedNames in okf (does not appear in WalkConcepts).

Superseded by D62: kb.Init no longer generates AGENTS.md. The filename remains in reservedNames/IsReserved (backward compatibility: KBs that still have it from a previous Init keep working, the file is simply ignored by the server).

D22 — Event-driven exporter: HTTP webhook + Go worker pool ✅ Removed

Removed (June 2026). The internal/exporter package and the source_ingest tool were deleted. Ingest is now the agent's direct responsibility via concept_write with no intermediaries. See D28 for the removal rationale.

D25 — KB layout: data/ conceptual root, services/ carve-out in ResolvePath

Decision. The conceptual root is kb.Root/data/ (DataRoot()); ResolvePath makes an explicit carve-out for services/*.md (base = kb.Root), and WalkConcepts also walks services/ so that search/graph/lint/index see the Services. Rationale. Without the carve-out, Services would be listed by WalkConcepts but unreachable for reading via ResolvePath — a silent bug; the carve-out stays minimal (one check on the first path segment). Details: docs/data-plane.md §Filesystem layout of a KB.

D28 — Removal of raw/, mcp/, source_ingest, scrub, exporter (June 2026)

Decision. The raw/ and mcp/ directories are removed from the KB structure. The internal/scrub and internal/exporter packages are deleted. The source_ingest MCP tool and the --exporter flag in main.go are removed. The configurator abandons mcp/wiki.yaml in favor of CLI flags (--name, --transport, --url, --auth, --token-env). Rationale. The complexity of source_ingest (copy into raw/, secret scrubbing, webhook exporter, content-hash provenance) did not hold up cost/benefit-wise for the small-to-medium KBs that are the primary target. The direct write cycle via concept_write is simpler, verifiable, and composable with existing LLM tools. The PII quality guarantee moves to the LLM-as-judge quality-gate at the ingest checkpoint. mcp/wiki.yaml added a file that every user had to keep aligned with their CLI flags — the defaults in DefaultConfig() cover the common case without extra files.

D62 — KB repo cleanup: no generated AGENTS.md/.gitignore, exclude via .git/info/exclude (WP6)

Context. kb.Init generated, besides the content skeleton, AGENTS.md (D19, soft agent contract) and .gitignore (.cartographer/). Both were noise: AGENTS.md presupposed an agent editing the KB directly on the filesystem (nonexistent scenario, the KB is always mediated by the server); .gitignore versioned a purely local need.

Decision. kb.Init no longer writes AGENTS.md or .gitignore. Excluding .cartographer/ moves to .git/info/exclude (ensureInfoExclude, git's local file, never versioned), called both by Init and by Open — so every existing KB self-migrates at the first Open with no operator intervention. Backward compatibility. AGENTS.md remains in reservedNames (D19): pre-D62 KBs with these files keep working unchanged, no migration removes them. Discarded alternatives. Removing existing AGENTS.md/.gitignore on Open (destructive on content potentially edited by hand); a .gitignore kept but unversioned via merge strategy (.git/info/exclude is already git's native mechanism for this). Details: docs/data-plane.md §Filesystem layout of a KB.

Status: implemented (2026-07-09), all WPs. Delta versus the plan below: (a) WP1 — write errors during the backlink rewrite abort the pass and are propagated as a tool error (only the index upserts stay best-effort); a single summary AppendLog for the whole batch call (not one per rewritten concept); move chains within the same batch (a→b + b→c) are not supported: validation reads the sources from disk before applying; (b) WP1 — new primitive kb.RewriteLinks(body, basePath, moveMap) alongside ExtractLinks, same resolution for both syntaxes; (c) WP4 — the "new dossier" check happens with os.Stat before the MkdirAll, so a pre-existing dossier is never touched; the dossier_missing_index lint is directory-based (new dedicated scope-match helper). The homelab-wiki remediation (stub index.md in the 7 implicit dossiers + index_rebuild) is an operational post-deploy step, not part of the code.

Context. Real session (2026-07-09): restructuring of the homelab-wiki KB — 47 concept_move calls to introduce the dossier level in entities/ and topics/. Total cost ~1M tokens, largely attributable to control-plane gaps, not to the agent: 1. concept_move does not rewrite backlinks → a subagent fixed ~269 [[...]] links by hand (592k tokens, 368 tool calls, 50 min) for work the server can do in one commit; 2. concept_move does not update the indexes (unlike write/patch/delete): stale FTS entries on the old IDs still present 8 days later, duplicated results in search; 3. no inventory tool: ~30 search calls with trick queries ("a e i o u", "di") to enumerate the concepts, with entries still missed on the first pass; 4. 47 moves = 47 commits and 47 locks, not atomic (interruption halfway = mixed KB with broken links); 5. moves to nested paths created 7 implicit dossiers (side effect of the MkdirAll in WriteConcept) without the index.md that dossier_create would have generated → index_get fails on those paths; moreover PathToID accepts arbitrary depth, while the data model (data-plane.md §Hierarchy) allows at most archive/dossier/concept; 6. (surfaced during planning) ExtractLinks (internal/kb/graph.go:14) parses only markdown links [testo](path.md), never the wiki-links [[id]] — which are the de facto syntax used throughout the homelab-wiki KB. graph_neighbors, the lint's broken_link check and orphan detection are therefore blind to the real links.

Decision. Structural refactoring is a first-class control-plane operation: the server already has all the data (files, links, indexes) and must absorb its mechanical cost.

WP0 — First-class wiki-links in ExtractLinks. Precondition of WP1: ExtractLinks also recognizes [[id]] and [[id#sezione]] alongside markdown links. Semantics: wiki-links are root-relative (the ID is the path from the KB root without .md, as in real usage: [[entities/smart-home/otbr]]), while markdown links remain relative to basePath. No alias form [[id|testo]] (added only if needed — post-implementation note: the first full lint on homelab-wiki revealed that the alias form IS used in some concepts; those links work for the reader but are invisible to graph/lint/backlink-rewrite — a natural candidate for a future WP: parse + rewrite of the alias form). Intended side effect: graph_neighbors, lint broken_link and orphan detection start seeing the real links — after the fix a lint full on homelab-wiki will give the true picture.

WP1 — Batch concept_move with backlink rewrite. - New parameter moves: [{source_id, target_id}] (backward compatible: single source_id/target_id remain valid as a batch of 1). Full validation of all entries before applying (sources exist, targets free, path guard); then application and a single commit for the whole batch. - Server-side backlink rewrite: a single WalkConcepts pass with the complete old→new map, rewriting both syntaxes (root-relative wiki-links [[old-id]]/[[old-id#sezione]] and markdown links resolved via ExtractLinks) in all concepts, including services/. Parameter rewrite_links (default true). The result lists the applied moves and the concepts touched by the rewrite. - The rewrite goes through the same write path as concepts (hashes, indexes) — no raw writes.

WP2 — Index consistency on move. toolConceptMove receives live + sqlIdx like concept_delete: removal of the old ID's entry and upsert of the new one for every move and for every concept rewritten by the backlink rewrite. A post-move search must not return dead IDs.

WP3 — Inventory: concept_list([scope], [limit]). Read-only tool that enumerates concepts (id, title, type) under a prefix, via WalkConcepts + frontmatter — the bounded equivalent of "ls -R". Registered agent-visible: it is the natural first step of any reconnaissance and replaces the abuse of search as an enumerator. index_get remains the curated route (progressive disclosure), concept_list the exhaustive one.

WP4 — Explicit dossiers and enforced depth. - When a write (concept_write/concept_move) implicitly creates a new dossier directory, the server also generates the index.md stub (type: Index, title from the name) — same content as CreateDossier. index_get must never fail on a real dossier. - Enforcement of the documented hierarchy on the write path: ConceptID in data/ with max 3 segments (archive/dossier/concept); explicit error beyond that. services/ unchanged (2 segments). lint check: dossier without index.md (for KBs born before the fix). - homelab-wiki remediation at activation: index.md in the 7 implicit dossiers (entities/{infra,ai-tools,smart-home,clients}, topics/{smart-home-protocols,infra,ai-tools}) and index_rebuild for the stale FTS entries.

WP5 — Documentation and closure. control-plane.md §MCP API (batch concept_move + rewrite_links, concept_list), data-plane.md (enforced depth, implicit dossiers), readonly.go/visibility.go + golden tests, server_test.go (atomic batch, rewrite with anchors, stale index, depth guard, index.md stub), this entry, and the release tracking state then in use.

Order. WP0 and WP2 are small and autonomous (can start immediately, even in parallel); WP1 depends on WP0 and is best done after WP2 in the same area; WP3 and WP4 independent; WP5 closes. All additive and backward compatible (WP0 widens graph/lint, no API change).

D74 — Import of external non-OKF wikis/KBs: kb-import skill + CLI scaffold + lint-driven curation

Status: implemented (2026-07-10).

Decision. Importing an external corpus (Obsidian vault, markdown folder, wiki export) is an agentic procedure — bundled skill kb-import (internal/skillbundle/bundled/kb-import/), sister of kb-create — consistent with D28: no server-side ingest tool. Two legs: a mechanical CLI scaffold for mass conversion with no LLM consumption, and an incremental agentic curation driven by the lint.

WP1 — imported_draft lint (warning). A concept with frontmatter status: imported produces an imported_draft finding. Implementation next to stale_claim in internal/lint/lint.go:131; tests in lint_test.go. The marker makes the curation debt visible and splittable across multiple sessions instead of a big-bang.

WP2 — cartographer import subcommand. cartographer import --source <dir> --kb <dir> [--archive <nome>] [--map <srcdir>=<archivio>] [--dry-run]: walks the source's .md files; for each file it synthesizes the minimal frontmatter (title from the first H1 or from the filename, status: imported) preserving any existing frontmatter and adding only the missing fields; maps the source's relative directories onto archive/dossier; writes via kb.Open+WriteConcept (OKF invariants for free, no per-file commit: a single final commit by the operator). --dry-run prints the mapping plan without writing. Dispatch in cmd/cartographer/main.go:73.

Links. Wiki-links [[...]] remain valid as-is (D72, first-class in ExtractLinks); relative markdown links are rewritten best-effort by the scaffold against the new layout; broken_link acts as a safety net for what slips through.

Deviations from the specification (implementation, 2026-07-10). - kb.WriteConcept requires a non-empty type (pre-existing OKF invariant, not discussed in D74's original text): when the source file has no type — neither pre-existing nor otherwise derivable — the scaffold synthesizes type: Note alongside title/status. A type already present in the source frontmatter is never touched. - The --map <srcdir>=<archivio> mapping is exact per-directory: it matches only the literal source directory (relative to --source, . for the root), with no inheritance on nested subdirectories — consistent with WP2's "mechanical, not intelligent scaffold" spirit; subdirectories not explicitly mapped fall back to --archive (flat default) or, in its absence, fail the whole command with the list of unmapped directories, before any write. - Slug collisions (two source files normalizing to the same destination concept ID) are resolved with an incremental numeric suffix (-2, -3, …) instead of failing the import. - Amended by D91. The optional final commit, map scaffold, and directory-as-expanded-concept import behavior are specified in D91; the default remains an uncommitted, flat mechanical pass.

Rationale. The mechanical conversion (frontmatter, layout, links) must not consume LLM tokens — on a large wiki it would be unsustainable; the semantic part (mapping onto the archives, dedup, substantive rewrites) stays with the agent with a human checkpoint on the mapping plan before any write. Discarded alternatives: a server-side MCP import tool (re-creates the source_ingest removed with D28); fully agentic import (cost proportional to the corpus, not to the part requiring judgment).

D77 — Atlas/Map/Journal hierarchy: the dossier becomes a state of the concept

Status: implemented (2026-07-10). homelab-wiki migration executed and verified (2026-07-10).

Context. Analysis of the homelab-wiki KB (66 concepts): the "dossier" level was used exclusively as a taxonomic category (entities/{ai-tools,infra,smart-home,clients}, topics/{...}) — zero authentic dossiers in the model's sense ("coherent unit on a topic"). Three causes: (1) archives defined by ontological type (entities/topics) that duplicate the type frontmatter and leave thematic categories homeless; (2) dossier defined only structurally (any subdir), no semantic invariant; (3) file→folder promotion left to the agent (rename+backlinks), hence never performed — and with the 3-segment depth cap, category-dossiers made the true dossier impossible.

Decision.

a) Lexicon (agent-facing surface, English — open project): Atlas = the KB; Map = thematic archive with mixed concept_types (Entity and Topic of the same domain coexist: the type is an attribute, not a position); Journal = chronological append-oriented registry archive (incidents, notes). The "dossier" disappears as a noun: it is a state of the concept — expanded concept, directory map/nome/ with index.md and satellites.

b) Descriptor _map.md (type: Map, kind: map|journal, default map; archive_type retired). Legacy _archive.md stays read-compat (treated as a Map with kind: map, never written again, legacy_archive_descriptor lint).

c) ID resolution with fallback: map/nome resolves to map/nome.md or, if absent, to map/nome/index.md — on the read and write paths. Thus concept_expand(id) (new tool) promotes a concept to an expanded concept without changing the ConceptID and without backlink rewrite (unlike concept_move). Both forms present = write error (expanded_ambiguous, also a lint check with error severity). WalkConcepts emits map/concept/index.md as concept map/concept (search/graph/lint see it). No inverse (concept_collapse): YAGNI. Max depth unchanged (3 segments, map/concept/child). Expansion also allowed in journals (heavy incident with attachments).

d) MCP surface v2 (breaking, → release v2.0.0): kb_overview→atlas_overview, archive_list→map_list (exposes kind), archive_create→map_create(kind); dossier_list/dossier_create removed with no alias (inventory: concept_list/index_get; growth: concept_expand). CLI import: --archive→--default-map (the --map name was already taken by the per-directory mapping — a deliberate deviation from the plan, which called for --map).

e) Deterministic lint guardrails (never LLM): category navigation is the job of curated indexes/search/graph, not the filesystem — expanded_as_category (>8 children mostly not linked to the concept's index), map_oversize (>50 concepts: thematic split, not subfolders; info severity, new), expanded_missing_index (rename of dossier_missing_index), expanded_ambiguous, legacy_archive_descriptor. Thresholds as named constants in internal/lint/lint.go.

f) Rename scope: MCP surface + data model + docs only. The internal/kb package, the --kb CLI flags, the kbs: config and the CARTOGRAPHER_* envs are not renamed (possible future D); the low-cost inconsistent internal identifiers (ListDossiers→ListExpanded, DossierCount→ExpandedCount) are.

Rationale. Every physical hierarchy is under pressure to become a taxonomy; the model must defend itself instead of relying on the agent's discipline. Making expansion an atomic, ID-preserving operation removes the cost that prevented authentic dossiers from being born; removing the noun removes the level that invited abuse; the lints flag the drift when it reappears. Existing KBs stay readable without migration (read-compat b/c): the homelab-wiki migration (merge entities+topics by theme → maps smart-home/infra/ai-tools/clients, incidents/notes journals) is operational, post-deploy, via batch concept_move.

D87 — Fence-aware heading detection: shared line iterator for ListHeadings/ExtractSection/SectionHashes

Status: implemented (2026-07-23).

Context. ListHeadings, ExtractSection and SectionHashes (internal/okf/okf.go) each scanned body lines independently, calling parseHeading with no code-fence state. A # line inside a fenced code block (e.g. a bash comment # Usage: ... inside a ```bash fence) was parsed as a real heading, surfacing as a spurious entry in the concept_read outline (internal/mcpserver/tools_read.go) and potentially splitting sections/hashes mid-fence.

Decision. A shared helper (headingEligibleLines) computes, once per body, which lines are eligible for heading parsing; all three functions consult it before calling parseHeading, so they stay mutually consistent by construction. Fence rules (pragmatic CommonMark subset, not the full spec):

  • a line whose trimmed content starts with ``` or ~~~ opens a fence (an info string after the marker, e.g. bash, is ignored);
  • the fence closes on the next line whose trimmed content starts with the same marker; markers do not cross-close (a ``` fence is not closed by ~~~);
  • an unclosed fence extends to the end of the document — the rest of the body is treated as fenced content, not as a fallback to normal heading parsing. This is a deliberate simplification: a malformed/unclosed fence is far more likely in a document that also has trailing garbage than one that "resumes" cleanly, and erring toward under-detection (fewer headings) is safer for section extraction than guessing where the fence should have closed.

Ripple. The D78 size-guard outline counts headings; documents with fenced # lines now report a smaller (correct) heading count — the changed count is the fix, not a regression.

Rationale. A single shared iterator, rather than three ad-hoc fence-aware loops, is what actually guarantees the outline/section/hash consumers agree — duplicating fence-tracking logic three times would only move the risk of drift from "no fence awareness" to "three subtly different fence-aware implementations."

D91 — Import convenience without widening the default write surface

Status: implemented (2026-07-24).

Decision. cartographer import remains uncommitted and flat by default, preserving D74's mechanical-review workflow. --commit makes exactly one final commit of only the paths written by the import; --message supplies its message and implies --commit. It uses the KB git lock and an isolated Git index seeded from HEAD, so pre-existing staged or unstaged work cannot enter the import commit. A partial batch still commits its successful writes and reports the errors.

Each absent destination map is created through CreateMap, adding the same _map.md/index.md/log.md scaffold as map_create; existing maps are not altered. --dir-as-concept promotes a source directory with index.md, or README.md if no index exists, into <map>/<dirname>/: its chosen index is written through the expanded-concept path and its sibling markdown files become satellites. The dry-run identifies each promotion. Without the flag, index.md retains the established reserved-name rejection and other files remain flat.

Rationale. An opt-in single commit supplies the expected convenience without making a mass import silently commit by default, and the isolated index makes that promise meaningful even in a dirty clone. Reusing map creation and expanded-concept resolution keeps the CLI output structurally identical to the MCP write path instead of adding another partial KB shape. Directory promotion is explicit because automatic hierarchy inference would turn D74's deliberately mechanical importer into a semantic mapper.

D106 — Concept-owned non-Markdown assets

Decision. An expanded concept may own regular non-Markdown files through the asset_* MCP family. Assets remain filesystem-and-git-history data: they carry no frontmatter or ConceptID and stay outside WalkConcepts, validation, search, and the concept graph. They follow an expanded owner's directory move; deleting an owner with assets requires explicit force.

Rationale. Expanded concepts already establish a stable owner directory without changing their ConceptID. Keeping attached evidence in that directory lets git preserve it across moves while the concept graph remains exclusively Markdown-to-Markdown, avoiding false search and lint results from binary or generated files.

Alternative rejected. Widening the provisioning artifact_* whitelist to data/** was rejected: provisioning artifacts and concept evidence have different validation, ownership, lifecycle, visibility, and allow_artifact_write semantics. A separate data-plane API preserves both boundaries.

D107 — Declarative lint contracts in map descriptors

Decision. A map may declare required_fields, additive required_fields.<Type>, and require_index_entry in _map.md. Lint reports a missing required field as an error and an incomplete curated index as a warning; malformed recognized entries are informational. The contract is optional, so no server-default vocabulary or schema is imposed. gate_check fails on the error, while validate and contradiction-only commit_gate retain their existing responsibilities.

Rationale. A descriptor is versioned, local to the map it governs, and uses the existing string/list frontmatter grammar. It gives deterministic, auditable enforcement without executable plugins, which would make lint arbitrary-code execution and undermine its no-LLM guarantee. Required field contracts belong to governance rather than the read-path validator so legacy KBs remain consumable. Domain-specific rules (for example, required section prose or naming conventions) remain KB skill policy: generalising them would require a rule language outside this data-plane contract. index_incomplete deliberately complements, rather than replaces, expanded_as_category: the former reports each missing curated entry; the latter detects a broader filesystem-taxonomy smell.

D109 — KB-owned concept templates and one-shot scaffolding

Decision. Templates are plain Markdown artifacts at templates/<slug>.md, maintained through the artifact lifecycle and discovered with template_list. They are not concepts: they have no ConceptID and stay outside WalkConcepts, search, lint and the graph. concept_new(template, id, vars) renders a validated template into a new concept with literal, single-pass substitution; it never overwrites or curates a map index.

Templates are deliberately excluded from the provisioning manifest, revision, lockfile, provider matrix and pruning. They are KB content for server-side page creation, not instructions or configuration to materialize into a client. This is the intentional divergence between the artifact whitelist and provisioning kinds.

Template variables use only {{identifier}}. The syntax is disjoint from D75's client-side {{repo:key}} and {{path:name}}: colon forms are rejected so a provision-time placeholder can never silently survive into a concept. Rendering has no logic, includes or repeated interpretation; inserted values are literal.

Alternatives rejected. Templates as concepts in a dedicated map would pollute the graph, indexes and lint; putting templates in a map descriptor would couple reusable shape to one location; adding a template parameter to concept_write would blur its full-content, if_match update contract. D107 remains complementary: templates prescribe a starting shape, while map contracts deterministically report required-field and curated-index conformance.