Skip to content

D278 — A source ledger of Source concepts revisits D28's removal of source ingestion

Amends D28. D28 removed source_ingest together with raw/, secret scrubbing, a webhook exporter and content-hash provenance. This reinstates only the bookkeeping.

Decision. A primary source the KB has absorbed is a concept of the conventional type Source, kept in a journal (sources by default). Two tools serve it: source_register (a write: dedup by sha256 when both sides have one, otherwise an exact locator; never by title) and source_list (read-only, pending first, oldest first, with a cited_by count). A concept cites a source by listing its ConceptID in provenance; marking a source ingested is a plain concept_patch. lint flags an ingested Source nobody cites (source_uncited), concept_delete treats a citation as a link, and kb_status counts sources by status.

Why. Without a ledger a KB cannot say whether a document was already ingested, what is still pending, where a claim came from, or what depends on a source that turned out wrong; the same document ingested twice produces near-duplicate pages. D28 was right about the cost, which was the pipeline, not the record.

What stays removed from D28. No server-side ingestion or transformation, no mandatory copy of the source (keeping it is optional, as an asset of the expanded concept), no scrubbing pipeline, no exporter. The server never fetches anything; sha256 is computed by the agent and is advisory.

Alternatives rejected. - A dedicated sources.yaml or database: a second store outside search, graph, visibility, history and lint, which a concept gets for free. - Re-adding source_ingest: reintroduces the pipeline D28 removed. - Deduplicating by title: two different documents share a title far more often than one document is titled twice. - A third tool to mark a source ingested: it is a frontmatter patch. - source_register creating the sources journal: maps are created by an explicit act everywhere else.

Consequences. source_register is whole-KB write scope (dedup must see every source), so a narrowed token writes pages with concept_write and cannot register; source_list is a collection read filtered by visibility. A provenance entry that equals an existing Source ID is a citation; anything else stays a plain note. The audit trail records map (register) and scope (list) only: title, locator and body are free text. source_uncited is per-concept and suppressible.