Testing strategy

Principle

Test the lowest stable boundary that proves the behavior:

• package behavior in colocated Go tests;
• protocol wiring with smoke tests;
• cross-component behavior with deterministic end-to-end scenarios.

The oracle is state and structured output, never prose generated by a model. LLM/agent behavior is evaluated during real usage rather than in CI because it is non-deterministic, costly and dependent on external providers.

Levels

Go tests

make test runs go test ./.... Tests live beside their packages and cover parsing, KB invariants, tools, authorization, git behavior, provisioning and client configuration.

make vet runs go vet ./... and make fmt-check fails on anything not gofmt-clean. All three are make gate, which is the single command CI runs.

Provisioning signature coverage includes deterministic Ed25519 envelopes, strict key parsing and identity separation, plus remote sync_pull verification and tampering rejection before Apply.

Server Git profile coverage (D117) uses local two-clone remotes plus an injected fake forge over httptest: branch startup and resume, PR create/reuse and duplicate handling, base rebases, conflict registry resolution, stale caller heads, review/check rejection, finalize leases, base-SHA races and post-merge reconciliation after an uncertain outcome. CI never calls GitHub. Release validation may additionally run an opt-in sandbox smoke against a disposable protected repository and token; that smoke must assert the protected base stays unchanged until the forge confirms the squash merge, and must exercise a timeout-after-merge restart.

CLI JSON tests decode stdout as JSON and assert that diagnostics remain on stderr. Golden help is limited to 80 columns. Bubble Tea view/update tests use 60, 80 and 120-column window messages, strip ANSI sequences before measuring line width, and cover healthy, unavailable, drift, connect retry, sync-all and disconnect confirmation states.

Documentation gates

internal/repodocs holds the deterministic checks on the repository's own documentation and agent-facing layout, and they are ordinary Go tests so make test runs them (D204). They exist because each of these failures is otherwise silent — no client, compiler or linter reports any of them:

• the AGENTS.md budgets: 120 lines of hand-written text, 12.000 characters for the whole file, and every Codex AGENTS.md chain under 32 KiB;
• the generated blocks are up to date — the decision index in docs/decisions.md and the code map in AGENTS.md. The generator is this test run with -update, which is all make decisions-index and make codemap do, so the writer and the checker cannot disagree about the format;
• every decision file is well formed, carries a known topic, holds no leftover template placeholder, and claims a number no other file claims;
• every D<n> cited anywhere in the repository resolves to a record, is a declared gap, or is reserved by an open plan (D208);
• every docs/… path cited anywhere exists — including in Go and shell comments, which no other check reads;
• every relative link in every tracked markdown file resolves, and none uses the retired #d<n> anchor form;
• each skill's frontmatter parses with the same spec-compliant YAML parser a client uses, and satisfies the strictest client's rules (D207);
• the skill bridges are symlinks in the git index, so the answer is the same on a Windows checkout as on CI;
• CONTRIBUTING.md's per-client table agrees with the provider matrix in internal/provisioning (D207);
• CLAUDE.md is exactly @AGENTS.md and nothing else, and the corpus these checks read actually contains the files they exist to read — a corpus definition fails by matching nothing and passing.

Stdio smoke

make smoke builds the binary, starts a temporary stdio server and verifies the MCP initialize handshake.

It holds stdin open past the request rather than piping a bare echo: over stdio the client owns the pipe's lifetime and the server tears the session down on EOF, so closing it immediately races the response out of existence (D168).

This is a fast local check and is not currently a separate CI step; the Go server tests cover the same protocol path more precisely.

HTTP smoke

make smoke-http runs test/smoke/http.sh. It starts the real binary with two temporary KBs, calls MCP through HTTP and exercises Map creation, concept writes/expansion and Atlas overview.

Tool names are qualified with the prefix /health reports for each KB, rather than assumed bare: since D153 a KB-name prefix is the default, so a bare name resolves to nothing. Each call asserts on isError — an unresolvable tool comes back as a JSON-RPC result whose text says "tool not found", which the script would otherwise print and count as a pass.

It runs in CI.

Deterministic end-to-end

make e2e runs the scenarios under test/e2e/. They exercise the compiled binary, HTTP server, CLI client, filesystem and real temporary git remotes together.

The canonical scenario catalog and direct-run flags live in test/e2e/README.md. The suite uses no LLM credentials and runs in CI.

Authorization is tested at two levels, on purpose. 14_rbac_visibility covers RBAC end-to-end at KB granularity, the only level serve currently accepts a policy for through env/CLI. The finer map/journal/type selectors that auth.roles compiles into are covered in Go (internal/mcpserver/policy_test.go, internal/auth/auth_test.go, internal/config/roles_test.go), because expressing them requires a YAML config file rather than the env-var form the scenarios use. Anything asserting non-disclosure belongs at whichever level can observe the raw response: a forbidden resource and a missing one must produce byte-identical output, and a filtered collection must not reveal hidden elements through a short page.

Audit failure paths are tested by fault injection. The whole contract of audit.mode is what happens when the sink is broken, so the write path is made to fail on purpose (audit.FailAppendsForTest) rather than waiting for a real disk error: best_effort must let the call through, required must reject it before the tool handler runs — that last assertion is the one that matters, since a log missing an operation that actually happened is worse than no log. 15_operational_audit closes the loop end-to-end by tampering with a recorded entry and requiring audit verify and audit export to fail on it.

Prefixed multi-KB is exercised with an arbitrary prefix. 16_prefixed_multikb sets tool_prefix to a string unrelated to the KB name, precisely so a client that re-derived the prefix from the KB name instead of discovering it from /health fails the scenario. It also asserts the negative — the bare tool name must not resolve on the prefixed KB — because the D102 promise is that prefixing is exact, not additive. Its last phase restarts the same two KBs with both mounted unprefixed and asserts the D144 startup warning names them, while tools/list stays unchanged on both endpoints: the warning is a diagnostic, never an implicit prefix.

The routed mount is asserted on the wire, not only in Go. 17_routed_multikb starts a real three-KB server with mcp.mount_mode: routed and drives it with curl: /health must advertise mount_mode and routed_path, /mcp/routed must list each tool exactly once with a kb argument and in far fewer bytes than the per-KB mounts summed, a call must reach the KB it names, a call without kb must be refused naming the mounted KBs, and ?kb= on that URL must be 400. It then asserts the additive promise from the other side — the per-KB endpoints still answer and their tools/list has gained no kb argument — and that connect writes exactly one MCP entry, pointed at /mcp/routed.

The scenario earns its keep: it is what caught that D153's derived kb-name prefix, which nobody configures explicitly, was making the routed mode fail at startup out of the box. No Go test could see it, because the defect lived in the wiring between the config default and the mount, not in either.

The workspace scope is asserted on real repositories. 18_workspace_projection is D193's mandatory acceptance criterion, and it is a scenario that cannot pass before the scope exists: two KBs holding a skill with the same name, two git repositories, one provider bound to a different KB in each. It asserts that each workspace receives its own perimeter — comparing the bodies, not just the paths, because reaching the wrong KB also produces a readable file — that no KB artifact is left in the global catalogue while Cartographer's own transversal bundled skills stay there, that an unbound workspace receives nothing, that git status is clean and .gitignore untouched, that unbinding one workspace prunes only its files, and that one workspace bound to both KBs is still refused as a collision.

It earned its keep on the first run, finding three defects no unit test could see: the bundled skills were being copied into every workspace, a CLAUDE.md Cartographer created itself was left untracked and dirtying the repository, and unbinding a workspace left its projected files behind forever because every later sync only iterates the declared projections.

Packaging (install.sh and the generated Cask)

make test-install runs the network-free suite under test/install/ plus test/install/goreleaser_guard.sh, and runs in CI after make e2e. The scenarios drive install.sh against a fake curl and a stateful fake installed binary that records its invocations and returns a scripted exit code, so the D121 contract is asserted without downloading a release or touching a real launchd/systemd service: an already-current version never invokes upgrade-repair, a stopped or absent service is never started, exit 1 still leaves the binary update successful, and exit 2 (or an unexpected code) fails visibly but only after the new binary is in place.

Since D192 the suite also covers the checksum path — a matching digest installs and says checksum OK, a mismatch fails without leaving a binary behind, a sha256sums.txt that exists but does not cover the asset is an error rather than a silent skip, and a release shipping no checksum file at all still installs — and the uninstall branches: nothing installed, a binary with no units, a unit present (refused, non-zero, nothing removed), --binary-only (proceeds and states what it left behind), and a partial state such as a timer without a service.

A separate, non-deterministic check lives in internal/provisioning/clientcompat_test.go: it runs the client's own discovery command and asserts it recognizes the destination Cartographer declares, for the two paths that diverge from the vendor documentation (see interoperability §Known divergences). It skips when the client is not installed, and also when the client cannot answer — it errors, times out, or prints nothing — because an unrelated client problem must not turn this into a red suite that everyone learns to ignore. Only a successful run whose output does not mention the declared directory is a signal.

The Cask install steps are checked as a repository template, never as the generated file in BeppeTemp/homebrew-tap: on the YAML of .goreleaser.yaml (comments excluded) the guard asserts a postflight_steps stanza that removes quarantine from the staged path, and the absence of hooks, of the deprecated postflight do and of upgrade-repair, which cannot work inside Homebrew's sandbox (D199). Its Ruby is not executed — that would need a real Homebrew and GoReleaser environment, which is out of the deterministic gate (see below).

What is deliberately not in CI

• Whether a particular model interprets an instruction well.
• Provider/model quality comparisons.
• Tests requiring production credentials or external private infrastructure.
• Manual UI appearance checks.

These belong to production validation or an explicit release exercise, not to the deterministic repository gate.

Before a pull request

make gate          # gofmt + vet + test, documentation gates included
make smoke-http
make e2e
make test-install

Exactly what CI runs, in the same order: make gate is the one place that defines "green", so a step added here has to be added to gate or to ci.yml, not to a list that only lives in prose.

Before a release

• CI is green on the release commit.
• make smoke succeeds for the packaged/local stdio path.
• Installation and upgrade are verified on the target platform.
• Private deployment rollout checks are performed through maintainer tooling.