Skip to content

Testing strategy

Principle

Test the lowest stable boundary that proves the behavior:

  • package behavior in colocated Go tests;
  • protocol wiring with smoke tests;
  • cross-component behavior with deterministic end-to-end scenarios.

The oracle is state and structured output, never prose generated by a model. LLM/agent behavior is evaluated during real usage rather than in CI because it is non-deterministic, costly and dependent on external providers.

Levels

Go tests

make test runs go test ./.... Tests live beside their packages and cover parsing, KB invariants, tools, authorization, git behavior, provisioning and client configuration.

make vet runs go vet ./.... Both are required in CI.

Provisioning signature coverage includes deterministic Ed25519 envelopes, strict key parsing and identity separation, plus remote sync_pull verification and tampering rejection before Apply.

Server Git profile coverage (D117) uses local two-clone remotes plus an injected fake forge over httptest: branch startup and resume, PR create/reuse and duplicate handling, base rebases, conflict registry resolution, stale caller heads, review/check rejection, finalize leases, base-SHA races and post-merge reconciliation after an uncertain outcome. CI never calls GitHub. Release validation may additionally run an opt-in sandbox smoke against a disposable protected repository and token; that smoke must assert the protected base stays unchanged until the forge confirms the squash merge, and must exercise a timeout-after-merge restart.

CLI JSON tests decode stdout as JSON and assert that diagnostics remain on stderr. Golden help is limited to 80 columns. Bubble Tea view/update tests use 60, 80 and 120-column window messages, strip ANSI sequences before measuring line width, and cover healthy, unavailable, drift, connect retry, sync-all and disconnect confirmation states.

Stdio smoke

make smoke builds the binary, starts a temporary stdio server and verifies the MCP initialize handshake.

This is a fast local check and is not currently a separate CI step; the Go server tests cover the same protocol path more precisely.

HTTP smoke

make smoke-http runs test/smoke/http.sh. It starts the real binary with two temporary KBs, calls MCP through HTTP and exercises Map creation, concept writes/expansion and Atlas overview.

It runs in CI.

Deterministic end-to-end

make e2e runs the scenarios under test/e2e/. They exercise the compiled binary, HTTP server, CLI client, filesystem and real temporary git remotes together.

The canonical scenario catalog and direct-run flags live in test/e2e/README.md. The suite uses no LLM credentials and runs in CI.

Authorization is tested at two levels, on purpose. 14_rbac_visibility covers RBAC end-to-end at KB granularity, the only level serve currently accepts a policy for through env/CLI. The finer map/journal/type selectors that auth.roles compiles into are covered in Go (internal/mcpserver/policy_test.go, internal/auth/auth_test.go, internal/config/roles_test.go), because expressing them requires a YAML config file rather than the env-var form the scenarios use. Anything asserting non-disclosure belongs at whichever level can observe the raw response: a forbidden resource and a missing one must produce byte-identical output, and a filtered collection must not reveal hidden elements through a short page.

Audit failure paths are tested by fault injection. The whole contract of audit.mode is what happens when the sink is broken, so the write path is made to fail on purpose (audit.FailAppendsForTest) rather than waiting for a real disk error: best_effort must let the call through, required must reject it before the tool handler runs — that last assertion is the one that matters, since a log missing an operation that actually happened is worse than no log. 15_operational_audit closes the loop end-to-end by tampering with a recorded entry and requiring audit verify and audit export to fail on it.

Prefixed multi-KB is exercised with an arbitrary prefix. 16_prefixed_multikb sets tool_prefix to a string unrelated to the KB name, precisely so a client that re-derived the prefix from the KB name instead of discovering it from /health fails the scenario. It also asserts the negative — the bare tool name must not resolve on the prefixed KB — because the D102 promise is that prefixing is exact, not additive.

What is deliberately not in CI

  • Whether a particular model interprets an instruction well.
  • Provider/model quality comparisons.
  • Tests requiring production credentials or external private infrastructure.
  • Manual UI appearance checks.

These belong to production validation or an explicit release exercise, not to the deterministic repository gate.

Before a pull request

make vet
make test
make smoke-http
make e2e

Before a release

  • CI is green on the release commit.
  • make smoke succeeds for the packaged/local stdio path.
  • Installation and upgrade are verified on the target platform.
  • Private deployment rollout checks are performed through maintainer tooling.