Version: v0.9.0a2

Integrator

Conformance Testing

5 min readProtocol implementer · ContributorMulti-backend

What this page covers

Two complementary conformance layers: machine-readable wire vectors for the HTTP API, and a behavioral suite that exercises the full fact/recall/graph/decay/federation/contradiction contract across SQLite, libSQL, and Postgres.

Audience: protocol implementers validating a Stigmem node; contributors adding or changing spec behaviour; operators of forks or alternative backend implementations.

Overview

Layer

Audience

What it tests

Wire conformance

spec contributors

HTTP wire format and response shape against machine-readable JSON vectors.

Multi-backend behavioral suite

backend implementers · CI

Full behavioral contract across SQLite, libSQL, and Postgres.

Wire conformance

The current wire-vector suite is the executable test contract for the v0.9.0a1 wire format.

The files still live under data/conformance/v1.0/ for compatibility with the pre-reset harness naming — treat that directory name as historical, not as a claim that v1.0 has shipped.

Running the suite

# From the repo root
uv run pytest node/tests/conformance/test_conformance_v1.py -v

The runner starts an in-process test node, executes every vector against it, and fails on any regression. Zero skips are enforced — if your change causes a vector to be skipped, CI will fail.

To run a single vector group:

uv run pytest node/tests/conformance/test_conformance_v1.py -v -k "garden"

CI

The Conformance workflow (conformance.yml) runs automatically on every push to main and on pull requests that touch node/, spec/, data/conformance/, or the workflow file.

Vector groups

File

Spec coverage

Notes

01_fact_assert.json

Spec-01, Spec-03

Fact assert wire format, all FactValue types.

02_fact_query.json

Spec-03, Spec-15

Query, single-fact GET, retraction.

03_wellknown.json

Spec-03

/.well-known/stigmem discovery endpoint.

04_gardens.json

Spec-02

Memory Garden CRUD, role management.

05_garden_facts.json

Spec-02, Spec-01

Garden-tagged fact writes and ACL enforcement.

Adding a new vector

Requirement

Every new spec section or wire-format change MUST include at least one new conformance vector. PRs that add or modify spec text without a corresponding vector will not be merged.

Add to an existing group file that covers the relevant spec section. If no group exists, create a new numbered file (e.g. 06_my_feature.json).
Follow the vector structure (see below).
Run uv run pytest node/tests/conformance/test_conformance_v1.py -v locally before opening a PR.

Vector structure:

{
  "spec_section": "§X.Y",
  "title": "Short group title",
  "description": "Optional longer description",
  "vectors": [
    {
      "id": "unique-kebab-id",
      "description": "What this vector tests",
      "method": "POST",
      "path": "/v1/facts",
      "body": { ... },
      "expected_status": 201,
      "expected_body_contains": { "entity": "stigmem://testnode/user/alice" },
      "expected_body_has_keys": ["id", "timestamp"]
    }
  ]
}

Assertion fields:

Field

Type

Checks

expected_status

int

HTTP response status code.

expected_body_contains

object

Top-level response keys equal these values.

expected_body_has_keys

string[]

Response object includes these keys (any value).

expected_nested

object

Dotted-path assertions into the response.

Do not add vectors with requires_auth: true.

Zero skips are enforced and auth-dependent scenarios belong under node/tests/auth/.

Multi-backend behavioral suite

The behavioral suite in node/src/stigmem_conformance/tests/ exercises the full behavioral contract against SQLite, libSQL, and Postgres.

In-process (simplest, no external services)

# SQLite (default — no dependencies)
python -m stigmem_conformance --backend sqlite

# libSQL (requires libsql-experimental)
pip install 'stigmem-node[libsql]'
python -m stigmem_conformance --backend libsql

Postgres

# Start a local Postgres instance (Docker)
docker run -d -e POSTGRES_PASSWORD=test -p 5432:5432 postgres:16

# Install dependencies
pip install 'stigmem-node[postgres,conformance]'

# Run
STIGMEM_TEST_PG_DSN="postgresql://postgres:test@localhost/postgres" \
python -m stigmem_conformance --backend postgres

Generate a Markdown report

python -m stigmem_conformance --backend sqlite --report conformance-sqlite.md

The report includes pass/fail counts, test details, and failure traces. Suitable for committing alongside a release or embedding in operator documentation.

Via pytest directly

# All backends (skips unavailable ones automatically)
uv run pytest node/src/stigmem_conformance/tests/ -v

# Pin to one backend
uv run pytest node/src/stigmem_conformance/tests/ --conformance-backend=sqlite -v

Test domains

Module

Tests

Skip conditions

test_facts.py

core

Fact assert, query, retraction, TTL, validation. No skips.

test_recall.py

core

Recall endpoint, token budget, scope isolation, response shape.

test_graph.py

core

Graph neighbors, depth traversal, scope filter, retraction cleanup.

test_decay_synthesis.py

core

Synthesize endpoint, decay sweeper, confidence ordering.

test_contradiction.py

core

Contradiction detection, conflict list, resolution.

test_embeddings.py

SQLite-only

Recall without vector index; vector tests. Skipped on libSQL/Postgres (sqlite-vec not applicable).

test_provenance.py

core

Audit trail, source attribution, recall_id/query_hash.

test_federation.py

core

Well-known endpoint, peer registration, pull API shape.

Interpreting the report

Status

Symbol

Meaning

passed

✅

Backend behaves correctly for this scenario.

failed

❌

Behavioral regression — must be fixed before shipping.

skipped

⏭

Feature not applicable to this backend (justified skip).

A report with zero failures and only justified skips means the backend is conformant.

Extending the suite

Create node/src/stigmem_conformance/tests/test_<domain>.py.
Use only the HTTP API — no direct DB access.
Use the conformance_client fixture (parametrized over all backends).
Add pytest.skip(reason="...") for backend-specific features with a clear justification string.
Update the test-domains table above.

# Example: new domain test
from .conftest import ConformanceClient

class TestMyFeature:
    def test_something(self, conformance_client: ConformanceClient) -> None:
        if conformance_client.backend == "postgres":
            pytest.skip("feature X not supported on Postgres (pg_vector needed)")
        r = conformance_client.client.post("/v1/my-endpoint", json={...})
        assert r.status_code == 200

CI matrix

The conformance-multi-backend job runs all three backends in parallel. A conformance-gate job ensures all three are green before a PR can merge. The Postgres backend is feature-flagged (via the postgres extras and STIGMEM_TEST_PG_DSN env var) and runs against a GitHub-hosted Postgres 16 service.

Spec citations

Every vector group file declares the modular spec it covers. Keep these citations up to date with the canonical spec composition in spec/PROTOCOL.md.

When the spec and a vector disagree, the spec is authoritative.

Update the vector and open a bug report for the reference node.

Overview​

Wire conformance​

Running the suite​

CI​

Vector groups​

Adding a new vector​

Multi-backend behavioral suite​

In-process (simplest, no external services)​

Postgres​

Generate a Markdown report​

Via pytest directly​

Test domains​

Interpreting the report​

Extending the suite​

CI matrix​

Spec citations​

Overview

Wire conformance

Running the suite

CI

Vector groups

Adding a new vector

Multi-backend behavioral suite

In-process (simplest, no external services)

Postgres

Generate a Markdown report

Via pytest directly

Test domains

Interpreting the report

Extending the suite

CI matrix

Spec citations