Enabling the privacy substrate

By default, deployments tokenize PII through a passthrough — the "token" is the cleartext value. That's deliberately permissive during onboarding so operators can prove the search corpus is correct before adding the cryptographic layer. With the privacy substrate enabled, the same wiring seam binds a real vault: PII gets encrypted at ingest, the global graph carries only opaque tokens, and dereferences happen through authenticated calls back to the regional vault.

This page covers how to flip your deployment.

Pre-flight

• The moolelo-privacy-substrate services (vault, residency-service, hris-collector) deployed and reachable from the search-side cluster. See the substrate's own deploy guide for the substrate side.
• HashiCorp Vault Transit secrets engine running and the substrate vault binary configured to use it (VAULT_KEK_BACKEND=vault-transit).
• Identity catalog populated for your tenants — the substrate's HRIS collector or the search-side identity-collector should have emitted at least one round of attribute updates so the residency- service knows where each subject lives.

What changes

Before (vault disabled)	After (vault enabled)
Tokens are cleartext.	Tokens are tok_<region>_<hex>.
Tokenizer.Mode() == "cleartext" on every audit row.	tokenizer_mode == "vault-aesgcm" on every audit row.
Iceberg entities/chunks tables hold raw PII.	Iceberg tables hold opaque tokens; PII lives only in the regional vault.
Subject erasure is a tombstone in the connector cursor table.	Subject erasure flips vault_tokens.erased=true; tokens become unresolvable; the global graph becomes a "redacted user" placeholder.
Detokenize is a no-op (the token is already the value).	Detokenize is an authenticated call back to the home-region vault; the caller's SPIFFE id + reason is logged.

Build the binaries with the vault tag

The vault-adapter is build-tag gated so community-tier images stay slim. Production images intended for substrate-enabled deployments must be built with -tags=vault_enabled.

go build -tags=vault_enabled -trimpath -ldflags='-s -w' ./cmd/...

Without the tag, vaultadapter.New() returns ErrVaultDisabled and the wiring layer falls back to passthrough — boot succeeds but no PII reaches the substrate. The CI release-gate verifies the tag is set on tagged release builds.

Apply the overlay

helm upgrade <release> ./deploy/helm/<chart> \
  -f deploy/values/base.yaml \
  -f deploy/values/with-vault.yaml

The overlay sets, per service:

MOOLELO_VAULT_ENDPOINTS=eu-central=http://moolelo-vault.moolelo-system:8443
MOOLELO_RESIDENCY_URL=http://moolelo-residency-service.moolelo-system:8090
MOOLELO_VAULT_SPIFFE_ID=spiffe://moolelo.dev/ns/moolelo-system/sa/<svc>

Multi-region operators populate MOOLELO_VAULT_ENDPOINTS with one region=url pair per region they run a vault in.

Per-tenant onboarding step

When you provision a new tenant via the webserver admin endpoint (POST /api/admin/tenants), the provisioner calls the substrate vault's /admin/ensure-tenant endpoint to pre-create the per-tenant KEK in HashiCorp Vault Transit. This step is idempotent and adds ~50 ms to the onboarding call. Without it, the first tokenize for a fresh tenant would fail with a missing-key error.

Audit verification

After the cutover, query.served audit rows in the audit warehouse include a tokenizer_mode field:

SELECT
  count(*) FILTER (WHERE details['tokenizer_mode'] = 'cleartext') AS cleartext_calls,
  count(*) FILTER (WHERE details['tokenizer_mode'] = 'vault-aesgcm') AS vault_calls
FROM audit.tenant_<id>.audit_queries
WHERE event_date >= current_date - INTERVAL '7' DAY;

A successful cutover shows cleartext_calls = 0 after the rollout window. If cleartext_calls stays non-zero, one of the binaries was built without -tags=vault_enabled or its env vars are empty; the wiring layer's boot-time warning log identifies which seam fell back.

Rollback

To roll back to the passthrough binding without redeploying images, unset MOOLELO_VAULT_ENDPOINTS on the affected services and restart. The wiring layer's vault-disabled fallback engages on the next boot; the warning log makes the demoted seam visible.

For a hard rollback (binary built with vault_enabled is in the path of an audit failure), redeploy without the tag — that guarantees no PII can reach the substrate from the affected binary.

Existing records when you cut over

Records ingested before the cutover have cleartext "tokens" already written to the global graph. Cutting over to vault doesn't retroactively encrypt them. A re-tokenize worker is on the roadmap that walks every existing chunk, decodes the cleartext, calls the vault to re-tokenize, and rewrites the chunk's tokens in place. Until it ships, operators who need the pre-cutover backlog encrypted re-ingest the affected sources.