ACL evaluator cutover

Out of the box, deployments serve queries through a permissive passthrough ACL evaluator — every retrieved candidate is returned to the caller. Switching to the real evaluator (backed by OpenFGA) filters served results against the upstream source-system permissions.

The cutover is operationally significant: enabling it on a corpus the projector hasn't fully populated will under-grant (queries return fewer hits than expected); enabling it before the connectors have published every record's ACL will likewise miss records.

Run the cutover in two stages.

Stage 1 — dry run

Before the cutover, prove the system is safe to flip. Apply the with-acl.values.yaml overlay with MOOLELO_ACL_DRY_RUN=true:

query-coordinator:
  env:
    QUERY_COORDINATOR_OPENFGA_URL: http://moolelo-openfga.moolelo-system:8080
    QUERY_COORDINATOR_OPENFGA_STORE_ID: <tenant store id from tenant_acl_config>
    MOOLELO_ACL_DRY_RUN: "true"

In dry-run mode the coordinator:

1. Runs the passthrough evaluator as the primary. Served queries are unchanged; users see the same results they saw before.
2. Runs the OpenFGA evaluator as the shadow, in parallel.
3. Emits one acl.dry_run_diff audit row whenever the two evaluators disagree on the allowed set for a query.

Inspect the diff rate via the audit warehouse:

SELECT
  count(*) FILTER (WHERE action = 'acl.dry_run_diff') AS diffs,
  count(*) FILTER (WHERE action = 'query.served')     AS served,
  100.0 * count(*) FILTER (WHERE action = 'acl.dry_run_diff')
       / nullif(count(*) FILTER (WHERE action = 'query.served'), 0) AS diff_rate_pct
FROM audit.tenant_<id>.audit_queries
WHERE event_date >= current_date - INTERVAL '7' DAY;

The cutover gate is < 0.1 % diff rate over a ≥ 7-day window. Higher rates mean either the projector hasn't caught up (most common) or some source-system permission isn't being modeled correctly (rare; investigate the specific entities flagged in would_grant_extra / would_revoke).

Stage 2 — flip

Once the diff rate is below the threshold:

1. Edit the overlay: MOOLELO_ACL_DRY_RUN: "false".
2. Apply via ArgoCD or helm upgrade. The query-coordinator picks up the new env on its next pod restart (rollout is Recreate strategy, so there's a short gap on a single replica — for HA tenants set replicaCount: 2+ before the flip).
3. Watch the audit warehouse for query.served outcome rates. Sustained increase in error outcome with stage=acl indicates the OpenFGA cluster is unreachable; roll back the overlay.

Rollback

If queries start returning empty result sets after the flip:

1. Re-apply the overlay with MOOLELO_ACL_DRY_RUN: "true". The coordinator falls back to passthrough on the next restart.
2. Inspect the per-query audit rows for the acl failure stage. The most common cause is a misconfigured store id that doesn't match what tenant_acl_config says.
3. After fixing, repeat Stage 1 (dry run) for at least 24 hours before flipping again — the corpus may have drifted while the rollback was in effect.

Per-tenant rollout

Multi-tenant deployments cut over one tenant at a time. The tenant_acl_config.strategy column persists each tenant's choice; the helm overlay reads the per-tenant values via the ApplicationSet generator, so different tenants can run in different modes during the rollout window.

A typical sequence: start with one design-partner tenant (lowest user count, highest tolerance for diff investigation), then expand to mid-tier tenants in cohorts of 5–10 over 2–3 weeks.

Strategies

The query-coordinator supports four ACL pre-filter strategies via QUERY_COORDINATOR_ACL_STRATEGY:

• search-then-check (default) — Vespa retrieves top-K; the coordinator's post-retrieval evaluator runs BatchCheck on the candidate set. Works for every tenant; one OpenFGA round-trip per query.
• list-then-filter — coordinator calls OpenFGA ListObjects to build a per-user allow-list, embeds it as a parent_entity_id IN (…) YQL clause, then retrieves. Best when the per-user allow-list is small (≤ 1 000 entities). The ListObjects cap is operator-tunable in OpenFGA's config.
• auto — currently maps to search-then-check; on the roadmap is an adaptive variant that picks based on observed corpus size and ACL churn.

Operators set the strategy at provisioning time via POST /api/admin/tenants or update via the same endpoint. The choice is persisted on tenant_acl_config.strategy and read by the coordinator at boot.

Per-source policy hooks

For permissions that don't fit OpenFGA's relation language cleanly (GitHub archived repos, Confluence anonymous-readable pages, SharePoint sensitivity labels above the user's clearance), the coordinator composes per-source policy hooks via pkg/aclcoordinator.WithHooks. Hooks may tighten the OpenFGA decision but never loosen it — the "more restrictive, never less" invariant is enforced at the wrapper level.

The plan caps the total number of hooks at five across all sources; if you find yourself adding a sixth, look at expressing it as an OpenFGA conditional tuple instead.