Supersession (deduplication)

Knowledge bases accumulate duplicates and stale versions. A wiki page is rewritten; the old version stays. A pull request closes an issue; the issue stays open in search. Three engineers each write their own runbook for the same incident; all three show up at the top of results.

Supersession is the platform's answer. Moʻolelo continuously identifies pairs of records where one has effectively replaced another, and tags the older one so retrieval can downrank it, filter it, or surface a "this was replaced by…" marker.

You don't have to do anything to get the basic behaviour — supersession runs automatically on every tenant. You configure how it shows up in results.

What triggers a supersession decision

Five signals feed the supersession engine. Each runs continuously across the corpus.

Signal	What it detects
Content hash	Two records whose post-tokenization chunk text is byte-equal. Conservative — only fires on exact duplicates.
MinHash	Near-duplicates by Jaccard similarity over shingles. Catches "copy-paste with light edits".
Embedding ANN	Semantic near-duplicates from bge-m3 vectors via Vespa nearest-neighbor. Catches rewrites that share no exact wording.
Reference graph	A transitive walk over the supersedes edge in the entity graph. If page A links to page B as "see updated version" and B links to C the same way, the engine resolves A → C in one hop.
CDC explicit	The source system told us. GitHub "PR #42 closes #7" is the canonical example. SharePoint and Confluence have their own equivalents.

Each decision lands in the supersession_decisions Iceberg table with the signal, a confidence score, the decided-at timestamp, and an operator-readable reason ("MinHash Jaccard 0.94", "PR #42 closed issue #7"). The decisions are auditable per tenant.

Decisions emitted by cdc-explicit are accepted at confidence 1.0 — source-system truth wins. Decisions from minhash or content-hash arrive at 0.85+ when they pass. Embedding-ANN decisions carry a lower confidence floor and are the most conservative.

Three ways to apply decisions

Each tenant picks one of three retrieval policies for how supersession decisions reach query results:

Policy	What the user sees
marker (default)	Both records appear at their original rank; the superseded one carries a superseded_by field pointing at the canonical record. The UI renders "This was replaced by [X]". Most auditable — nothing is hidden.
rerank	Both records appear, but the superseded one's score is multiplied by a tenant-configurable factor (default 0.5). It sinks in rank but stays visible.
filter	The superseded record is dropped from the result set entirely. Cleanest UI; least auditable.

Pick marker if your users need to know what was replaced. Pick filter for assistant-style interfaces where the user shouldn't see ghosts. rerank is the middle ground.

Switch policies with a single tenant-config write — no redeploy:

UPDATE tenant_supersession_config
SET retrieval_policy = 'rerank',
    rerank_factor    = 0.4
WHERE tenant_id = 'acme';

The query coordinator picks up the change on its next config-refresh tick (default 60 s; tunable per environment).

Authority hierarchies

Two records are near-duplicates. Which one wins? The engine consults the tenant's authority hierarchy — an ordered list of (source, record_type, priority) tuples that says "a Confluence page beats a SharePoint document beats a GitHub issue comment".

A hierarchy looks like this:

INSERT INTO tenant_authority_rules (tenant_id, source, record_type, priority) VALUES
  ('acme', 'confluence',  'page',         100),
  ('acme', 'sharepoint',  'document',      80),
  ('acme', 'github',      'pull_request',  60),
  ('acme', 'github',      'issue',         40);

Higher priority wins. Two records with the same priority tie — the engine keeps both rather than picking arbitrarily, and the retrieval policy applies to neither. Sane defaults ship for every connector; override per tenant when the defaults don't match the org's source-of-truth structure.

How to verify it's working

The admin webserver surfaces a per-tenant supersession dashboard:

GET /api/admin/tenants/{tenant}/supersession/recent

Returns the last N decisions with signal, confidence, reason, and the two record IDs involved. Use this to sanity-check the engine on a new tenant or after switching authority rules.

For evidence in compliance audits, query the supersession_decisions Iceberg table directly via Trino or DuckDB — every decision is recorded with its tenant, signal, confidence, and reason. See Audit warehouse.

Disabling supersession per tenant

If a tenant explicitly doesn't want any deduplication (the corpus is genuinely meant to carry every version, e.g. a regulatory archive):

UPDATE tenant_supersession_config
SET enabled = false
WHERE tenant_id = 'acme';

The engine still records candidate pairs in the audit warehouse but does not project decisions into Vespa. You can flip the switch back on at any time; the engine catches up from the last processed offset.