Querying via Iceberg

The durable record store is a set of Apache Iceberg tables managed by Project Nessie. Any Iceberg-aware tool can query them. This page shows the most common ones.

What's in there

Four tables in the ingest warehouse:

• chunks — the parsed, embedded, and tokenized chunks that back the search index. Useful for analytical questions over the search corpus, and for re-projecting Vespa from a durable source if its index needs to be rebuilt.
• entities — extracted entity rows (one per source record). Tokenized fields only — cleartext PII never lives here.
• relations — directed edges between entities (authored_by, mentions, references, etc.). Powers graph analytics.
• acl_authority — the source-system authority chain that justified ingesting each record. Useful for compliance queries ("show me who could see this record at this moment").

The pre-tokenization payload (the verbatim source content with cleartext PII) is not in Iceberg. It lives only in Pulsar with ~30-day retention as the short-term replay buffer. After the ingest pipeline tokenizes each chunk, the tokenized form lands in chunks; the cleartext is discarded from Iceberg at rest. This keeps the entire Iceberg warehouse residency-clean — token-only, no cleartext PII to manage residency boundaries around.

Audit warehouse tables (audit.queries, audit.tokenizations, audit.access_decisions) are in a separate Iceberg catalog. See the audit warehouse docs for that.

Connecting Trino

CREATE CATALOG moolelo USING iceberg
WITH (
  "iceberg.catalog.type" = 'rest',
  "iceberg.rest-catalog.uri" = 'https://<your-nessie-host>:19120/iceberg',
  "iceberg.rest-catalog.warehouse" = '<warehouse-name>'
);

If your warehouse is on AWS S3, add s3.region, s3.endpoint (when using S3-compatible storage), s3.access-key-id, s3.secret-access-key. For Google Cloud Storage, the Nessie catalog handles credentials server-side and Trino just talks Iceberg REST.

Once the catalog is registered:

USE moolelo.moolelo;

SELECT count(*)
FROM chunks
WHERE source = 'github' AND tenant_id = 'default';

SELECT source, count(*) AS chunks_count
FROM chunks
WHERE created_at > current_timestamp - INTERVAL '7' DAY
GROUP BY source
ORDER BY chunks_count DESC;

Connecting DuckDB

INSTALL iceberg;
LOAD iceberg;

ATTACH 'iceberg' AS moolelo (
  TYPE iceberg,
  ENDPOINT 'https://<your-nessie-host>:19120/iceberg'
);

SELECT count(*) FROM moolelo.moolelo.chunks;

DuckDB's Iceberg support is read-only. For write-back analytical jobs, use Trino or Spark.

Connecting Spark

val spark = SparkSession.builder()
  .config("spark.sql.catalog.moolelo", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.moolelo.type", "rest")
  .config("spark.sql.catalog.moolelo.uri", "https://<your-nessie-host>:19120/iceberg")
  .config("spark.sql.catalog.moolelo.warehouse", "<warehouse-name>")
  .getOrCreate()

spark.sql("SELECT count(*) FROM moolelo.moolelo.chunks").show()

Connecting PyIceberg

from pyiceberg.catalog import load_catalog

catalog = load_catalog("moolelo", uri="https://<your-nessie-host>:19120/iceberg")
table = catalog.load_table(("moolelo", "chunks"))

# Scan into Pandas
df = table.scan(
  row_filter="source = 'github' AND created_at > '2026-05-01T00:00:00Z'"
).to_pandas()

Schema reference

The full schema of every table is discoverable from your query engine — DESCRIBE moolelo.moolelo.chunks in Trino or table.schema() in PyIceberg. The fields you'll use most often on chunks:

• tenant_id, source — partition + filter columns.
• chunk_id, parent_entity_id — natural identifiers; parent_entity_id joins to entities.
• text — the tokenized chunk content. PII spans appear as tok_<region>_<hex> references; pass them to the regional vault to detokenize if you need the cleartext.
• embedding — packed little-endian float32 dense vector (1024-d bge-m3 today). Reshape with np.frombuffer(row['embedding'], dtype=np.float32) if you need it.
• structural_metadata — map<string,string> of chunk-position context (heading path, file path, line range, etc.).
• created_at — ingestion timestamp.

Snapshot semantics

Iceberg tables are immutable at the snapshot level — every write produces a new snapshot. You can run time-travel queries against any historical snapshot:

-- Trino: query as of a specific timestamp
SELECT count(*) FROM moolelo.chunks FOR TIMESTAMP AS OF TIMESTAMP '2026-04-01 00:00:00 UTC';

-- Trino: query as of a specific snapshot id
SELECT count(*) FROM moolelo.chunks FOR VERSION AS OF 12345;

This is useful for debugging ("what did the corpus look like before yesterday's ingest?") and for compliance ("show me the corpus at the moment a query was answered").

Permissions

The Iceberg layer doesn't enforce per-user permissions on its own — anyone with read access to the catalog can read the underlying tables. If your tenant requires user-level access control on the analytical layer, you have two options:

1. Project-level access: configure your Trino / Spark cluster's user model to gate access to the moolelo catalog.
2. Application-level filtering: query through an application that joins chunks (or entities) with the acl_authority table and applies the calling user's authority chain.

For the search use case (Vespa), Moʻolelo applies authority chains automatically. The Iceberg layer is intentionally more permissive — it's a data plane for analytical workloads, not a substitute for the search index.

Re-projecting Vespa from Iceberg

If you need to rebuild the Vespa index — schema migration, content-node loss, intentional reindex — the chunks table is the durable source. The vespa-projector consumes from the Pulsar <region>.chunks.tokenized topic in steady state, but a one-shot re-projector that scans chunks and writes to Vespa is a clean recovery path that doesn't require re-fetching from source systems or re-running the parser. Plan for this when sizing your Iceberg storage — keep enough chunk history that a full Vespa rebuild is always cheaper than re-ingesting.