Troubleshooting

Common operational issues, organized by symptom. For per-source troubleshooting, see the connector pages: GitHub, Confluence, SharePoint.

A service won't start

license: invalid

The license key signature didn't verify.

• The file is corrupted — sign in to your account at moolelo.dev/account and re-download the key, or rotate it.
• The key was issued against a different signing root than your binary expects. This happens when you mix licenses across major-version generations of Moʻolelo. Contact your account team.
• The file is empty — check the Kubernetes Secret content with kubectl get secret -n moolelo-system moolelo-license-key -o yaml.

no license file configured

MOOLELO_LICENSE_PATH is unset, or points to a file that doesn't exist in the container.

Verify the Helm values include the license block, and confirm the volume mount on the Pod:

kubectl exec -n moolelo-system deploy/connector -- ls -la /etc/moolelo/

license expired N hours ago

You're past the 7-day grace window. Roll a renewed license key (see License key management) and restart the affected services.

Pod in CrashLoopBackOff with no helpful logs

The pod is crashing before logging is initialized. Common causes:

• Missing required env var (see the per-service env-var reference).
• The container can't reach Pulsar / Postgres / Vespa / Nessie at startup. Check Network Policies and DNS resolution from inside the cluster.

Records aren't appearing

The connector is running but no records published

1. Check the connector logs for the active mode:

   kubectl logs -n moolelo-system deploy/connector | grep "controller: mode change"

   If you see mode: disabled (or no mode line at all), the tenant config isn't enabling the source. UPSERT the connector_configs row with the right mode.

2. For backfill mode, check the cursor:

   psql "$CONNECTOR_PG_DSN" -c \
     "SELECT * FROM connector_cursors WHERE source = 'github';"

   If updated_at is stuck, the source's API is rate-limiting you. The connector backs off and retries; this can take hours on heavily-restricted tokens.

3. For webhook mode, check that the source side is actually sending webhooks. GitHub repository → Settings → Webhooks → Recent Deliveries shows whether your URL is being called and what the response was.

Records published but not appearing in search

Walk the pipeline backwards:

1. Iceberg has the chunks?

   -- Via Trino
   SELECT count(*) FROM moolelo.chunks WHERE source = 'github';

   If 0, the iceberg-sink isn't keeping up — check its logs for commit failures. (Note: pre-tokenization payloads do not persist to Iceberg; if you need to inspect the verbatim source bytes for a record, check Pulsar within the 30-day retention window.)

2. Vespa has the chunks?

   curl 'https://<your-vespa-host>:8080/document/v1/moolelo/chunk/docid?selection=true&wantedDocumentCount=1'

   If the response shows zero chunks, the projection pipeline (dispatcher → tokenizer-driver → vespa-projector) is stuck somewhere. Check Pulsar lag on each topic.

3. Pulsar lag stalls?

   kubectl exec -n moolelo-system pulsar-broker-0 -- \
     pulsar-admin topics stats persistent://moolelo/default/eu-central.chunks.embedded

   Look at subscriptions.<sub-name>.msgBacklog. A growing backlog means the consumer can't keep up; scale it (KEDA usually does this automatically) or investigate downstream errors.

Search results look wrong

Results are missing records that should match

Three common causes:

1. The records aren't indexed yet. Backfill takes hours for a large corpus. Check the Vespa chunk count is increasing.
2. The query is filtering them out. YQL filter clauses are strict — tenant_id contains "default" matches only the literal string. Inspect your query.
3. OpenFGA denies the user on the chunk's parent entity. The query-coordinator post-filters Vespa candidates through OpenFGA BatchCheck — a chunk Vespa returns can still be dropped if the user has no viewer relation on its parent_entity_id. Inspect the OpenFGA decision:

   curl -X POST https://<openfga-host>/stores/<store_id>/check \
     -H "Content-Type: application/json" \
     -d '{"tuple_key":{"user":"user:alice","relation":"viewer","object":"entity:<parent_entity_id>"}}'

Results are ranked weirdly

Switch the rank profile and compare:

• bm25 — pure keyword match. If results look better here than under default, the embedding model isn't a good fit for your corpus's domain.
• semantic — pure embedding similarity. If results look better here, your queries don't share much vocabulary with your documents (try this with conversational queries).
• hybrid with query(alpha) tuned — most flexible.

For systematic improvement, use the Vespa eval harness against a labeled gold set.

Result text shows tokens instead of cleartext

The Vespa chunk text holds vault tokens (the actual personally-identifiable strings live in regional vaults). Your search frontend needs to call the regional vault to detokenize before rendering — it's not Moʻolelo's job to detokenize on the search response.

If you don't have personally-identifiable data tokenized at all (single-region deployment with passthrough configuration), the vault returns the cleartext directly. If you see tok_xxx strings in your search results without a multi-region setup, check your vault adapter's configuration.

Performance issues

Backfill is slow

Backfill speed is gated by the source system's API. GitHub limits authenticated requests to 5,000/hour by default; Confluence is similar; SharePoint via Microsoft Graph is more generous but still bounded.

You cannot make backfill faster than the source allows. What you can do:

• For GitHub: use a GitHub App instead of a PAT (GitHub Apps get higher per-installation rate limits).
• For Confluence: split a large space into multiple connector entries pointing at different child pages.
• For SharePoint: use delta tokens (the connector does this automatically) so subsequent backfills only fetch changes.

Embedding is slow

If you're on the in-cluster, no-Ray topology and embedding is your bottleneck:

• Bump MOOLELO_EMBEDDER_BATCH from the default 16 to 32 or 64. Larger batches are more efficient on CPU.
• Scale the embedder Deployment horizontally — KEDA autoscales on Pulsar lag, but the parser/embedder Deployments don't have a lag signal; you can either scale manually or set up an HPA on CPU.
• Move to the KubeRay + KServe topology with vLLM on GPUs (10x+ throughput improvement).

Vespa queries are slow

Check:

• Vespa container CPU saturation. Add container nodes if pegged.
• Vespa content node disk IOPS. The HNSW index needs fast random reads; SSDs are required, not HDDs.
• The query's complexity. Compound YQL filters with many ORs scale linearly; flatten where possible.

Data integrity issues

Iceberg snapshot summary out of sync with Pulsar offset

The iceberg-sink stores the last-committed Pulsar message ID in the Iceberg snapshot summary. On restart, it seeks the consumer past that offset. If for any reason the snapshot summary is corrupt or unparseable, the sink falls back to the broker's saved cursor — which means it might re-process messages.

Re-processing is safe (the sink is idempotent on natural keys). You'll see duplicate records in Iceberg only if the message IDs are different across re-deliveries (which they aren't — Pulsar message IDs are stable).

Search index out of sync with Iceberg

Vespa is a projection of Iceberg. If they diverge — say, you manually deleted chunks from Vespa, or Vespa lost a content node and recovered from a stale backup — you can rebuild the projection by triggering a backfill on the relevant connectors.

A full rebuild from Iceberg without re-fetching from sources isn't currently supported (it's on the roadmap). For now, the simplest "rebuild Vespa" path is: re-run the backfill.

When to escalate

Escalate to your Moʻolelo support contact when:

• A pod crashes with logs that don't match anything on this page.
• An upgrade between Moʻolelo versions fails partway through.
• A license key issue isn't resolved by re-installing the key.
• Data residency rules appear to be violated (e.g., personal data appearing in a region it shouldn't be in).

Always include the relevant pod logs (kubectl logs --previous for crashed pods), your Moʻolelo version, and the Helm values of the affected service.