Observability

Every Moʻolelo service emits OpenTelemetry traces and metrics, plus structured JSON logs. This page covers what's available and how to consume it.

Logs

All services log to stdout in single-line JSON. Every line carries:

• time — ISO-8601 timestamp.
• level — INFO, WARN, ERROR, etc.
• service — the binary name (e.g., iceberg-sink).
• msg — the human-readable message.
• Plus typed fields per event (e.g., tenant, region, chunk_id, error).

A typical log line:

{
  "time": "2026-05-09T14:23:11.412Z",
  "level": "INFO",
  "service": "iceberg-sink",
  "msg": "icebergsink: committed batch",
  "records": 127,
  "last_message_id": "1234567:42:0:0"
}

Pipe stdout to your log aggregator of choice — Loki, Elasticsearch, Datadog, Splunk, anything that ingests JSON logs.

Metrics

Services emit Prometheus-compatible metrics via OpenTelemetry. Metric names follow moolelo_<service>_<metric>. The metrics every consumer service emits:

Metric	Type	Meaning
moolelo_<svc>_messages_processed_total	counter	Successfully processed messages. Labels: result ∈ ack
moolelo_<svc>_messages_errors_total	counter	Failed messages. Labels: class ∈ transient, permanent, ack
moolelo_<svc>_messages_inflight	gauge	Currently in-flight (between Receive and Ack/Nack)
moolelo_<svc>_message_duration_seconds	histogram	Per-message handler latency

Service-specific metrics:

Service	Metric	Meaning
iceberg-sink	moolelo_iceberg_sink_records_committed_total	Records flushed to Iceberg
vespa-projector	moolelo_vespa_projector_documents_upserted_total	Documents written to Vespa
entity-extractor-dispatcher	moolelo_entity_extractor_dispatcher_chunks_emitted_total	Chunk envelopes published downstream

Configure your Prometheus to scrape http://<pod-ip>:8081/metrics on every Moʻolelo service. The chart values include Prometheus-compatible service annotations.

Traces

Every Pulsar message handled, every external call (parser, embedder, Vespa, Iceberg/Nessie), and every meaningful unit of work emits an OpenTelemetry span.

Span names follow <service>.<operation>:

• iceberg-sink.commit
• entity-extractor-dispatcher.handle
• connector-github.backfill-resource

Standard span attributes on operations that have them:

• tenant, region, source — for cross-cutting context.
• entity_id, chunk_id, record_id — for correlation with specific records.

Configure the OTLP exporter via OTEL_EXPORTER_OTLP_ENDPOINT:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

The Helm charts surface these as values:

observability:
  otlpEndpoint: http://otel-collector.observability:4317

Pipe traces to Tempo, Jaeger, Zipkin, Honeycomb, Datadog, or any other OTLP-compatible backend.

Health endpoints

Every service exposes:

• GET /healthz — liveness; returns 200 once the process is up.
• GET /readyz — readiness; returns 200 once the service is ready to accept work.

Use these in your Kubernetes probes (the Helm charts wire them automatically) and in your external uptime monitoring.

What to alert on

Recommended starting alerts:

Signal	Threshold	Why
moolelo_<svc>_messages_errors_total{class="transient"} rate over 5 min	> 10× the previous hour's rate	Indicates an upstream dependency is unhealthy
moolelo_<svc>_messages_errors_total{class="permanent"} rate over 5 min	> 0 sustained	Records being routed to DLQ — investigate the malformed payloads
moolelo_iceberg_sink_message_duration_seconds p95 over 5 min	> 60 s	Iceberg commits are slow; likely object-storage latency or Nessie issues
moolelo_vespa_projector_messages_inflight	> 1000 sustained	Vespa upserts can't keep up with input rate
/readyz failures	> 1 minute sustained	Pod is unhealthy; Kubernetes will restart it but worth alerting
Audit rows with reason ∈ {cluster_cap_hard, tenant_cap_hard} over 5 min	> 0 sustained	MAU seat cap reached — new users are being denied with HTTP 429. Either upgrade the cap, flip seat_enforcement to soft, or widen the exclude patterns.
Audit rows with reason ∈ {cluster_cap_soft, tenant_cap_soft} over 1 hour	> 0 sustained	Soft enforcement is admitting over-cap users — surface in the admin banner and use as the trigger to size the renewal correctly.
Coordinator log line seats: master unreachable rate over 5 min	> 0 sustained	The fail-closed gate is firing — regional coordinator can't reach the master's /api/internal/seat-claim. New users are being refused with ResourceExhausted; cached subjects keep working. Check master health + NetworkPolicy egress.
moolelo_webserver_seat_claim_duration_seconds p95 over 5 min (when published)	> 100 ms	Master-side Postgres latency on the seat ledger is high; risks blowing the coordinator's 3 s HTTP timeout under load.

Audit log

Every served query produces one audit row written to ClickHouse (with the same data also persisted to the audit_* Iceberg tables for long-term retention).

For compliance dashboards, query the ClickHouse replica:

SELECT
  caller, action, outcome, count(*) AS n
FROM moolelo.audit
WHERE timestamp > now() - INTERVAL 1 DAY
GROUP BY caller, action, outcome
ORDER BY n DESC;

For historical investigation past ClickHouse's retention window, query the Iceberg audit_* tables via Trino — same shape, longer history.

The audit pipeline is documented in Security model.

Sample dashboards

A starter Grafana dashboard JSON is shipped under deploy/grafana/ (when available) covering:

• Per-service request rate, error rate, latency.
• Pulsar consumer lag per topic.
• Iceberg sink commit rate and snapshot summary lag.
• Vespa query latency and indexing lag.
• License key status and grace-window indicator.

Import it into your Grafana instance and adjust the data source bindings.