License key management

Every Moʻolelo service refuses to start without a verified license key. Verification is offline against the public key embedded in each binary — no service ever phones home in steady state.

What's in a license key

A license key is a JWT-shaped envelope (<base64url-payload>.<base64url-signature>) carrying:

• Tier — community, enterprise, or regulated.
• Issued-to — your organization name as it appears on your account.
• Issued-at and expires-at — Unix timestamps.
• Limits — maximum seats (max_employees, the MAU cap), maximum events per day, maximum regions, maximum tenants, and mau_window_days (rolling window the seat counter uses, default 30).
• Seat enforcement — "hard" (default) refuses new subjects past the cap; "soft" admits them but emits an audit row and surfaces a banner. Existing subjects already in the window are never disrupted.
• Features — a list of capability flags the license unlocks. Currently three: multi_tenant, multi_region, vault_substrate.

The signature is ed25519 against the website's signing root. Tampered or unsigned envelopes are rejected at startup.

Getting your first license

1. Register at moolelo.dev/register. Sign-in is by email magic link — no password.
2. From your account dashboard click Generate Community license. The site signs a fresh envelope and shows the new entry under "Your licenses."
3. Download the key file. It will be named moolelo-<id>.key.
4. Install it (see below).

The Community tier is free and unlocks single-tenant single-region production use at any organization. The license is time-bounded — when your one-year term is up you renew from the same dashboard, also for free.

The Commercial tier unlocks multi-tenant, multi-region, and the vault-substrate feature flags. To request one, use the Talk to us link on the pricing page.

Installing a license key

There are two supported shapes. Pick one per deployment.

Option A — license authority (default, recommended)

Upload the envelope once to the webserver; every other service fetches it from there over HTTP, verifies the signature locally with its compiled-in public key, and re-fetches periodically to pick up rotations.

The webserver boots in bootstrap mode when no license is loaded — only the license + health endpoints are reachable; the rest of the admin and feature surface returns 503 until an envelope is uploaded. A small static bootstrap token authorises the upload call (set as a Kubernetes Secret).

# 1. Generate the bootstrap token and store it as a Secret.
TOKEN=$(openssl rand -hex 32)
kubectl create secret generic webserver-bootstrap-token \
  --namespace moolelo-system \
  --from-literal=token="$TOKEN"

# 2. Enable the Secret in the webserver Helm release.
helm upgrade --install moolelo-webserver deploy/helm/webserver \
  --set bootstrapTokenSecret.enabled=true

# 3. Port-forward and upload the envelope.
kubectl port-forward -n moolelo-system svc/moolelo-webserver 8088:8088 &
curl -X POST http://localhost:8088/api/admin/license \
  -H "X-Moolelo-Bootstrap-Token: $TOKEN" \
  -H "Content-Type: text/plain" \
  --data-binary @/path/to/moolelo-<id>.key

A successful upload returns the parsed tier, issued-to, and expires-at. The webserver flips to active state, and the other services — connector, iceberg-sink, query-coordinator, etc. — boot cleanly with their MOOLELO_CONTROL_PLANE_URL env var (http://moolelo-webserver.moolelo-system:8088) pointing at the master webserver. From that one URL each service auto-discovers two things: the signed license envelope (/api/internal/license) and the regions config (/api/internal/regions, which seeds the vault endpoint map). See Control plane below.

The bootstrap token is single-purpose: it returns 401 for any upload attempt once a valid license is loaded. Rotate the underlying Secret to invalidate.

By default the authority persists the envelope in Postgres (the existing tenant-config database). To back it with HashiCorp Vault or Azure Key Vault instead, set MOOLELO_LICENSE_SOURCE on the webserver:

env:
  MOOLELO_LICENSE_SOURCE: vault+https://vault.example.com/v1/secret/data/moolelo/license?field=envelope
  # OR
  MOOLELO_LICENSE_SOURCE: azkv://moolelo-prod.vault.azure.net/moolelo-license

When backed by an external KV, the upload API is rejected — manage the envelope through the KV's own API. The authority re-reads from the source every 5 minutes (MOOLELO_LICENSE_REFRESH_EVERY overrides) so rotations propagate without restart.

Option B — file mount on every pod

The classic shape: ship the envelope as a Kubernetes Secret mounted into every service's pod. No license-authority hop, no bootstrap dance.

kubectl create secret generic moolelo-license-key \
  --namespace moolelo-system \
  --from-file=license.key=/path/to/moolelo-<id>.key

Then set MOOLELO_LICENSE_PATH=/etc/moolelo/license.key on every service (including the webserver, which then skips the bootstrap path) and mount the Secret at that path. MOOLELO_LICENSE_URL and MOOLELO_LICENSE_PATH are mutually exclusive — services that have both set will refuse to boot.

Useful when:

• You have a single-binary deployment (no separate webserver to host the authority).
• Your secret-distribution story is already mature and you'd rather not add another hop.
• You want every service to be independent at startup with no dependency on the webserver.

Control plane

The master region runs a control plane that every other service in the cluster (and every replica region) pulls its boot configuration from. Once an operator sets MOOLELO_CONTROL_PLANE_URL on a service, two endpoints come into play:

Endpoint	Purpose	Consumed by
GET /api/internal/license	Signed license envelope	Every service's pkg/license/fetcher
GET /api/internal/regions	Region declarations (id, jurisdiction, vault URL, notes)	Every service's pkg/wiring via pkg/regionclient
GET /api/internal/seat-config	Hash key + exclude patterns + MAU window + caps	Each query coordinator's pkg/seatclient
POST /api/internal/seat-claim	Authoritative per-subject cap check	Each query coordinator's RemoteSeatGate on every new-subject Search

Concretely, an iceberg-sink running in a replica region with just:

env:
  MOOLELO_SERVICE: iceberg-sink
  MOOLELO_REGION: us-east
  MOOLELO_CONTROL_PLANE_URL: http://moolelo-webserver.master-region.example.com:8088

will fetch the license envelope, fetch the regions list, populate its in-memory vault endpoint map from the regions whose vault_url is non-empty, and verify the license locally against its compiled-in public key — all at startup. No per-service MOOLELO_VAULT_ENDPOINTS env var, no per-service MOOLELO_LICENSE_URL. The control plane is the single source of truth.

Manual overrides still work (MOOLELO_LICENSE_PATH for an offline license, MOOLELO_VAULT_ENDPOINTS for an explicit region map), but they're mutually exclusive with the control-plane URL — set one or the other, never both. LoadConfig rejects the ambiguous combinations at boot:

Pair	Result
MOOLELO_LICENSE_PATH + MOOLELO_LICENSE_URL	Boot refused: pick one
MOOLELO_CONTROL_PLANE_URL + MOOLELO_VAULT_ENDPOINTS	Boot refused: control plane already provides the map

The master webserver itself doesn't set MOOLELO_CONTROL_PLANE_URL (it IS the control plane). It uses MOOLELO_LICENSE_SOURCE for the envelope and reads admin-declared regions directly from the platform_regions table.

Vespa replication between the master and replica regions is a separate operational concern — the control plane carries the declarative source of truth, but rebuilding a replica's Vespa cluster from the master Pulsar is out-of-band setup.

What happens at startup

When a Moʻolelo service starts:

1. It reads the license key from the configured path.
2. It verifies the ed25519 signature against the public key embedded in the binary.
3. It parses the claims — tier, expiry, limits, features.
4. It applies the 7-day grace window past expires_at.
5. It checks any feature flags the binary's wiring needs against the license's features list — for example, configuring MOOLELO_VAULT_ENDPOINTS requires vault_substrate, and provisioning a non-default tenant requires multi_tenant.

If signature verification, parsing, or a feature gate fails, the service refuses to start with a clear error in its logs. Operators see this immediately via kubectl get pods showing CrashLoopBackOff.

What happens at expiry

Time vs expires_at	Behavior
Before expiry	Normal operation.
Up to 7 days past	Service continues operating; logs a warning at every startup; admin UI surfaces a banner.
After 7-day grace	Ingestion stops (connectors halt). Queries against existing data continue working — your users don't see disruption. New license file required to resume ingestion.

The grace window gives operators time to roll a renewed key without an emergency.

Renewing a key

Licenses are time-bounded. The renewal model is also the air-gap-correct revocation lever: nothing on the platform side phones home; we revoke by declining to issue your next envelope.

When you're approaching expires_at:

1. Sign in to moolelo.dev/account.

2. Click Renew on the license you want to renew. We sign a fresh envelope; the previous one stays valid until its own expires_at so deployments have a clean transfer window.

3. Download the new envelope.

4. Replace the contents of the moolelo-license-key secret:

   kubectl create secret generic moolelo-license-key \
     --namespace moolelo-system \
     --from-file=license.key=/path/to/new-license.key \
     --dry-run=client -o yaml | kubectl apply -f -

5. Restart the services to pick up the new key:

   kubectl rollout restart deployment -n moolelo-system \
     -l moolelo.io/component

If your deployment uses External Secrets Operator + Vault, drop the new key into Vault — the secret rotates and pods restart automatically per your operator's reload policy.

Renew vs Rotate vs Revoke

The dashboard exposes three actions per license:

• Renew — issue a fresh envelope; keep the old one valid until its own expiry. Use this for planned end-of-term renewal. The two envelopes overlap, so deployments roll the new key on their schedule.
• Rotate — issue a fresh envelope AND revoke the old one immediately. Use this for compromise scenarios: a key has leaked and the existing envelope must stop working at the next platform restart cycle.
• Revoke — flip the existing envelope to revoked without issuing a replacement. Re-download is blocked. The deployed envelope keeps working until its own expires_at (no runtime phone-home), so this is effectively "schedule expiry of an unused key."

Limit enforcement

Limits are checked at the points where they matter:

• Max seats (max_employees) — counted as the MAU (monthly active users): distinct identities that hit the query coordinator inside the rolling mau_window_days window (default 30 days). The master webserver holds the single cluster-wide active_subjects ledger; every regional coordinator's RemoteSeatGate hashes the OIDC subject locally and POSTs the hash to /api/internal/seat-claim on the master. The master runs the atomic cap check + ledger write and returns Allow/Deny. Coordinator caches admit decisions in memory for 1 hour to keep the master from being hit on every Search. If the master is unreachable, new claims are denied (fail-closed) — search refuses with gRPC ResourceExhausted / HTTP 429 over MCP. Soft enforcement (set on the signed claim) admits new subjects past the cap but emits an audit row and surfaces an admin-UI banner.
• Max events per day — checked by the iceberg-sink on every batch commit. Past the cap, a warning is logged but ingestion continues.
• Max regions — checked at startup. Past the cap, the service refuses to start.
• Max tenants — checked by the webserver's tenant provisioner. Past the cap, provisioning a new tenant is refused.

Two operator-facing seat knobs live on the admin UI (and the underlying admin API):

Endpoint	Purpose
GET /api/admin/seats/exclude, PUT /api/admin/seats/exclude	Comma-separated glob exclude patterns (e.g. svc-*,bot-*,*@bots.example.com). Matched against the raw subject before hashing — matching subjects never enter the ledger and don't count against the cap.
GET /api/admin/seats/tenants, PUT /api/admin/seats/tenants/{tenant}, DELETE /api/admin/seats/tenants/{tenant}	Per-tenant seat caps. Carve the cluster cap into per-tenant allocations. Missing row = no per-tenant cap; only the cluster cap applies.
GET /api/admin/seats/usage	Per-tenant MAU breakdown for the admin tenants page.

For Community-tier deployments, the license ships with seat_enforcement: "soft" — the platform admits over-limit users with a banner. Enterprise / Regulated tiers default to "hard" and the contract terms specify what a sustained over-limit triggers (typically a true-up bill, not a service interruption).

Inspecting your license

To inspect what license a service is using, check its startup logs:

kubectl logs -n moolelo-system deploy/connector | grep license:
license: tier=enterprise expires=2027-12-31 in_grace=false features=[multi_tenant,multi_region]

The in_grace=true flag tells you whether the license is past expiry but inside the grace window.

Air-gap deployments

License keys work the same in air-gap deployments — the public key is embedded in every Moʻolelo binary at build time, and verification is offline. There is no online check-in, no SaaS dependency, no telemetry leak.

The only operational difference: when your renewal arrives, you have to manually transfer the file into the air-gap network through your existing secure channel. This is the same transfer path you already use for image mirrors, Helm chart bundles, and ML model weights. Plan a renewal touch-point ~30 days before each expires_at.

There is no automated rotation in air-gap setups, by design.

Troubleshooting

• Service crashes on startup with license: invalid. The key signature didn't verify. Either the file is corrupted (re-download), or the key was issued against a different signing root than your binary expects (you have a key from another deployment generation — contact your account team).
• Service crashes with no license source configured. Neither MOOLELO_LICENSE_PATH nor MOOLELO_LICENSE_URL is set. Pick one — authority mode (the default Helm shape) sets MOOLELO_LICENSE_URL; file mode sets MOOLELO_LICENSE_PATH.
• Service crashes with boot retries exhausted. Authority mode but the webserver's /api/internal/license is unreachable. Confirm the webserver pod is running, the in-cluster DNS resolves, and the NetworkPolicy egress rules permit TCP 8088 to the moolelo-system namespace.
• Webserver returns 503: webserver bootstrap mode. No license uploaded yet. Run the upload curl from the Installing a license key section.
• Webserver upload returns 401 unauthorized. The bootstrap token is missing, wrong, or the authority already flipped to active mode (where the token endpoint is rejected). If a license is genuinely needed mid-process, use the regular admin trust chain (post-boot) or restart the webserver to re-enter bootstrap mode.
• Service crashes with license: feature not licensed. Your deployment is configured to use a Commercial-only capability (multi-tenant, multi-region, vault-substrate) but the license doesn't carry that feature flag. Either upgrade the license tier or remove the configuration.
• Service starts but logs InGrace=true repeatedly. Your license is past expires_at. Renew within the next 7 days.
• Ingestion stopped suddenly. Past the 7-day grace window. Renew and restart the connector + iceberg-sink services.
• MCP / Search returns ResourceExhausted (HTTP 429) for new users. The MAU seat cap has been reached and seat_enforcement is "hard". Either upgrade to a larger cap, switch the claim to "soft", exclude the affected subject pattern (PUT /api/admin/seats/exclude), or wait for inactive subjects to fall out of the rolling window.
• MCP / Search returns ResourceExhausted with master unreachable in the coordinator log. The regional coordinator can't reach POST /api/internal/seat-claim on the master. This is the fail-closed posture — check master webserver health, NetworkPolicy egress, and QUERY_COORDINATOR_CONTROL_PLANE_URL on the coordinator. Existing subjects whose admit decision is still cached (1 hour TTL) keep working; only new subjects refuse.