Secret management
Moʻolelo treats every sensitive configuration value — connector OAuth secrets, refresh tokens, GitHub PATs, identity-source credentials — as a reference to a value held in an external key vault, not as a literal stored in Postgres or pasted into the admin UI.
The platform supports four vault backends out of the box. Operators register one or more secret providers through the admin UI; every reference string then resolves through them at use site, with a small in-process cache so repeated lookups don't hammer the vault.
Supported backends
| Backend | Auth chain |
|---|---|
| HashiCorp Vault (KV v1 + v2) | Kubernetes SA auth → AppRole → token (auto-detected) |
| Azure Key Vault | DefaultAzureCredential (Workload Identity → Managed Identity → env SP → az CLI) |
| AWS Secrets Manager | AWS SDK default chain (IRSA → instance profile → env → shared file) |
| Google Cloud Secret Manager | Application Default Credentials (Workload Identity → SA key → env) |
Every backend uses its cloud SDK's default credential chain. On managed Kubernetes — AKS, EKS, GKE, OpenShift on a public cloud — pods authenticate via workload identity with zero credential in the environment. The pod's identity is the bootstrap credential.
On pure Docker, Docker Swarm, and unmanaged VMs the operator falls back to env-var credentials. That's the only "bootstrap secret" required — and it's the platform's only chicken-and-egg secret regardless of how many connector + identity providers are configured.
Reference format
<provider-id>://<path>[#field]Examples:
vault-prod://secret/data/connectors/github#token
azkv-prod://moolelo-secrets/confluence-refresh-token
aws-prod://prod/moolelo/sharepoint-secret#client_secret
gcp-prod://projects/moolelo/secrets/sharepoint-secret/versions/latestThe <provider-id> is the id the operator assigned when registering the provider in the admin UI. The <path> is backend-specific (Vault KV path; Azure secret name [optionally with /<version>]; AWS SecretId or ARN; GCP short-id or full resource name). The optional #field selects one key when the stored value is JSON-shaped — useful for storing OAuth bundles ({"client_id": "...", "client_secret": "...", "refresh_token": "..."}) as a single secret.
Literal vs reference auto-detection
Every sensitive field auto-detects whether you supplied a literal value or a vault reference. Backwards compat is preserved — existing literal credentials continue to work unchanged.
The disambiguation rule is strict to avoid false positives:
A value is treated as a reference only when both of the following hold:
- It has the syntactic shape
<scheme>://<path>where<scheme>is lowercase alphanumerics +-/_. - The
<scheme>is a currently-registered provider id in/secret-providers.
This means plain URLs like postgres://user:pw@db/moolelo, https://login.microsoftonline.com/…, or pulsar://broker:6650 pass through as literals even though they technically match the syntactic shape — because their scheme isn't a registered provider id. You can paste these into env vars or admin-UI fields and they work exactly as before.
Coverage
✓ = supports both literal and vault reference shapes.
| Sensitive field | Resolution |
|---|---|
| Connector OAuth client_secret / refresh_token / PATs (GitHub, Confluence, Jira, SharePoint) | ✓ |
| Identity-collector OAuth + tokens (Atlassian, Entra, GitHub) | ✓ |
Webhook signing secrets (WEBHOOK_INGRESS_*_SECRET) | ✓ at boot (rotation requires pod restart) |
Webserver admin session signer (MOOLELO_ADMIN_SESSION_SECRET) | ✓ at boot |
OIDC client secret (MOOLELO_OIDC_CLIENT_SECRET) | ✓ at boot |
OpenFGA pre-shared token (WEBSERVER_OPENFGA_TOKEN, QUERY_COORDINATOR_OPENFGA_TOKEN, ACL_PROJECTOR_OPENFGA_TOKEN) | ✓ at boot |
NebulaGraph passwords (QUERY_COORDINATOR_NEBULAGRAPH_PASSWORD, …_SUPERSESSION_NEBULAGRAPH_PASSWORD) | ✓ at boot |
Postgres DSN (WEBSERVER_POSTGRES_DSN, CONNECTOR_PG_DSN, IDENTITY_COLLECTOR_PG_DSN, ACL_PROJECTOR_PG_DSN, QUERY_COORDINATOR_SUPERSESSION_POSTGRES_URL) | ✓ at boot |
For each of these, paste either:
- A literal:
postgres://moolelo:hunter2@db/moolelo,ghp_xxx…, etc. - A reference:
vault-prod://secret/data/db-creds#dsn,azkv-prod://moolelo-secrets/db-dsn, etc.
Env vars that MUST stay literal (the bootstrap floor)
Even when every other secret is sourced from a vault, a small set of env vars must remain literal. These are the bootstrap credentials needed BEFORE the secret-provider machinery is reachable; making them reference-able would be a chicken-and-egg.
| Env var | Service | Why it can't be a reference |
|---|---|---|
MOOLELO_ADMIN_BOOTSTRAP_TOKEN | webserver | Gates the admin surface (including /secret-providers) BEFORE any provider can be configured. The operator needs this to log in the first time and add the first provider. |
WEBSERVER_POSTGRES_DSN | webserver | The master reads secret_providers rows from its OWN Postgres. The DSN therefore must be literal — the resolver can't help fetch the DSN before Postgres is connected. (Other services' DSNs reach the master over the control plane, so they CAN be references.) |
MOOLELO_LICENSE_SOURCE | webserver | Defines where the license envelope lives (Postgres / Vault / Azure KV / file). The resolver can't read it before the license is verified at boot, so the source URL is consumed before the resolver is constructed. |
MOOLELO_CONTROL_PLANE_URL | every data-plane service | Tells the binary where the master is. It IS the bootstrap pointer — every other resolution depends on it. |
Pulsar / Vespa / object-storage endpoints (*_PULSAR_URL, *_VESPA_BASE_URL, S3 endpoints) | various | These are non-secret endpoint URLs. Auth to these services is via mTLS / IAM / workload identity (set out-of-band via Helm values), not via the URL string. |
Cloud-provider bootstrap creds for the secret-provider's OWN auth (e.g. AZURE_CLIENT_ID, AWS_ACCESS_KEY_ID, VAULT_TOKEN) | data-plane pods, only when workload identity isn't available | The one ‘seed' secret that gets the provider talking to its backend. On managed-K8s with workload identity these aren't needed. |
That's it. Everything else — every OAuth secret, every refresh token, every PAT, every database password, every signing key — can be a vault reference on top of the master-Postgres + bootstrap-token + control-plane-URL floor.
Provider's own auth — workload identity vs env-var
The provider's own credential (the one it uses to talk to Vault / KV / Secrets Manager / Secret Manager) is the only secret that can't be sourced from itself. The platform handles this by leaning on cloud workload identity wherever it exists:
| Deployment | HashiCorp Vault | Azure Key Vault | AWS Secrets Manager | GCP Secret Manager |
|---|---|---|---|---|
| AKS | K8s auth method | AKS Workload Identity | OIDC federation (rare) | OIDC federation (rare) |
| EKS | K8s auth or AWS IAM auth | OIDC federation (rare) | EKS IRSA | OIDC federation (rare) |
| GKE | K8s auth method | OIDC federation (rare) | OIDC federation (rare) | GKE Workload Identity |
| OpenShift on cloud | K8s auth method | Workload Identity via OIDC | IRSA via OIDC | Workload Identity via OIDC |
| Managed container hosts (ACI / ECS / Cloud Run / App Runner) | AppRole / token | Managed Identity | Instance / Task role | Runtime SA |
| Docker / Swarm / unmanaged VMs | AppRole or token (env) | Service Principal (env) | Access key / secret (env) | SA-key JSON file or env |
The Helm overlays in deploy/values/with-aks-workload-identity.yaml, with-eks-irsa.yaml, and with-gke-workload-identity.yaml annotate the data-plane pods' ServiceAccounts with the right cloud-specific labels + annotations. See Configuring secret providers for the full setup walk-through.
Caching + rotation
Every resolved value is cached in-process for 5 minutes by default (secrets.DefaultCacheTTL). Rotations propagate within 5 minutes without a service restart.
Provider-config changes (a new provider added, an endpoint URL changed) propagate within 5 minutes as well — every data-plane service polls /api/internal/secret-config on a refresh ticker. When the wire payload's fingerprint changes the in-memory Registry is rebuilt atomically; in-flight requests using the old Registry finish unaffected.
For faster rotation: lower MOOLELO_SECRETS_CACHE_TTL (env var). At the trade-off of more frequent vault calls.
For immediate rotation: rolling-restart the affected data-plane Deployments. The Registry is rebuilt from scratch + the cache is empty, so the next request resolves through the vault.
Audit + observability
Every Resolve call records a structured log line with provider_id, path, and outcome (cached / fetched / not-found / unreachable). The resolved value is never logged. Operators can index these in Loki and alert on resolve-failure rates per provider to catch outages or misconfigurations before they propagate to user-facing failures.
Failure semantics
| Condition | Outcome |
|---|---|
Reference uses an unknown provider-id | Resolve returns ErrUnknownProvider; the connector / identity backfill fails at the source-system auth call with a clear "credentials not resolvable" message |
| Provider is unreachable (network / IAM denied) | Resolve returns the provider's error wrapped; same downstream surface |
| Reference is malformed | ErrInvalidReference at decode time |
Field selector #field not present in resolved secret | ErrFieldMissing |
Plain literal (no :// prefix) | Pass-through unchanged — backwards-compat with existing configs |
The fail-modes are deliberately fail-closed at use site: a misconfigured reference will not silently allow a connector to start with bogus credentials. The source system rejects the call, the connector logs the resolver chain, and the operator fixes it from the admin UI.