Configuring secret providers

This page walks an operator through registering a vault provider so connector + identity-source credentials are sourced from there instead of pasted into the admin UI.

The conceptual model is in Secret management; this page is the operational setup.

Step 1 — set up workload identity (managed Kubernetes)

Pick the section matching your platform.

AKS Workload Identity

# 1. Create a User-Assigned Managed Identity (UAMI).
az identity create --name moolelo-data-plane --resource-group <rg> --location <region>

# 2. Grant the UAMI access to Key Vault secrets.
az keyvault set-policy --name <kv-name> --secret-permissions get list \
  --object-id $(az identity show --name moolelo-data-plane --resource-group <rg> --query principalId -o tsv)

# 3. Federate the AKS ServiceAccount → UAMI.
ISSUER=$(az aks show --name <cluster> --resource-group <rg> --query oidcIssuerProfile.issuerUrl -o tsv)
for chart in connector identity-collector acl-projector iceberg-sink; do
  az identity federated-credential create \
    --name moolelo-$chart \
    --identity-name moolelo-data-plane \
    --resource-group <rg> \
    --issuer $ISSUER \
    --subject "system:serviceaccount:moolelo-system:moolelo-$chart" \
    --audiences api://AzureADTokenExchange
done

# 4. Deploy each chart with the AKS overlay.
CLIENT_ID=$(az identity show --name moolelo-data-plane --resource-group <rg> --query clientId -o tsv)
helm install moolelo-connector ./deploy/helm/connector \
  -f ./deploy/values/base.yaml \
  -f ./deploy/values/with-aks-workload-identity.yaml \
  --set serviceAccount.annotations.azure\\.workload\\.identity/client-id=$CLIENT_ID

EKS IRSA

# 1. Associate an OIDC provider with the cluster (idempotent).
eksctl utils associate-iam-oidc-provider --cluster <cluster> --approve

# 2. Create the IAM role + policy mapped to the SA.
eksctl create iamserviceaccount \
  --name moolelo-connector \
  --namespace moolelo-system \
  --cluster <cluster> \
  --attach-policy-arn arn:aws:iam::aws:policy/SecretsManagerReadOnly \
  --override-existing-serviceaccounts --approve

# 3. Deploy with the EKS overlay, supplying the role ARN.
ROLE_ARN=$(aws iam get-role --role-name eksctl-<cluster>-addon-iamserviceaccount-moolelo-system-moolelo-connector --query Role.Arn --output text)
helm install moolelo-connector ./deploy/helm/connector \
  -f ./deploy/values/base.yaml \
  -f ./deploy/values/with-eks-irsa.yaml \
  --set serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn=$ROLE_ARN

GKE Workload Identity

# 1. Cluster-wide WI must be enabled on the cluster + node pool.

# 2. Create the GSA + grant Secret Manager access.
PROJECT=<project-id>
gcloud iam service-accounts create moolelo-data-plane --project=$PROJECT
gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:moolelo-data-plane@$PROJECT.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

# 3. Bind the KSA → GSA.
for chart in connector identity-collector acl-projector iceberg-sink; do
  gcloud iam service-accounts add-iam-policy-binding \
    moolelo-data-plane@$PROJECT.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT.svc.id.goog[moolelo-system/moolelo-$chart]"
done

# 4. Deploy with the GKE overlay.
helm install moolelo-connector ./deploy/helm/connector \
  -f ./deploy/values/base.yaml \
  -f ./deploy/values/with-gke-workload-identity.yaml \
  --set serviceAccount.annotations.iam\\.gke\\.io/gcp-service-account=moolelo-data-plane@$PROJECT.iam.gserviceaccount.com

HashiCorp Vault (Kubernetes auth, any cluster)

# 1. Enable the kubernetes auth method on Vault.
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# 2. Create a Vault policy + role bound to the data-plane ServiceAccounts.
vault policy write moolelo-data-plane - <<EOF
path "secret/data/connectors/*"  { capabilities = ["read"] }
path "secret/data/identity/*"    { capabilities = ["read"] }
EOF
vault write auth/kubernetes/role/moolelo-data-plane \
  bound_service_account_names="moolelo-connector,moolelo-identity-collector,moolelo-acl-projector,moolelo-iceberg-sink" \
  bound_service_account_namespaces="moolelo-system" \
  policies=moolelo-data-plane \
  ttl=1h

# 3. No Helm overlay needed — the standard ServiceAccount + the
#    pod's projected SA token (mounted by Kubernetes by default)
#    are enough. Just register the provider in the admin UI with
#    auth_mode=k8s + k8s_role=moolelo-data-plane.

Docker / Docker Swarm / unmanaged VMs

Workload identity isn't available; the provider's bootstrap credential comes from an env var:

• Vault: VAULT_TOKEN, or VAULT_APPROLE_ROLE_ID + VAULT_APPROLE_SECRET_ID
• Azure KV: AZURE_CLIENT_ID + AZURE_TENANT_ID + AZURE_CLIENT_SECRET (Service Principal)
• AWS: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY
• GCP: GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json (or set the inline JSON)

Set these on the data-plane containers via your usual secret-distribution mechanism (Docker secrets, Compose env_file, Swarm config). The provider configs in the admin UI reference the env var names, never the values — so the Postgres row that gets distributed via the control plane stays free of literal secrets.

Step 2 — register the provider in the admin UI

1. Open /secret-providers and click New provider.
2. Pick the backend (Vault / Azure KV / AWS Secrets Manager / GCP Secret Manager).
3. Fill in the endpoint fields. For Vault: address (URL) + auth_mode (auto is fine in most cases) + k8s_role if using K8s auth. For cloud backends: region (AWS) / vault_name (Azure) / project_id (GCP).
4. Save. The provider row is written to Postgres and distributed via /api/internal/secret-config within seconds. Every data-plane service that's already running rebuilds its in-memory Registry on the next refresh tick (default 5 min) — or you can roll the pods if you want immediate effect.

Step 3 — switch a connector field from literal to vault

1. Open the connector edit page (/connectors/<tenant>/<source>).
2. For each password-typed field (OAuth client secret, refresh token, PAT) toggle the 🔒 vault ref chip next to the label.
3. Pick the provider from the dropdown and paste the path (e.g. secret/data/connectors/github#token for Vault KV-v2 with a JSON-shaped value).
4. Save. The row is upserted in Postgres; the connector picks up the change via LISTEN/NOTIFY within seconds.

The next time the connector calls the source system it resolves the reference through the provider, hits the source with the resolved value, and caches the resolved value for the configured cache TTL. Rotate the secret in the vault; within 5 minutes the next API call uses the new value.

Verifying

# 1. Confirm the provider row.
kubectl -n moolelo-system port-forward svc/moolelo-webserver 8088:8088 &
curl -sS http://localhost:8088/api/internal/secret-config | jq .

# 2. Confirm the data-plane service is consuming it.
kubectl -n moolelo-system logs deploy/moolelo-connector | grep -i secretclient
# Expect: "secretclient: refresh ok (fingerprint=...)" or similar.

# 3. Run a backfill cycle and verify the source-system auth works.
#    Failures will surface as "resolve oauth.refresh_token" or
#    "AzureCredential.GetToken: ManagedIdentityCredential authentication failed" in the connector logs.

Troubleshooting

Symptom	Likely cause
secrets: unknown provider "vault-prod" in connector logs	The reference uses a provider id that's not in /secret-providers. Either add the provider or fix the reference.
azidentity: ManagedIdentityCredential authentication failed	The pod's UAMI / federated credential isn't set up. Check azure.workload.identity/client-id annotation + the federated-credential subject matches system:serviceaccount:<ns>:<sa-name>.
IRSA: WebIdentityTokenFile env var not set	The pod's ServiceAccount isn't IRSA-annotated. Re-run eksctl create iamserviceaccount and roll the Deployment.
google: could not find default credentials	GKE Workload Identity binding missing. Verify the iam.gke.io/gcp-service-account annotation + the GSA → KSA binding.
vault: 403 Forbidden	The Vault role doesn't grant access to the requested path. Check the policy attached to auth/kubernetes/role/<role>.
Connector continues using the old secret value after rotation	Within cache TTL (default 5 min). Either wait or roll the Deployment.