Choosing a deployment topology

The parser and embedder services — the parts that turn documents into searchable chunks — can run in three places. The rest of Moʻolelo is the same in all three.

This page is for operators evaluating which topology fits their organization. If you've already picked one, see Install.

At a glance

Topology	Where the parser runs	Where the embedder runs	When it's the right choice
In-cluster, no Ray	A small CPU-only Kubernetes Deployment	A small CPU-only Kubernetes Deployment	Small to medium tenants. No Ray cluster, no GPU pool, no operator dependencies.
In-cluster with KubeRay + KServe	KubeRay-managed Ray Serve	KServe InferenceService (vLLM, on GPUs)	Mid- to large-size tenants who already operate KubeRay + KServe and want maximum throughput
External	Ray cluster on VMs (Azure VMSS, EC2 fleet)	KServe in another cluster, or self-hosted vLLM on GPU VMs	Enterprises with split-stack deployments — search in one cluster, ML in another (or on-prem ML, cloud search)

The Moʻolelo dispatcher binary is identical in all three. Only the URLs and the embedder protocol change.

In-cluster, no Ray

Two small CPU-only Deployments live alongside the rest of Moʻolelo. The embedder uses sentence-transformers with the BAAI/bge-m3 model on CPU. The parser runs Docling and tree-sitter inline.

Throughput: in the range of 10–20 chunks per second per embedder replica. Scales linearly with replicas.

Operational cost: the simplest of the three — no operator installs, no GPU node pool, no cross-cluster networking. Two Deployments, two Services, two NetworkPolicies. Helm install is one command.

When to pick this:

• You're onboarding your first tenant and want the smallest possible footprint.
• You don't operate a GPU pool and don't want to start now.
• Your corpus is in the low millions of chunks and your query latency requirements are not sub-second.
• You're running an air-gap deployment where adding KServe + vLLM weights to the air-gap mirror is expensive.

helm install ml deploy/helm/ml-tier1-embedded --namespace moolelo-system

The dispatcher chart's defaults already point at the in-cluster service names this chart creates — no overrides needed.

In-cluster with KubeRay + KServe

The parser runs as a Ray Serve deployment managed by KubeRay. The embedder runs as a KServe InferenceService backed by vLLM on GPUs.

Throughput: typically 100+ chunks per second per embedder replica with vLLM on a single GPU. Ray Serve handles request batching automatically.

Operational cost: requires the KubeRay operator and the KServe operator installed cluster-wide (or in a dedicated namespace they manage). Both are well-established CNCF/LF AI&Data projects with active operator communities.

When to pick this:

• You already operate Ray and/or KServe for other ML workloads.
• You have GPU nodes available and want to amortize the cost across the search workload.
• Your corpus is in the tens of millions of chunks and you need throughput more than simplicity.
• You want autoscaling beyond what KEDA can do — Ray Serve's autoscaler handles per-deployment metrics that KEDA can't see.

# Pre-requisites (one-time, cluster-wide)
helm install kuberay-operator kuberay/kuberay-operator --namespace kuberay-system --create-namespace
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.14.0/kserve.yaml

# Install the topology
helm install ml deploy/helm/ml-tier2-kuberay \
  --namespace moolelo-ml --create-namespace \
  --set parser.image=<your-registry>/moolelo-ml-ray:0.1.0 \
  --set embedder.image=<your-registry>/moolelo-ml-kserve:0.1.0

# Update the dispatcher to talk to the new endpoints
helm upgrade entity-extractor-dispatcher \
  deploy/helm/entity-extractor-dispatcher \
  --namespace moolelo-system \
  --set parser.url=http://moolelo-parser-serve-svc.moolelo-ml:8000 \
  --set embedder.url=http://bge-m3-predictor.moolelo-ml:80 \
  --set embedder.protocol=kserve-v2 \
  --set embedder.modelID=bge-m3

External

The ML services run somewhere outside your search cluster. Common patterns:

• A dedicated Ray cluster on Azure VM Scale Sets, deployed via Bicep / Terraform / Ansible.
• An EC2 GPU fleet running vLLM behind an internal load balancer.
• An existing AKS cluster used for other ML, with KServe already configured.

The ml-tier3-external chart installs nothing in the search cluster — it just records the agreed URLs in a ConfigMap so multiple operators can see the same source of truth.

Throughput: depends entirely on your external deployment. The Moʻolelo dispatcher batches requests; the upstream limits are the model server's.

Operational cost: zero in the search cluster, but you're responsible for the external Ray/KServe deployment's day-2. If your organization already has a "ML platform team" running Ray clusters for other purposes, this is the topology that lets them do the same for Moʻolelo.

When to pick this:

• Your security model requires search and ML to live in separate clusters or even separate networks.
• An ML platform team in your org already operates Ray and wants to consolidate.
• You're running search on-prem but the ML stack lives in cloud (or vice versa).
• You want to share a single embedding model server across multiple Moʻolelo deployments.

helm install ml deploy/helm/ml-tier3-external --namespace moolelo-system \
  --set parser.url=https://parser.ml.your-domain.com \
  --set embedder.url=https://bge-m3.ml.your-domain.com \
  --set embedder.modelID=bge-m3

Switching topologies later

You can switch topologies without re-ingesting your corpus, with one important caveat: if the new embedder produces different embeddings (different model, different normalization, different version), the existing search index becomes incompatible.

A safe topology change:

1. Stand up the new ML topology alongside the old one.
2. Trigger a backfill on every connector — this re-embeds every record using the new topology.
3. Once Vespa shows the new chunks (you can confirm via the chunk's embedder_model metadata), retire the old topology.

If you're staying on the same model and embedder backend (just changing where it runs), no re-embed is needed.

Hardware sizing

For the in-cluster, no-Ray topology:

Tenant size	Parser CPU	Embedder CPU	Embedder RAM
< 100k records	0.5 cores	1 core	2 GB
100k – 1M records	1 core	2 cores	4 GB
1M – 10M records	2 cores per replica, scale horizontally	2 cores per replica, scale horizontally	4 GB per replica
> 10M records	Consider moving to KubeRay + KServe topology

For the KubeRay + KServe topology, single A10 / A100 GPU per embedder replica is typical; Ray Serve handles batching for you.

For the external topology, sizing is whatever your external deployment provides — the Moʻolelo dispatcher will fan out parallel requests up to DISPATCHER_MAX_CHUNKS_PER_EMBED per call.