Architecture Decision Record
ADR 014: vLLM for Served Generation, Alongside Ollama
A second inference runtime for throughput — vLLM serving a quantized 30B tensor-parallel across both GPUs, with Ollama kept for embeddings and flexible experimentation.
Status: Accepted — rollout in progress · Date: Aug 2026 · ← All ADRs
Context
ADR 011 established local inference on Ollama, and that was the right call to start: trivial model management, one-line model swaps, and more than good enough for interactive chat and document Q&A.
Then the workloads changed shape. Agentic loops and — especially — Cortexa’s evaluation harness aren’t latency-bound the way a single chat is; they’re throughput-bound. A full eval run is ~150 sequential model calls, all serialized on one GPU, and Ollama serves essentially one request at a time. Two things I wanted, Ollama doesn’t do well:
- Concurrency — many in-flight requests batched together, not queued one behind another.
- Both GPUs on one model — tensor parallelism across the pair of RTX 3060s, rather than one card doing the work.
Decision
Run vLLM as the served-generation runtime, and keep Ollama for everything else — a split by job, not a straight replacement.
- vLLM (
vllm/vllm-openai:v0.26.0) servesQwen3-Coder-30B-A3B-Instruct-AWQ(asqwen3-coder-30b), tensor-parallel across both RTX 3060s (--tensor-parallel-size=2), AWQ-quantized to fit a 30B MoE in 24 GB. OpenAI-compatible API atvllm-service.vllm.svc.cluster.local:8000, externally atvllm.private.gawarecki.us. - Ollama stays, split by role. A CPU-only
ollama-cpunow serves embeddings (nomic-embed-text); the GPU Ollama deployment is scaled to zero and brought up only for bulk re-embeds. Because vLLM and GPU-Ollama each claim both cards and GPU time-slicing was rejected, only one GPU generation workload runs at a time — the two runtimes divide the work, they don’t share the silicon.
Reasoning
- Throughput via continuous batching. vLLM’s PagedAttention +
--max-num-seqs=16keeps the GPUs busy across concurrent requests instead of idling between serialized calls — directly aimed at the eval-harness and multi-agent workloads. - Full GPU utilization.
--tensor-parallel-size=2puts both 3060s on one model; AWQ quantization is what makes a 30B fit at all on 24 GB. - OpenAI-compatible API. A drop-in
/v1surface that standard SDKs and clients expect. The external gateway is provisioned specifically so Cortexa can reach it. - Ollama earned its keep for the rest. Embeddings and quick model experimentation want Ollama’s flexibility; generation wants vLLM’s throughput. Keeping both is cheaper than pretending one tool is best at both.
Tradeoffs
- Less model flexibility — which is why Ollama stays. vLLM is effectively one model per server, with a heavy startup (torch.compile, a ~20 GB first-run download, ~90 s init). Swapping models is nothing like Ollama’s one-liner. The coexistence isn’t hedging; it’s using each tool where it’s strong.
- The two can’t share the GPUs. Both want both cards, and time-slicing was rejected, so GPU generation is vLLM or GPU-Ollama — never both at once. Embeddings had to move to CPU to free the cards for vLLM.
- More knobs, and sharp ones.
--gpu-memory-utilization=0.93(0.90 OOMed during CUDA-graph capture on 12 GB cards),NCCL_P2P_DISABLE=1(no NVLink on consumer 3060s), and a 1 GiB/dev/shmbecause the 64 MB default breaks NCCL all-reduce. Ollama needed none of this. - Tuning advice is hardware-specific, and mine disagreed with the internet. I enabled n-gram speculative decoding — a widely recommended vLLM throughput win — and measured it making generation 3.6× slower on this rig, so I reverted it. Speculative decoding trades extra compute for fewer sequential steps; on two mid-range consumer cards already saturated by tensor parallelism, that trade is backwards. A reminder that “known good” settings are known good on someone else’s hardware.
- Weights on hostPath, not the SAN. Talos
/var/mntis read-only and the Ollama UserVolume was too small, so model weights live on ahostPathat/var/lib/vllm-models(~96 GiB ephemeral on the node). A node rebuild re-downloads ~20 GB — a deliberate, documented departure from the platform’s usual PVC-on-Synology pattern (ADR 005). - No egress policy yet. The
vllmnamespace has no Cilium lane (ADR 008). It needs outbound access to pull weights from HuggingFace on first run, but “no policy” means unrestricted egress — the right fix is a Lane-B FQDN allowlist for the HF endpoints, and it isn’t in place.
Outcome
The layered stack this runs on is drawn in the GPU inference stack diagram.
vLLM is live and serving qwen3-coder-30b across both cards. But this is honestly recorded as a migration in progress (Aug 2026), not a finished cutover. The generation consumers — Open WebUI, paperless-ai, karakeep — still point at Ollama endpoints and haven’t been repointed to vLLM’s OpenAI API, so some currently have no generation backend during the transition. The external gateway is wired for Cortexa; the internal repointing (and the egress lane) is the open work.
I’m documenting it mid-flight on purpose. The tidy version — “migrated to vLLM” — would be a cleaner sentence and a less honest one. This is what the change actually looks like a few days in.