kvfleet
A KV-cache-aware routing control plane for self-hosted and hybrid LLM fleets, built around explainable policy, locality, health, and fallback decisions.
01
Problem
LLM routing is often reduced to choosing a model, while production fleets also need to preserve cache locality, respect tenant and compliance constraints, react to endpoint health, and recover without hiding why a route changed.
Constraints that shaped the design
- Remain useful as both an embedded Python library and an OpenAI-compatible gateway.
- Keep route decisions inspectable instead of hiding them behind a single score.
- Balance CPU-side decision cost against much larger GPU-side recomputation cost.
02
Architecture
- 01
Normalize and fingerprint
Build stable request, conversation, and prefix signals from incoming work.
- 02
Filter and rank
Apply policy and health gates before ranking eligible endpoints across multiple objectives.
- 03
Execute and explain
Route with fallback behavior while exposing the factors behind the decision.
03
Design decisions
- Keep conversation affinity and prefix affinity as separate signals.
- Apply hard policy constraints before soft ranking so an attractive score cannot bypass eligibility.
- Offer gateway mode to reduce migration cost for OpenAI-compatible clients.
04
Alternatives considered
- A model-only router that ignores replicas, endpoint locality, and fleet health.
- A single opaque routing score with no inspectable decision factors.
- Assuming process-local route memory is sufficient for every horizontally scaled deployment.
05
Testing and results
- The public package and repository expose the implementation and release history.
- The technical article walks through the routing pipeline, gateway mode, design trade-offs, and current limits.
- The article explicitly identifies process-local affinity and prefix memory as a scaling boundary.
Outcome
kvfleet packages cache-aware, policy-aware routing into a reusable control plane and pairs the code with a long-form architectural explanation.
06
Limitations
- Route memory is currently process-local unless the deployer supplies a shared design.
- Horizontal deployments still need deliberate ingress locality or shared affinity state.
07
Lessons learned
- The best fleet route is constrained by locality, policy, health, economics, and recovery at the same time.
- A router becomes easier to operate when every decision has a legible explanation.