Platform Engineering for LLM Applications
Open Source For You, Tuesday, October 13th, 2026
LLM features break platform assumptions on quality, cost and trust; a model gateway is the one place to enforce controls.
Drawing on consulting work, the author describes a logistics firm whose AI spend jumped from Rs 42,000 to Rs 1.1 million in two months because an agent looped without a retry ceiling, which went unnoticed until tracing was added.
Most of the platform (Kubernetes, ArgoCD, Prometheus) stays the same, but three things fail silently.
Quality fails because a well-formed 200 response can carry stale answers, cost shows up a month late on the bill, and trust erodes because prompts mix your instructions with untrusted text.
The fix is a model gateway such as LiteLLM in front of every model call to enforce budgets, routing and observability.