What are the core components of a production LLMOps stack?

Six components tend to show up in every mature LLMOps setup, each solving a distinct failure mode. A prompt registry centralizes templates, their versions, and their evaluation history so nobody is editing prompts in an untracked scratch file. A gateway sits as the single entry point for all LLM calls, handling routing across providers, rate limiting, authentication, and cost tracking in one place. An evaluation pipeline runs automatically on every prompt or model change rather than relying on someone eyeballing a few outputs. An observability layer traces every call with its full prompt, response, latency, and cost. A guardrail system filters both input and output for PII, injection attempts, and policy violations. And a cost-control layer — routing, caching, prompt compression — exists specifically because without it, LLM bills can multiply overnight with no single obvious cause.

The gateway is worth calling out as the control plane of the whole stack: it’s the one place that sees every request, which makes it the natural place to enforce rate limits, track spend, and fail over to a backup provider — without it, there’s effectively no centralized visibility into how the system is actually being used.