LLMOps: run LLMs in production without chaos

LLMOps: production for token systems

LLMOps covers the practices used to build, deploy, observe and improve LLM-based applications. In practice, it means fewer "AI strategy" slides and more replayable pipelines: versioned prompts, evals, logs, budgets and incident handling.

The difference with classic tabular MLOps comes down to the object. The output is language, so it is fuzzy. The cost is often counted per token. External dependencies (model APIs, tools, MCP) move outside your control. And the "dataset" is sometimes a living document corpus.

Without LLMOps, every dev keeps a prompt in a pad, the surprise bill lands at the end of the month, and nobody can explain why quality dropped on Tuesday.

Eval: the only compass

To steer an LLM system, you need three complementary evaluation layers:

1. Offline unit tests: gold cases and deterministic assertions on the JSON or the structure
2. Bounded LLM judges, with healthy distrust and a human-reviewed sample
3. Online monitoring: thumbs, human edits and incidents

Start small, with 30 critical business cases, and block the merge as soon as regression exceeds a threshold. A model that is "better on a public bench" but breaks your 30 cases is not an upgrade.

For a RAG, separate the retrieval eval from the generation eval. For an agent, evaluate the full trajectory (the tools it called), not only the final text.

Versioning: prompts, models, tools

Everything that changes the system's behavior deserves a versioned identifier:

• The system prompt and the skills
• Parameters such as temperature
• The model, pinned to a precise snapshot
• The tools and MCP servers
• The RAG corpus

Log those identifiers on every request. And ban "we changed the prompt in prod without a ticket": a prompt follows the same discipline as code, with a PR, a review and a possible rollback.

Feature flags complete the setup: you enable a new prompt on 5% of traffic, or on an internal workspace, before going to 100%.

Observability and cost

Traces are the base of debugging. Record retrieve, generation and tool latency, tokens in and out, provider errors and retries, with OpenTelemetry when you can. Without traces, debugging an agent amounts to reading logs at random.

The budget needs steering too: set a ceiling per team, per project and per day, with alerts and dashboards. Multi-agents and ultra modes multiply context windows, so cost is no longer linear with "one chat".

On online quality, track the human edit rate, the rejects and the AI-related support tickets. Those are your real KPIs, not the vendor leaderboard.

Guardrails and deployment

Guardrails cover most of the risk:

• Schema validation on outputs
• PII filters
• A tool allowlist
• Human review (HITL) on risky actions
• A sandbox for execution

For supervision design, lean on the human-in-the-loop guide. On the deployment side, separate environments (dev, staging, prod), keep secrets out of the code, and set timeouts and circuit breakers to absorb provider lags. An agent without a timeout is an open bill.

Finally, prepare two incident runbooks: one for a "quality drop" (last change, logs, prompt or model rollback) and one for a "cost spike" (top routes, multi-agent, tool loops).

Minimal LLMOps for an SME (checklist)

The base fits in seven points:

1. One owner is named for the AI system
2. 30 eval cases run in CI
3. Prompts and models are pinned
4. Logs and traces cover 100% of prod traffic
5. A token budget exists, with an alert
6. Human review protects risky actions
7. A weekly 30-minute review tracks quality and cost

An enterprise LLMOps platform can wait. What matters on day 1 is writing these seven points down and keeping them.

If you want to set an LLMOps base on a real agent or RAG, we can scope it together in 20-40 minutes.