MIT research widely reported in 2025 found roughly 95% of generative AI pilots fail to scale into production impact. That number gets quoted a lot, usually to argue AI is overhyped. I read it differently: most of those pilots did not fail because the model was bad. They failed because nobody built the operational layer around it.
LLMOps is that layer. It is less glamorous than agent design and it is the actual difference between a demo and a product.
What is genuinely different from normal DevOps
Most DevOps practice transfers directly. Four things do not:
- Non-determinism. The same input can produce different output. Your test strategy cannot rely on exact-match assertions, which is why evaluation replaces conventional unit testing for the model-facing parts.
- The dependency changes underneath you. A provider can update a model behind the same API name. Your code did not change; your behaviour did. Pin model versions explicitly where the provider allows it, and treat an unpinned model as an unpinned dependency.
- Usage-based cost that scales with success. Traditional infrastructure cost is broadly predictable. Token cost scales with usage and can 10x from a retry loop shipped on a Friday.
- Prompts are deployable artefacts. They change behaviour, they need versioning, and they need rollback — but they are not compiled, so nothing forces you to treat them properly.
The deployment stack in practice
FastAPI is the default serving layer for Python AI services, and for good reasons: native async (essential, since you are I/O-bound on API calls), automatic OpenAPI docs, Pydantic validation built in, and straightforward streaming support for token-by-token responses.
Docker packages it with pinned dependencies. This single practice eliminates the largest category of "worked locally, failed in staging" incidents.
Where it runs matters less than teams think. Cloud Run, ECS, App Runner, a Kubernetes cluster you already operate — all fine. Choose based on what your team can operate at 2am, not on architectural elegance. For Malaysian organisations, region selection is often the binding constraint: PDPA and, for financial institutions, BNM RMiT expectations make "which region processes this data" a compliance question rather than a latency optimisation.
Versioning: four things that must be pinned
When behaviour changes, you need to know which of these moved:
- Model version — pinned explicitly, not "latest".
- Prompt version — in Git, tagged, and recorded on every trace.
- Retrieval index version — re-embedding a corpus with a different model changes answers even with an identical prompt.
- Application code version — the ordinary one.
If your traces record all four, then "quality dropped on Tuesday" is a five-minute investigation. If they record none, it is a week of guessing.
CI/CD when tests are probabilistic
A workable pipeline shape:
- Lint and unit tests for deterministic code paths — parsing, routing, validation, tool wrappers.
- Eval suite as a quality gate — run your dataset, post scores on the pull request, block the merge on regression past a threshold. This is where prompt changes get the same scrutiny as code changes.
- Canary deploy — route a small traffic percentage to the new version, compare online metrics.
- Automated rollback — triggered on error-rate or quality-score regression, not on someone noticing.
Step two is the one that separates teams who ship confidently from teams who ship nervously.
Cost control before it becomes a board conversation
- Attribute cost per request, per feature, per tenant. Aggregate monthly spend tells you that you have a problem, not where it is.
- Set hard caps. Per-user, per-tenant, per-day. A runaway loop should hit a ceiling, not an invoice.
- Cache aggressively. Prompt caching on repeated context, and semantic caching for repeated questions, both cut real spend. In support workloads the same twenty questions dominate the distribution.
- Right-size the model per task. Classification and routing rarely need your most capable model. Reserve it for the reasoning that justifies it.
- Alert on trend, not just threshold. A 40% week-on-week rise matters even if you are still under budget.
What to page someone about
Distinguish alerts that need a human at 2am from dashboards someone reviews on Monday.
Page: error rate above threshold, p95 latency past SLA, cost spike beyond a hard cap, guardrail block rate spiking (usually an attack or a broken upstream input), provider outage.
Dashboard: quality score trends, token usage patterns, cost per feature, refusal and escalation rates, retrieval relevance drift.
Teams that page on quality scores burn out fast — those numbers are noisy by nature. Page on availability and cost; review quality on a cadence.
The minimum viable production setup
If you are moving your first system to production this quarter: FastAPI in Docker with pinned dependencies, prompts versioned in Git and stamped on every trace, an eval suite gating pull requests, OpenTelemetry tracing to a self-hosted collector, hard cost caps with alerting, guardrails on input and output, and a tested kill switch operations can flip without an engineer.
That is perhaps two weeks of work for someone who has done it before. It is also the difference between joining the 95% and joining the 5%.
Our AI Engineering programme covers deployment, evaluation, observability and cost control for production AI — HRD Corp SBL-KHAS claimable for eligible Malaysian employers.
