Here is a pattern I see in almost every Malaysian team that has shipped an AI feature. Someone tweaks a system prompt on Tuesday. They paste in three test inputs, read the outputs, decide it "feels better," and merge. On Friday, support tickets spike for a case nobody thought to test.
The gap between those two moments is where evaluation lives. And in 2026, with agentic systems making multi-step decisions on live customer data, that gap is no longer a quality problem — it is a business risk.
Evaluation is not benchmarking
First, a distinction that trips up a lot of teams. Benchmarks (MMLU, GPQA, SWE-bench) tell you how a model performs on standardised academic tasks. They are useful when choosing a base model. They tell you almost nothing about whether your application works.
Evals are task-specific tests built on your data, measuring your success criteria. Anthropic's own guidance on defining success criteria is blunt about this: start by specifying what "good" means in measurable terms before you write a single test. Not "accurate responses" — something like "correctly extracts the invoice total in 98% of cases, and never fabricates a number that is not in the source document."
That second clause matters more than the first. In finance and healthcare workflows, a confident wrong answer costs far more than an "I don't know."
The three places evaluation runs
A mature setup runs evals at three distinct points, and they answer different questions.
1. Offline evals — against a curated dataset
A fixed set of inputs with known-good outputs. You run this whenever you change a prompt, swap a model, or adjust retrieval. It answers: did this change make things better or worse on cases we already understand?
The hard part is not the tooling — it is building the dataset. Start with 50–100 examples drawn from real production traffic, not invented ones. Include the awkward cases: the ambiguous invoice, the customer who writes in mixed Malay and English, the document with a scanning artefact. A clean dataset of easy cases will pass every test and catch nothing.
2. CI evals — before the merge
The same dataset, wired into your pull-request pipeline as a quality gate. Prompt changes get reviewed like code changes: a score summary posted on the PR, and a merge blocked if a key metric regresses past a threshold.
This is the single highest-leverage practice on this list, and the one most teams skip. It converts evaluation from something a diligent engineer remembers to do into something the pipeline enforces.
3. Online evals — against live traffic
Sampling real production requests and scoring them continuously. This catches drift that offline evals structurally cannot: new user behaviours, seasonal patterns, an upstream document format that quietly changed.
You cannot have ground-truth labels for live traffic, so online evals lean on reference-free metrics — groundedness, faithfulness to retrieved context, refusal rate, latency, cost per resolved request — plus a sampled human review queue.
Which metrics are worth tracking
Metric sprawl is real. Teams instrument twenty scores, look at none of them, and learn nothing. A defensible starting set for a RAG or agent system:
- Groundedness / faithfulness — is every claim in the answer supported by the retrieved context? This is your hallucination detector and usually the most important single number.
- Context relevance — did retrieval actually surface the right documents? Separating this from generation quality is the main reason RAGAS exists as a RAG-specific library. When answers are bad, this metric tells you whether to fix the retriever or the prompt.
- Answer correctness — against a reference answer, where you have one.
- Task completion — for agents, did the multi-step workflow reach the goal? Anthropic's guidance on agent evals makes the point that scoring the final outcome is often more meaningful than scoring each intermediate step, because there are many valid paths to a correct result.
- Refusal and escalation rate — how often does the system correctly decline or hand off? A system that never refuses is not safe; it is overconfident.
- Cost and latency per request — quality metrics without cost metrics produce systems nobody can afford to run.
Choosing tooling without over-buying
The landscape sorts into three rough categories, and most teams need fewer tools than vendors suggest.
Metric libraries — RAGAS for RAG-specific retrieval-versus-generation metrics, DeepEval for pytest-style assertions that slot into existing test suites. These are libraries, not platforms: they compute scores and leave orchestration to you.
Eval platforms — LangSmith, Braintrust, Arize Phoenix. These add dataset management, experiment tracking, side-by-side comparison, and CI integration. The genuine value is turning a production failure into a regression test case with one click, so your dataset compounds instead of going stale.
Human review — irreducible. Every automated metric needs periodic calibration against human judgement, or you end up optimising a number that has quietly stopped correlating with quality.
If you are starting from zero: write 50 test cases in a spreadsheet, score them with a simple script, and wire it into CI. That beats an expensive platform with no dataset behind it.
Where teams get this wrong
- Evaluating only the happy path. Your dataset needs adversarial and edge cases or it will pass while production fails.
- Averaging away the failures. A 92% mean score can hide a category that fails 100% of the time. Segment by input type, language, and customer tier.
- Never revisiting the dataset. Production changes. A dataset built in January is measuring a system that no longer exists by July.
- Treating the judge as ground truth. Automated graders carry measurable biases — enough that it deserves its own treatment, which we give it in the LLM-as-a-judge article.
The honest starting point
You do not need a platform, a dedicated eval engineer, or a taxonomy of thirty metrics. You need fifty real examples, three metrics you actually look at, and a CI gate that stops a regression from reaching customers.
Teams that build that in a week outperform teams that spend a quarter designing the perfect evaluation architecture and never ship it. Evaluation compounds — but only once it exists.
For Malaysian teams building production AI with evaluation discipline built in, our AI Engineering programme covers evals, RAG, observability and deployment hands-on — HRD Corp SBL-KHAS claimable for eligible employers.
