What is the difference between logging and tracing for AI systems?

Logging produces a flat stream of independent events. Tracing produces a nested tree: one trace represents an end-to-end request, and each span within it represents one operation (an LLM call, a tool call, a retrieval step) nested under its parent. For a multi-step agent, the tree structure is what lets you answer 'which step was slow and which step produced the wrong output' — a question flat logs cannot answer without heavy manual correlation.

Are the OpenTelemetry GenAI semantic conventions stable?

Not yet. As of 2026 they are marked Development status and live in a dedicated semantic-conventions-genai repository that extends the core OpenTelemetry spec. Attribute names can still change between releases, so pin your semconv version and read release notes when upgrading. Even in Development status they are worth adopting, because the alternative is proprietary instrumentation that locks you to a single vendor.

Should I self-host my LLM observability platform?

For Malaysian organisations handling PDPA-regulated personal data — and particularly financial institutions under BNM RMiT expectations — self-hosting is often the deciding factor rather than a preference, because traces can contain prompt content with customer identifiers. Langfuse is MIT-licensed and fully self-hostable without feature gating, which is why it is a common default in regulated environments.

Can I use more than one observability tool at once?

Yes, and it is a common production pattern. Because OpenTelemetry-compatible instrumentation lives in your application layer rather than being tied to a backend, you can send the same spans to multiple destinations. A frequent setup pairs Langfuse for operational telemetry (cost, latency, prompt versions) with Arize Phoenix for RAG-specific analysis (retrieval quality, faithfulness, hallucination detection).

Should I capture full prompt and completion content in traces?

Deliberately, and not by default. The OpenTelemetry GenAI conventions make content capture explicitly opt-in for good reason: prompts in Malaysian business contexts routinely contain names, NRIC numbers, account details and medical or financial specifics. Enabling full capture without a redaction layer creates a second, less-governed copy of your most sensitive data inside an observability tool. Start with metadata and token counts, and add content capture only once redaction is in place.

LLM Observability: Tracing AI Systems You Cannot Debug by Reading Code

LLM Observability: Tracing Systems You Cannot Debug by Reading Code

A traditional bug is reproducible. An agent that took eleven steps and returned nonsense is not. Observability is how you recover the ability to answer "what actually happened?"

By Marcus Chia 2026-08-11 11 min read

Debugging traditional software has a comforting property: the same input produces the same failure. You reproduce it, you step through it, you fix it.

Now consider a support agent that received a customer message, decided to call a retrieval tool, got back four documents, decided one was relevant, called an order-status API, misread the response, and produced a confidently wrong answer. Same prompt tomorrow might work fine. There is no stack trace. There is no line of code that is wrong.

This is the problem observability solves. Not "is the service up" — but "what did this system actually do, step by step, for this specific request?"

Traces, spans, and why logs are not enough

Logging gives you a flat stream of events. For a multi-step agent, that is close to useless — you get fragments with no structure connecting them.

Tracing gives you the tree. A trace is one end-to-end request. A span is one operation within it, nested under its parent. A single agent request might produce a trace containing:

Root span: the user request (2.4s total)
- Span: retrieval call (180ms, 4 documents returned)
- Span: LLM call — planning (600ms, 1,200 input tokens, 80 output tokens)
- Span: tool call — order lookup API (900ms, HTTP 200)
- Span: LLM call — final answer (700ms, 2,100 input tokens, 210 output tokens)

Now the question "why was this slow and wrong?" has an answer you can actually read. The order lookup took 900ms and returned a payload the model then misinterpreted — visible in one screen instead of inferred from scattered log lines.

OpenTelemetry GenAI conventions: instrument once

The most consequential development for this space is not any single vendor's product. It is standardisation.

OpenTelemetry — already the standard for distributed tracing in conventional systems — now has dedicated GenAI semantic conventions, maintained in a separate repository that extends the core spec. These define standard attribute names for the things every LLM call has: the model invoked, input and output token counts, the operation type, and — when you explicitly opt in — the prompt and completion content itself.

The practical payoff is vendor portability. If your application emits spans following the GenAI conventions, your instrumentation lives in your application layer and is not welded to any observability backend. Switching from one platform to another becomes a configuration change rather than a re-instrumentation project.

Worth being precise about status: the GenAI conventions are still marked Development rather than stable, so attribute names can shift between releases. Pin your semconv version and read the release notes. That is a far smaller tax than proprietary lock-in.

You will also encounter OpenInference (Apache 2.0, originating from Arize), which defines span kinds and attribute namespaces for LLM applications and predates parts of the OTel work. Several platforms speak both.

What the platforms actually differ on

Once instrumentation is standardised, platform choice becomes a question of operating model rather than data format.

Langfuse — MIT-licensed and fully self-hostable with no feature gating. For Malaysian organisations under PDPA constraints, or in financial services where BNM RMiT expectations make sending prompt content to a foreign SaaS a difficult conversation, self-hosting is often the deciding factor rather than a preference.

Arize Phoenix — grew out of Arize's ML observability heritage, local-first and notebook-friendly, built tightly around OpenTelemetry and OpenInference. Strongest on RAG-specific analysis: retrieval quality, faithfulness scoring, hallucination detection.

LangSmith — the fastest path to working traces if you are already building on LangChain or LangGraph, at the cost of the tightest coupling to one ecosystem.

A pattern that shows up repeatedly in production: Langfuse for operational telemetry (cost, latency, prompt versions, request tracing) alongside Phoenix for retrieval and faithfulness analysis. Because both consume OTel-compatible spans, running them together is not the integration burden it sounds like.

The four signals worth instrumenting first

Full request traces — every LLM call, tool call and retrieval step, nested, with timings. This is the foundation everything else builds on.
Token counts and cost per request — attributed to a user, tenant or feature. Without this, an AI feature's unit economics are invisible until the monthly invoice arrives.
Latency percentiles, not averages — p50, p95 and p99, broken out by step. Mean latency hides the tail that users actually complain about.
Quality scores attached to traces — the bridge between observability and evaluation. When a groundedness score is low, you want to click straight through to the trace and see which retrieved chunk caused it.

That last point is the one worth internalising. Evaluation tells you that quality dropped. Observability tells you why. Teams that keep them in separate tools spend their time copying request IDs between browser tabs.

Content capture and the PDPA question

The GenAI conventions treat capturing prompt and completion content as explicitly opt-in, and that design choice reflects a real risk. Prompts in a Malaysian business context routinely contain customer names, NRIC numbers, account details, and medical or financial specifics.

Turning on full content capture without a redaction layer means creating a second, less-governed copy of your most sensitive data inside an observability tool. That is a decision worth making deliberately — we cover the specifics in what to log and what never to log.

Where to start

Instrument with an OpenTelemetry-compatible SDK following the GenAI conventions. Send spans to a self-hosted Langfuse instance. Capture metadata and token counts from day one; leave full content capture off until you have a redaction policy. Attach eval scores to traces so quality and behaviour live in the same place.

Most teams discover in their first week of real traces that something has been quietly broken for months. That is not a failure of the team — it is what happens when a system has been running without eyes on it.

Our AI Engineering programme covers tracing, evaluation and production monitoring hands-on — HRD Corp SBL-KHAS claimable for eligible Malaysian employers.

Build AI systems that hold up in production

Evaluation, observability and guardrails are what separate a demo from a system your business can depend on. AITraining2U runs hands-on, HRD Corp SBL-KHAS claimable AI training for Malaysian organisations — tool-agnostic and mapped to your actual stack.