Your RAG Incident Has No Trace. Don't Touch Retrieval Yet

A customer says your RAG assistant cited an obsolete policy. You search the observability backend for the execution. Nothing.

That looks like an observability dead end. It is actually a routing decision.

The tempting response is to change retrieval, raise the sampling rate, or log every prompt. But “no trace” is not a RAG diagnosis. It can describe four evidence failures, each with a different owner and repair. Change the model now and you may ship risk without making the next incident any easier to investigate.

First, preserve what survived

Before widening the search, record the customer-visible symptom, expected behaviour, product surface, environment, event window and timezone. Preserve any authorised tenant-scoped session, interaction, request, job, or result reference exactly.

Record the exact observability workspace and environment you searched, including region where relevant. Note the deployment and telemetry configuration active at the time. Those details turn an empty result into something you can test.

Keep raw prompts and retrieved documents in protected systems. Customer records and credentials belong there too. Keep raw traces behind their existing access controls. OWASP's logging guidance recommends special handling for secrets and sensitive data, with restricted access to logs. More content is not the same as better evidence.

Now ask which of four paths produced the blank search.

1. The execution was never recorded

Check the sampling policy and instrumentation deployed during the event window. If request-adjacent metadata safely preserves a W3C traceparent, inspect its sampled flag without overstating it.

The Trace Context specification says a set sampled flag means the caller may have recorded data. It does not prove that a collector accepted the data or that a backend retained it. OpenTelemetry's Tracing SDK specification also distinguishes recording from sampling: a non-recording span discards its attributes and events.

Classify the incident as probably unsampled only when configuration or surviving metadata supports it.

Repair: change recording or sampling for the consequential workflow, or emit a small non-sensitive diagnostic envelope for every user-visible result.

2. The telemetry disappeared after recording

If the execution should have been recorded, inspect the collector around the event window. OpenTelemetry's internal telemetry exposes accepted, refused, send-failed, and queue signals. These are aggregate health indicators. They cannot prove what happened to one request, and a send failure may later succeed after a retry.

Then compare the backend's retention policy with the incident time and reporting delay. A trace that expired and a trace that never reached storage look identical in search, but create different engineering work. The Collector troubleshooting guide covers faults in configuration or networking. It also describes capacity and destination problems that can interrupt the pipeline.

Repair: fix the receiver, exporter, queue, retry, alerting, or retention policy supported by the evidence. Keep the result probable when only aggregate counters support it.

3. The trace exists, but the investigator cannot see it

Repeat the lookup through the authorised path for the exact environment and observability project. Include region and tenant scope where relevant. “Not visible to me” is not the same state as “not retained.”

This is an access and runbook failure, not permission to give everyone broader production access.

Repair: provide a least-privilege on-call lookup path and name the selected workspace. Record environment or access failures explicitly in the handoff.

4. The trace exists, but the report cannot identify it

Search the bounded window using the safe product references that survived. Then inspect where an interaction, request, job, or result reference should connect to the distributed trace.

OpenTelemetry's context propagation documentation explains how services inject and extract trace and parent identifiers across boundaries. Missing propagation, an asynchronous job, a retry, or a missing product-level result ID can leave real traces that cannot be matched confidently to the displayed answer.

The closest timestamp is not a match. Record why a candidate belongs to the report and what remains uncertain.

Repair: restore propagation across the broken boundary and attach a safe, opaque interaction or result reference to the output and feedback path.

One symptom, four backlog items

The same blank search can now produce four legitimate tickets:

• the root execution was not sampled;
• the exporter queue failed during the event window;
• the investigator searched staging instead of the authorised production project; or
• candidate executions exist, but none carries the result reference shown to the user.

None explains why the policy was obsolete. Each explains why the team cannot investigate that answer and what must change before the next report.

If no path has supporting evidence, keep the classification unknown. Do not choose the most convenient cause.

Repair observability before retrieval

“Log everything next time” can expose prompt content or retrieved text. It also risks capturing credentials or customer data while leaving the original observability failure untouched.

Prefer the smallest durable correlation layer. Start with event time and product surface. Add the environment and an opaque interaction or result reference. Record the deployment version and trace availability state. The Support-to-Trace Checklist can help structure that handoff.

Once the report is matched to an execution, A Wrong RAG Answer Is Not Yet a Retrieval Bug follows the evidence through retrieval and generation to display. Until then, changing retrieval is guesswork.

FaultTrail is building automated support-to-evidence triage around this boundary. It does not yet accept reports, retrieve telemetry, or connect to customer systems. If this failure pattern keeps blocking your team, describe the workflow and systems involved, without sending tickets, traces, prompts, credentials, personal data, or confidential customer content.