Support-to-Trace Checklist for AI Product Incidents

Use this checklist before escalating an AI-related customer complaint. It is intentionally tool-neutral: a trace might live in LangSmith, Langfuse, Braintrust, Helicone, OpenTelemetry, or your own logs.

Before escalating

• Preserve the customer's original wording.
• Record what the customer was trying to achieve.
• State expected and actual behaviour separately.
• Record customer impact, frequency, and urgency.
• Capture the approximate timestamp and timezone.
• Capture an anonymised account/user and request/session identifier.
• Record whether the issue can be reproduced.
• Remove secrets, credentials, personal data, and unnecessary customer content.

Pull from AI observability

• Trace or run URL/ID.
• Exact model/provider/version.
• Prompt and configuration version.
• Input and relevant conversation state.
• Retrieval query, selected chunks, document IDs, and versions.
• Tool definitions, calls, arguments, results, and errors.
• Token usage, latency, retries, fallbacks, and timeouts.
• Evaluation, score, or feedback attached to the run.

Pull from product systems

• Product release and feature flags.
• Tenant settings and permissions.
• Product/session events around the failure.
• Application errors and relevant log lines.
• Related incidents, support tickets, or engineering issues.
• Data freshness or indexing status where retrieval is involved.

Attach to the engineering issue

• One-sentence customer symptom.
• Customer and business impact.
• Expected versus actual behaviour.
• Reproduction input or workflow.
• Evidence links with access instructions.
• Suspected primary failure type, confidence, and reasoning.
• Secondary possibilities.
• Explicit missing information.
• Suggested next investigation step.
• Regression-test or eval candidate when the expected result is clear.

Send back for more context when

• There is no usable timestamp, account, request, session, or trace identifier.
• The expected outcome is unclear.
• The report contains only a screenshot of a response with no input or context.
• The issue may involve permissions or cross-tenant data and has not entered the security process.
• Support would have to guess the model, prompt version, trace, or root cause.

Keep diagnosis with engineering. Support should preserve the customer symptom and enough identifiers to investigate it without immediately asking for the same missing context.