AI Bug Report Template for LLM, RAG, and Agent Products

“The AI was wrong” describes a symptom. It does not tell an engineer which execution to inspect, what the product should have done, or whether the problem came from retrieval, a tool, application state, configuration, or something else entirely.

The answer is not to make every support ticket enormous. A useful AI bug report needs a small set of facts that lets engineering locate the event and understand the failure. Diagnostic detail can follow once someone has found the relevant trace, logs, and source records.

The template below follows that order. The first section is the minimum handoff. Everything after it is optional context: include it when the information is available and relevant.

Copyable AI bug report template

# [AI failure] <specific user-visible symptom>

Status: Needs Evidence / Ready for Engineering

## Minimum engineering handoff

### Source Report

- Original wording: <quote the customer or support report>
- Source: <channel and ticket/thread link>
- Reported at: <timestamp and timezone>
- Affected account/user: <internal or anonymised identifier>
- Product area or workflow: <where the failure occurred>

### Impact

- User intent: <what the person was trying to accomplish>
- Consequence: <what was blocked, degraded, or put at risk>
- Frequency: once / intermittent / repeatable / unknown
- Urgency: low / medium / high, because <reason>

### Expected behaviour

<What should the product have returned or done? Link the policy, record,
specification, or other source of truth when one exists.>

### Actual behaviour

<What did the product return or do instead? Quote the output when possible.>

### Correlation details

- Event time window and timezone:
- Account, tenant, or workspace ID:
- Session, conversation, request, or run ID:
- Exact user input, if available:

### Evidence

Observed Evidence:

- <link or identifier> — <what it establishes>

Evidence still needed:

- [ ] <missing item> — <question it would help answer>

### Failure Hypotheses

1. <possible explanation>
   - Failure class: retrieval / prompt or configuration / model / tool selection
     / tool execution / tool-result handling / state or memory / permission
     / latency or timeout / policy or refusal / expectation mismatch / unknown
   - Confidence: low / medium / high
   - Evidence for:
   - Evidence against or unresolved:
   - What would confirm or reject it:

Confirmed Root Cause: Not established / <cause plus decisive Evidence or
engineering verification>

## Optional diagnostic context

### Conversation and input state

- Relevant conversation history:
- Uploaded files, selected records, filters, or user state:
- Reproduction steps:
- Reproduction result: consistent / intermittent / not reproduced / not attempted

### AI execution

- Trace or run link:
- Model, provider, and version:
- Prompt, system instruction, or configuration version:
- Input and output safety/policy result:
- Token, latency, timeout, or retry details:

### Retrieval Evidence

- Query and any query transformation:
- Retrieved documents/chunks and scores:
- Knowledge-base or document version:
- Source-of-truth record:

### Tool and application Evidence

- Tools requested or called:
- Arguments supplied:
- Results or errors returned:
- Relevant application logs:
- Tenant settings, permissions, feature flags, or cached state:
- Recent deploys or configuration changes:

### Related work

- Similar incidents or search terms:
- Existing issues:
- Owning team:
- Suggested labels:

## Regression test or eval candidate

Add this section only when expected and actual behaviour are specific enough to
make a meaningful assertion.

- Test name:
- Input:
- Setup and required state:
- Expected behaviour:
- Failure assertion:
- Evidence fixture or source-of-truth record:

What the minimum handoff must accomplish

The minimum section answers three questions.

First: what did the user experience? Preserve the Source Report in the reporter's own words, then state the expected and actual behaviour separately. This prevents a support summary—or an engineer's early theory—from quietly replacing the original symptom.

Second: which execution should engineering inspect? A timestamp is rarely enough by itself. Pair the narrowest reliable time window with an account, tenant, session, request, conversation, or run identifier. These correlation details turn a broad search through telemetry into a bounded investigation.

Third: what do we know, and what are we only considering? Evidence should say what each linked artifact establishes. A trace can show that a tool was not called; a source record can show the correct state at a particular time. Neither fact, on its own, necessarily explains why the behaviour occurred. Keep possible explanations under Failure Hypotheses until decisive Evidence or engineering verification supports a Confirmed Root Cause.

If one of those questions cannot be answered, leave it visibly unresolved. Unknown is a useful engineering fact. A plausible guess presented as fact is not.

Evidence should carry an argument

A list of links forces the next person to repeat the triage. In a fictional invoice incident, the difference could look like this:

• Weak: Trace: run_01J...
• Better: Trace run_01J... — matches the affected account and time window; shows no invoice lookup tool call before the answer.
• Weak: Database record attached.
• Better: Invoice snapshot — records status=paid at 13:42 UTC, eleven minutes before the reported response.

The stronger versions still do not claim a root cause. They identify observations that support or contradict a Failure Hypothesis. For example, the missing tool call is consistent with a tool-selection or prompt/configuration problem, but it does not distinguish between them without more Evidence.

How to move from Source Report to Repro Pack

1. Copy the original Source Report before interpreting it.
2. Record expected and actual behaviour in terms an engineer can compare.
3. Add the identifiers needed to locate the relevant execution.
4. Link authorised Evidence; do not paste credentials, secrets, or unnecessary customer data into the issue.
5. Write unknown for missing facts and name the Evidence needed to resolve them.
6. Rank Failure Hypotheses, including contradicting Evidence and a clear confirmation condition.
7. Add an eval or regression candidate only when the expected outcome can be asserted reliably.

Once reviewed, this handoff becomes a Repro Pack: the Source Report, correlated Evidence, bounded Failure Hypotheses, missing information, and proposed engineering work in one place.

A practical readiness check

Mark the report Ready for Engineering when it contains:

• a clear user-visible symptom and impact;
• distinct expected and actual behaviour;
• enough correlation detail to locate the execution, or a precise request for what is missing;
• at least one relevant piece of Evidence, with an explanation of what it establishes; and
• uncertainty expressed as open questions or Failure Hypotheses rather than implied fact.

If the report has no usable identifier, time window, or evidence path, mark it Needs Evidence and send the focused questions back to the person who can answer them. A shorter, honest report is more useful than a complete-looking one filled with assumptions.

Continue the investigation

See the failure taxonomy when you need to separate plausible mechanisms, then compare the result with the fictional stale-invoice Repro Pack.

FaultTrail is building this support-to-evidence workflow with small AI SaaS teams. If it resembles a recurring problem in your product, explore the early pilot. Start by describing your workflow and systems; do not send tickets, traces, credentials, personal information, or confidential customer content.