Sample Repro Pack: A Stale AI Invoice Answer

Fictional example: The company, customer, trace IDs, and product details below are invented. Do not treat them as a real incident or customer claim.

The Source Report

Customer says the assistant gave the wrong answer about an invoice. It told them the invoice was overdue, but finance says it was paid last week. The answer worked before. Can engineering look?

Support also supplied:

• Affected workspace: anonymised workspace_42.
• Approximate report time: 10 July 2026, 09:30 AWST.
• Product area: billing assistant.
• No trace link was attached.

This is a credible customer symptom, but it is not yet a diagnosis. The report does not identify the invoice, quote the exchange, or show what the billing system returned at the time. “Finance says it was paid” and “the answer worked before” are useful leads, not verified Evidence.

Sending the ticket straight to engineering would force the next person to reconstruct the incident before they could investigate it. A Repro Pack makes that uncertainty explicit.

The worked Repro Pack

1. Reported symptom

At approximately 09:30 AWST on 10 July 2026, someone in workspace_42 reportedly asked the billing assistant about an invoice. The assistant described it as overdue. The customer’s finance team believed the invoice had been paid during the previous week.

The exact question, response, invoice identifier, and authoritative billing record are not attached. It is therefore not yet possible to reproduce the interaction or verify which state was correct when the answer was generated.

2. Expected and actual behaviour

Expected: When asked for an invoice status, the assistant should use the invoice record authorised for the current workspace and report its current state. If the record includes a payment date, the answer may include it.

Reported actual: The assistant said the invoice was overdue. The customer says the invoice had already been paid.

This wording matters. The customer’s account tells us there is a contradiction worth investigating; it does not establish that the assistant saw a paid record and ignored it.

3. Impact and immediate risk

• A wrong invoice status can trigger inappropriate collection activity or duplicate communication.
• A billing assistant that contradicts the finance system loses operator trust, even if the incident is isolated.
• The number of affected workspaces and invoices is unknown.
• Tenant context must be checked early. A record from another workspace would be materially more serious than a stale record from the correct one.

This is an investigation priority, not a severity declaration. The available report does not establish frequency, exposure, or whether the source data was correct at the time.

4. Evidence currently available

Evidence	What it supports	What it does not prove
workspace_42	Narrows correlation to one tenant	That the assistant used the correct tenant context
Approximate time, with timezone	Provides a bounded trace-search window	Which execution produced the reported answer
Product area: billing assistant	Narrows the service or feature to inspect	Which retrieval path, tool, prompt, or model ran
Customer’s “paid last week” statement	Establishes the expected state reported by the customer	That the authoritative record was already paid when the interaction occurred
No trace link	Explains why execution Evidence is absent	That no trace exists

At this point, all available Evidence comes from the Source Report. No execution trace, tool result, retrieved record, source-of-truth record, or release evidence has been correlated.

5. Missing information

Ask support for the smallest set of details that can identify the execution:

1. The exact question and complete assistant answer.
2. The anonymised invoice identifier.
3. A more precise interaction time, if available.
4. A conversation, interaction, request, session, or trace identifier.
5. The current invoice status and payment timestamp shown by the authorised billing source.
6. Whether the answer cited a date or source.
7. Whether the same question now produces the same result.

Once the execution is located, retrieve:

• the request and final response;
• the account and invoice identifiers used during the run;
• invoice lookup arguments, status, raw result, and timestamp;
• retrieved records, including document or index versions and update times;
• cache reads, fallback events, and invalidation history;
• the context supplied to response generation;
• prompt, model, service, and deployment versions; and
• a healthy comparison run or similar reports, if either exists.

6. Failure Hypotheses

These explanations are deliberately bounded. Their order should change when new Evidence arrives.

Failure Hypothesis	Why it is plausible	What would support or contradict it
Stale retrieval or cache state — low confidence	The reported answer conflicts with the customer’s understanding of the current invoice state.	Supported if the run consumed an older unpaid record while the authoritative record was already paid. Contradicted if every supplied record was current and marked paid.
Invoice lookup was omitted or failed	The assistant may have answered without consulting the current billing source.	Supported by an absent, failed, or timed-out lookup followed by an answer stated as fact. Contradicted by a successful lookup containing the correct invoice state.
Tool result was mishandled	A correct lookup can still be transformed or interpreted incorrectly before generation.	Supported if the tool returned paid but downstream context or application state changed it to overdue.
Wrong workspace or invoice context	The ticket does not yet show which tenant and invoice the execution actually queried.	Supported if request identifiers and returned record identifiers do not match. Contradicted when account scope and invoice identity agree throughout the execution.
Unsupported generation	The final answer may contradict otherwise correct Evidence.	Supported if retrieval and tool Evidence both say paid, the model received that state, and the generated answer says overdue.
Source-data or expectation mismatch	The customer and assistant may have consulted different systems or points in time.	Supported if the authorised billing source still showed overdue during the interaction, or if “paid” referred to a different invoice.

The leading Failure Hypothesis is stale retrieval or cache state because it fits the reported symptom, not because the Source Report proves it. No Confirmed Root Cause should be recorded yet.

7. What would confirm the cause?

A category label is not confirmation. A Confirmed Root Cause requires Evidence that exposes the mechanism or engineering verification that reproduces it.

Examples for this incident:

• Stale retrieval: the trace shows an unpaid document version was supplied after the authoritative invoice had changed to paid, and the same version mismatch reproduces the answer.
• Tool-result handling: the lookup returned paid, while a visible transformation mapped that value to overdue before response generation.
• Wrong tenant context: the request belongs to workspace_42, but the queried or cached record carries a different workspace identifier.
• Unsupported generation: the generation input contains an unambiguous paid status and no conflicting record, while the resulting answer says overdue. This confirms a failure in that execution, not a general model regression.

If the execution cannot be identified or its Evidence was not retained, the correct conclusion is insufficient Evidence. The Repro Pack should say what is missing rather than promote the most plausible explanation to fact.

8. Engineering issue draft

Title: Investigate billing assistant report of an overdue status for a reportedly paid invoice

Reported symptom

A customer in anonymised workspace_42 reports that the billing assistant described an invoice as overdue at approximately 09:30 AWST on 10 July 2026. The customer’s finance team says the invoice was paid during the previous week.

Current Evidence

• Source Report with workspace, approximate time, and product area.
• No exact exchange, invoice identifier, trace, billing record, or tool result attached.

Failure Hypotheses

Stale retrieval or cache state is the current low-confidence lead. Lookup omission or failure, tool-result handling, incorrect tenant context, unsupported generation, and a source-data mismatch remain viable.

Next investigation step

Identify the interaction using the workspace and time window. Then compare the authoritative invoice state at that time with the tenant context, retrieved records, lookup result, generation input, and final response.

Do not classify this as a prompt or model failure until the data path, tool path, tenant scope, and response handling have been checked.

9. Regression candidate—not yet ready

Do not create a regression test from the customer’s recollection alone. First confirm the original input, authorised invoice state, and mechanism that produced the contradiction.

Once those are known, a minimal case could be:

• Setup: an invoice for the test workspace is marked paid with a known payment timestamp.
• Input: ask the billing assistant for that invoice’s status.
• Expected: the answer reports paid, does not call the invoice overdue, and remains scoped to the correct workspace.
• Instrumentation assertion: the execution uses the expected invoice record and preserves its status through tool handling and generation.
• Failure condition: the answer contradicts the authoritative record, or the run queries a different tenant or invoice.

The regression should target the Confirmed Root Cause when one exists. A broad “answer should be correct” check may catch the symptom while leaving the faulty component ambiguous.

Why this is better than the original ticket

The Repro Pack does not pretend the cause has already been found. It gives engineering:

• the Source Report without silently upgrading reported details into facts;
• a compact inventory of available and missing Evidence;
• ranked Failure Hypotheses with clear ways to test them;
• a threshold for calling something a Confirmed Root Cause;
• an issue draft with a concrete first investigation step; and
• a regression shape that can be completed after the mechanism is known.

That is the useful handoff: not certainty, but a shorter and more disciplined path to it.

Build your own Repro Pack

Start with the AI bug report template and use the failure taxonomy to keep competing explanations visible.

FaultTrail is building this support-to-evidence workflow with small AI SaaS teams. If it resembles a recurring problem in your product, explore the early pilot. Start by describing your workflow and systems; do not send tickets, traces, credentials, personal information, or confidential customer content.