41dots

LLM evaluation: trace a wrong price to the failing step

Reproduce and repair a stale catalog cache, then check the lookup and answer separately with a local Python example.

A catalog assistant quotes £19 for a mug that costs £24. Before changing its prompt, compare the catalog record, the lookup result, and the answer. If the lookup returned £19, a model that repeats it has followed its evidence and still given the customer the wrong price. If the lookup returned £24, the price changed later in the flow.

Keep three checks separate: whether the tool met its contract, whether the answer repeated the tool’s product and price, and whether the final answer matched the expected catalog record. A final-price score marks both £19 answers wrong without distinguishing a stale lookup from a later misquote.

Four saved traces, three checks

Consider a synthetic catalog example. At the time of the request, product MUG-01 costs 2400 in GBP minor units (£24) in catalog-v2. An older record, catalog-v1, lists 1900 (£19). Integer minor units let the example compare prices without floating-point rounding; the currency remains part of every comparison.

A trace is a record of the steps taken to answer a request, including tool inputs and outputs. Here we reduce each saved trace to a lookup result and a structured answer claim: product ID, currency, and price. The reference is the catalog snapshot expected at request time. Comparing an old request with today’s price would test a different question.

Saved caseTool matches referenceAnswer matches toolAnswer matches reference
Old £19 lookup, £19 answerFailPassFail
Correct £24 lookup, £19 answerPassFailFail
Correct £24 lookup, £24 answerPassPassPass
Old £19 lookup, £24 answerFailFailPass

The first row passes answer fidelity: the answer repeats the tool accurately. That says nothing about whether the tool returned the right price. Investigate the catalog query, cache, or snapshot selection before treating a prompt change as the repair.

The second row places the mismatch after the lookup. Inspect the model output and any formatting step that produced the displayed answer. The comparison locates a boundary; it does not identify the responsible line of code.

The fourth row is easy to miss when evaluation checks only the final answer. The customer sees the expected price, yet the lookup remains stale. The answer could have come from another context item or a guess; these saved fields cannot establish its cause. Keep the lookup failure visible so an accidental correct answer cannot certify the data path.

Evaluating components separately is an established approach. LangSmith’s agent evaluation tutorial distinguishes final-response, trajectory, and single-step checks. This example makes one consequence concrete: answer fidelity and final correctness can disagree in either direction.

Run the comparison locally

Download the Python price-trace example, save it as price-trace-regression.py, and run:

python3 price-trace-regression.py

It requires Python 3, uses only the standard library, and makes no network calls or file writes. It first scores synthetic saved outputs, then runs a small catalog lookup before and after a cache-key repair. A deterministic function copies the lookup into an answer; it does not call a model, parse answer prose, or measure an assistant’s failure rate.

The evaluator is small enough to inspect:

def evaluate(reference, tool, answer):
    keys = ("sku", "currency", "price_minor")
    return {
        "tool_matches_reference": all(tool[k] == reference[k] for k in (*keys, "version")),
        "answer_matches_tool": all(answer[k] == tool[k] for k in keys),
        "answer_matches_reference": all(answer[k] == reference[k] for k in keys),
    }

The lookup contract includes the catalog version. The answer comparisons cover the visible claim, which does not include a version. Consequently, a tool with the right product and price but the wrong version fails the tool check while its answer can pass both answer checks. A matching version is a provenance check, not proof that the underlying data is fresh.

The script prints three Boolean scores for each table row, then reports:

PASS: four trace cases and three contract mutations

The three additional checks change the product ID, currency, and catalog version independently. Comparing only price_minor would accept the wrong product, the wrong currency, or the wrong catalog version whenever the numeric price stayed the same. The passing script confirms that the evaluator distinguishes these fixtures; it does not mean every saved trace represents a successful application run.

Reproduce the stale lookup and replay the repair

The second half of the download creates the first table row by running a lookup with an in-memory cache. It warms the cache with the £19 record from catalog-v1, then requests the £24 record from catalog-v2 for the same SKU. Both snapshots are synthetic; there is no live catalog or model in this exercise.

The broken lookup uses only the SKU as its cache key. The second request therefore hits the old entry without reading the new snapshot. The answer function copies that stale result, producing these scores:

cache_before: tool=False, fidelity=True, final=False

The repair changes the key so each catalog snapshot gets a separate entry:

# Before
key = sku
# After
key = (sku, catalog["version"])

The script repeats the same sequence with a fresh cache and the repaired key. The second request now reads catalog-v2, returns £24, and passes all three checks. It also verifies that another request for catalog-v2 reuses the cache, and that returning to catalog-v1 cannot replace the newer entry. These checks distinguish the repair from simply disabling caching.

Finally, it pairs the repaired lookup with a deliberately wrong £19 answer. The lookup passes while fidelity and final correctness fail. Fixing the data path has not made the later answer check redundant. The run ends with:

PASS: cache failure reproduced, repaired and replayed

Try changing the repaired key back to sku: the replay raises an AssertionError. That failure is the regression this example preserves.

This key works because the request supplies an immutable catalog snapshot. In a live system, first establish how the request obtains the correct snapshot or revision: adding a version field cannot fix a stale version selector. Customer price tier, currency, locale, or tax treatment may also change the result and therefore the cache key. Keep old entries only as long as the application’s retention policy requires; this small in-memory example has no eviction policy.

For a real incident, retain the request, tool arguments, returned record, answer, and expected record valid at that time. If the reference cannot be reconstructed, leave correctness unresolved. Replay those inputs through the changed application and score its new outputs. The local cache exercise demonstrates a data-path repair; testing a model’s answers still requires running that model with the saved context.

In this single-source example, an answer must use the lookup result. An assistant allowed to reconcile several sources needs a different fidelity rule: record which source supports each claim and how conflicts should be resolved. Copying one tool’s value is too narrow for that task. LangSmith’s evaluation types documentation explains using curated reference examples for offline tests; choose the assertions around the application’s actual contract.

Our e-commerce AI platform case study describes work with LangSmith, LiteLLM, and unit and integration tests. The catalog fixture above is a small exercise for choosing what those tests should distinguish.

All engineering guides

Have a project in mind?hello@41dots.com ↗