A human reviewer needs to know what an agent was trying to do when judging a particular turn. Raw input and output fields can conceal the conversation that made an answer appropriate or mistaken. Evaluation quality therefore depends partly on the interface through which the evidence is presented.
LangChain’s October 26–November 2 LangSmith update opens annotation-queue items in their conversation trajectory, positioned at the turn being graded. It also introduces custom queue views that can be previewed before saving. The custom views change presentation only and cannot call the LangSmith API. The new trajectory view is read-only; editing inputs or outputs before adding a run to a dataset uses the classic view. Read the primary source account.
Context helps, but presentation still matters
Our analysis is that showing the surrounding trajectory can reduce avoidable disagreement about what the model knew or had already attempted. A response that looks incomplete in isolation may be a reasonable continuation. Conversely, a plausible final sentence may follow an incorrect tool action that matters more than the wording.
Custom views introduce a useful tradeoff. Removing irrelevant fields can make review faster, but hiding a tool call or earlier turn can also remove the evidence needed to explain a failure. The right view depends on the grading task. Teams should not assume a cleaner display is automatically a more faithful representation of the behavior under review.
The separation between display and editing is especially important for evaluation provenance. A reviewer should be able to tell whether they are inspecting a recorded run or a modified example prepared for a dataset. If those states become ambiguous, later comparisons may be difficult to interpret even when the scoring rubric is unchanged.
Test the review process itself
The immediate product improvement is a more contextual route through annotation work. Teams can assess whether reviewers reach the relevant evidence more quickly and whether disagreements become easier to explain. The update does not establish a measured improvement in grading accuracy or inter-reviewer agreement across all tasks.
Longer term, better review interfaces could make human feedback more useful for model and agent iteration. That requires consistent rubrics, visible provenance and a way to inspect cases where the chosen view obscures something important. Interface design is part of the evaluation system rather than a neutral wrapper around it.
The next practical check is a small set of known cases reviewed in both the default and proposed custom view. If reviewers can locate the decisive evidence, explain their judgments and preserve the original record, the new presentation can improve the workflow without weakening its auditability.
Source organizations: LangChain.
References
Questions, news tips or corrections? Contact [email protected].


