LangSmith Trace Review: From a Flagged Run to the User's Session

When a LangSmith trace of your agent gets flagged — an error, a low eval score, an end-user thumbs-down — the question is always the same: did the agent produce a bad result, or did the end-user hit friction the agent had nothing to do with? This recipe wires up an unattended loop that answers it for you. The moment a run is flagged, it opens the flagged end-user’s Subtext session, has a triage agent reason over both the trace and the session, and posts the finding to Slack (or wherever your team watches) before anyone has to notice. It’s the automation version of the LangSmith integration workflow: instead of a person pulling the subtext_url and pasting a prompt, a triage agent does it on every flagged run.

What you need

Your own LangChain/LangGraph agent, serving real end-users, already threading each end-user’s Subtext session URL into its trace metadata as subtext_url (see the LangSmith integration — this is the load-bearing step; no subtext_url on the trace means no session to review later).
A LangSmith project with tracing enabled for that agent.
A working Subtext MCP endpoint.
The official langsmith-mcp-server MCP (see the LangSmith integration), or the LangSmith SDK if your triage agent’s runtime doesn’t support MCP.
Somewhere to run a webhook handler — a serverless function or a small backend service reachable from LangSmith.
Wherever you want the finding to land — Slack, a ticket queue, an internal channel.

Set it up

Thread the end-user session into trace metadata

Attach subtext_url under metadata on the root run for every request your agent serves (covered in the LangSmith integration). Get this in place before anything else — it’s the one step everything downstream depends on.

(Optional) score runs with an online evaluator

If you want a quality signal beyond errors and explicit end-user feedback, add a LangSmith online evaluator — an LLM-as-judge run against each new trace — that writes a feedback key, e.g. review_quality, scoring whether the agent’s output actually looks right. Richer signal than error/thumbs-down alone, at the cost of a model call per traced run. Skip it for a first pass if error + thumbs-down is enough.

Create a LangSmith Automation

In your LangSmith tracing project, click + New in the top right, then New Automation. (Existing rules and their logs live under the project’s Automations tab.)Filter — an FQL query over the condition you care about, e.g.:

eq(status, "error")

or(eq(status, "error"), and(has(feedback_key, "user_score"), lt(feedback_score, 1)))

If you added an evaluator in Step 2, add has(feedback_key, "review_quality") so the rule waits for the score to land before firing — LangSmith’s docs call out this evaluator/webhook race explicitly; without the wait, the Automation can fire before the async evaluator finishes.Sampling rate — 1.0 to review every matching run; lower it if flagged-run volume is high and you’d rather triage a sample.Action — Webhook, pointed at your handler’s URL. Add a ?secret= query param (or an encrypted custom header) and verify it in the handler before doing anything else — fail closed.

Stand up the webhook handler

The one hard constraint: LangSmith’s webhook contract requires your endpoint to respond within 5 seconds, or delivery is declared failed (5xx retries twice with backoff; 4xx doesn’t retry; the response body is ignored). A real Subtext review takes 30–60s+, so the handler cannot review inline — acknowledge fast, then do the work asynchronously.The payload LangSmith POSTs:

{
  "rule_id": "...",
  "start_time": "...",
  "end_time": "...",
  "runs": [{ "id": "...", "trace_id": "...", "...": "..." }],
  "feedback_stats": { "...": "..." }
}

Run dicts in the payload are lightweight — fetch the full run by id via the SDK or langsmith-mcp-server if you need metadata reliably.Handler shape (any runtime that supports “ack now, keep working”):

# A background/serverless function, or a queue-backed worker — the shape is the
# same: verify, ack, then do the real work off the request thread.
def handle_webhook(request):
    if request.args.get("secret") != WEBHOOK_SECRET:
        return Response(status=401)

    payload = request.get_json()
    enqueue_async(review_flagged_runs, payload["runs"])  # don't await this
    return Response(status=202)  # ack within the 5s window

def review_flagged_runs(runs):
    for run_ref in runs:
        run = langsmith_client.read_run(run_ref["id"])
        # subtext_url lives in the trace metadata. LangChain/LangGraph cascades
        # run metadata to child runs, so a flagged child span usually carries it
        # too — but a filter like eq(status, "error") can match a child span, and not
        # every setup cascades, so fall back to the trace's root run (its id ==
        # trace_id) when the flagged run itself doesn't have it.
        subtext_url = (run.extra or {}).get("metadata", {}).get("subtext_url")
        if not subtext_url and run.trace_id and run.trace_id != run.id:
            root = langsmith_client.read_run(run.trace_id)
            subtext_url = (root.extra or {}).get("metadata", {}).get("subtext_url")
        if not subtext_url:
            continue
        finding = run_triage_agent(run, subtext_url)
        post_finding(finding)  # Slack, ticket, wherever the team watches

Two ways to run the triage agent itself:

As a background LangGraph run (LangChain-native) — if your triage agent is itself a LangGraph deployment, start a non-streamed/blocking run from the handler (a .../runs/wait call or the SDK equivalent) and read its final output. Keeps the triage agent on the same platform as the agent being reviewed.
As a serverless background job (portable) — a plain agent loop in the handler’s own runtime that holds both MCP connectors directly. No LangGraph dependency; the simpler option if the agent you’re reviewing isn’t itself LangGraph-based.

Either way, the triage agent holds both MCPs: the langsmith-mcp-server (fetch_runs(trace_id=...) — what your agent actually did) and the Subtext MCP (review-open(session_url=subtext_url) — what the end-user actually experienced). It reasons over both and decides: did the agent produce a bad result, or did the end-user hit friction the agent had nothing to do with? That’s the finding worth posting.

Triage prompt sketch:

This LangSmith run was flagged (<error | low score | thumbs-down>). Trace: <trace_id>.
End-user session: <subtext_url>.

Use the LangSmith tools to see what the agent actually did on this run. Use the Subtext tools to
open the end-user's session and see what they actually experienced. Decide: did the agent produce
a bad result, or did the end-user hit friction the agent had nothing to do with? Give a
one-sentence verdict plus the evidence from both sources, then post it.

One agent serving two paths. If a single deployed agent serves both your public end-user traffic and doubles as the triage agent, don’t attach the LangSmith MCP to that shared deployment — a prompt injection planted in an end-user’s own content could coerce a trace-fetching tool into enumerating other end-users’ traces. Gate trace access to the triage path only (a separate route, a separate credential, or — safest — fetch the trace server-side in the handler and hand the triage run a read-only digest instead of a live LangSmith tool). If your triage agent is an internal, trusted, backend-only agent with no public-facing twin — the common case — this caveat doesn’t apply: hand it both MCPs directly, per the shape above.

On-demand vs. cron

Recommended: on-demand, via the webhook above. Timely — the team hears about a bad run in the same window it happened — no polling, and it’s what LangSmith Automations support natively. Fallback: daily cron. A scheduled job that queries LangSmith once a day for flagged traces in the last 24h (list_runs(filter=...), the same filter as the Automation) and batch-reviews them, for teams that would rather get one daily digest than a notification per run. Same triage-agent shape, just invoked on a schedule instead of a webhook.

Test it

Force a flagged run

Throw an error in your agent, or manually log a bad feedback score on a real run via the SDK.

Confirm the Automation fires

Check its run history in the LangSmith UI, and confirm your webhook receives the POST within the 5s window.

Confirm the async review actually ran

Check your handler’s logs for the fetch_runs / review-open tool calls, and confirm the finding landed in Slack (or your target destination).

You’ve wired it correctly when the finding names something from both sources — a concrete detail from the trace and a concrete detail from the session — not a paraphrase of one or the other.

LangSmith integration — attach subtext_url to your traces and add the langsmith-mcp-server.
Session Review overview — what your triage agent does once a session is open.

Get Started

Install by Framework

Session Review

Privacy

Integrations

Recipes

API reference

LangSmith Trace Review: From a Flagged Run to the User's Session

What you need

Set it up

On-demand vs. cron

Test it

​What you need

​Set it up

​On-demand vs. cron

​Test it

​Related

What you need

Set it up

On-demand vs. cron

Test it

Related