> ## Documentation Index
> Fetch the complete documentation index at: https://subtext.fullstory.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LangSmith Trace Review: From a Flagged Run to the User's Session

> When a LangSmith trace of your agent gets flagged, automatically open the end-user's Subtext session and have a triage agent reason over both — before a human has to notice.

When a LangSmith trace of your agent gets flagged — an error, a low eval score, an end-user thumbs-down — the question is always the same: did the agent produce a bad result, or did the end-user hit friction the agent had nothing to do with? This recipe wires up an unattended loop that answers it for you. The moment a run is flagged, it opens the flagged end-user's Subtext session, has a triage agent reason over both the trace and the session, and posts the finding to Slack (or wherever your team watches) before anyone has to notice.

It's the automation version of the [LangSmith integration](/docs/integrations/langsmith) workflow: instead of a person pulling the `subtext_url` and pasting a prompt, a triage agent does it on every flagged run.

## What you need

* Your own LangChain/LangGraph agent, serving real end-users, already threading each end-user's Subtext session URL into its trace metadata as `subtext_url` (see the [LangSmith integration](/docs/integrations/langsmith) — this is the load-bearing step; no `subtext_url` on the trace means no session to review later).
* A LangSmith project with tracing enabled for that agent.
* A working **Subtext MCP** endpoint.
* The official **`langsmith-mcp-server`** MCP (see the [LangSmith integration](/docs/integrations/langsmith)), or the LangSmith SDK if your triage agent's runtime doesn't support MCP.
* Somewhere to run a webhook handler — a serverless function or a small backend service reachable from LangSmith.
* Wherever you want the finding to land — Slack, a ticket queue, an internal channel.

## Set it up

<Steps>
  <Step title="Thread the end-user session into trace metadata">
    Attach `subtext_url` under `metadata` on the root run for every request your agent serves (covered in the [LangSmith integration](/docs/integrations/langsmith)). Get this in place before anything else — it's the one step everything downstream depends on.
  </Step>

  <Step title="(Optional) score runs with an online evaluator">
    If you want a quality signal beyond errors and explicit end-user feedback, add a LangSmith online evaluator — an LLM-as-judge run against each new trace — that writes a feedback key, e.g. `review_quality`, scoring whether the agent's output actually looks right. Richer signal than error/thumbs-down alone, at the cost of a model call per traced run. Skip it for a first pass if error + thumbs-down is enough.
  </Step>

  <Step title="Create a LangSmith Automation">
    In your LangSmith tracing project, click **+ New** in the top right, then **New Automation**. (Existing rules and their logs live under the project's **Automations** tab.)

    **Filter** — an FQL query over the condition you care about, e.g.:

    <CodeGroup>
      ```text Errors only theme={null}
      eq(status, "error")
      ```

      ```text Errors + thumbs-down theme={null}
      or(eq(status, "error"), and(has(feedback_key, "user_score"), lt(feedback_score, 1)))
      ```
    </CodeGroup>

    If you added an evaluator in Step 2, add `has(feedback_key, "review_quality")` so the rule waits for the score to land before firing — LangSmith's docs call out this evaluator/webhook race explicitly; without the wait, the Automation can fire before the async evaluator finishes.

    **Sampling rate** — `1.0` to review every matching run; lower it if flagged-run volume is high and you'd rather triage a sample.

    **Action** — **Webhook**, pointed at your handler's URL. Add a `?secret=` query param (or an encrypted custom header) and verify it in the handler before doing anything else — fail closed.
  </Step>

  <Step title="Stand up the webhook handler">
    **The one hard constraint:** LangSmith's webhook contract requires your endpoint to respond within **5 seconds**, or delivery is declared failed (5xx retries twice with backoff; 4xx doesn't retry; the response body is ignored). A real Subtext review takes 30–60s+, so the handler **cannot review inline** — acknowledge fast, then do the work asynchronously.

    The payload LangSmith POSTs:

    ```json theme={null}
    {
      "rule_id": "...",
      "start_time": "...",
      "end_time": "...",
      "runs": [{ "id": "...", "trace_id": "...", "...": "..." }],
      "feedback_stats": { "...": "..." }
    }
    ```

    Run dicts in the payload are lightweight — fetch the full run by id via the SDK or `langsmith-mcp-server` if you need `metadata` reliably.

    **Handler shape** (any runtime that supports "ack now, keep working"):

    ```python theme={null}
    # A background/serverless function, or a queue-backed worker — the shape is the
    # same: verify, ack, then do the real work off the request thread.
    def handle_webhook(request):
        if request.args.get("secret") != WEBHOOK_SECRET:
            return Response(status=401)

        payload = request.get_json()
        enqueue_async(review_flagged_runs, payload["runs"])  # don't await this
        return Response(status=202)  # ack within the 5s window

    def review_flagged_runs(runs):
        for run_ref in runs:
            run = langsmith_client.read_run(run_ref["id"])
            # subtext_url lives in the trace metadata. LangChain/LangGraph cascades
            # run metadata to child runs, so a flagged child span usually carries it
            # too — but a filter like eq(status, "error") can match a child span, and not
            # every setup cascades, so fall back to the trace's root run (its id ==
            # trace_id) when the flagged run itself doesn't have it.
            subtext_url = (run.extra or {}).get("metadata", {}).get("subtext_url")
            if not subtext_url and run.trace_id and run.trace_id != run.id:
                root = langsmith_client.read_run(run.trace_id)
                subtext_url = (root.extra or {}).get("metadata", {}).get("subtext_url")
            if not subtext_url:
                continue
            finding = run_triage_agent(run, subtext_url)
            post_finding(finding)  # Slack, ticket, wherever the team watches
    ```

    Two ways to run the triage agent itself:

    * **As a background LangGraph run** (LangChain-native) — if your triage agent is itself a LangGraph deployment, start a non-streamed/blocking run from the handler (a `.../runs/wait` call or the SDK equivalent) and read its final output. Keeps the triage agent on the same platform as the agent being reviewed.
    * **As a serverless background job** (portable) — a plain agent loop in the handler's own runtime that holds both MCP connectors directly. No LangGraph dependency; the simpler option if the agent you're reviewing isn't itself LangGraph-based.

    Either way, **the triage agent holds both MCPs**: the `langsmith-mcp-server` (`fetch_runs(trace_id=...)` — what your agent actually did) and the Subtext MCP (`review-open(session_url=subtext_url)` — what the end-user actually experienced). It reasons over both and decides: did the agent produce a bad result, or did the end-user hit friction the agent had nothing to do with? That's the finding worth posting.

    ```text theme={null}
    Triage prompt sketch:

    This LangSmith run was flagged (<error | low score | thumbs-down>). Trace: <trace_id>.
    End-user session: <subtext_url>.

    Use the LangSmith tools to see what the agent actually did on this run. Use the Subtext tools to
    open the end-user's session and see what they actually experienced. Decide: did the agent produce
    a bad result, or did the end-user hit friction the agent had nothing to do with? Give a
    one-sentence verdict plus the evidence from both sources, then post it.
    ```

    <Warning>
      **One agent serving two paths.** If a *single* deployed agent serves both your public end-user traffic and doubles as the triage agent, don't attach the LangSmith MCP to that shared deployment — a prompt injection planted in an end-user's own content could coerce a trace-fetching tool into enumerating other end-users' traces. Gate trace access to the triage path only (a separate route, a separate credential, or — safest — fetch the trace server-side in the handler and hand the triage run a read-only digest instead of a live LangSmith tool). If your triage agent is an internal, trusted, backend-only agent with no public-facing twin — the common case — this caveat doesn't apply: hand it both MCPs directly, per the shape above.
    </Warning>
  </Step>
</Steps>

## On-demand vs. cron

**Recommended: on-demand, via the webhook above.** Timely — the team hears about a bad run in the same window it happened — no polling, and it's what LangSmith Automations support natively.

**Fallback: daily cron.** A scheduled job that queries LangSmith once a day for flagged traces in the last 24h (`list_runs(filter=...)`, the same filter as the Automation) and batch-reviews them, for teams that would rather get one daily digest than a notification per run. Same triage-agent shape, just invoked on a schedule instead of a webhook.

## Test it

<Steps>
  <Step title="Force a flagged run">
    Throw an error in your agent, or manually log a bad feedback score on a real run via the SDK.
  </Step>

  <Step title="Confirm the Automation fires">
    Check its run history in the LangSmith UI, and confirm your webhook receives the POST within the 5s window.
  </Step>

  <Step title="Confirm the async review actually ran">
    Check your handler's logs for the `fetch_runs` / `review-open` tool calls, and confirm the finding landed in Slack (or your target destination).
  </Step>
</Steps>

You've wired it correctly when the finding names something from *both* sources — a concrete detail from the trace and a concrete detail from the session — not a paraphrase of one or the other.

## Related

* [LangSmith integration](/docs/integrations/langsmith) — attach `subtext_url` to your traces and add the `langsmith-mcp-server`.
* [Session Review overview](/docs/session-review/overview) — what your triage agent does once a session is open.