LangSmith and Subtext: From a Signal to the Session

LangSmith tells you that a run misbehaved: it errored, scored low on an eval, or a human thumbs-downed it. It can’t tell you whether the person on the other end was actually blocked, confused, or fine. Subtext can. Attach the Subtext session URL to your LangSmith trace metadata and every flagged run becomes a jumping-off point — your agent reads the subtext_url, opens that person’s real session, and walks the decisive moments with screenshots, the component tree, network, and console. LangSmith detects; Subtext diagnoses.

This guide assumes the Subtext capture snippet is installed and the langsmith SDK is installed with tracing enabled (LANGSMITH_TRACING=true, LANGSMITH_API_KEY set), or that you run on LangSmith Deployment (formerly LangGraph Platform).

Attach the session URL

LangSmith is the exception to Subtext’s flat-property convention. Most integrations attach subtext_url as a top-level property on a user, session, or event record. A LangSmith run has no such column — its context lives in metadata, a dict attached at run-create time. Attach subtext_url there, nested under metadata, on the root run (the top of the trace) so anything reading the trace’s metadata sees it without walking every child run. Which surface you use depends on how your agent runs. The URL itself originates in the browser. Capture it with FS('getSession', { format: 'url.now' }) — it returns null until the session has started, so read it after init (and re-read after a re-identify), then send it to your backend with the request that starts the agent run:

const subtextUrl = FS('getSession', { format: 'url.now' })
// include subtextUrl in the request body that kicks off the agent run

`@traceable` (Python decorator)

A static metadata= on the decorator is fixed at import time, which is no use for a per-end-user URL. Pass it per call instead, via langsmith_extra:

from langsmith import traceable

@traceable
def run_agent(user_input: str):
    ...

def handle_request(user_input: str, subtext_url: str | None):
    return run_agent(
        user_input,
        langsmith_extra={"metadata": {"subtext_url": subtext_url}} if subtext_url else {},
    )

`RunTree` (manual instrumentation)

from langsmith.run_trees import RunTree

run = RunTree(
    name="agent_run",
    run_type="chain",
    inputs={"input": user_input},
    extra={"metadata": {"subtext_url": subtext_url}},
)

`config.metadata` (LangChain / LangGraph runnables)

The common case: your agent is a LangChain or LangGraph runnable invoked once per end-user request. Thread the session URL through the invoke config — the active LangChainTracer reads config["metadata"] and writes it onto the run automatically, no direct tracer call needed.

result = agent.invoke(
    {"messages": [...]},
    config={"metadata": {"subtext_url": subtext_url}},
)

const result = await agent.invoke({ messages: [...] }, { metadata: { subtext_url: subtextUrl } })

LangSmith Deployment (deployed assistants)

If your agent runs as a LangSmith Deployment, attach the same key in the run-create call’s top-level metadata field, whether from the SDK or REST directly. This is the surface a backend or proxy uses when it — not the browser — holds both the end-user’s session identity and the call that starts the agent run.

from langgraph_sdk import get_client

client = get_client(url=DEPLOYMENT_URL)
run = await client.runs.create(
    thread_id,
    assistant_id,
    input={"messages": [...]},
    metadata={"subtext_url": subtext_url},
)

POST /threads/{thread_id}/runs
{
  "assistant_id": "agent",
  "input": { "messages": [...] },
  "metadata": { "subtext_url": "<subtext_url>" }
}

Don’t expect subtext_url to appear as a filterable top-level column — it’s nested under metadata, so a query needs FQL like eq(metadata_key, "subtext_url"), not a bare property filter. Don’t attach it only on a child run: a person reading the root-run metadata panel, or an agent fetching the trace by id, won’t see it there. And don’t store a raw session id where the URL belongs — attach the clickable viewer URL, since that is what gets opened directly. The value written at run-create is frozen, so if the same end-user drives a multi-turn thread, re-attach the current subtext_url on each new run — in the invoke config["metadata"] for runnables, or the run-create metadata field on LangSmith Deployment.

From signal to session

However you first hear about a bad run in LangSmith — a flagged trace, an eval regression, an error spike — the path is the same: get to the subtext_url in the run’s metadata, then hand it to your agent for a Detect-vs-Diagnose read. If you land on a child run and don’t see it, jump to the root run, where it’s attached.

A trace flagged low-quality in the UI

Scanning the traces table (or an annotation queue) in your LangSmith project, you spot a run someone tagged low-quality, or thumbs-downed from your product’s feedback widget.Open the run, go to the Metadata panel, and copy the subtext_url value. If you’re on a child run and don’t see it, jump to the root run.

This LangSmith run got flagged low-quality (thumbs-down from the end-user). Here's their session:
<PASTE subtext_url>

Open it and give me a Detect-vs-Diagnose read. Did the agent actually produce a bad outcome, or
did the person hit something else — a slow page, a confusing next step, an unrelated bug? Point to
the exact moment they reacted, and tell me whether this is an agent-quality issue or a
product-friction issue.

An eval-dashboard regression

An online evaluator’s score (say answer_correctness, or your own review_quality feedback key) drops week-over-week on the LangSmith experiments dashboard.Filter the project’s runs by the low-scoring feedback key, open a handful of the worst-scoring runs, and copy subtext_url off each one’s metadata.

Our eval score for <feedback_key> dropped this week. Here are sessions from the lowest-scoring
runs: <PASTE 3-4 subtext_urls>

Review them and find the common thread. Is the eval catching a real regression in what the agent
produced, or are these sessions where the person's own input was already unusual? Give me one
synthesized hypothesis for the drop.

An error spike on a chain

A spike in errored runs on a specific chain or node — LangSmith’s own monitoring, or an Automation-driven Slack/webhook alert, flags it.Filter runs by error for the affected chain in the window, open a few, and copy subtext_url off each.

Errors spiked on <chain name> in the last 15 minutes. Here are sessions from a few of the errored
runs: <PASTE 3-4 subtext_urls>

Walk each session up to the point the error would have surfaced. Did the person actually see a
broken experience, or did the agent recover or retry silently? Tell me if this needs a rollback or
if it's contained.

For a spike or regression across several runs, ask your agent to roll the findings into one proof document (doc-create) with a screenshot per session and a single shared link — then paste the synthesis back onto the run as a LangSmith comment or annotation, so the score or error carries a human explanation, not just a number.

For agents

An autonomous agent can run the whole loop against LangSmith directly: discover the flagged runs, extract the subtext_url from trace metadata, and hand each session to the Subtext MCP. LangChain ships an official MCP server, langsmith-mcp-server — run it locally over stdio (pip install langsmith-mcp-server or uvx langsmith-mcp-server) or in Docker, authenticated with LANGSMITH_API_KEY. Repo: github.com/langchain-ai/langsmith-mcp-server; docs: docs.langchain.com/langsmith/langsmith-mcp-server.

Discover flagged runs

With the MCP, fetch_runs is the discovery tool — fetch a trace tree by trace_id, filtering by error, run_type, or a raw FQL (Filter Query Language) query. Without the MCP, Client.list_runs takes the same FQL filter:

from langsmith import Client

client = Client()

# errored runs
errored = client.list_runs(project_name="your-project", filter='eq(status, "error")')

# thumbs-down / low score on a feedback key you log (e.g. user_score, review_quality)
low_score = client.list_runs(
    project_name="your-project",
    filter='and(eq(feedback_key, "user_score"), lt(feedback_score, 1))',
)

See the trace query syntax reference for the full filter grammar.

Extract the session URL

Read subtext_url off the flagged run’s root-run metadata. A run fetched from deep in the trace tree may not carry it, so walk up to the root run (no parent_run_id) if the child run you matched doesn’t have it:

for run in errored:
    metadata = run.extra.get("metadata", {})
    subtext_url = metadata.get("subtext_url")
    if subtext_url:
        ...  # hand off below

Hand off to Subtext

For each URL, call review-open(session_url=<subtext_url>) on the Subtext MCP, then step through at the flagged moment for a Detect-vs-Diagnose read. Batch several runs’ URLs to review a spike or regression together for the common cause.

subtext_url only round-trips if it was attached to the root run at trace-create time (see Attach the session URL). It’s metadata, not a top-level column, so discover by error, score, or feedback first, then read subtext_url off the result. Treat a missing value as “not instrumented,” not an error.

Session Review overview — what your agent does once a session is open.
Install the capture snippet — required before any session URL exists to attach.

Get Started

Install by Framework

Session Review

Privacy

Integrations

Recipes

API reference

LangSmith and Subtext: From a Signal to the Session

Attach the session URL

`@traceable` (Python decorator)

`RunTree` (manual instrumentation)

`config.metadata` (LangChain / LangGraph runnables)

LangSmith Deployment (deployed assistants)

From signal to session

For agents

​Attach the session URL

​@traceable (Python decorator)

​RunTree (manual instrumentation)

​config.metadata (LangChain / LangGraph runnables)

​LangSmith Deployment (deployed assistants)

​From signal to session

​For agents

​Related

Attach the session URL

`@traceable` (Python decorator)

`RunTree` (manual instrumentation)

`config.metadata` (LangChain / LangGraph runnables)

LangSmith Deployment (deployed assistants)

From signal to session

For agents

Related