Add docs for LangSmith Polly

src/docs.json

@@ -844,6 +844,7 @@
                 {
                   "group": "Additional resources",
                   "pages": [
+                    "langsmith/polly",
                     {
                       "group": "Releases & changelogs",
                       "pages": [

src/langsmith/create-a-prompt.mdx

@@ -36,6 +36,10 @@ Adding an output schema to your prompt will get output in a structured format. L
 
 You can also add a tool by clicking the `+ Tool` button at the bottom of the prompt editor. See [here](/langsmith/use-tools) for more information on how to use tools.
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** in the Playground to generate tools, create output schemas, and optimize your prompts with AI assistance.
+</Callout>
+
 ## Run the prompt
 
 Click "Start" to run the prompt.

src/langsmith/observability-concepts.mdx

@@ -55,6 +55,10 @@ To group traces into threads, you pass a special metadata key (`session_id`, `th
     alt="Thread representing a sequence of traces in a multi-turn conversation."
 />
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** to analyze traces, runs, and threads. Polly helps you understand agent performance, debug issues, and gain insights from conversation threads without manually digging through data.
+</Callout>
+
 ## Projects
 
 A _project_ is a collection of traces. You can think of a project as a container for all the traces that are related to a single application or service. You can have multiple projects, and each project can have multiple traces.

src/langsmith/observability-quickstart.mdx

@@ -379,6 +379,10 @@ Here are some topics you might want to explore next:
 - [Trace a RAG application](/langsmith/observability-llm-tutorial) is a full tutorial, which adds observability to an application from development through to production.
 - [Sending traces to a specific project](/langsmith/log-traces-to-project) changes the destination project of your traces.
 
+<Callout type="info" icon="bird">
+After logging traces, use **[Polly](/langsmith/polly)** to analyze them and get AI-powered insights into your application's performance.
+</Callout>
+
 ## Video guide
 <iframe
   className="w-full aspect-video rounded-xl"

src/langsmith/observability.mdx

@@ -66,4 +66,8 @@ The following sections help you set up and use tracing, monitoring, and observab
 
 For terminology definitions and core concepts, refer to [Observability concepts](/langsmith/observability-concepts).
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)**, LangSmith's AI assistant, to analyze traces and get AI-powered insights into your application's performance.
+</Callout>
+
 <HostingSetup/>

src/langsmith/polly.mdx

@@ -0,0 +1,127 @@
+---
+title: LangSmith Polly
+sidebarTitle: Polly (Beta)
+---
+
+<Callout color="#4F46E5">
+**Polly is in beta.** Your [feedback](https://forum.langchain.com) on Polly is invaluable as the team refines its capabilities.
+</Callout>
+
+**LangSmith Polly** is an AI assistant embedded directly in your LangSmith [workspace](/langsmith/administration-overview#workspaces) to help you analyze and understand your application data.
+
+Polly helps you gain insight from your traces, conversation threads, and prompts without having to dig through data manually. By asking natural language questions, you can quickly understand agent performance, debug issues, and analyze user sentiment.
+
+<img src="/langsmith/images/polly.png" alt="LangSmith Polly icon" style={{float: 'left', marginRight: '20px', marginTop: '-1px', marginBottom: '20px', maxWidth: '100px'}} /> Polly appears in the right-hand bottom corner of the following locations within [LangSmith UI](https://smith.langchain.com), optimized for different use cases:
+
+- [Trace pages](#tracing-page)
+- [Thread views](#thread-views)
+- [Prompt Playground](#prompt-playground)
+
+### Tracing page
+
+On an individual [trace](/langsmith/observability-concepts#traces), Polly pulls in the context of the page and analyzes the [run](/langsmith/observability-concepts#runs). Polly reads the run data and trajectory to help you understand what happened and identify areas for improvement.
+
+To ask Polly about your tracing:
+
+1. In your **Tracing Projects**, click on a trace to view its details page.
+1. Select a run in the trace.
+1. Open Polly in the right-hand corner of the page to ask questions relating to this run.
+1. Ask Polly a question about your data. You can use the sample questions or you might ask questions like:
+
+    - "Is there anything that the agent could have done better here?"
+    - "Why did this run fail?"
+    - "What took the most time in this trace?"
+    - "What errors occurred during this run?"
+    - "Summarize what happened in this trace"
+
+When analyzing runs, Polly will examine the full trace context, including [run metadata](/langsmith/observability-concepts#metadata), inputs, outputs, intermediate steps, and configuration to provide actionable insights. This helps you diagnose issues without manually expanding each step in the trace tree or cross-referencing multiple runs.
+
+### Thread view
+
+Under the **Threads** tab, Polly analyzes conversation [threads](/langsmith/observability-concepts#threads) by pulling in relevant information about the user interaction. This helps you understand user sentiment and conversation outcomes.
+
+To ask Polly about your threads:
+
+1. Select a thread.
+1. Open Polly in the right-hand corner of the page to ask questions relating to this thread.
+1. Ask Polly a question about the conversation thread. You might ask questions like:
+
+    - "Did the user seem frustrated?"
+    - "What issues is the user experiencing?"
+    - "How did this conversation resolve?"
+    - "Was the user's problem solved?"
+    - "What was the main topic of this thread?"
+
+Use Polly in thread view to gain insights into how users are interacting with your application. Understand conversation outcomes and whether issues were resolved, identify common user pain points, and track user sentiment through thread analysis. This helps you improve user experience by understanding what's working and what needs improvement in your application's responses.
+
+### Prompt Playground
+
+When you open a [prompt](/langsmith/prompt-engineering-concepts#prompt-in-langsmith) in the [Playground](/langsmith/prompt-engineering-concepts#prompt-playground), Polly can help you edit and improve your prompts based on your instructions. Polly reads the prompt and makes suggested edits.
+
+To ask Polly about your prompt:
+
+1. Enter the **Playground** from the left-hand navigation or trace view.
+1. Select a prompt to experiment with.
+1. Open Polly in the right-hand corner of the page to work on this prompt.
+1. You can use one of the automated options that Polly suggests:
+    - <Icon icon="play" /> **Optimize prompt**: Polly will analyze the current prompt and make edits to the prompt with a summary of the changes.
+    - <Icon icon="wrench" /> **Generate a tool**: Give details to Polly on the tool you would like to add. It will generate a tool for your prompt template. It can also help you modify existing tooling or system messages about tooling. Then, have Polly test tool configurations with reviews of sample output from the model using the tool.
+    - <Icon icon="brackets-curly" /> **Generate an output schema**: Polly will create a JSON schema that defines the structure of the output you want the model to generate. This is useful when you need the model to return data in a specific format. Select this option, and then provide Polly with the type of data, fields/properties, and any other constraints you might need.
+
+    Or, you might ask your own questions, like:
+
+    - "Make it respond in Italian"
+    - "Add more context about the user's role"
+    - "Make the tone more professional"
+    - "Simplify the instructions"
+    - "Add examples to the prompt"
+
+    <img
+    className="block dark:hidden"
+    src="/langsmith/images/polly-prompt-tool.png"
+    alt="Prompt Playground showing Polly chat in the sidebar with information on a generated tool."
+    />
+
+    <img
+    className="hidden dark:block"
+    src="/langsmith/images/polly-prompt-tool-dark.png"
+    alt="Prompt Playground showing Polly chat in the sidebar with information on a generated tool."
+    />
+
+## What's next
+
+Learn more about the features that Polly helps you explore:
+
+<CardGroup cols={2}>
+  <Card
+    title="Observability"
+    icon="magnifying-glass"
+    href="/langsmith/observability"
+  >
+    Learn more about tracing and monitoring your LLM applications
+  </Card>
+
+  <Card
+    title="Threads"
+    icon="comments"
+    href="/langsmith/threads"
+  >
+    Understand how threads work in LangSmith
+  </Card>
+
+  <Card
+    title="Prompt Engineering"
+    icon="wand-magic-sparkles"
+    href="/langsmith/prompt-engineering"
+  >
+    Create and iterate on prompts in the playground
+  </Card>
+
+  <Card
+    title="Evaluation"
+    icon="clipboard-check"
+    href="/langsmith/evaluation"
+  >
+    Evaluate and test your applications systematically
+  </Card>
+</CardGroup>

src/langsmith/prompt-engineering-concepts.mdx

@@ -131,6 +131,10 @@ In the playground you can:
 * Run the prompt through the model
 * Observe the outputs
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** in the Playground to optimize prompts, generate tools, and create output schemas with AI assistance.
+</Callout>
+
 ## Testing multiple prompts
 
 You can add more prompts to your playground to easily compare outputs and decide which version is better:

src/langsmith/prompt-engineering-quickstart.mdx

@@ -350,6 +350,10 @@ To improve your prompts:
 - Learn how to set up the Playground to [Test multi-turn conversations](/langsmith/multiple-messages) in this tutorial.
 - Learn how to test your prompt's performance over a dataset instead of individual examples, refer to [Run an evaluation from the Prompt Playground](/langsmith/run-evaluation-from-prompt-playground).
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** in the Playground to help optimize your prompts, generate tools, and create output schemas.
+</Callout>
+
 ## Video guide
 <iframe
   className="w-full aspect-video rounded-xl"

src/langsmith/prompt-engineering.mdx

@@ -65,4 +65,8 @@ The following sections help you create, manage, and optimize your prompts:
 
 </Columns>
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** in the Prompt Playground to optimize prompts, generate tools, and create output schemas with AI-powered assistance.
+</Callout>
+
 <HostingSetup/>

src/langsmith/run-evaluation-from-prompt-playground.mdx

@@ -11,6 +11,10 @@ If you prefer to run experiments in code, visit [run an evaluation using the SDK
 
 ![](/langsmith/images/playground-experiment.gif)
 
+<Callout type="info" icon="bird">
+**[Polly](/langsmith/polly)** is available in the Playground to help you optimize prompts before running evaluations.
+</Callout>
+
 ## Create an experiment in the prompt playground[​](#create-an-experiment-in-the-prompt-playground "Direct link to Create an experiment in the prompt playground")
 
 1. **Navigate to the playground** by clicking **Playground** in the sidebar.

src/langsmith/threads.mdx

@@ -302,6 +302,9 @@ You can view threads by clicking on the **Threads** tab in any project details p
 />
 </div>
 
+<Callout type="info" icon="bird">
+Use **[Polly](/langsmith/polly)** in thread views to analyze conversation threads, understand user sentiment, identify pain points, and track whether issues were resolved.
+</Callout>
 
 ### View a thread