feat(assistant): trace-level scorers + server-side tool execution with needsApproval (supabase#45654)

## Motivation

When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.

An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:

> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.

We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.

## Changes

Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.

Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.

Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.

Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response

## Verification

In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.

In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.

In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))

<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>

I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.

**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval

Supercedes supabase#45556 and
supabase#45339

Closes AI-473

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.

* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
  * Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: mattrossman

apps/studio/components/ui/AIAssistantPanel/AIAssistant.tsx

@@ -1,6 +1,6 @@
 import type { UIMessage as MessageType } from '@ai-sdk/react'
 import { useChat } from '@ai-sdk/react'
-import { lastAssistantMessageIsCompleteWithToolCalls } from 'ai'
+import { lastAssistantMessageIsCompleteWithApprovalResponses } from 'ai'
 import { LOCAL_STORAGE_KEYS, useFlag } from 'common'
 import { useParams, useSearchParamsShallow } from 'common/hooks'
 import { AnimatePresence, motion } from 'framer-motion'
@@ -158,15 +158,15 @@ export const AIAssistant = ({ className }: AIAssistantProps) => {
     error,
     sendMessage,
     setMessages,
-    addToolResult,
+    addToolApprovalResponse,
     stop,
     regenerate,
   } = useChat({
     id: snap.activeChatId,
     ...(snap.activeChatId && snap.chatInstances[snap.activeChatId]
       ? { chat: snap.chatInstances[snap.activeChatId] }
       : {}),
-    sendAutomaticallyWhen: lastAssistantMessageIsCompleteWithToolCalls,
+    sendAutomaticallyWhen: lastAssistantMessageIsCompleteWithApprovalResponses,
     onError: onErrorChat,
   })
 
@@ -281,7 +281,7 @@ export const AIAssistant = ({ className }: AIAssistantProps) => {
             message={message}
             isLoading={chatStatus === 'submitted' || chatStatus === 'streaming'}
             readOnly={message.role === 'user'}
-            addToolResult={addToolResult}
+            addToolApprovalResponse={addToolApprovalResponse}
             onDelete={deleteMessageFromHere}
             onEdit={editMessage}
             isAfterEditedMessage={isAfterEditedMessage}
@@ -300,7 +300,7 @@ export const AIAssistant = ({ className }: AIAssistantProps) => {
       cancelEdit,
       editingMessageId,
       chatStatus,
-      addToolResult,
+      addToolApprovalResponse,
       handleRateMessage,
       messageRatings,
     ]

apps/studio/components/ui/AIAssistantPanel/DisplayBlockRenderer.tsx

@@ -1,6 +1,7 @@
 import { acceptUntrustedSql, type UntrustedSqlFragment } from '@supabase/pg-meta'
 import { PermissionAction } from '@supabase/shared-types/out/constants'
 import { useQueryClient } from '@tanstack/react-query'
+import type { ToolUIPart } from 'ai'
 import { useParams } from 'common'
 import { useRouter } from 'next/router'
 import { useRef, useState, type DragEvent, type PropsWithChildren } from 'react'
@@ -31,13 +32,19 @@ interface DisplayBlockRendererProps {
     yAxis?: string
   }
   initialResults?: unknown
-  onResults?: (args: { messageId: string; results: unknown }) => void
+  /** Called when locally running SQL fails before or during client-side execution. */
   onError?: (args: { messageId: string; errorText: string }) => void
-  toolState?: 'input-streaming' | 'input-available' | 'output-available' | 'output-error'
+  /** Responds affirmatively to an AI SDK tool approval request; does not run SQL directly. */
+  onApprove?: () => void
+  /** Responds negatively to an AI SDK tool approval request; does not run SQL directly. */
+  onDeny?: () => void
+  /** AI SDK tool state used to show approval UI for pending tool calls. */
+  toolState?: ToolUIPart['state']
   isLastPart?: boolean
   isLastMessage?: boolean
   showConfirmFooter?: boolean
   onChartConfigChange?: (chartConfig: ChartConfig) => void
+  /** Called when the user clicks the query block play button to run SQL locally. */
   onQueryRun?: (queryType: 'select' | 'mutation') => void
 }
 
@@ -46,8 +53,9 @@ export const DisplayBlockRenderer = ({
   toolCallId,
   initialArgs,
   initialResults,
-  onResults,
   onError,
+  onApprove,
+  onDeny,
   toolState,
   isLastPart = false,
   isLastMessage = false,
@@ -169,10 +177,6 @@ export const DisplayBlockRenderer = ({
         onSuccess: (data) => {
           setRows(Array.isArray(data.result) ? data.result : undefined)
           setIsWriteQuery(queryType === 'mutation' || initialArgs.isWriteQuery || false)
-          onResults?.({
-            messageId,
-            results: Array.isArray(data.result) ? data.result : undefined,
-          })
           if (queryType === 'mutation') {
             queryClient.invalidateQueries({ queryKey: lintKeys.lint(ref) })
             queryClient.invalidateQueries({ queryKey: entityTypeKeys.list(ref) })
@@ -219,13 +223,13 @@ export const DisplayBlockRenderer = ({
     )
   }
 
-  const resolvedHasDecision = initialResults !== undefined || rows !== undefined
   const shouldShowConfirmFooter =
     showConfirmFooter &&
-    !resolvedHasDecision &&
-    toolState === 'input-available' &&
+    toolState === 'approval-requested' &&
     isLastPart &&
-    isLastMessage
+    isLastMessage &&
+    !!onApprove &&
+    !!onDeny
 
   return (
     <div className="display-block w-auto overflow-x-hidden">
@@ -252,12 +256,8 @@ export const DisplayBlockRenderer = ({
             cancelLabel="Skip"
             confirmLabel={executeSqlLoading ? 'Running...' : 'Run Query'}
             isLoading={executeSqlLoading}
-            onCancel={async () => {
-              onResults?.({ messageId, results: 'User skipped running the query' })
-            }}
-            onConfirm={() => {
-              handleExecute(isWriteQuery ? 'mutation' : 'select')
-            }}
+            onCancel={onDeny}
+            onConfirm={onApprove}
           />
         </div>
       )}

apps/studio/components/ui/AIAssistantPanel/EdgeFunctionRenderer.test.tsx

@@ -0,0 +1,136 @@
+import { screen } from '@testing-library/react'
+import userEvent from '@testing-library/user-event'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+
+import { EdgeFunctionRenderer } from './EdgeFunctionRenderer'
+import { render } from '@/tests/helpers'
+
+const {
+  mockSendEvent,
+  mockUseEdgeFunctionQuery,
+  mockUseParams,
+  mockUseProjectSettingsV2Query,
+  mockUseSelectedOrganizationQuery,
+} = vi.hoisted(() => ({
+  mockSendEvent: vi.fn(),
+  mockUseEdgeFunctionQuery: vi.fn(),
+  mockUseParams: vi.fn(),
+  mockUseProjectSettingsV2Query: vi.fn(),
+  mockUseSelectedOrganizationQuery: vi.fn(),
+}))
+
+vi.mock('common', async () => {
+  const actual = await vi.importActual<typeof import('common')>('common')
+
+  return {
+    ...actual,
+    useParams: mockUseParams,
+  }
+})
+
+vi.mock('@/data/config/project-settings-v2-query', () => ({
+  useProjectSettingsV2Query: mockUseProjectSettingsV2Query,
+}))
+
+vi.mock('@/data/edge-functions/edge-function-query', () => ({
+  useEdgeFunctionQuery: mockUseEdgeFunctionQuery,
+}))
+
+vi.mock('@/data/telemetry/send-event-mutation', () => ({
+  useSendEventMutation: () => ({ mutate: mockSendEvent }),
+}))
+
+vi.mock('@/hooks/misc/useSelectedOrganization', () => ({
+  useSelectedOrganizationQuery: mockUseSelectedOrganizationQuery,
+}))
+
+vi.mock('../EdgeFunctionBlock/EdgeFunctionBlock', () => ({
+  EdgeFunctionBlock: ({
+    showReplaceWarning,
+    onCancelReplace,
+    onConfirmReplace,
+  }: {
+    showReplaceWarning?: boolean
+    onCancelReplace?: () => void
+    onConfirmReplace?: () => void
+  }) => (
+    <div>
+      {showReplaceWarning && (
+        <div>
+          <p>An edge function with this name already exists.</p>
+          <button onClick={onCancelReplace}>Cancel</button>
+          <button onClick={onConfirmReplace}>Replace function</button>
+        </div>
+      )}
+    </div>
+  ),
+}))
+
+vi.mock('./ConfirmFooter', () => ({
+  ConfirmFooter: ({
+    confirmLabel,
+    onConfirm,
+  }: {
+    confirmLabel?: string
+    onConfirm?: () => void
+  }) => <button onClick={onConfirm}>{confirmLabel ?? 'Confirm'}</button>,
+}))
+
+describe('EdgeFunctionRenderer', () => {
+  beforeEach(() => {
+    mockSendEvent.mockReset()
+    mockUseEdgeFunctionQuery.mockReset()
+    mockUseParams.mockReturnValue({ ref: 'project-ref' })
+    mockUseProjectSettingsV2Query.mockReturnValue({ data: undefined })
+    mockUseSelectedOrganizationQuery.mockReturnValue({ data: { slug: 'org-slug' } })
+  })
+
+  it('only deploys an existing function from the replace warning confirmation', async () => {
+    const user = userEvent.setup()
+    const onApprove = vi.fn()
+
+    mockUseEdgeFunctionQuery.mockReturnValue({ data: { slug: 'hello-world' } })
+
+    render(
+      <EdgeFunctionRenderer
+        label="Deploy Edge Function"
+        code="Deno.serve(() => new Response('ok'))"
+        functionName="hello-world"
+        onApprove={onApprove}
+      />
+    )
+
+    await user.click(screen.getByRole('button', { name: 'Deploy' }))
+    expect(screen.getByText('An edge function with this name already exists.')).toBeInTheDocument()
+    expect(onApprove).not.toHaveBeenCalled()
+
+    await user.click(screen.getByRole('button', { name: 'Deploy' }))
+    expect(onApprove).not.toHaveBeenCalled()
+    expect(mockSendEvent).not.toHaveBeenCalled()
+
+    await user.click(screen.getByRole('button', { name: 'Replace function' }))
+    expect(onApprove).toHaveBeenCalledTimes(1)
+    expect(mockSendEvent).toHaveBeenCalledTimes(1)
+  })
+
+  it('deploys immediately when no existing function is found', async () => {
+    const user = userEvent.setup()
+    const onApprove = vi.fn()
+
+    mockUseEdgeFunctionQuery.mockReturnValue({ data: undefined })
+
+    render(
+      <EdgeFunctionRenderer
+        label="Deploy Edge Function"
+        code="Deno.serve(() => new Response('ok'))"
+        functionName="hello-world"
+        onApprove={onApprove}
+      />
+    )
+
+    await user.click(screen.getByRole('button', { name: 'Deploy' }))
+
+    expect(onApprove).toHaveBeenCalledTimes(1)
+    expect(mockSendEvent).toHaveBeenCalledTimes(1)
+  })
+})