feat: observe-act flow + stagehand v4 upgrade

.changeset/browser-act-replay.md

@@ -0,0 +1,13 @@
+---
+"@upstash/box": minor
+---
+
+feat: replay a resolved browser action with `tab.act(action)` (no LLM, no key)
+
+`observe()` now returns each element's suggested `method` and `arguments`
+alongside `selector`, and `act()` accepts a pre-resolved action
+(`BrowserObserveElement` or `BrowserActAction`) in addition to a natural-language
+string. Passing an action replays it deterministically: no LLM call, no tokens,
+and no model provider key required. Resolve a step once with `observe()`, cache
+the returned action, and replay it across pages or runs. A new `BrowserAction`
+type (`BrowserObserveElement | BrowserActAction`) is exported.

.changeset/remove-browser-run.md

@@ -0,0 +1,7 @@
+---
+"@upstash/box": minor
+---
+
+**Breaking:** remove `tab.run()` (the autonomous multi-step browser agent) and the `BrowserRunOptions` / `BrowserRunResult` / `BrowserRunStep` types.
+
+Stagehand v4 removed the underlying agent primitive, so the DOM-aware browser now exposes `observe`, `act`, and `extract` only. For multi-step goals, drive `act` / `observe` from your own loop (replay a resolved step with `act(action)` for no-LLM, no-key execution), or connect over CDP with Playwright / Puppeteer.

packages/python-sdk/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to `upstash-box` (Python) are documented here.
 
 ## Unreleased
 
+- **Removed** `tab.run()` (the autonomous multi-step browser agent) and the
+  `BrowserRunResult` / `BrowserRunStep` types. Stagehand v4 removed the agent
+  primitive, so the browser exposes `observe` / `act` / `extract` only. For
+  multi-step goals, drive `act` / `observe` from your own loop (replay a
+  resolved step with `act(action)`), or drive Playwright / Puppeteer over CDP.
+  Mirrors `@upstash/box`.
+- `tab.act(action)` — replay a pre-resolved action from `observe()`
+  deterministically, with no LLM call, no tokens, and no model provider key
+  required (pass a `BrowserObserveElement` or `BrowserActAction` instead of a
+  string; `model` is ignored in that form). `BrowserObserveElement` now also
+  carries `method` and `arguments`. Mirrors `act(action)` in `@upstash/box`.
 - `browser.recordings.download(recording_id, path=...)` — save a recording's
   video to a local file (streamed to disk, parent directories created as
   needed) and return the path written. Recordings download as MP4; recordings

packages/python-sdk/PARITY.md

@@ -51,7 +51,6 @@ JS `Run`/`StreamRun` → Python `Run`/`StreamRun` (+ `AsyncRun`/`AsyncStreamRun`
 | `goto`, `content`, `observe`, `act`, `close` | same |
 | `screenshot({type, fullPage})` | `screenshot(encoding=, full_page=)` — see drift table |
 | `extract(instruction, schema, options?)` | `extract(instruction, schema, *, model=None)` |
-| `run(prompt, options?)` | `run(prompt, *, schema=None, max_steps=None, model=None)` |
 | `liveViewUrl` | `live_view_url` |
 | `id`, `url`, `title` | `id`, `url`, `title` |
 
@@ -94,7 +93,6 @@ statics `create`, `from_snapshot`, `get_by_name`, `delete_boxes`,
 | timeouts in **milliseconds** | Matches the JS SDK units. |
 | agent `options` keys are **snake_case** (Python) vs camelCase (JS) | Pythonic public API; the SDK converts to the backend's per-harness casing (Claude Code / OpenCode → camelCase, Codex → snake_case). |
 | `StreamRun.aclose()` needed for `detached` on early break | Python doesn't run generator `finally` on `break` (JS `for await` does). |
-| `BrowserRunOptions.prompt` (JS, deprecated) | Legacy `run({prompt})` overload — not ported; Python takes the prompt as the first argument only. |
 | Browser `schema` = Pydantic model or raw dict (Python) vs Zod (JS) | Same `ResponseSchema` contract as `agent.run`; raw dicts skip client-side validation. |
 | `screenshot` `type: "png"\|"base64"` (JS) → `encoding: "bytes"\|"base64"` (Python) | Python returns native `bytes`; `encoding` matches `files.read` naming. |
 | `browser` on `from_snapshot` (Python) | Python's shared create-body builder forwards `browser=True` on `from_snapshot`; JS `fromSnapshot` currently omits it (JS gap). |

packages/python-sdk/README.md

@@ -260,7 +260,6 @@ class Headline(BaseModel):
 data = await tab.extract("Get the top headline", Headline)
 actions = await tab.observe("What can I click?")
 await tab.act("Click the first headline")
-result = await tab.run("Find the top comment and summarize it", max_steps=10)
 
 print(await tab.live_view_url())  # watch the tab live
 print(await box.browser.cdp_url())  # connect Playwright/Puppeteer over CDP

packages/python-sdk/examples/browser.py

@@ -1,7 +1,7 @@
 """Headless browser — box.browser.* on a lightweight box.
 
 Creating a box with ``browser=True`` provisions Chromium; all page operations
-(content, screenshots, AI extract/act/run) work headless. Mirrors the JS
+(content, screenshots, AI extract/act) work headless. Mirrors the JS
 examples ``browser.ts`` / ``headless-browser.ts``.
 
 Run: python examples/browser.py  (needs UPSTASH_BOX_API_KEY)

packages/python-sdk/tests/_async/test_box_browser.py

@@ -261,87 +261,62 @@ async def test_act_executes_one_action():
     await box.aclose()
 
 
-class Person(BaseModel):
-    name: str
-    headline: str
-    profile_url: str
-
+@respx.mock
+async def test_act_rejects_action_without_selector():
+    from upstash_box import BrowserObserveElement
 
-class People(BaseModel):
-    people: list[Person]
+    box = await make_async_box(respx.mock)
+    with pytest.raises(BoxError):
+        await box.browser.get_tab("tab-2").act(BrowserObserveElement(description="unresolved"))
+    await box.aclose()
 
 
 @respx.mock
-async def test_run_with_schema_validated_structured_output():
+async def test_act_replays_pre_resolved_action():
+    from upstash_box import BrowserActAction
+
     box = await make_async_box(respx.mock)
-    run = respx.post(f"{BASE}/browser/run").mock(
+    act = respx.post(f"{BASE}/browser/act").mock(
         return_value=httpx.Response(
             200,
             json={
-                "result": "Found five people",
-                "data": {
-                    "people": [
-                        {
-                            "name": f"Founder {i + 1}",
-                            "headline": "AI founder in Berlin",
-                            "profile_url": f"https://linkedin.com/in/founder-{i + 1}",
-                        }
-                        for i in range(5)
-                    ]
-                },
-                "completed": True,
-                "steps": [{"step": 1, "action": "search", "url": "https://linkedin.com/search"}],
-                "step_count": 1,
-                "input_tokens": 100,
-                "output_tokens": 25,
+                "success": True,
+                "message": "done",
+                "action_description": "Sign in",
+                "actions": [],
+                "input_tokens": 0,
+                "output_tokens": 0,
             },
         )
     )
 
-    result = await box.browser.get_tab("tab-2").run(
-        "Find five AI founders in Berlin", schema=People, max_steps=25
-    )
-
-    assert isinstance(result.data, People)
-    assert len(result.data.people) == 5
-    assert result.completed is True
-    assert result.steps[0].action == "search"
-    body = last_json_body(run)
-    assert body["prompt"] == "Find five AI founders in Berlin"
-    assert body["tab"] == "tab-2"
-    assert body["max_steps"] == 25
-    assert body["schema"]["type"] == "object"
-    await box.aclose()
-
-
-@respx.mock
-async def test_run_without_schema():
-    box = await make_async_box(respx.mock)
-    run = respx.post(f"{BASE}/browser/run").mock(
-        return_value=httpx.Response(
-            200, json={"result": "done", "completed": True, "steps": [], "step_count": 3}
-        )
+    action = BrowserActAction(
+        selector="xpath=/html/body/button", description="Sign in", method="click", arguments=[]
     )
+    result = await box.browser.get_tab("tab-2").act(action)
 
-    result = await box.browser.get_tab("tab-1").run("Do the thing")
-
-    assert result.data is None
-    assert result.result == "done"
-    assert result.completed is True
-    assert result.step_count == 3
-    assert last_json_body(run) == {"prompt": "Do the thing", "tab": "tab-1"}
+    assert result.success is True
+    assert result.input_tokens == 0
+    # Posts a pre-resolved action, never an instruction.
+    assert last_json_body(act) == {
+        "action": {
+            "selector": "xpath=/html/body/button",
+            "description": "Sign in",
+            "method": "click",
+            "arguments": [],
+        },
+        "tab": "tab-2",
+    }
     await box.aclose()
 
 
 @respx.mock
-async def test_rejects_non_schema_for_extract_and_run():
+async def test_rejects_non_schema_for_extract():
     box = await make_async_box(respx.mock)
     tab = box.browser.get_tab("tab-1")
 
     with pytest.raises(BoxError, match="extract requires"):
         await tab.extract("get data", "not-a-schema")  # type: ignore[arg-type]
-    with pytest.raises(BoxError, match="run requires"):
-        await tab.run("go", schema="not-a-schema")  # type: ignore[arg-type]
     await box.aclose()
 
 

packages/python-sdk/upstash_box/__init__.py

@@ -66,8 +66,6 @@
     BrowserObserveResult,
     BrowserRecording,
     BrowserRecordingMarker,
-    BrowserRunResult,
-    BrowserRunStep,
     Chunk,
     ClaudeCode,
     ClaudeCodeAgentOptions,
@@ -219,8 +217,6 @@
     "BrowserObserveResult",
     "BrowserRecording",
     "BrowserRecordingMarker",
-    "BrowserRunResult",
-    "BrowserRunStep",
     "EphemeralBoxData",
     "FileEntry",
     "GitCommitResult",

packages/python-sdk/upstash_box/_async/client.py

@@ -42,11 +42,12 @@
     BoxData,
     BoxGetOptions,
     BoxRunData,
+    BrowserActAction,
     BrowserActResult,
     BrowserContent,
+    BrowserObserveElement,
     BrowserObserveResult,
     BrowserRecording,
-    BrowserRunResult,
     Chunk,
     CodeLanguage,
     CustomHarnessConfig,
@@ -589,55 +590,32 @@ async def observe(
         )
         return BrowserObserveResult.model_validate({"elements": resp.get("elements") or []})
 
-    async def act(self, instruction: str, *, model: Optional[str] = None) -> BrowserActResult:
-        """Resolve and execute one natural-language action on this tab (metered)."""
-        body: Dict[str, Any] = {"instruction": instruction, "tab": self.id}
-        if model:
-            body["model"] = model
-        resp = await self._box._request(
-            "POST",
-            f"/v2/box/{self._box.id}/browser/act",
-            body=body,
-            timeout=180000,
-        )
-        return BrowserActResult.model_validate(resp)
-
-    async def run(
+    async def act(
         self,
-        prompt: str,
+        instruction: Union[str, BrowserObserveElement, BrowserActAction],
         *,
-        schema: Optional[ResponseSchema] = None,
-        max_steps: Optional[int] = None,
         model: Optional[str] = None,
-    ) -> BrowserRunResult:
-        """Autonomously complete a multi-step task on this tab (metered).
+    ) -> BrowserActResult:
+        """Resolve and execute one action on this tab.
 
-        Runs a DOM-aware browser agent (Stagehand) inside the box: it reads the
-        page, acts, and repeats until done. ``max_steps`` defaults to 15
-        (max 30). Needs a key for the model's provider on the box or account.
+        Pass a string (LLM-resolved, metered) or a pre-resolved ``observe()``
+        action to replay it with no LLM call and no key (``model`` ignored).
         """
-        json_schema = common.to_json_schema(schema) if schema is not None else None
-        if schema is not None and json_schema is None:
-            raise BoxError("run requires a pydantic model class or a JSON-schema dict")
-        body: Dict[str, Any] = {"prompt": prompt, "tab": self.id}
-        if json_schema is not None:
-            body["schema"] = json_schema
-        if max_steps:
-            body["max_steps"] = max_steps
-        if model:
-            body["model"] = model
+        if isinstance(instruction, str):
+            body: Dict[str, Any] = {"instruction": instruction, "tab": self.id}
+            if model:
+                body["model"] = model
+        else:
+            if not instruction.selector:
+                raise BoxError("act(action) requires a selector; observe() did not resolve one")
+            body = {"action": instruction.model_dump(exclude_none=True), "tab": self.id}
         resp = await self._box._request(
             "POST",
-            f"/v2/box/{self._box.id}/browser/run",
+            f"/v2/box/{self._box.id}/browser/act",
             body=body,
-            timeout=600000,
-        )
-        data = (
-            common.validate_structured_data(schema, resp.get("data"))
-            if schema is not None
-            else None
+            timeout=180000,
         )
-        return BrowserRunResult.model_validate({**resp, "data": data})
+        return BrowserActResult.model_validate(resp)
 
     async def live_view_url(self) -> str:
         """Live-view URL for this tab (authenticated via a token in the URL).

packages/python-sdk/upstash_box/_sync/client.py

@@ -41,11 +41,12 @@
     BoxData,
     BoxGetOptions,
     BoxRunData,
+    BrowserActAction,
     BrowserActResult,
     BrowserContent,
+    BrowserObserveElement,
     BrowserObserveResult,
     BrowserRecording,
-    BrowserRunResult,
     Chunk,
     CodeLanguage,
     CustomHarnessConfig,
@@ -582,55 +583,32 @@ def observe(self, instruction: str, *, model: Optional[str] = None) -> BrowserOb
         )
         return BrowserObserveResult.model_validate({"elements": resp.get("elements") or []})
 
-    def act(self, instruction: str, *, model: Optional[str] = None) -> BrowserActResult:
-        """Resolve and execute one natural-language action on this tab (metered)."""
-        body: Dict[str, Any] = {"instruction": instruction, "tab": self.id}
-        if model:
-            body["model"] = model
-        resp = self._box._request(
-            "POST",
-            f"/v2/box/{self._box.id}/browser/act",
-            body=body,
-            timeout=180000,
-        )
-        return BrowserActResult.model_validate(resp)
-
-    def run(
+    def act(
         self,
-        prompt: str,
+        instruction: Union[str, BrowserObserveElement, BrowserActAction],
         *,
-        schema: Optional[ResponseSchema] = None,
-        max_steps: Optional[int] = None,
         model: Optional[str] = None,
-    ) -> BrowserRunResult:
-        """Autonomously complete a multi-step task on this tab (metered).
+    ) -> BrowserActResult:
+        """Resolve and execute one action on this tab.
 
-        Runs a DOM-aware browser agent (Stagehand) inside the box: it reads the
-        page, acts, and repeats until done. ``max_steps`` defaults to 15
-        (max 30). Needs a key for the model's provider on the box or account.
+        Pass a string (LLM-resolved, metered) or a pre-resolved ``observe()``
+        action to replay it with no LLM call and no key (``model`` ignored).
         """
-        json_schema = common.to_json_schema(schema) if schema is not None else None
-        if schema is not None and json_schema is None:
-            raise BoxError("run requires a pydantic model class or a JSON-schema dict")
-        body: Dict[str, Any] = {"prompt": prompt, "tab": self.id}
-        if json_schema is not None:
-            body["schema"] = json_schema
-        if max_steps:
-            body["max_steps"] = max_steps
-        if model:
-            body["model"] = model
+        if isinstance(instruction, str):
+            body: Dict[str, Any] = {"instruction": instruction, "tab": self.id}
+            if model:
+                body["model"] = model
+        else:
+            if not instruction.selector:
+                raise BoxError("act(action) requires a selector; observe() did not resolve one")
+            body = {"action": instruction.model_dump(exclude_none=True), "tab": self.id}
         resp = self._box._request(
             "POST",
-            f"/v2/box/{self._box.id}/browser/run",
+            f"/v2/box/{self._box.id}/browser/act",
             body=body,
-            timeout=600000,
-        )
-        data = (
-            common.validate_structured_data(schema, resp.get("data"))
-            if schema is not None
-            else None
+            timeout=180000,
         )
-        return BrowserRunResult.model_validate({**resp, "data": data})
+        return BrowserActResult.model_validate(resp)
 
     def live_view_url(self) -> str:
         """Live-view URL for this tab (authenticated via a token in the URL).