Add ZhiPu AI support with CustomOpenAIClient improvements

[shrey150]

Summary

• Change CustomOpenAIClient response format from json_schema to json_object for broader compatibility with models like ZhiPu
• Add prompt-based JSON schema instructions for models without native structured output support
• Export CustomOpenAIClient from public types for easier developer access

import { Stagehand, CustomOpenAIClient } from "@browserbasehq/stagehand";
import OpenAI from "openai";

const stagehand = new Stagehand({
  env: "BROWSERBASE",
  disableAPI: true,
  llmClient: new CustomOpenAIClient({
    modelName: "glm-4.6v",
    client: new OpenAI({
      apiKey: "your-zhipu-key",
      baseURL: "https://open.bigmodel.cn/api/paas/v4",
    }),
  }),
});

Testing

Verified with ZhiPu glm-4.6v model using Browserbase integration with successful extract, act, and observe operations.

---

Summary by cubic

Added support for ZhiPu models by switching CustomOpenAIClient to json_object responses and adding a schema-instruction prompt for structured JSON. Also exported CustomOpenAIClient from public types for easier use.

• Bug Fixes
  • Use shared validateZodSchema for field-level errors, add request-scoped logging, handle invalid JSON with retries, and throw CreateChatCompletionResponseError with the specific message.

^{Written for commit e2d07a8. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: e2d07a8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

No issues found across 2 files

[greptile-apps]

Greptile Summary

This PR enhances ZhiPu AI support by switching CustomOpenAIClient from OpenAI's json_schema to the more widely compatible json_object response format, with prompt-based schema instructions for better compatibility with models that lack native structured output support.

• Response Format Change: Switched from json_schema to json_object with schema provided in prompt instructions, enabling support for ZhiPu and similar OpenAI-compatible providers
• Error Handling Improvements: Wrapped JSON.parse() in try-catch to properly handle invalid JSON responses (common with json_object format) and distinguish between parse errors and validation errors
• Code Quality: Replaced local validateZodSchema with shared utility from lib/utils, improving maintainability and consistency across the codebase
• Public API Export: Exported CustomOpenAIClient from public types for easier developer access
• Retry Logic: Maintains automatic retries for both parse and validation failures, improving reliability with less predictable models
• Error Messaging: Enhanced error reporting with detailed logging of different error types and request context

Confidence Score: 5/5

• This PR is safe to merge with no functional or security concerns
• The PR addresses critical issues from previous review feedback. The changes properly wrap JSON.parse() in try-catch blocks, handle both parse and validation errors consistently, and use the shared validation utility. The response format switch from json_schema to json_object is appropriate for broader compatibility and is well-supported by the enhanced error handling. Code changes are minimal, focused, and follow existing patterns in the codebase. No edge cases or vulnerabilities introduced.
• No files require special attention

Important Files Changed

Filename	Overview
packages/core/examples/external_clients/customOpenAI.ts	Switched response format from json_schema to json_object for broader compatibility with models like ZhiPu. Added prompt-based JSON schema instructions. Replaced local validation with shared utility. Improved error handling with proper try-catch around JSON parsing and schema validation, with distinction between parse errors and validation errors. Consistent error wrapping for all failure modes.
packages/core/lib/v3/types/public/index.ts	Exported CustomOpenAIClient from public types module, making it easily accessible to developers using the SDK. This is a straightforward and safe export addition.

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant Client as CustomOpenAIClient
    participant OpenAI as OpenAI API<br/>(ZhiPu/Ollama)
    participant Validate as Validator

    App->>Client: createChatCompletion(options)
    activate Client
    
    alt response_model provided
        Note over Client: Format schema as JSON<br/>Add to messages
    end
    
    Client->>OpenAI: POST chat/completions<br/>response_format: json_object
    activate OpenAI
    OpenAI-->>Client: Response with content
    deactivate OpenAI
    
    alt response_model provided
        Client->>Client: JSON.parse(content)
        alt Parse OK
            Client->>Validate: validateZodSchema(data)
            activate Validate
            alt Valid
                Validate-->>Client: Success
                Client-->>App: Return { data, usage }
            else Invalid
                Validate-->>Client: ZodSchemaValidationError
                deactivate Validate
                alt retries > 0
                    Note over Client: Retry (retries-1)
                    Client->>OpenAI: Retry call
                else retries exhausted
                    Note over Client: Wrap error +<br/>detailed logging
                    Client-->>App: CreateChatCompletionResponseError
                end
            end
        else Parse fails (SyntaxError)
            alt retries > 0
                Note over Client: Retry (retries-1)
                Client->>OpenAI: Retry call
            else retries exhausted
                Note over Client: Error: Failed to parse JSON
                Client-->>App: CreateChatCompletionResponseError
            end
        end
    else No response_model
        Client-->>App: Return { data: content, usage }
    end
    
    deactivate Client

Loading

[greptile-apps]

Additional Comments (2)

1. packages/core/examples/external_clients/customOpenAI.ts, line 27-34 (link)

   style: CustomOpenAIClient defines its own validateZodSchema function, but the codebase already exports a centralized validateZodSchema utility from packages/core/lib/utils.ts that throws detailed ZodSchemaValidationError. Consider using the shared utility for consistency and better error reporting, which would give clearer validation failure messages to users.

2. packages/core/examples/external_clients/customOpenAI.ts, line 225-235 (link)

   logic: The local validateZodSchema silently returns false on validation failure, losing detailed error information. When validation fails, users get a generic "Invalid response schema" error instead of specific field-level validation details. This differs significantly from OpenAIClient's approach using the shared validateZodSchema which throws ZodSchemaValidationError with detailed error information.

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[shrey150]

@greptileai

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[shrey150]

@greptileai

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[shrey150]

@greptileai

[yangnuowei88]

I am looking forward to this feature in china