Fix MotherDuck hanging on concurrent scripts, bump to 0.18.0

[MaxHalford]

Summary

• Fix MotherDuck threading: MotherDuckClient used the module-level duckdb connection which isn't thread-safe when multiple scripts execute concurrently via ThreadPoolExecutor. Now uses a persistent connection with per-job cursors, matching the DuckLakeClient pattern.
• Bump version to 0.18.0
• Bump sqlglot to >=30.2
• Update motherduck example with correct LEA_WAREHOUSE=motherduck config (was using the old broken LEA_WAREHOUSE=duckdb approach)

Closes #55

Test plan

• 71 unit tests pass
• Manually tested against MotherDuck — all 9 jaffle shop scripts run successfully in ~5 seconds

🤖 Generated with Claude Code

---

Summary by cubic

Fixes MotherDuck hangs with concurrent scripts by giving MotherDuckClient a persistent DuckDB connection and per-job cursors. Releases v0.18.0, upgrades sqlglot to >=30.2, and unifies DuckLake secret env vars into a single LEA_DUCKLAKE_SECRET (breaking).

• Bug Fixes

  • Make MotherDuckClient thread-safe with a persistent connection and per-job cursors.
  • Add active database handling; the conductor sets it after CREATE/USE for MotherDuck.
  • Update MotherDuck example: use LEA_WAREHOUSE=motherduck, LEA_MOTHERDUCK_DATABASE, and run with --scripts.

• Migration

  • Replace provider-specific DuckLake env vars (LEA_DUCKLAKE_*) with LEA_DUCKLAKE_SECRET; for quack use LEA_QUACK_DUCKLAKE_SECRET.

^{Written for commit 64c3fcc. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="examples/motherduck/README.md">

<violation number="1" location="examples/motherduck/README.md:17">
P2: Initialize the `jaffle_shop` submodule before symlinking it; otherwise this example fails on a normal clone because the CSV seeds are missing.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Conductor as Conductor (Orchestrator)
    participant MDClient as MotherDuckClient
    participant DuckDB as duckdb (Module/Instance)
    participant Cursor as Job Cursor
    participant Cloud as MotherDuck Service

    Note over Conductor,Cloud: Session Initialization
    
    Conductor->>MDClient: NEW: Instantiate Client
    MDClient->>DuckDB: NEW: connect() (Persistent Connection)
    DuckDB-->>MDClient: _conn
    
    Conductor->>MDClient: prepare_session(write_dataset)
    MDClient->>DuckDB: execute("CREATE DATABASE...")
    MDClient->>DuckDB: execute("USE <dataset>")
    
    Conductor->>MDClient: NEW: set_active_database(write_dataset)
    Note right of MDClient: Stores active DB name for future cursors

    Note over Conductor,Cloud: Concurrent Execution (ThreadPoolExecutor)

    rect rgb(240, 240, 240)
        Note over Conductor,Cursor: Parallel Task (for each SQL script)
        Conductor->>MDClient: make_job_config(script)
        MDClient->>DuckDB: NEW: _conn.cursor()
        DuckDB-->>MDClient: cursor
        
        opt _active_database is set
            MDClient->>Cursor: NEW: execute("USE <active_db>")
        end
        
        MDClient-->>Conductor: DuckDBJob (includes cursor)
        
        Conductor->>Cursor: execute(script.query)
        Cursor->>Cloud: Remote SQL execution
        Cloud-->>Cursor: Results
        Cursor-->>Conductor: Job Complete
    end

    Note over Conductor,Cloud: CHANGED: Using per-job cursors prevents thread-safety hangs<br/>previously caused by shared module-level connection.

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}