Skip to content

fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility#249

Open
ensky wants to merge 3 commits into
benvinegar:mainfrom
ensky-corp:upstream/fix-arrow-codegen
Open

fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility#249
ensky wants to merge 3 commits into
benvinegar:mainfrom
ensky-corp:upstream/fix-arrow-codegen

Conversation

@ensky
Copy link
Copy Markdown

@ensky ensky commented May 12, 2026

Problem

The daily-rollup scheduled handler (extractAsArrow in
packages/server/workers/lib/arrow.ts) cannot succeed on Cloudflare
Workers as currently written.

apache-arrow's tableFromJSON() internally walks records and
triggers vectorFromArraymakeBuildernew StructBuilder
new BuildercreateIsValidFunction, which calls
new Function(...) to generate validity-check code at runtime. The
Cloudflare Workers runtime forbids this:

EvalError: Code generation from strings disallowed for this context
    at new Function (<anonymous>)
    at createIsValidFunction
    at new Builder
    at new StructBuilder
    at makeBuilder
    at vectorFromArray
    at tableFromJSON
    at extractAsArrow

I observed this on a real counterscale deployment with CF Workers
compatibility_date = "2025-08-08". The cron has thrown on every fire
since the worker was deployed; no Arrow file has ever been written to
the daily-rollups R2 bucket.

Fix

Replace tableFromJSON(records) with a small recordsToTable helper
that pivots records to columnar form and builds Utf8 / Float64
Arrow Data directly via makeData, then assembles them into a
RecordBatch. This bypasses the Builder/codegen path entirely.

Column types are inferred per column from the first non-null sample
(number → Float64, otherwise → Utf8). This preserves the same column
shape tableFromJSON was producing for this consumer — the records
written here only ever have string and number fields.

Public API of extractAsArrow is unchanged (same return shape, same
filename pattern, same R2 write).

Tests

Adds workers/lib/__tests__/arrow.test.ts:

  • Round-trips a mixed string/number record set through
    tableToIPC / tableFromIPC and asserts values match.
  • Handles null / undefined values.
  • Returns an empty Table for empty input.
  • Proxies globalThis.Function and asserts no new Function(...)
    call happens during table construction or IPC serialization — a
    unit-level guarantee that the Workers runtime restriction is
    satisfied.

Notes

Happy to adjust style/structure to match the project's conventions.
The change is small and self-contained to workers/lib/arrow.ts plus
the new unit test.

ensky and others added 3 commits May 12, 2026 01:23
Co-Authored-By: Paperclip <noreply@paperclip.ing>
docs: fix typo in README ("Analaytics" -> "Analytics")
apache-arrow's `tableFromJSON()` (via `vectorFromArray` → `makeBuilder`
→ `new StructBuilder` → `createIsValidFunction`) calls `new Function(...)`
to generate validity checkers. Cloudflare Workers' runtime forbids
runtime code generation:

    EvalError: Code generation from strings disallowed for this context
        at new Function (<anonymous>)
        at createIsValidFunction
        ...
        at tableFromJSON
        at extractAsArrow

The daily-rollup cron has thrown on every fire since deployment under
this restriction, so the R2 `counterscale-daily-rollups` bucket has
never received an object.

Replace `tableFromJSON(records)` with a small `recordsToTable` helper
that builds `Utf8`/`Float64` `Data` directly via `makeData` and assembles
them into a `RecordBatch` — bypassing the Builder/codegen path entirely.

Adds a vitest covering:
  - round-trip through `tableToIPC` / `tableFromIPC`
  - null/undefined handling
  - empty-input edge case
  - a guard that proxies `globalThis.Function` and asserts no
    `new Function(...)` call happens during table construction or IPC
    serialization (proves the CF Workers compatibility property at unit
    level)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant