fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility#249
Open
ensky wants to merge 3 commits into
Open
fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility#249ensky wants to merge 3 commits into
ensky wants to merge 3 commits into
Conversation
Co-Authored-By: Paperclip <noreply@paperclip.ing>
docs: fix typo in README ("Analaytics" -> "Analytics")
apache-arrow's `tableFromJSON()` (via `vectorFromArray` → `makeBuilder`
→ `new StructBuilder` → `createIsValidFunction`) calls `new Function(...)`
to generate validity checkers. Cloudflare Workers' runtime forbids
runtime code generation:
EvalError: Code generation from strings disallowed for this context
at new Function (<anonymous>)
at createIsValidFunction
...
at tableFromJSON
at extractAsArrow
The daily-rollup cron has thrown on every fire since deployment under
this restriction, so the R2 `counterscale-daily-rollups` bucket has
never received an object.
Replace `tableFromJSON(records)` with a small `recordsToTable` helper
that builds `Utf8`/`Float64` `Data` directly via `makeData` and assembles
them into a `RecordBatch` — bypassing the Builder/codegen path entirely.
Adds a vitest covering:
- round-trip through `tableToIPC` / `tableFromIPC`
- null/undefined handling
- empty-input edge case
- a guard that proxies `globalThis.Function` and asserts no
`new Function(...)` call happens during table construction or IPC
serialization (proves the CF Workers compatibility property at unit
level)
Co-authored-by: Paperclip <noreply@paperclip.ing>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The daily-rollup scheduled handler (
extractAsArrowinpackages/server/workers/lib/arrow.ts) cannot succeed on CloudflareWorkers as currently written.
apache-arrow'stableFromJSON()internally walks records andtriggers
vectorFromArray→makeBuilder→new StructBuilder→new Builder→createIsValidFunction, which callsnew Function(...)to generate validity-check code at runtime. TheCloudflare Workers runtime forbids this:
I observed this on a real counterscale deployment with CF Workers
compatibility_date = "2025-08-08". The cron has thrown on every firesince the worker was deployed; no Arrow file has ever been written to
the daily-rollups R2 bucket.
Fix
Replace
tableFromJSON(records)with a smallrecordsToTablehelperthat pivots records to columnar form and builds
Utf8/Float64Arrow
Datadirectly viamakeData, then assembles them into aRecordBatch. This bypasses the Builder/codegen path entirely.Column types are inferred per column from the first non-null sample
(
number→ Float64, otherwise → Utf8). This preserves the same columnshape
tableFromJSONwas producing for this consumer — the recordswritten here only ever have string and number fields.
Public API of
extractAsArrowis unchanged (same return shape, samefilename pattern, same R2 write).
Tests
Adds
workers/lib/__tests__/arrow.test.ts:tableToIPC/tableFromIPCand asserts values match.null/undefinedvalues.globalThis.Functionand asserts nonew Function(...)call happens during table construction or IPC serialization — a
unit-level guarantee that the Workers runtime restriction is
satisfied.
Notes
Happy to adjust style/structure to match the project's conventions.
The change is small and self-contained to
workers/lib/arrow.tsplusthe new unit test.