Skip to content

PERF: read_csv fast paths for nullable and pyarrow-backed EA dtypes#66279

Draft
jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:perf-read_csv-ea
Draft

PERF: read_csv fast paths for nullable and pyarrow-backed EA dtypes#66279
jbrockmendel wants to merge 2 commits into
pandas-dev:mainfrom
jbrockmendel:perf-read_csv-ea

Conversation

@jbrockmendel

Copy link
Copy Markdown
Member

Requesting an ExtensionArray dtype explicitly (e.g. dtype="Int64", dtype="int64[pyarrow]", dtype="boolean") is currently 5–9x slower than the equivalent dtype_backend inference, because the c engine materializes an intermediate ndarray[object] of Python strings and then re-parses it (via to_numeric / _from_sequence_of_strings). This routes those dtypes through the nogil C converters instead.

  • nullable Integer/Floating (e.g. Int64, Float64): parse with the same C converters dtype_backend="numpy_nullable" uses, building the NA mask from the actual NA positions rather than the integer sentinel, so legitimate int64-min / uint64-max values are preserved (the sentinel-based inference path masks them).
  • pyarrow-backed numeric (e.g. int64[pyarrow]): the same C-converter parse, then wrapped as pyarrow — this keeps explicit int64[pyarrow] parsing identical to dtype_backend="pyarrow" rather than adopting pyarrow's more lenient string casting (which would e.g. accept "0x1F"). Temporal / boolean / string ArrowDtype build a pyarrow large_string array straight from the buffers and hand it to _from_sequence_of_strings (same conversion engine as before).
  • nullable boolean (boolean): a nogil pass recognizing the true/false sets plus the "1"/"1.0"/"0"/"0.0" spellings, testing the true set before the false set to match map_string's ordering (so it stays correct even with overlapping user true_values/false_values).

Every fast path falls back to the generic string path on anything it cannot confidently handle (non-numeric data, ints wider than uint64, invalid UTF-8, an unrecognized boolean token, a dtype the requested type can't hold), so behavior — including raised errors — is unchanged. Also makes ArrowExtensionArray._from_sequence_of_strings compute its isna mask lazily so a pyarrow-array input isn't scanned unnecessarily.

Timings (1M rows × 10 cols, single-threaded), before → after:

dtype before after
Int64 / Float64 3419 / 3092 ms 399 / 614 ms
int64[pyarrow] / double[pyarrow] 3515 / 3268 ms 408 / 635 ms
bool[pyarrow] 811 ms 498 ms
boolean 1556 ms 312 ms

Validated with per-family differential tests (random + adversarial inputs, custom na_values, the parallel-read path) against the legacy path; whole result and raised errors match.

Route explicitly-requested ExtensionArray dtypes through the nogil C
converters instead of materializing an intermediate ndarray[object] of
strings and re-parsing it via to_numeric / _from_sequence_of_strings:

- nullable Integer/Floating (e.g. dtype="Int64"): parse via the same C
  converters dtype_backend="numpy_nullable" uses, building the NA mask from
  the actual NA positions so int64-min / uint64-max are preserved.
- pyarrow-backed numeric (e.g. dtype="int64[pyarrow]"): same C-converter
  parse, then wrap as pyarrow -- keeping it consistent with
  dtype_backend="pyarrow" (rather than adopting pyarrow's more lenient
  string casting). Temporal/boolean/string ArrowDtype build a pyarrow
  large_string array straight from the buffers for _from_sequence_of_strings.
- nullable boolean (dtype="boolean"): recognize the true/false sets plus the
  "1"/"1.0"/"0"/"0.0" spellings in a nogil pass.

Every fast path falls back to the generic string path on anything it cannot
confidently handle, so behavior (including raised errors) is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jbrockmendel jbrockmendel added Performance Memory or execution speed performance IO CSV read_csv, to_csv labels Jul 10, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

IO CSV read_csv, to_csv Performance Memory or execution speed performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant