PERF: read_csv fast paths for nullable and pyarrow-backed EA dtypes#66279
Draft
jbrockmendel wants to merge 2 commits into
Draft
PERF: read_csv fast paths for nullable and pyarrow-backed EA dtypes#66279jbrockmendel wants to merge 2 commits into
jbrockmendel wants to merge 2 commits into
Conversation
Route explicitly-requested ExtensionArray dtypes through the nogil C converters instead of materializing an intermediate ndarray[object] of strings and re-parsing it via to_numeric / _from_sequence_of_strings: - nullable Integer/Floating (e.g. dtype="Int64"): parse via the same C converters dtype_backend="numpy_nullable" uses, building the NA mask from the actual NA positions so int64-min / uint64-max are preserved. - pyarrow-backed numeric (e.g. dtype="int64[pyarrow]"): same C-converter parse, then wrap as pyarrow -- keeping it consistent with dtype_backend="pyarrow" (rather than adopting pyarrow's more lenient string casting). Temporal/boolean/string ArrowDtype build a pyarrow large_string array straight from the buffers for _from_sequence_of_strings. - nullable boolean (dtype="boolean"): recognize the true/false sets plus the "1"/"1.0"/"0"/"0.0" spellings in a nogil pass. Every fast path falls back to the generic string path on anything it cannot confidently handle, so behavior (including raised errors) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requesting an ExtensionArray dtype explicitly (e.g.
dtype="Int64",dtype="int64[pyarrow]",dtype="boolean") is currently 5–9x slower than the equivalentdtype_backendinference, because thecengine materializes an intermediatendarray[object]of Python strings and then re-parses it (viato_numeric/_from_sequence_of_strings). This routes those dtypes through the nogil C converters instead.Int64,Float64): parse with the same C convertersdtype_backend="numpy_nullable"uses, building the NA mask from the actual NA positions rather than the integer sentinel, so legitimateint64-min /uint64-max values are preserved (the sentinel-based inference path masks them).int64[pyarrow]): the same C-converter parse, then wrapped as pyarrow — this keeps explicitint64[pyarrow]parsing identical todtype_backend="pyarrow"rather than adopting pyarrow's more lenient string casting (which would e.g. accept"0x1F"). Temporal / boolean / stringArrowDtypebuild a pyarrowlarge_stringarray straight from the buffers and hand it to_from_sequence_of_strings(same conversion engine as before).boolean): a nogil pass recognizing the true/false sets plus the"1"/"1.0"/"0"/"0.0"spellings, testing the true set before the false set to matchmap_string's ordering (so it stays correct even with overlapping usertrue_values/false_values).Every fast path falls back to the generic string path on anything it cannot confidently handle (non-numeric data, ints wider than
uint64, invalid UTF-8, an unrecognized boolean token, a dtype the requested type can't hold), so behavior — including raised errors — is unchanged. Also makesArrowExtensionArray._from_sequence_of_stringscompute itsisnamask lazily so a pyarrow-array input isn't scanned unnecessarily.Timings (1M rows × 10 cols, single-threaded), before → after:
Int64/Float64int64[pyarrow]/double[pyarrow]bool[pyarrow]booleanValidated with per-family differential tests (random + adversarial inputs, custom
na_values, the parallel-read path) against the legacy path; whole result and raised errors match.