PERF: GIL-free loop, fixed-layout fast path and same-date cache for the read_csv parse_dates fastpath#66278
Draft
jbrockmendel wants to merge 13 commits into
Draft
Conversation
For columns named in parse_dates, the C tokenizer now parses each ISO8601 field from its char buffer straight into a datetime64 output, skipping the intermediate ndarray[object] of Python strings that date_converter would otherwise feed to to_datetime. Naive and tz-aware inputs with consistent offsets are handled, mixed resolutions trigger a single recursive reparse at the widest unit, and any non-ISO or mixed-offset column falls back to the existing object-string path. Benchmarks at 500k rows: ISO naive 222ms -> 88ms, tz-aware 167ms -> 62ms. Non-ISO formats are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- require a consistent format when date_format is None, matching to_datetime's infer-from-first-element behavior; mixed ISO layouts parse per-row only with date_format="ISO8601" - in low_memory mode, keep raw-byte receipts per chunk so a fallback in any chunk restores the whole column to the object-string path (previously parsed chunks could be silently re-stringified from Timestamps when a later chunk fell back) - guard the tz-to-UTC shift against int64 wraparound near the datetime64 bounds; fall back so the slow path raises as before - fall back on all-NA columns so the dtype matches the slow path - skip marking implicit-index files; noconvert indices cannot line up - tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…imes # Conflicts: # doc/source/whatsnew/v3.1.0.rst
# Conflicts: # doc/source/whatsnew/v3.1.0.rst # pandas/_libs/parsers.pyx
# Conflicts: # doc/source/whatsnew/v3.1.0.rst
# Conflicts: # doc/source/whatsnew/v3.1.0.rst
_box_arena_utf8 referenced kh_get_strbox (not in the khash cimport list) and PyUnicode_Decode (not imported); both were undeclared-name Cython errors. Add the cimport and switch to the already-imported PyUnicode_DecodeUTF8 helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # pandas/_libs/parsers.pyx
…he parse_dates fastpath The per-row loop in _datetime_box_utf8 is now pure C and runs without the GIL, so parse_dates columns convert in parallel in the threaded read path. The two dominant fixed-width layouts (YYYY-MM-DD and YYYY-MM-DD[ T]HH:MM:SS) parse through a dedicated validator instead of the general ISO parser, consecutive timestamps sharing a calendar date reuse the date's epoch conversion, and format-consistency checks between fixed-layout rows collapse to an O(1) layout-code comparison. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jbrockmendel
added
Performance
…imes # Conflicts: # doc/source/whatsnew/v3.1.0.rst # pandas/_libs/parsers.pyx
…ogil # Conflicts: # pandas/_libs/parsers.pyx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, stacked on GH-65353 (branched from
perf-read_csv-datetimes; the diff shows that PR's commits until it merges — only the top commit is new).The per-row loop in
_datetime_box_utf8is now pure C and runs without the GIL (the prerequisite for parallelizingparse_datesreads as a follow-up to GH-66275). The two dominant fixed-width layouts (YYYY-MM-DDandYYYY-MM-DD[ T]HH:MM:SS) parse through a dedicated validator instead of the general ISO parser, consecutive timestamps sharing a calendar date reuse the date's epoch conversion, and format-consistency checks between fixed-layout rows collapse to an O(1) layout-code comparison. The receipts/all-or-nothing fallback semantics of the base PR are preserved;get_supported_resobecomesnoexcept nogilin tslibs.Benchmark vs the base branch (M3 Pro, 1M rows × 4 ISO datetime columns, parse_dates, serial, interleaved A/B): 291.3 → 133.6 ms (−54%).
🤖 Generated with Claude Code