feat!(datafusion): enable parallel file scanning with eager task bucketing#2298

Open

toutane wants to merge 32 commits into

apache:mainfrom

toutane:draft/partitioned-file-scanning-contribution

Commits on Jun 16, 2026

feat(datafusion): add IcebergPartitionedTableProvider and IcebergPartitionedScan for parallel file scanning
toutane
committed
docs(datafusion): update comment in IcebergPartitionedScan
toutane
committed
Update crates/integrations/datafusion/src/table/mod.rs

toutane
and
timsaucer
committed
fix(datafusion): reject non-empty children in IcebergPartitionedScan::with_new_children
toutane
committed
fix(datafusion): use ArrowReaderBuilder existing configuration path
toutane
committed
feat(datafusion): bucket FileScanTasks across target_partitions with identity-hash partitioning

authored and
toutane
committed
feat(datafusion): mark identity-partition filters as Exact pushdown

authored and
toutane
committed
feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection

authored and
toutane
committed
Revert "feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection"
toutane
committed
Revert "feat(datafusion): mark identity-partition filters as Exact pushdown"
toutane
committed
refactor(datafusion): merge IcebergPartitionedTableProvider into IcebergTableProvider
toutane
committed
refactor(datafusion): polish scan API and add bucketing tests
toutane
committed
test(sqllogictest): update EXPLAIN snapshots for eager bucketing output
toutane
committed
fix(datafusion): resolve conflicts
toutane
committed
refactor(datafusion): read preplanned tasks with ArrowReaderBuilder
toutane
committed
refactor(datafusion): distinguish lazy and eager scan buckets
toutane
committed
refactor(datafusion): introduce IcebergTableScanBuilder
toutane
committed
perf(datafusion): avoid cloning task buckets during execute
toutane
committed
refactor(datafusion): centralize scan stream error mapping
toutane
committed
test(datafusion): verify identity bucket hashes match repartitioning
toutane
committed
fix(datafusion): reuse eager scan predicates during execution
toutane
committed
perf(datafusion): batch identity partition hashing
toutane
committed
refactor(datafusion): share Iceberg scan config
toutane
committed
fix(datafusion): use unknown partitioning for empty eager scans
toutane
committed
test(datafusion): cover hash partitioning fallback cases
toutane
committed
fix(datafusion): give table runtime to ArrowReaderBuilder::new()
toutane
committed
fix(datafusion): give table runtime to table_with_additional_partition_spec
toutane
committed
fix(datafusion): make check green
toutane
committed
refactor(datafusion): reuse iceberg::util::available_parallelism
toutane
committed
docs(datafusion): note count-based bucketing limitation
toutane
committed
docs(datafusion): explain conservative multi-spec bucketing gate
toutane
committed
fix: generate-public-api
toutane
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!(datafusion): enable parallel file scanning with eager task bucketing#2298

feat!(datafusion): enable parallel file scanning with eager task bucketing#2298
toutane wants to merge 32 commits into
apache:mainfrom
toutane:draft/partitioned-file-scanning-contribution

Commits on Jun 16, 2026

feat(datafusion): add IcebergPartitionedTableProvider and IcebergPartitionedScan for parallel file scanning

docs(datafusion): update comment in IcebergPartitionedScan

Update crates/integrations/datafusion/src/table/mod.rs

fix(datafusion): reject non-empty children in IcebergPartitionedScan::with_new_children

fix(datafusion): use ArrowReaderBuilder existing configuration path

feat(datafusion): bucket FileScanTasks across target_partitions with identity-hash partitioning

feat(datafusion): mark identity-partition filters as Exact pushdown

feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection

Revert "feat(datafusion): allow Exact pushdown across spec evolution via per-column intersection"

Revert "feat(datafusion): mark identity-partition filters as Exact pushdown"

refactor(datafusion): merge IcebergPartitionedTableProvider into IcebergTableProvider

refactor(datafusion): polish scan API and add bucketing tests

test(sqllogictest): update EXPLAIN snapshots for eager bucketing output

fix(datafusion): resolve conflicts

refactor(datafusion): read preplanned tasks with ArrowReaderBuilder

refactor(datafusion): distinguish lazy and eager scan buckets

refactor(datafusion): introduce IcebergTableScanBuilder

perf(datafusion): avoid cloning task buckets during execute

refactor(datafusion): centralize scan stream error mapping

test(datafusion): verify identity bucket hashes match repartitioning

fix(datafusion): reuse eager scan predicates during execution

perf(datafusion): batch identity partition hashing

refactor(datafusion): share Iceberg scan config

fix(datafusion): use unknown partitioning for empty eager scans

test(datafusion): cover hash partitioning fallback cases

fix(datafusion): give table runtime to ArrowReaderBuilder::new()

fix(datafusion): give table runtime to table_with_additional_partition_spec

fix(datafusion): make check green

refactor(datafusion): reuse iceberg::util::available_parallelism

docs(datafusion): note count-based bucketing limitation

docs(datafusion): explain conservative multi-spec bucketing gate

fix: generate-public-api