Scale sample-backed range partitioning
Added sample-backed range-partition scaling for adaptive and distributed planners, carrying retained boundaries through optimizer rewrites, repartitioning, joins, protobuf, and the DataFusion FFI.
apache/datafusion · #24766
Distributed query-planning feature
DataFusion can now scale sample-backed range partitioning while preserving its higher-resolution boundary data across optimizer rewrites and execution boundaries.
Problem
RangePartitioning stored only active partition boundaries, so planners could reduce parallelism but could not later scale it back up without discarding useful range information. That limitation made adaptive and distributed planning more likely to lose an existing range layout or introduce avoidable repartitioning.
Approach
Retains a higher-resolution sample set behind RangePartitioning, exposes bounded scaling and count APIs, and integrates them into EnsureRequirements, RepartitionExec, joins, unions, protobuf, and FFI. Unsupported target counts fall back to key partitioning instead of manufacturing invalid range boundaries.
Impact and scope
- Lets adaptive and distributed planners change task counts while preserving sampled range boundaries and ordering semantics.
- Keeps compatible range layouts available through projections, joins, unions, repartitioning, and serialization boundaries, reducing unnecessary loss of useful partitioning information.
- A DataFusion contributor reported successful integration in datafusion-distributed and ParadeDB before the feature was merged.
- The change is an API and ABI evolution: FFI consumers must rebuild, and the available benchmarks do not establish a general performance improvement or parity.
Validation
- Added an exchange regression that scales 2 to 4 to 1 to 4 partitions and verifies exact routing, ordering, retained samples, configuration, and fresh execution state and metrics.
- Added 23 optimizer integration regressions plus join, union, projection, equality, interleave-safety, protobuf, FFI, and NULL-handling coverage.
- Passed the 12,022-test Rust workspace suite, all 522 SQL logic-test files, all 155 all-feature FFI tests, formatting, Clippy with warnings denied, and the full workspace lint script.
- A contributor independently exercised the feature in two downstream integrations; the final upstream review approved the authored change, and the merge closed GH-24712.