Merged upstreamPerformanceMerged Aug 11, 2026
Avoid buffering unbounded repartition output indefinitely
Fixed DataFusion RepartitionExec so unbounded inputs can emit available partial batches instead of waiting indefinitely for the configured batch size.
apache/datafusion · #24193
Problem
Producer-side coalescing flushed residual batches only when all input senders finished. For an unbounded input that never finishes, rows that were already available could be withheld indefinitely, making incremental streaming plans appear to produce no output.
Approach
Skipped producer-side batch coalescing when the input is unbounded while preserving existing coalescing behavior for bounded inputs and preserve-order execution.
Impact and scope
- Improves DataFusion streaming latency by allowing ready rows to move downstream without waiting for a full batch.
- Keeps bounded batch workloads on the existing throughput-oriented coalescing path.
- Documents a concrete point fix while maintainers track broader streaming SQL ergonomics for DataFusion.
Validation
- Added a regression test with an unbounded source that emits one partial batch and then remains open.
- Verified the regression fails on the prior implementation and passes with the fix.
- Ran formatting, datafusion-physical-plan tests, strict clippy, and the extended AGENTS.md workspace command including all 503 sqllogictest files.