Apache Software Foundation / DataFusion
Merged upstreamPerformanceMerged Aug 11, 2026

Avoid buffering unbounded repartition output indefinitely

Fixed DataFusion RepartitionExec so unbounded inputs can emit available partial batches instead of waiting indefinitely for the configured batch size.

apache/datafusion · #24193

Problem

Producer-side coalescing flushed residual batches only when all input senders finished. For an unbounded input that never finishes, rows that were already available could be withheld indefinitely, making incremental streaming plans appear to produce no output.

Approach

Skipped producer-side batch coalescing when the input is unbounded while preserving existing coalescing behavior for bounded inputs and preserve-order execution.

Impact and scope

  • Improves DataFusion streaming latency by allowing ready rows to move downstream without waiting for a full batch.
  • Keeps bounded batch workloads on the existing throughput-oriented coalescing path.
  • Documents a concrete point fix while maintainers track broader streaming SQL ergonomics for DataFusion.

Validation

  • Added a regression test with an unbounded source that emits one partial batch and then remains open.
  • Verified the regression fails on the prior implementation and passes with the fix.
  • Ran formatting, datafusion-physical-plan tests, strict clippy, and the extended AGENTS.md workspace command including all 503 sqllogictest files.