Count UnnestExec output batches after splitting
Fixed DataFusion UnnestExec baseline output metrics so output_batches is counted at the consumer-visible stream boundary after oversized results are split.
apache/datafusion · #24482
Problem
UnnestExec recorded baseline output metrics before BatchSplitStream divided oversized results. A query that emitted three batches such as [10, 10, 5] rows could report output_batches=1, making execution metrics disagree with batches observed by downstream operators and users.
Approach
Moved baseline output observation to the final stream boundary by wrapping BatchSplitStream with the existing ObservedStream, kept elapsed_compute inside UnnestStream where the unnest work happens, and added a regression test for emitted batch sizes plus output_batches and output_rows metrics.
Impact and scope
- Makes UnnestExec execution metrics match the number of batches actually emitted to consumers.
- Improves observability for physical-plan diagnostics without changing query results, row ordering, batch boundaries, plans, or public APIs.
- Keeps compute-time measurement scoped to the unnest operation while moving output metrics to the correct final boundary.
Validation
- Added a regression test that fails before the fix with output_batches=1 and passes afterward with output_batches=3.
- All 13 unnest tests, 1,760 datafusion-physical-plan unit tests, documentation tests, and 107 datafusion-cli tests passed.
- The extended AGENTS.md workspace command passed, including all 503 SQL logic test files; formatting, strict clippy, and rustdoc with warnings denied also passed.