Merged upstreamCorrectnessMerged May 30, 2026
Handle empty batches in Python worker counters
Fixed a Python worker counter path so empty batches do not raise ZeroDivisionError during byte-size accounting.
apache/beam · #38748
Problem
Apache Beam's Python worker operation counters divided estimated batch byte size by batch length. When ReadFromCSV or another batched path produced an empty batch, batch_length could be zero and the worker counter update failed before the pipeline could continue.
Approach
Added an early return when the batch converter reports zero elements. The element counter still records the empty batch length, but the mean byte-size counter is left unchanged because there are no elements to average.
Impact and scope
- Improves Python SDK worker robustness for real data pipelines where empty batches can appear during batched CSV/DataFrame processing.
- Prevents a low-level counter failure from interrupting otherwise valid Dataflow/Beam jobs.
- Keeps metrics behavior principled by avoiding an average byte-size update when there are no elements to average.
Validation
- Added OperationCountersTest.test_update_empty_batch for the zero-length batch path.
- Verified the worker counter test module covering OperationCounters behavior.
- Checked nearby Beam Python batch typehint tests and repository whitespace checks from the PR workflow.