Reject unsupported Substrait aggregation phases
Validates aggregation phases before argument translation, accepts complete calls, and returns clear errors for explicit intermediate phases and unknown protobuf enum values.
apache/datafusion · #25045
Query-correctness fix
DataFusion now rejects explicit unsupported Substrait aggregate and window phases instead of silently evaluating intermediate-state requests as complete functions.
Problem
The Substrait consumer ignored AggregateFunction.phase. An INITIAL_TO_INTERMEDIATE average could execute as an ordinary complete aggregate and return 1.5 rather than the requested intermediate state; rooted plans could instead fail later with an unrelated schema-naming error.
Approach
Accepts INITIAL_TO_RESULT and the historical UNSPECIFIED compatibility value, rejects every explicit unsupported intermediate phase, and applies the same gate to aggregate and window expressions before translation obscures the real problem.
Impact and scope
- Prevents silently incorrect query results when an external Substrait plan explicitly requests unsupported intermediate aggregation semantics.
- Produces direct phase errors instead of allowing later output-schema failures to mask the unsupported operation.
- Preserves plans emitted by released DataFusion versions, which historically used UNSPECIFIED for complete calls.
- UNSPECIFIED remains ambiguous because the Substrait specification maps it to INTERMEDIATE_TO_RESULT; plans from other producers using that meaning can still pass as complete calls.
Validation
- The original behavior was reproduced with an INITIAL_TO_INTERMEDIATE average over values 1 and 2 returning the complete result 1.5.
- Tests cover supported and unsupported phases, rooted and unrooted aggregates, unknown wire enum values, and actual window output.
- The Substrait integration target passed 213 tests, the extended workspace passed 11,267 Rust tests and all 511 SQL logic-test files, and full Clippy and workspace lint passed.
- Thirty-eight hosted checks passed with no failures; an independent contributor revalidated the compatibility clarification and focused integration tests, and the verified merge commit credits Goutam Adwant.