Reuse column-normalization context across expressions
Introduced a private lazy normalization context reused by expression batches, sort normalization, projection construction, and wildcard expansion while preserving qualified-column fast paths.
apache/datafusion · #25010
Query-planning performance improvement
Repeated column normalization now shares immutable plan context across wide expression lists instead of traversing the same plan for every column.
Problem
Each unqualified column repeatedly collected fallback schemas and traversed the input plan for USING columns. Wide projections multiplied the same plan-dependent work, creating a superlinear normalization cost in planning and logical-plan decoding paths.
Approach
Computes fallback schemas and USING-column context once when needed, then reuses it across columns. Already-qualified columns, literals, and other expressions that do not require normalization remain on the existing lazy fast path.
Impact and scope
- Reduces repeated plan traversal for wide unqualified expression lists without changing public APIs or intended name-resolution behavior.
- Shares the optimization across normalization call sites, including validated projections, wildcard expansion, and sort expressions.
- Independent benchmarks reported unqualified 2,000-expression normalization improving from 74.7 ms to 6.3 ms (11.9×) and 2,000-column projection construction from 94.2 ms to 25.2 ms (3.74×).
- Constructor-based protobuf plan decoding remains separate; these measurements cover normalization and projection construction, not complete serialized-plan decoding.
Validation
- Correctness tests cover schema precedence, USING joins, ambiguity/error ordering, sort options, and lazy handling of qualified columns and literals.
- Focused expression and SQL tests, the extended workspace command, all 511 SQL logic-test files, expression-crate Clippy, and the documented workspace lint script passed.
- The full GitHub matrix passed, including Rust tests, SQL logic tests, benchmarks, Clippy, docs, MSRV, security audit, CodeQL, macOS ARM64, and WebAssembly checks.
- A DataFusion contributor approved the change and independently published benchmark comparisons after merge. The verified merge commit credits Goutam Adwant. A separate all-features Clippy run had previously encountered an existing PostgreSQL lint outside the changed source.