Make schema pruning idempotent after VARIANT pushdown
Fixed an Apache Spark SQL optimizer interaction between VARIANT pushdown and schema pruning, with the authored patch applied directly to Spark's canonical history by a project committer.
apache/spark · #57809
Optimizer correctness fix
Spark's early scan-pushdown batch now reaches a stable plan on its first application after VARIANT pushdown.
Problem
`PushVariantIntoScan` could make a reconstruction projection unnecessary after earlier schema-pruning passes had finished. Reapplying the `Once` optimizer batch then removed that projection, so testing-mode idempotence checks observed a different second plan and failed.
Approach
Runs `SchemaPruning` once more after `PushVariantIntoScan`, allowing the first optimizer pass to remove the redundant reconstruction projection and reach the same fixed point as subsequent applications.
Impact and scope
- Restores deterministic optimizer behavior for Parquet V1 queries combining metadata columns, filtered VARIANT values, and nondeterministic operators.
- Eliminates a redundant reconstruction projection while leaving production query results and public APIs unchanged.
- Adds a durable regression for a rule-ordering interaction in Spark SQL's early scan-pushdown pipeline.
Validation
- The regression was confirmed to fail before the optimizer change and pass afterward.
- Passed the focused Parquet V1 suite, a 349-test group covering schema pruning and VARIANT pushdown, Scalastyle, and diff validation.
- The canonical Apache Spark commit names Goutam Adwant as author and references PR #57809; both submitted commits are GitHub-verified, and two reviewers approved the patch.