Apache Software Foundation / DataFusion
Merged upstreamPerformanceSelected contributionMerged Aug 15, 2026

Union nested Parquet leaves across projection accesses

Optimized DataFusion Parquet nested schema pruning so multiple accesses to the same nested root can read the union of required leaves instead of falling back to the full root column.

apache/datafusion · #24130

Problem

When one projected nested column was consumed through multiple narrowing casts, or through both a narrowing cast and get_field, the Parquet read plan fell back to reading the entire root column even though the required leaf set was computable.

Approach

Accumulated and deduplicated leaf offsets across narrowing casts and get_field accesses, derived the emitted Arrow type from the merged leaf set, and retained full-read fallback behavior for unsupported partial wrapper shapes.

Impact and scope

  • Reduces unnecessary Parquet I/O for queries with multiple nested accesses to the same root column.
  • Preserves existing SQL/API behavior while improving scan efficiency for supported nested projection patterns.
  • Keeps correctness by unioning all required leaves rather than clipping to one consumer or reading the whole root.

Validation

  • Added coverage for disjoint, overlapping, and repeated casts, cast plus get_field, and casts diverging below List<Struct>.
  • SQL logic tests verify returned values and confirm merged projections scan fewer bytes than a full-root read.
  • Ran datasource-parquet library tests, parquet_nested_schema_pruning sqllogictests, formatting, and strict clippy for the changed package.