Apache Software Foundation / Auron
Merged upstreamCorrectnessMerged Aug 16, 2026

Prune Iceberg changelog tasks by metadata predicates

Optimized Auron native Iceberg changelog scans so eligible metadata predicates prune AddedRowsScanTask instances before native scan planning.

apache/auron · #2456

Problem

Iceberg changelog metadata values are constant per changelog task, but Auron still planned every supported AddedRowsScanTask in a changelog range even when predicates on _change_type, _commit_snapshot_id, or _change_ordinal proved a task could not match.

Approach

Associated supported metadata-only filters with direct Iceberg changelog scans, matched attributes by expression ID, prepared extension plans before the columnar transition, pruned tasks for = and IN predicates combined with AND, and preserved filter tags when Spark rebuilt BatchScanExec for runtime filters.

Impact and scope

  • Allows eligible native Iceberg changelog scans to plan and read fewer insert tasks without public API or configuration changes.
  • Avoids unnecessary work when metadata predicates can prove changelog tasks cannot match.
  • Keeps correctness by leaving OR, NOT, mixed data-column predicates, unsafe shapes, and non-AddedRowsScanTask tasks on the existing path.

Validation

  • AuronIcebergIntegrationSuite passed 45 tests on Spark 3.5 with Iceberg 1.10.1 and the native engine.
  • dev/reformat --check passed Spark 3.5 formatting and compile/test-compile across Spark 3.0-3.4 and Spark 4.0-4.1.
  • Regression coverage includes supported and unsupported predicates, aggregate/limit/nondeterministic barriers, expression-ID mismatches, pre-columnar staging, runtime-filter copies, dynamic partition pruning, and non-insert task retention.