Merged upstreamCorrectnessMerged Aug 16, 2026
Prune Iceberg changelog tasks by metadata predicates
Optimized Auron native Iceberg changelog scans so eligible metadata predicates prune AddedRowsScanTask instances before native scan planning.
apache/auron · #2456
Problem
Iceberg changelog metadata values are constant per changelog task, but Auron still planned every supported AddedRowsScanTask in a changelog range even when predicates on _change_type, _commit_snapshot_id, or _change_ordinal proved a task could not match.
Approach
Associated supported metadata-only filters with direct Iceberg changelog scans, matched attributes by expression ID, prepared extension plans before the columnar transition, pruned tasks for = and IN predicates combined with AND, and preserved filter tags when Spark rebuilt BatchScanExec for runtime filters.
Impact and scope
- Allows eligible native Iceberg changelog scans to plan and read fewer insert tasks without public API or configuration changes.
- Avoids unnecessary work when metadata predicates can prove changelog tasks cannot match.
- Keeps correctness by leaving OR, NOT, mixed data-column predicates, unsafe shapes, and non-AddedRowsScanTask tasks on the existing path.
Validation
- AuronIcebergIntegrationSuite passed 45 tests on Spark 3.5 with Iceberg 1.10.1 and the native engine.
- dev/reformat --check passed Spark 3.5 formatting and compile/test-compile across Spark 3.0-3.4 and Spark 4.0-4.1.
- Regression coverage includes supported and unsupported predicates, aggregate/limit/nondeterministic barriers, expression-ID mismatches, pre-columnar staging, runtime-filter copies, dynamic partition pruning, and non-insert task retention.