Reuse extension function implementations
Fixed Auron extension scalar UDF identity reuse so repeated equivalent functions can compare equal under DataFusion pointer-equality semantics and share cached evaluation results.
apache/auron · #2467
Problem
After the DataFusion 49 upgrade, Auron created a fresh function-implementation Arc for every extension UDF lookup. Logically identical functions therefore had different identities, so DataFusion SimpleScalarUDF equality failed and the cached expression evaluator could not deduplicate repeated expressions.
Approach
Introduced shared lazy function implementations with OnceLock, kept the existing direct match-based lookup path, returned cloned Arcs for subsequent lookups without adding a runtime map or lock lookup, and added regression coverage for implementation reuse, function isolation, and DataFusion UDF equality.
Impact and scope
- Restores expression-cache reuse for repeated Auron extension-function expressions after the DataFusion 49 equality behavior change.
- Avoids duplicate evaluation work for equivalent native extension UDFs without changing public APIs or configuration.
- Keeps separate functions isolated while making repeated lookups of the same function share one implementation identity.
Validation
- Added a regression test for shared implementation identity, separate-function isolation, and DataFusion UDF equality.
- Ran datafusion-ext-functions targeted and full library tests, plus auron-planner and datafusion-ext-plans library tests.
- Ran strict clippy for datafusion-ext-functions and dev/reformat --check across required dependency profiles and Spark compatibility builds.