Merged upstreamCorrectnessMerged Aug 18, 2026
Match Spark abs overflow errors
Aligned datafusion-spark ANSI abs overflow behavior with Spark 4.2's canonical ARITHMETIC_OVERFLOW message for scalar and array inputs.
apache/datafusion · #24156
Problem
With ANSI mode enabled, datafusion-spark raised DataFusion/Arrow overflow messages for integral abs overflow, and scalar and array paths produced different text. Spark 4.2 uses one canonical ARITHMETIC_OVERFLOW message, so the compatibility layer was inconsistent with Spark and with itself.
Approach
Added a Spark-local checked abs array kernel, unified scalar and array overflow text around the Spark 4.2 error contract, left core DataFusion abs behavior unchanged, and updated SQL logic tests for all signed integer widths and both input shapes.
Impact and scope
- Improves Spark SQL compatibility for users relying on ANSI overflow behavior and stable Spark-style diagnostics.
- Keeps Spark-specific error semantics localized to datafusion-spark instead of changing core DataFusion users' error messages.
- Creates a cleaner precedent for future Spark compatibility fixes where shared core kernels cannot express Spark-specific contracts.
Validation
- Added Spark SQL logic coverage for signed integer abs overflow across scalar and array paths.
- Ran datafusion-spark formatting, strict clippy, all-feature package tests, and focused Spark abs sqllogictests.
- Ran broader CI-profile workspace tests with avro, json, backtrace, extended tests, recursive protection, and parquet encryption features.