Merged upstreamFeatureMerged Aug 14, 2026
Add opt-in Spark functions to DataFusion CLI
Added an explicit --spark mode to datafusion-cli so users can explore Spark-compatible SQL syntax and functions interactively without changing default CLI behavior.
apache/datafusion · #24179
Problem
datafusion-cli did not register the datafusion-spark function library, so Spark-compatible functions failed interactively. Registering them by default was unsafe because Spark functions can override core DataFusion functions with different behavior.
Approach
Added an opt-in --spark flag, selected the Spark SQL dialect during session construction, registered Spark scalar, aggregate, and window functions only when enabled, and documented the override behavior.
Impact and scope
- Makes DataFusion's Spark compatibility layer directly usable from the CLI for exploration, debugging, and examples.
- Preserves backward-compatible default CLI behavior by keeping Spark registration explicit instead of default-on.
- Documents the function override risk so users understand when the Spark function library is active.
Validation
- Added CLI integration coverage proving Spark features require opt-in and default behavior stays unchanged.
- Ran datafusion-cli integration tests, all datafusion-cli targets, formatting, strict clippy, and doc prettier checks.
- Merged after maintainer approvals, including a final note calling the approved PR a great resolution for the issue.