Apache Software Foundation / DataFusion
Merged upstreamFeatureMerged Aug 14, 2026

Add opt-in Spark functions to DataFusion CLI

Added an explicit --spark mode to datafusion-cli so users can explore Spark-compatible SQL syntax and functions interactively without changing default CLI behavior.

apache/datafusion · #24179

Problem

datafusion-cli did not register the datafusion-spark function library, so Spark-compatible functions failed interactively. Registering them by default was unsafe because Spark functions can override core DataFusion functions with different behavior.

Approach

Added an opt-in --spark flag, selected the Spark SQL dialect during session construction, registered Spark scalar, aggregate, and window functions only when enabled, and documented the override behavior.

Impact and scope

  • Makes DataFusion's Spark compatibility layer directly usable from the CLI for exploration, debugging, and examples.
  • Preserves backward-compatible default CLI behavior by keeping Spark registration explicit instead of default-on.
  • Documents the function override risk so users understand when the Spark function library is active.

Validation

  • Added CLI integration coverage proving Spark features require opt-in and default behavior stays unchanged.
  • Ran datafusion-cli integration tests, all datafusion-cli targets, formatting, strict clippy, and doc prettier checks.
  • Merged after maintainer approvals, including a final note calling the approved PR a great resolution for the issue.