Apache Software Foundation / DataFusion
Merged upstreamDocumentationMerged Aug 11, 2026

Explain Parquet content-defined chunking

Added user-facing DataFusion documentation for the experimental Parquet content-defined chunking writer feature, including when to use it and how to configure it.

apache/datafusion · #24155

Problem

Parquet content-defined chunking existed as an experimental writer capability, but DataFusion users lacked a guide explaining appropriate use cases, storage-system requirements, configuration, limitations, and writer-path tradeoffs.

Approach

Created a dedicated user guide, added single-file SQL and Rust examples, documented session-level configuration and tuning guidance, updated Parquet format options, and linked the guide from the documentation index.

Impact and scope

  • Makes an advanced Parquet writer feature discoverable and usable for storage or transfer systems that benefit from duplicate byte-range reuse.
  • Clarifies that content-defined chunking improves deduplication opportunities but does not perform deduplication itself.
  • Explains sequential writer requirements, directory-output behavior, stable ordering, defaults, and tuning tradeoffs from a user perspective.

Validation

  • Verified the documented behavior with the parquet_cdc.slt sqllogictest.
  • Ran formatting, strict clippy, doc prettier checks, and git diff whitespace checks.
  • Merged after maintainer approval called out the quick revision.