Merged upstreamDocumentationMerged Aug 11, 2026
Explain Parquet content-defined chunking
Added user-facing DataFusion documentation for the experimental Parquet content-defined chunking writer feature, including when to use it and how to configure it.
apache/datafusion · #24155
Problem
Parquet content-defined chunking existed as an experimental writer capability, but DataFusion users lacked a guide explaining appropriate use cases, storage-system requirements, configuration, limitations, and writer-path tradeoffs.
Approach
Created a dedicated user guide, added single-file SQL and Rust examples, documented session-level configuration and tuning guidance, updated Parquet format options, and linked the guide from the documentation index.
Impact and scope
- Makes an advanced Parquet writer feature discoverable and usable for storage or transfer systems that benefit from duplicate byte-range reuse.
- Clarifies that content-defined chunking improves deduplication opportunities but does not perform deduplication itself.
- Explains sequential writer requirements, directory-output behavior, stable ordering, defaults, and tuning tradeoffs from a user perspective.
Validation
- Verified the documented behavior with the parquet_cdc.slt sqllogictest.
- Ran formatting, strict clippy, doc prettier checks, and git diff whitespace checks.
- Merged after maintainer approval called out the quick revision.