Apache Software Foundation / Arrow Rust
Merged upstreamCorrectnessMerged Jul 2, 2026

Reject dictionary-encoded dictionary values in IPC writers

Fixed Apache Arrow Rust IPC writers so direct dictionary-of-dictionary schemas fail early with a clear error instead of writing IPC data that readers cannot decode.

apache/arrow-rs · #10230

Problem

Arrow IPC could write record batches whose dictionary values were themselves dictionary-encoded. The resulting stream later failed during read with a buffer metadata mismatch, making the real schema support problem hard to diagnose.

Approach

Added IPC schema validation to file and stream writer constructors, guarded low-level dictionary encoding as a backstop, and returned InvalidArgumentError before any IPC bytes are written.

Impact and scope

  • Prevents invalid Arrow IPC payloads from being emitted for a known unsupported schema shape.
  • Moves failure to writer construction with an actionable error instead of a later reader-side metadata mismatch.
  • Covers both IPC stream and file writers, preserving API shape while tightening validation.

Validation

  • Added regression coverage proving stream and file writers reject dictionary-of-dictionary schemas and leave buffers empty.
  • PR validation included arrow-ipc tests with default and all-features configurations.
  • Also ran formatting and Clippy checks for arrow-ipc targets before upstream merge.