Apache Software Foundation / Beam
Merged upstreamMaintenanceMerged Aug 12, 2026

Remove beam_fn_api requirement for KafkaIO dynamic reads

Removed Apache Beam KafkaIO's obsolete beam_fn_api expansion gate for dynamic reads while keeping dynamic-read execution classified as SDF-only.

apache/beam · #39735

Problem

KafkaIO dynamic reads failed during pipeline expansion unless the obsolete beam_fn_api experiment was enabled, even though KafkaIO's current dynamic-read path already uses the SDF implementation selection.

Approach

Removed the expansion-time beam_fn_api requirement and relied on the existing KafkaIOReadImplementationCompatibility path to keep dynamic reads SDF-only without changing runner fallback behavior.

Impact and scope

  • Unblocks KafkaIO dynamic-read expansion without requiring users to enable an obsolete experiment flag.
  • Preserves Beam's existing SDF-only compatibility decision for dynamic reads and avoids changing fallback behavior.
  • Documents the behavior in the 2.77.0 release notes so Java SDK users can see the KafkaIO change.

Validation

  • Added a regression test proving dynamic reads remain SDF-only and expand without beam_fn_api enabled.
  • Ran KafkaIO tests, Kafka 3.9.2 compatibility tests, Spotless, and the full KafkaIO check path.
  • Merged after Apache Beam maintainer approval and closed GH-29998.