Apache Software Foundation / Pinot
Merged upstreamFeatureMerged Aug 18, 2026

Support Kafka ConfigProviders in realtime table configs

Added Kafka ConfigProvider support for Apache Pinot realtime table stream configs so SSL secret references can survive Pinot config resolution and Kafka client property filtering.

apache/pinot · #19202

Problem

Realtime Kafka tables could not use file-based Kafka ConfigProviders for SSL passwords because Pinot interpreted ${provider:...} values as Pinot variable expressions and then filtered config.providers.* keys out before constructing Kafka consumers and AdminClients.

Approach

Escaped declared provider references during table-config variable resolution, admitted Kafka's config.providers namespace through consumer and AdminClient filtering, applied the behavior to Kafka 3.x and Kafka 4.x connectors, and covered legacy, multi-stream, scoped provider, and real Kafka client construction paths.

Impact and scope

  • Lets operators externalize Kafka SSL secrets with mounted provider files instead of storing plaintext password values in Pinot table configs.
  • Improves secret-rotation ergonomics because recreated Kafka clients can reread provider-backed values without requiring table-config password rewrites.
  • Keeps Pinot's normal environment and system-property substitution intact while limiting provider preservation to aliases declared in the containing config object.

Validation

  • Pinot SPI tests passed, including focused ConfigUtils coverage for provider-reference preservation.
  • Kafka 3.x and Kafka 4.x connector tests construct KafkaConsumer and shared AdminClient paths with provider-backed SSL values.
  • Spotless, checkstyle, and license checks passed for pinot-spi and both Kafka connector modules; a maintainer linked follow-up documentation after merge.