Mitigate stale realtime segmentStoppedConsuming notification
Fixed Apache Pinot realtime initialization cleanup so a stale failed segment manager does not send segmentStoppedConsuming after a newer manager is already registered.
apache/pinot · #18914
Problem
For realtime tables with single-permit segment consumption, a failed initialization attempt could schedule a delayed stop notification. If a retry succeeded first, the stale notification could flip the segment back to OFFLINE while a healthy in-memory consumer still held the semaphore, blocking the successor segment indefinitely.
Approach
Added an initialization-error-only guard that checks the currently registered segment manager before posting segmentStoppedConsuming. If another manager is already registered for the same segment, the stale failed attempt skips the notification while the normal consumer-error stop path remains unchanged.
Impact and scope
- Reduces the probability of realtime ingestion stalls caused by stale initialization cleanup.
- Protects single-permit consumer coordination used by partial-upsert and non-pauseless realtime tables.
- Preserves the regular postStopConsumedMsg behavior for healthy consumer error paths.
Validation
- Added coverage that skips the initialization-error stop message when a different manager is registered.
- Added coverage that still sends the stop message when the current manager is registered.
- Added coverage that the normal postStopConsumedMsg path does not use the stale-manager guard.