Add a Google Pub/Sub source connector
Implemented the Google Pub/Sub source side for SeaTunnel, including subscription reads, checkpoint-aware acknowledgement, authentication, formats, documentation, and emulator-backed integration coverage.
apache/seatunnel · #11989
New streaming connector
SeaTunnel can now consume Google Pub/Sub subscriptions with checkpoint-coupled acknowledgements and at-least-once delivery guarantees.
Problem
SeaTunnel's existing Google Pub/Sub connector supported sink delivery but could not ingest subscription messages. A source implementation needed to coordinate acknowledgements with SeaTunnel checkpoints so failures after emission would leave messages available for redelivery instead of losing them.
Approach
Added a single logical subscription split, JSON and delimited-text decoding, Application Default Credentials, service-account keys, emulator support, and acknowledgement only after the checkpoint containing each record completes. Subscriber, deserialization, and acknowledgement failures propagate to the source task, with explicit handling for aborted and overlapping checkpoints.
Impact and scope
- Adds a production-oriented Pub/Sub ingestion path to SeaTunnel's connector ecosystem.
- Aligns external acknowledgement with checkpoint completion to provide clear at-least-once delivery semantics.
- Keeps unacknowledged messages eligible for redelivery when subscriber, parsing, task, checkpoint, or acknowledgement failures occur.
Validation
- Added configuration, factory, reader, checkpoint, deserialization, subscriber, and acknowledgement failure coverage on JDK 8 and JDK 11.
- Added Pub/Sub emulator E2E coverage that consumes through Zeta and verifies checkpoint completion after emission; all eight E2E tests passed.
- Both authored commits and the merge commit are GitHub-verified; the feature received approvals from two maintainers.