Add an Azure Queue Storage source
Implemented a streaming Azure Queue Storage source with checkpoint-aware message ownership, multiple authentication modes, payload formats, and Azurite integration coverage.
apache/seatunnel · #12006
New cloud messaging connector
SeaTunnel can now consume Azure Queue Storage with checkpoint-coupled deletion, visibility renewal, and at-least-once delivery.
Problem
The existing connector could publish messages but not consume them. A source needed to prevent premature deletion, renew visibility while checkpoints were pending, recover from aborted or overlapping checkpoints, and bound retained messages safely.
Approach
Deletes messages only after the containing checkpoint completes, renews visibility meanwhile, retains aborted-checkpoint messages for later completion, bounds in-flight state, releases malformed or unprocessed messages, and propagates receive, delete, and renewal failures.
Impact and scope
- Adds production-oriented Azure queue ingestion while preserving existing sink behavior.
- Coordinates external deletion with SeaTunnel checkpoints to provide explicit at-least-once semantics.
- Supports connection strings, shared keys, SAS tokens, JSON, delimited text, and plain or Base64 payloads.
Validation
- Added broad coverage for checkpoint completion, aborts, overlaps, retries, visibility renewal, malformed payloads, shutdown, and authentication.
- Passed 36 connector tests on JDK 8 and JDK 11 plus Azurite E2E compilation and the final Apache Build.
- Both authored commits and the GitHub-verified merge credit Goutam Adwant; upstream approval landed the feature.