Apache Software Foundation / SeaTunnel
Merged upstreamFeatureMerged Aug 14, 2026

Enable continuous discovery for S3 and OSS file sources

Enabled Apache SeaTunnel S3File and OssFile sources to expose polling-based continuous discovery and update-mode options for streaming object-storage ingestion.

apache/seatunnel · #11789

Problem

The shared file source already supported polling-based continuous discovery, but the S3File and OssFile factories did not expose the required options, leaving object-storage sources behind the local, HDFS, FTP, and SFTP paths.

Approach

Exposed continuous discovery, update comparison, and post-sync options in the S3 and OSS factories; required target_path for update sync mode; added validation tests, docs, and a focused MinIO E2E test.

Impact and scope

  • Lets object-storage jobs discover new or changed S3 and OSS objects continuously while preserving the existing once/full defaults.
  • Moves SeaTunnel's object-storage CDC roadmap forward with a polling-based first slice before event notifications, changelog output, and lazy content fetching.
  • Gives users documented English and Chinese examples for streaming object-store synchronization.

Validation

  • Added S3File and OssFile factory tests for option exposure and target_path validation under update mode.
  • Added a MinIO E2E test that copies an initial object, uploads a later object after startup, discovers it, copies it, and cancels the streaming job cleanly.
  • Verified S3File and OssFile connector tests locally, focused S3 E2E with Testcontainers, and merged after approval confirmed green CI.