Apache Software Foundation / SeaTunnel
Merged improvementCorrectnessMerged Jul 21, 2026

Defer SFTP binary file discovery

Moved SftpFile binary once-mode file discovery out of REST submit-time source construction and into split enumerator startup.

apache/seatunnel · #11377

Problem

SftpFile binary jobs could enumerate and filter large remote directories while source config was being created, delaying REST submit-job responses before a jobId was returned and risking gateway timeouts.

Approach

Added a scoped deferred-discovery path for SftpFile binary once mode, kept eager discovery for formats that still need files during schema/config construction, and resolved deferred paths when the split enumerator opens.

Impact and scope

  • Allows REST submit to return the jobId before expensive SFTP binary file discovery runs.
  • Keeps non-binary and eager discovery behavior compatible for existing file connector paths.
  • Adds discovery timing visibility with plugin, path, file count, and elapsed-time logging.

Validation

  • Added unit coverage proving deferred config construction, existing eager behavior, enumerator-open discovery, and temporary resource cleanup.
  • PR validation covered focused file-source tests and connector-file base and SFTP package builds.
  • Maintainer approval accepted the scoped lifecycle change after source review and CI gates.