Merged improvementCorrectnessMerged Jul 21, 2026
Defer SFTP binary file discovery
Moved SftpFile binary once-mode file discovery out of REST submit-time source construction and into split enumerator startup.
apache/seatunnel · #11377
Problem
SftpFile binary jobs could enumerate and filter large remote directories while source config was being created, delaying REST submit-job responses before a jobId was returned and risking gateway timeouts.
Approach
Added a scoped deferred-discovery path for SftpFile binary once mode, kept eager discovery for formats that still need files during schema/config construction, and resolved deferred paths when the split enumerator opens.
Impact and scope
- Allows REST submit to return the jobId before expensive SFTP binary file discovery runs.
- Keeps non-binary and eager discovery behavior compatible for existing file connector paths.
- Adds discovery timing visibility with plugin, path, file count, and elapsed-time logging.
Validation
- Added unit coverage proving deferred config construction, existing eager behavior, enumerator-open discovery, and temporary resource cleanup.
- PR validation covered focused file-source tests and connector-file base and SFTP package builds.
- Maintainer approval accepted the scoped lifecycle change after source review and CI gates.