Add a Deep Lake sink connector
Implemented SeaTunnel's first Deep Lake sink using the managed parameterized batch-query API for scalar, text, binary, vector, and supported array fields.
apache/seatunnel · #11852
New vector-data connector
SeaTunnel can now write batch or streaming rows to managed Deep Lake datasets through parameterized REST SQL without Python or native worker dependencies.
Problem
SeaTunnel lacked a native path to managed Deep Lake workspaces. A connector needed schema controls, safe parameterized writes, correct vector and binary conversion, primary-key mapping, and unambiguous failure behavior without adding Python or native runtime requirements.
Approach
Added append-only batch and streaming writes, create/validate/ignore schema modes, eager schema validation, parameterized SQL, field and primary-key mapping, checkpoint flushing, and conservative handling that avoids automatic retries after ambiguous write failures.
Impact and scope
- Connects SeaTunnel pipelines to a vector-aware data platform used for AI datasets and retrieval workloads.
- Supports vectors and nested numeric arrays while rejecting unsupported shapes before data is sent.
- Provides at-least-once delivery with explicit append-only semantics and no hidden duplicate-producing retry after uncertain writes.
Validation
- Added 14 tests for SQL, schemas, types, REST authentication, row conversion, checkpoints, close behavior, and failure handling on Java 8 and Java 11.
- Used deterministic local REST contract tests and passed the final Apache Build.
- Four authored commits and the GitHub-verified merge credit Goutam Adwant; upstream approval closed GH-11309.