Add an opt-in paraphrase benchmark suite
Added a separately selectable 12-prompt paraphrase suite for routing, CDC prerequisites, and connector options without changing the 100-task default benchmark.
apache/seatunnel · #12241
Evaluation infrastructure
SeaTunnel CLI changes can be checked against alternative task wording while retaining the same assertions and execution fixtures.
Problem
The benchmark supported only fixed prompt wording, so contributors lacked a reviewed way to detect regressions when equivalent requests were phrased differently.
Approach
Each variant inherits its canonical task contract and fixtures, has a distinct ID and saved parent provenance, and pins the parent fingerprint. Invalid selections, malformed variants, and parent drift fail before provider setup; existing scoring and comparison paths are reused.
Impact and scope
- Adds reproducible wording-variation checks without replacing or silently expanding the historical baseline.
- Prevents paraphrases from changing evaluation requirements by disallowing assertion and fixture overrides.
- The corpus is public, not an unseen holdout; offline tests do not establish model accuracy, generalization, or production impact.
Validation
- Fifty-two offline cases cover selection, inheritance, mutation isolation, malformed definitions, parent drift, scoring parity, provenance, skipped gates, and revision comparison.
- The complete CLI suite passed 181 tests and three subtests on Python 3.10 and 3.11; all 12 prompts were checked against their canonical requirements and the 100 baseline definitions remained identical.
- GitHub Build, Lint, and Python checks passed. No model calls or engine jobs were run, and the Java check compiled and packaged without executing the full Java test suite. The verified merge commit credits Goutam Adwant.