Compare saved AI CLI benchmark revisions
Built an offline comparator that pairs model, task, and trial identities across two saved benchmark runs and reports first-attempt and repair-budget transitions alongside aggregate deltas.
apache/seatunnel · #12192
Evaluation tooling feature
SeaTunnel contributors can now compare saved AI CLI benchmark runs without allowing aggregate gains to conceal task-level regressions.
Problem
The existing report summarized one run at a time. A higher overall pass rate could therefore hide a specific task moving from pass to fail, while incompatible settings or incomplete evidence could distort comparison denominators.
Approach
Adds task fingerprints, strict compatibility checks, explicit exclusion reasons, deterministic Markdown output, and a non-overwriting CLI. Missing, changed, skipped, contradictory, or incomplete evidence is excluded from both sides instead of being guessed as success or failure.
Impact and scope
- Makes regressions visible even when the aggregate benchmark score improves.
- Creates an auditable comparison boundary for model settings, task definitions, gates, trials, and repair budgets.
- Runs entirely offline with the Python standard library and leaves existing single-run reports unchanged.
Validation
- Added 46 comparison cases covering hidden regressions, incompatibilities, incomplete gates, generation failures, duplicate identities, deterministic output, escaping, and file preservation.
- The complete CLI suite passed on Python 3.10 and 3.11 with 129 tests and three subtests per runtime; Black, Ruff, Spotless, and the Java verification build also passed locally.
- A SeaTunnel reviewer approved the additive implementation; the GitHub-verified merge commit credits Goutam Adwant and closed GH-12189.