Add repeatable Pinot connector E2E coverage
Replaced the Pinot connector's manual shell verification with an automated end-to-end suite that validates JSON field mapping and ten-message batch ingestion.
apache/iggy · #3922
Connector reliability improvement
Apache Iggy now continuously verifies real Pinot ingestion through a Gradle-integrated JUnit and Testcontainers workflow.
Problem
The previous script relied on fixed ports, fixed sleeps, and a mutable Pinot image, ran outside Gradle, and could report table creation success even when Pinot had rejected the connector workflow.
Approach
Built a bounded JUnit/Testcontainers test with in-memory Pinot configuration, dynamic host-port forwarding for the branch-built Iggy server, deterministic image handling, Gradle input tracking, service-log diagnostics, and CI test-result publication.
Impact and scope
- Turns a manual connector check into repeatable regression coverage within the project's standard Java build.
- Exercises the documented Iggy-to-Pinot ingestion path rather than validating only isolated connector code.
- Makes failures diagnosable through bounded polling, container logs, and surfaced JUnit results instead of fixed delays.
Validation
- Passed the Pinot connector check, repository verification, and the E2E suite against a branch-built external server.
- All final commit checks succeeded, and Codecov reported every modified coverable line as tested.
- The authored implementation and GitHub-verified merge credit Goutam Adwant; three upstream approvals landed the change and closed GH-2598.