Merged featureFeatureMerged Jul 23, 2026
Add Anthropic Messages API support
Added api.type anthropic_messages support to Kubernetes SIGs inference-perf for Anthropic-compatible benchmarking and trace replay.
Problem
inference-perf supported OpenAI-style completion and chat APIs, but not Anthropic's /v1/messages contract. That limited benchmarking fidelity for Claude-style agentic workloads, tool-use round trips, and OTel trace replay through Anthropic-compatible serving stacks.
Approach
Implemented an Anthropic Messages API adapter with request serialization, system prompt and tool schema conversion, tool_use and tool_result mapping, non-streaming and streaming response parsing, client/datagen wiring, docs, CLI flag updates, and regression coverage.
Impact and scope
- Extends inference-perf to benchmark Anthropic-compatible endpoints alongside OpenAI-compatible endpoints.
- Improves replay fidelity for agentic traces that naturally use Anthropic message, tool-use, and tool-result semantics.
- Adds reusable support across mock data generation, ShareGPT conversion, vLLM/mock clients, and OTel replay paths.
Validation
- Added request-formatting, response-parsing, config, client-header, streaming, and replay tool-use capture coverage.
- Addressed reviewer feedback around streaming behavior and tool-call coverage before approval.
- PR validation covered ruff format/check, strict mypy, focused pytest suites, OTel replay tests, and CLI docs sync.