Kubernetes SIGs / inference-perf
Merged featureFeatureMerged Jul 23, 2026

Add Anthropic Messages API support

Added api.type anthropic_messages support to Kubernetes SIGs inference-perf for Anthropic-compatible benchmarking and trace replay.

kubernetes-sigs/inference-perf · #575

Problem

inference-perf supported OpenAI-style completion and chat APIs, but not Anthropic's /v1/messages contract. That limited benchmarking fidelity for Claude-style agentic workloads, tool-use round trips, and OTel trace replay through Anthropic-compatible serving stacks.

Approach

Implemented an Anthropic Messages API adapter with request serialization, system prompt and tool schema conversion, tool_use and tool_result mapping, non-streaming and streaming response parsing, client/datagen wiring, docs, CLI flag updates, and regression coverage.

Impact and scope

  • Extends inference-perf to benchmark Anthropic-compatible endpoints alongside OpenAI-compatible endpoints.
  • Improves replay fidelity for agentic traces that naturally use Anthropic message, tool-use, and tool-result semantics.
  • Adds reusable support across mock data generation, ShareGPT conversion, vLLM/mock clients, and OTel replay paths.

Validation

  • Added request-formatting, response-parsing, config, client-header, streaming, and replay tool-use capture coverage.
  • Addressed reviewer feedback around streaming behavior and tool-call coverage before approval.
  • PR validation covered ruff format/check, strict mypy, focused pytest suites, OTel replay tests, and CLI docs sync.