API reference

API reference · API

A shadow test's report

GET/v1/shadow/{id}/reportAPI key

Per candidate: the agreement rate with its sample count and 95% Wilson interval, replay cost against what the originals were billed, latency p50 and p95 beside the original's, failures, and ready (enough comparisons and the interval's lower end at the threshold; shadow.candidate_ready fires once). A 404 at every path while the operator has not turned shadow testing on.

Parameters

Path and query parameters
FieldTypeWhat it is
id (path)requiredstring

Answer 200

The report.

Fields of the 200 answer
FieldTypeWhat it is
idstring
object"shadow.report"
modelstring
statusstring
thresholdnumber
min_samplesinteger
confidencenumber
candidatesarray of object
candidates[].modelstring
candidates[].suggestedboolean
candidates[].replaysobject
candidates[].replays.doneinteger
candidates[].replays.failedinteger
candidates[].replays.pendinginteger
candidates[].replays.skippedinteger
candidates[].agreementobject
candidates[].agreement.methodstring
candidates[].agreement.comparedintegerThe sample count.
candidates[].agreement.agreedinteger
candidates[].agreement.ratenumber or null
candidates[].agreement.intervalobject or nullThe 95% Wilson score interval; null with nothing compared.
candidates[].agreement.interval.lownumber
candidates[].agreement.interval.highnumber
candidates[].agreement.original_ratenumber or nullThe suite method: the original model's pass rate on the suite.
candidates[].readyboolean
candidates[].notified_atinteger or null
candidates[].costobject
candidates[].cost.replay_microsintegerWhat the replays were billed.
candidates[].cost.original_microsintegerWhat the same requests were billed on the original.
candidates[].cost.rationumber or null
candidates[].cost.compare_microsintegerEmbeddings or the judge, billed.
candidates[].latency_msobject
candidates[].latency_ms.p50integer or null
candidates[].latency_ms.p95integer or null
candidates[].original_latency_msobject
candidates[].original_latency_ms.p50integer or null
candidates[].original_latency_ms.p95integer or null
candidates[].failuresarray of object
candidates[].switchedboolean

Errors

  • 401OpenAI shaped error.
  • 404OpenAI shaped error.

Every error answers in one shape, with its type, code and the field it names. The whole list is in the docs' Errors section.

Antimatter