API reference

API reference · API

Shadow test a model on this key's own captured traffic

POST/v1/shadowAPI key

Chat scope. Only requests this key already keeps under its flight recorder capture are replayed; a key with capture off is refused with 409 and code capture_required. Each replay is an ordinary request billed as this key, never cached, never captured, never counted in rankings or reliability (its usage row is source shadow, session shadow:<test id>). An agent's key is refused. A 404 at every path while the operator has not turned shadow testing on.

Request body

application/json, required

Request body fields
FieldTypeWhat it is
namestringAt most 100 characters.
modelrequiredstringThe model whose traffic is shadowed: captured requests this model answered with a 2xx. At most 200 characters.
candidatesrequiredarray of string or "suggest"1 to 5 model ids, or "suggest" for the cheapest comparable models (no dearer on either rate, the original's context, its tool calling and JSON held).
suggest_countintegerFrom 1 to 5.
sample_raterequirednumberThe share of captured requests replayed, at most 0.1. A capture is in the sample by a hash of the test and its trace id. At most 0.1.
daily_caprequiredintegerThe most captured requests sampled in one UTC day, at most 1000. From 1 to 1000.
monthly_ceiling_microsrequiredintegerRequired. The most every replay and its comparison may be billed in a UTC month; each replay's worst case is held against what is left before it is sent. From 1 to 9007199254740991.
compareobjectauto (JSON field by field, else normalized text) unless set; embedding with embedding_model (cosine at cosine_threshold, 0.9 unless set); judge with a judge_model you name; suite with suite_id and suite_budget_micros.
compare.methodrequiredstringOne of "auto", "embedding", "judge", "suite".
compare.embedding_modelstringAt most 200 characters.
compare.judge_modelstringAt most 200 characters.
compare.suite_idstringAt most 100 characters.
compare.suite_budget_microsintegerAt most 9007199254740991.
compare.cosine_thresholdnumberFrom 0 to 1.
thresholdnumberThe agreement the 95% interval's lower end must reach for a candidate to be ready, 0.9 unless set. From 0 to 1.
min_samplesintegerComparisons a candidate needs before it is ready, 30 unless set. From 10 to 100000.
key_idinteger or nullThe dashboard only: the key whose traffic is tested, or null for every personal key. A key over /v1/shadow tests its own.

Example

{
  "model": "antimatter/auto",
  "candidates": [
    "string"
  ],
  "sample_rate": 0,
  "daily_cap": 1,
  "monthly_ceiling_micros": 1
}

Answer 201

The test, with suite_run when a suite was attached.

Fields of the 201 answer
FieldTypeWhat it is
idstringshd_ and 24 hex digits.
object"shadow.test"
namestring
modelstring
scopestringOne of "key", "account".
key_idinteger or null
statusstringOne of "active", "stopped".
stopped_reasonstring or null
held_untilinteger or nullUnix seconds: the month's ceiling is reached, nothing runs before this.
sample_ratenumber
daily_capinteger
sampled_todayinteger
monthly_ceiling_microsinteger
month_spent_microsinteger
month_reserved_microsinteger
spent_microsinteger
compareobject
thresholdnumber
min_samplesinteger
candidatesarray of object
candidates[].modelstring
candidates[].suggestedboolean
switchobject or nullThe switch standing, if any: the candidate and the keys it routes.
switch.tostring
switch.key_idsarray of integer
switch.switched_atinteger
switch.autobooleanMade by the gateway under auto savings.
autoobjectAuto savings: whether the gateway may switch this test itself once a candidate is ready.
auto.enabledboolean
auto.enabled_atinteger or null
auto.switched_atinteger or null
auto.switched_tostring or null
auto.off_reasonstring or nullWhy it is off when it was on: turned off, reverted, or why a switch failed.
savingsarray of objectThe newest weekly reports (Monday to Monday, UTC) of what a standing switch saved: the requests it moved, read from their usage rows, what they were billed, and what they would have cost on the original at the report's measured cost ratio.
savings[].week_startinteger
savings[].fromstring
savings[].tostring
savings[].autoboolean
savings[].requestsinteger
savings[].cost_microsinteger
savings[].baseline_microsinteger
savings[].saved_microsinteger
savings[].rationumber
created_atinteger
updated_atinteger

Errors

  • 400OpenAI shaped error.
  • 401OpenAI shaped error.
  • 403OpenAI shaped error.
  • 404OpenAI shaped error.
  • 409OpenAI shaped error.
  • 429Rate limited, or a spent budget or spend cap. The type distinguishes rate_limit_error, policy_exceeded, and spend_cap_exceeded.
  • 503OpenAI shaped error.

Every error answers in one shape, with its type, code and the field it names. The whole list is in the docs' Errors section.

Antimatter