Antimatter

PUT/v1/evals/{id}API key

Chat scope. The whole suite; its revision moves when the cases changed, and evidence from an older revision stops routing until the suite runs again.

Parameters

Path and query parameters
Field	Type	What it is
id (path)required	string	The suite id (ev_...).

Request body

application/json, required

Request body fields
Field	Type	What it is
namerequired	string
pass_threshold	number	The weighted share of cases a model must pass to route, 0 to 1. 0.9 unless set.
casesrequired	array of object
cases[].name	string
cases[].prompt	string
cases[].system	string
cases[].messages	array of object
cases[].messages[].role	string	One of "system", "user", "assistant".
cases[].messages[].content	string
cases[].max_tokens	integer
cases[].temperature	number
cases[].response_format	object
cases[].checkrequired	object	How the answer is graded: one of exact, contains, regex, json_schema, number. exact compares the trimmed answer (ignore_case optional); contains needs every string in value; regex tests pattern with flags from i, m and s on a linear time engine (RE2), which refuses lookaround and backreferences, and is never run on an answer over 10000 characters; json_schema parses the answer as JSON and holds it to schema with the strict JSON engine; number reads the last number in the answer and allows tolerance, absolute or, with relative, a share of value. There is no model graded check.
cases[].check.typerequired	string	One of "exact", "contains", "regex", "json_schema", "number".
cases[].check.value	any
cases[].check.ignore_case	boolean
cases[].check.pattern	string
cases[].check.flags	string
cases[].check.schema	object
cases[].check.tolerance	number
cases[].check.relative	boolean
cases[].weight	number
schedule	object or null	weekly: true checks the suite once a week and runs it again, inside budget_micros, only when the cases changed, a tested price moved, or a cheaper untested model arrived. The weekly run is a request of the key that last saved the suite (or of the account, when the dashboard saved it), checked again each week.
schedule.weekly	boolean
schedule.budget_micros	integer

Example

{
  "name": "string",
  "pass_threshold": 0,
  "cases": [
    {
      "prompt": "Say hello in five words.",
      "check": {
        "type": "exact"
      }
    }
  ],
  "schedule": {
    "weekly": false,
    "budget_micros": 0
  }
}

Answer 200

The suite.

Fields of the 200 answer
Field	Type	What it is
id	string
object	"eval"
name	string
pass_threshold	number
revision	integer	Moves whenever the cases change; evidence from another revision is not routed on.
case_count	integer
route_model	string	The model id that routes on this suite: antimatter/auto:suite/<id>.
schedule	object or null
schedule.weekly	boolean
schedule.budget_micros	integer
schedule.last_checked_at	integer or null
schedule.paid_by	string	key: each weekly run is the request of the key that last saved the suite. account: the dashboard saved it, and the personal account pays. One of "key", "account".
schedule.stopped	string or null	Why the schedule stopped (the key that pays for it was revoked, expired or lost the chat scope). Saving the suite again resumes it.
created_at	integer
updated_at	integer
cases	array of object
cases[].name	string
cases[].prompt	string
cases[].system	string
cases[].messages	array of object
cases[].messages[].role	string	One of "system", "user", "assistant".
cases[].messages[].content	string
cases[].max_tokens	integer
cases[].temperature	number
cases[].response_format	object
cases[].checkrequired	object	How the answer is graded: one of exact, contains, regex, json_schema, number. exact compares the trimmed answer (ignore_case optional); contains needs every string in value; regex tests pattern with flags from i, m and s on a linear time engine (RE2), which refuses lookaround and backreferences, and is never run on an answer over 10000 characters; json_schema parses the answer as JSON and holds it to schema with the strict JSON engine; number reads the last number in the answer and allows tolerance, absolute or, with relative, a share of value. There is no model graded check.
cases[].check.typerequired	string	One of "exact", "contains", "regex", "json_schema", "number".
cases[].check.value	any
cases[].check.ignore_case	boolean
cases[].check.pattern	string
cases[].check.flags	string
cases[].check.schema	object
cases[].check.tolerance	number
cases[].check.relative	boolean
cases[].weight	number
route	object	What the route would choose now: the models whose latest complete run passed, cheapest first at today's billed card applied to their mean tokens per case, and the models left out with why. A request carries the first 3 as its chain.
route.model	string
route.pick	string or null
route.passing	array of object
route.excluded	array of object
evidence	array of object
suggestions	array of object	Reasons to run again: never_run, cases_changed, prices_changed (with models), new_models (cheaper priced text models the last run never tried).
suggestions[].reason	string
last_run	object or null
active_run	object or null

Errors

400OpenAI shaped error.
401OpenAI shaped error.
403OpenAI shaped error.
404OpenAI shaped error.

Every error answers in one shape, with its type, code and the field it names. The whole list is in the docs' Errors section.

Try it

The same call with curl

curl -X PUT https://useantimatter.com/v1/evals/{id} \
  -H "Authorization: Bearer $ANTIMATTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"string","pass_threshold":0,"cases":[{"prompt":"Say hello in five words.","check":{"type":"exact"}}],"schedule":{"weekly":false,"budget_micros":0}}'