API reference · Account
One test suite, its route and its evidence
GET/api/suites/{id}Signed in session
Its cases; what antimatter/auto:suite/<id> would choose now and why each other model is left out; the evidence per model; and the reasons to run it again.
Parameters
| Field | Type | What it is |
|---|---|---|
| id (path)required | string | The suite id (ev_...). |
Answer 200
The suite.
| Field | Type | What it is |
|---|---|---|
| id | string | |
| object | "eval" | |
| name | string | |
| pass_threshold | number | |
| revision | integer | Moves whenever the cases change; evidence from another revision is not routed on. |
| case_count | integer | |
| route_model | string | The model id that routes on this suite: antimatter/auto:suite/<id>. |
| schedule | object or null | |
| schedule. | boolean | |
| schedule. | integer | |
| schedule. | integer or null | |
| schedule. | string | key: each weekly run is the request of the key that last saved the suite. account: the dashboard saved it, and the personal account pays. One of "key", "account". |
| schedule. | string or null | Why the schedule stopped (the key that pays for it was revoked, expired or lost the chat scope). Saving the suite again resumes it. |
| created_at | integer | |
| updated_at | integer | |
| cases | array of object | |
| cases[] | string | |
| cases[] | string | |
| cases[] | string | |
| cases[] | array of object | |
| cases[] | string | One of "system", "user", "assistant". |
| cases[] | string | |
| cases[] | integer | |
| cases[] | number | |
| cases[] | object | |
| cases[] | object | How the answer is graded: one of exact, contains, regex, json_schema, number. exact compares the trimmed answer (ignore_case optional); contains needs every string in value; regex tests pattern with flags from i, m and s on a linear time engine (RE2), which refuses lookaround and backreferences, and is never run on an answer over 10000 characters; json_schema parses the answer as JSON and holds it to schema with the strict JSON engine; number reads the last number in the answer and allows tolerance, absolute or, with relative, a share of value. There is no model graded check. |
| cases[] | string | One of "exact", "contains", "regex", "json_schema", "number". |
| cases[] | any | |
| cases[] | boolean | |
| cases[] | string | |
| cases[] | string | |
| cases[] | object | |
| cases[] | number | |
| cases[] | boolean | |
| cases[] | number | |
| route | object | What the route would choose now: the models whose latest complete run passed, cheapest first at today's billed card applied to their mean tokens per case, and the models left out with why. A request carries the first 3 as its chain. |
| route. | string | |
| route. | string or null | |
| route. | array of object | |
| route. | array of object | |
| evidence | array of object | |
| suggestions | array of object | Reasons to run again: never_run, cases_changed, prices_changed (with models), new_models (cheaper priced text models the last run never tried). |
| suggestions[] | string | |
| last_run | object or null | |
| active_run | object or null |
Errors
- 401OpenAI shaped error.
- 404OpenAI shaped error.
Every error answers in one shape, with its type, code and the field it names. The whole list is in the docs' Errors section.