API reference · API
One run: pass rates, cost, latency, the frontier and the failures
GET/v1/evals/{id}/runs/{runId}API key
Per model: weighted pass rate, billed cost per case, median latency, and whether it is on the cost and pass rate frontier; the cheapest model graded on every case at or over ?threshold= (0 to 1, the suite's unless given); and the failures with the model's output. Reading an unfinished run moves it on.
Parameters
| Field | Type | What it is |
|---|---|---|
| id (path)required | string | The suite id (ev_...). |
| runId (path)required | string | The run id (evrun_...). |
| threshold (query) | number |
Answer 200
The run.
| Field | Type | What it is |
|---|---|---|
| id | string | |
| object | "eval.run" | |
| suite_id | string | |
| status | string | One of "queued", "running", "completed", "budget_exhausted", "cancelled". |
| trigger | string | One of "manual", "scheduled". |
| revision | integer | |
| models | array of string | |
| case_count | integer | |
| budget_micros | integer | |
| spent_micros | integer | |
| reserved_micros | integer | Worst cases held for cases in flight. |
| progress | object | |
| progress. | integer | |
| progress. | integer | |
| created_at | integer | |
| started_at | integer or null | |
| finished_at | integer or null | |
| threshold | number | |
| cheapest_passing | string or null | The cheapest model graded on every case at or over the threshold. |
| summary | array of object | |
| summary[] | string | |
| summary[] | number or null | Weighted share of graded cases passed. |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | integer | |
| summary[] | boolean | |
| summary[] | integer | |
| summary[] | integer or null | |
| summary[] | integer or null | |
| summary[] | boolean | No other model is as cheap or cheaper and passes as much or more. |
| failures | array of object | At most 200, each with the model's output clipped to 4000 characters. |
| failure_count | integer |
Errors
- 400OpenAI shaped error.
- 401OpenAI shaped error.
- 404OpenAI shaped error.
Every error answers in one shape, with its type, code and the field it names. The whole list is in the docs' Errors section.