lean-bench

2.3. Required vs optional fields🔗

2.3.1. Required for both kinds🔗

key

type

meaning

schema_version

integer

Always 1 for v1 rows.

kind

string

"parametric" or "fixed". Readers MUST reject a row whose kind mismatches the expected kind for the parser.

function

string

Fully-qualified Lean name of the registered benchmark.

total_nanos

integer

Wall time in nanoseconds for the measured work.

status

string

One of "ok", "timed_out", "killed_at_cap", "error".

2.3.2. Required for parametric only🔗

key

type

meaning

param

integer

Ladder parameter the function was invoked with.

inner_repeats

integer

Auto-tuned inner repeat count. 0 on synthesized error rows.

2.3.3. Required for fixed only🔗

key

type

meaning

repeat_index

integer

0-based index across the configured repeats.

2.3.4. Optional fields (both kinds)🔗

These are emitted today. Readers MUST treat absence as if the field were null.

key

type

meaning

result_hash

string | null

Hex-prefixed 0xDEADBEEF literal (case-insensitive). null when the function's return type lacks Hashable.

error

string | null

Free-form message for status == "error". null otherwise.

env

object | null

Reproducibility metadata captured at child startup — see the Environment metadata section below. Emitted on every row by current writers.

alloc_bytes

integer | null

Total bytes allocated by the Lean runtime over the child's lifetime. Always null today — Lean 4 has no portable in-process API to read this. The key is reserved so the field can land additively when a future Lean release exposes allocation counters.

peak_rss_kb

integer | null

Peak resident-set size of the child process, in kibibytes (1024 bytes). On Linux read from /proc/self/status (VmHWM:); null on macOS, Windows, and any other platform without /proc.

2.3.5. Optional fields (parametric only)🔗

key

type

meaning

per_call_nanos

number | null

total_nanos / inner_repeats precomputed. Derived; readers MAY recompute it from total_nanos and inner_repeats when absent. Emitted as null on synthesized error / killed-at-cap rows where inner_repeats == 0 (a real division-by-zero), to keep every emitted row valid JSON.

cache_mode

string

"warm" (auto-tuned inner repeats inside one child, the v0.1 default) or "cold" (single untuned invocation; the parent respawns the child for every ladder rung). See the Cache-modes section of the Advanced page.

2.3.6. Reserved-for-future-use fields🔗

The following keys are reserved by the issues that prompted this schema and will land as additive (non-breaking) fields. They are listed here so concurrent feature work claims a stable name from the start:

No keys are currently reserved — every previously-reserved name has either landed (alloc_bytes / peak_rss_kb from issue #6 are now real optional fields, listed above) or been released (budget_status from issue #9 was not needed; see below).

Producers MUST NOT silently start emitting new keys with the same name as any landed field except as described in that field's landing PR.

The CI-budget mode landed in issue #9 surfaces budgeted-run state at the export-document level rather than per-row. There is no row-level budget_status field: a benchmark that's skipped because the budget was exhausted before it started doesn't produce any rows at all, and a benchmark whose ladder was cut short between rungs records budgetTruncated on the export's per-result object (see Export budget summary, below). The reserved name budget_status was carried by earlier drafts of this schema doc but is now released; future feature work is free to claim it.