lean-bench

3.6. What this library is not🔗

  • A statistical benchmarking framework. No CIs, no t-tests, no outlier rejection. The default is one batch per param; bumping --outer-trials reports per-param median / min / max / spread but is not a substitute for proper statistical analysis.

  • A profiler. Per-call time is wall time; we don't break it down by function or measure allocations. (Allocation tracking is on the v0.2 roadmap, F3.)

  • A measure of compile-time, kernel-reduction-time, or #eval time. Only compiled-code wall time.