What the grid runs on, and one thing worth knowing about the timings.
Everything runs locally on a Apple M5 Pro Mac
(24 GB, macOS 26.5.2). Two harnesses drive the models: Claude Code 2.1.220 for Opus 5 and Fable 5, and
Codex CLI0.145.0 on the
codex-p profile for the GPT models. Video work is
ffmpeg 8.1.2; the generator is Python 3.14.6; the benchmark
orchestrator is Rust 1.96.1.
Each run gets its own directory under /private/tmp/vidtask holding
nothing but clip.mp4, with a 30-minute ceiling and no cap on turns or tokens. How to approach the
video is entirely the model's choice.
One caveat on that ceiling for this grid: it killed the agent process but not the ffmpeg and python children it had spawned, and those inherit the output pipe — so a few runs kept going well past the limit, the longest reaching 52 minutes. The harness now signals the whole process group, but that landed after this grid started, so treat the ceiling as soft here. Actual durations are recorded per run either way.
The accuracy pass runs 8 agents at once, all of them driving ffmpeg and Python over 1800-frame video. Wall-clock under that load measures contention as much as it measures the model, so treat the avg t column as a rough shape rather than model speed. Every row records its concurrent-run count and the machine's load average, and clean timings come from a separate sequential pass instead of being quietly mixed into the same average.