RealCLI is a paired benchmark of real-world CLI tasks that measures both what agents can do on their own and how well they work with people. Every task is run twice — once autonomously (RealCLI-Exec) and once with a scripted user in the loop (RealCLI-Collab) — in the same containerized environment with the same verifier.
The agent receives the full task specification up front and works alone. Scored by Rexec — pass rate against a deterministic verifier (or VLM judge for visual tasks).
The agent starts from an incomplete request; a scripted user simulator reveals information and injects probes (clarifications, corrections, constraint changes, authorization decisions). Scored by Rcollab (pass rate), Scollab (collaboration quality across 5 dimensions), and their product Jcollab = Rcollab × Scollab.