# Zero2Repo Benchmark Finds Top AI Coding Agent Builds 15 of 17 Repos, Weakest's Best Run Manages 8

URL: https://edufabtech.com/news/zero2repo-benchmark-coding-agents-build-repositories-from-scratch  
Section: News · Artificial Intelligence  
Published: 2026-10-02 · Updated: 2026-11-16  
Publisher: EduFabTech (https://edufabtech.com)

> A new benchmark gives AI coding agents a requirements document and an empty folder instead of a bug to patch. The best agent finished 15 of 17 projects; the weakest's best run finished 8.

A benchmark released this week asks a question that most coding-agent evaluations have quietly avoided: can these systems build a piece of software from nothing, rather than patch one that already exists? Zero2Repo, posted to arXiv on September 29, 2026 by a 26-author team led by researchers at Gradient Data and McGill University, hands an agent a product requirements document, an interface contract, and an empty workspace, and checks whether what comes out actually works.

The headline result is narrow but pointed. Across the benchmark's 17 released tasks, the strongest agent tested — OpenAI's GPT-6 Astra running inside the Codex harness — completed 15. The weakest agent's best run, xAI's Grok 4.7 High running inside Cursor CLI, completed 8. These are not toy exercises: the tasks are rebuilt from real, version-pinned open-source projects, meaning frontier models had a reasonable chance of having seen the original code during training, yet a fully correct repository was still not something every agent could reliably produce.

![A bar chart compares pass rates across the three tested configurations: GPT-6 Astra/Codex at 90.9%, Claude Opus 5.5/Claude Code at 81.8%, and Grok 4.7 High/Cursor CLI at 63.6%.](https://edufabtech.com/uploads/2026/10/zero2repo-benchmark-coding-agents-build-repositories-from-scratch-2.png)
*A bar chart compares pass rates across the three tested configurations: GPT-6 Astra/Codex at 90.9%, Claude Opus 5.5/Claude Code at 81.8%, and Grok 4.7 High/Cursor CLI at 63.6%. — EduFabTech · Own work*

## How the test was built

Most existing coding benchmarks fall into one of two categories. Tools like SWE-bench hand an agent an existing repository with a bug and ask for a patch; tools like HumanEval ask for a single isolated function. Neither resembles the workflow the paper's authors say many users now lean on: describe what you want, and let the agent build the whole project. Zero2Repo targets that gap specifically, covering parsers, protocols, command-line tools, security utilities, and software that depends on external resources, written in Python, TypeScript, Go, and C++.

Each task is produced by a pipeline that converts a real open-source project into a behavioral specification, a reproducible container environment, and a hidden acceptance-test suite. Before release, every task must pass a reference implementation and reject deliberately broken ones, so the tests are checked against both correct and incorrect code, not written once and assumed to work. Agents run inside isolated containers with no access to the original repository or to the hidden tests, and scoring is strictly binary: a task counts as solved only if every hidden test passes, with no partial credit and no language model acting as judge.

## What the agents actually got wrong

The paper's results table, covering each model's strongest run, looks like this:

| Model | Harness | Tasks solved | Pass rate |
| --- | --- | --- | --- |
| GPT-6 Astra | Codex | 15 / 17 | 88.2% |
| Claude Opus 5.5 | OpenCode | 14 / 17 | 82.4% |
| Grok 4.7 High | Cursor CLI | 8 / 17 | 47.1% |

The more telling number sits beneath the pass/fail count. On every task an agent failed, its submission still passed between 90% and 99% of the hidden tests, and for the two strongest agents, most of the failing tests trace back to a single omitted edge case or a low-frequency rule buried in the specification, rather than an entire missing subsystem. In other words, the agents were not failing to understand the software they were asked to build — they were leaving small, specific holes in it, the kind that a binary pass/fail test suite catches and a casual read of the code would not.

![A side-by-side comparison shows how a failed task can still pass 95.8% of hidden tests, contrasted with the 100% binary threshold actually required to score a solve.](https://edufabtech.com/uploads/2026/10/zero2repo-benchmark-coding-agents-build-repositories-from-scratch-3.png)
*A side-by-side comparison shows how a failed task can still pass 95.8% of hidden tests, contrasted with the 100% binary threshold actually required to score a solve. — EduFabTech · Own work*

## Why a count of 17 still matters

The sample is still small by design. The project's maintainers, who publish the benchmark's code and task data separately from the paper, have grown the suite from its original 11 tasks to the current 17 and say they plan to expand further, toward 30–50 tasks with broader language coverage. That makes the percentages above provisional rather than definitive — a handful of additional tasks could still shift any single model's score by several points.

What the result does establish, with the kind of granularity a benchmark is supposed to provide, is a specific failure mode. Tools that already report strong scores on patch-style benchmarks are not automatically equipped for end-to-end construction, and the gap between "mostly working" and "fully working" in that setting is concentrated in exactly the edge cases that hidden, deterministic tests are built to catch. For teams deciding how much unsupervised scope to hand a coding agent — a single function, a bug fix, or an entire new service — that distinction between near-complete and complete is the one the benchmark was built to measure.

## Sources

1. Pei Yang, Tianyu Shi, Yuhang Yao, et al.. (2026). Zero2Repo: Can Coding Agents Build Repositories from Scratch?. arXiv preprint https://arxiv.org/abs/2609.38269
2. OpenEdgeHQ. (2026). Zero2Repo: from a spec to a working repo (benchmark code and task data). GitHub https://github.com/OpenEdgeHQ/Zero2Repo
3. OpenAI. (2026). GPT-6 Astra: the next generation in intelligence for work. OpenAI https://openai.com/index/gpt-6-astra-next-generation-work/
4. Anthropic. (2026). Claude Opus 5.5. Anthropic https://www.anthropic.com/claude/opus
5. xAI. (2026). Introducing Grok 4.7. xAI https://x.ai/news/grok-4-7

---
Source page: https://edufabtech.com/news/zero2repo-benchmark-coding-agents-build-repositories-from-scratch  
Quote with a link to the source page. © EduFabTech.
