Snowflake's engineering team has described SnowPR, a private benchmark meant to answer a question public leaderboards cannot: which coding agent performs best on the code Snowflake engineers actually change. According to the company's engineering blog, public suites such as SWE-bench draw from a small set of open source Python projects, and model developers increasingly tune for them, which weakens the signal. Snowflake's monorepo is a very different environment, with about 63,000 nontest source files and 17 million lines of nontest code, mostly Java and C++, all built as one Bazel graph.

What SnowPR contains

The benchmark has 65 tasks across 11 product areas. Each is a real pull request that was reviewed and merged. Sixty-two come from the monorepo and three from a separate Go repository for developer tools. The agent receives the pre-change code and a description of the work, and the hidden tests that shipped with the original PR decide pass or fail, executed through Snowflake's own Bazel builds on a remote build farm. Fifty-two tasks change the Java control plane, including the SQL compiler and metadata layer (ten tasks each). Eight touch the C++ execution engine, two cover hybrid tables across Java and C++, and three involve Go tooling. The team predicted 43 tasks as hard, 19 as medium and three as easy.

Why it is harder than public sets

Snowflake says its repository has about 21 times more nontest files and 39 times more lines than the average SWE-bench repository. Reference fixes average 4.0 files and 262 lines, versus 1.7 files and 33 lines in SWE-bench, and 34 tasks touch two or more modules. Tests often live far from the code being changed: in 33 tasks the graded test sits in a shared tree separate from the target module, and 30 tasks need two or more Bazel targets to grade. In 39 tasks the tests launch their own FoundationDB instance, which stores Snowflake's metadata, and check transactional behavior. A C++ task needs about 21,700 build actions and 32 GB of memory before the agent begins. Because all tasks come from private history, none can appear in public training data.

Construction and safeguards

Snowflake emphasizes hardening. In its largest mining pass, simple rules reduced 13,965 merged changes from one month to 4,218 that modify Java source and add at least one behavior test. An open-weight judge running on the company's GPUs then scored candidates for hardness, defined as reasoning depth along a code path rather than patch size, and for gradeability, rejecting PRs with proxy tests, exact-wording checks or requirements a description cannot specify. Human authors validate the remaining tasks. Each must have enough new and regression tests, pass with the real fix and fail without it, survive 11 automatic checks and withstand an adversarial review. All stages except writing the description are scripted, so the benchmark can follow Snowflake's continuous integration.

Agents and results

SnowPR doubles as a test bench for Snowflake's own agent, Cortex Code. The team compared its two modes, code mode (file, shell and code-search tools) and REM mode (REPL Execution Model), with Cursor, OpenAI's Codex and Anthropic's Claude Code, using the same models where possible. REM mode exposes file, shell, search and SQL operations as Python functions inside a single REPL tool and is internal only for now. Each agent tried each task three times; pass@1 is the share of passing trials and pass@3 the share of tasks solved at least once. The reported comparison covers seven agents: opus-5 in four harnesses and haiku-5.5 in three.

With opus-5, Cortex Code in REM mode matched the pass rate of code mode, Claude Code and Cursor at the lowest cost per trial, and solved the most tasks of any agent, 62 of 65. With haiku-5.5, it passed more trials than Claude Code, at $0.24 per trial. Cost includes subagents and is billed or token-priced. The post also notes that changes spanning Java and C++ passed 55% of trials, against 80% for Java alone, showing that polyglot monorepo work remains a weak point.

Implications

The work reflects a growing pattern in which large engineering organizations build private, repository-specific evaluations rather than trusting public leaderboards. For vendors, the results suggest harness design, such as REM mode's code-as-tool interface, can affect cost efficiency as much as the underlying model. Because REM mode is not yet customer-available and the results come from the vendor's own benchmark, they should be read as indicative rather than independent. Still, the methodology, including execution-based grading, contamination avoidance and adversarial validation, offers a template for enterprise buyers comparing coding agents.