SWE-bench Multimodal Benchmark on Vals.ai: Closing Network Access Cuts Muse Spark Scores by 21 to 35 Points
· vals.ai
Vals.ai describes how it evaluates models on SWE-bench Multimodal, a set of 480 test tasks from eleven JavaScript repositories where GitHub issues depend on screenshots or recordings. Models run in mini-swe-agent, each in an isolated sandbox, with images downloaded beforehand and the upstream harness grading the final diff. Because instance ids reveal the upstream pull request number, agents with web access can locate the official fix, so every published run blocks internet access. The change lowered Muse Spark 1.2, 1.3 and 1.3 Max by 21 to 35 percentage points, with the 1.3 Max score falling from 77.92% to 43.33%, showing how much network leakage can inflate results.
Vals.ai has published the methodology behind its SWE-bench Multimodal leaderboard, and the most notable element is a controlled comparison showing how much agent results change when internet access is removed. The comparison suggests that benchmark scores for coding agents can be heavily distorted by a simple data leak.
What the benchmark measures
SWE-bench Multimodal, built by John Yang, Carlos Jimenez, Ofir Press and colleagues at Stanford University and Princeton University, extends the well-known SWE-bench idea of resolving real GitHub issues. It focuses on JavaScript repositories where the bug report relies on visual material. Examples include a chart with wrong axis labels, a diagram editor that loses a connection, and a syntax highlighter that colors a keyword wrongly. A screenshot or short recording conveys information that the written description omits.
The task is therefore harder than text-only issue resolution. A model must interpret the image, trace the symptom to the responsible code inside an unfamiliar front-end project, and write a patch that passes the repository's own tests. The test split contains 480 tasks drawn from eleven repositories.
How Vals.ai runs the evaluation
Vals.ai uses mini-swe-agent, the minimal reference agent from the SWE-agent team. The model receives the issue text and a shell, then works turn by turn until it decides it has finished. Each task executes in its own sandbox. Once the agent finishes, the upstream SWE-bench harness grades the final diff by running the repository's tests.
Because issues point to their screenshots through links, Vals.ai downloads each image ahead of time and attaches it to the first message together with the issue text. Each model is run once, and the reported error bars are the binomial standard error. With a single run per model and 480 tasks, differences of a few points between models should be read with caution.
The network access problem
The key methodological decision concerns internet access. A SWE-bench Multimodal instance id embeds the number of the upstream pull request that fixed the issue. That makes the reference patch easy to find for any agent that can browse or fetch from the web. An agent with general network access can look up the real fix instead of solving the problem itself.
To prevent this, every published Vals.ai run uses an agent without internet access. Screenshots are fetched before execution and read from local disk. The effect of closing the channel was large, with each tested model losing between 21 and 35 percentage points. Vals.ai's diagnostic figures for the Muse Spark family were:
- Muse Spark 1.2: 62.71% with network access, 41.25% without
- Muse Spark 1.3: 69.17% with network access, 42.08% without
- Muse Spark 1.3 Max: 77.92% with network access, 43.33% without
Analysis and implications
Two patterns stand out. First, the networked scores rise steadily across the three model variants, from roughly 63% to 78%, while the offline scores are nearly flat, ranging from about 41% to 43%. This suggests that much of the apparent gain in the networked setting reflects better ability to exploit external lookups, not stronger visual debugging or code reasoning. Under the offline protocol, the three models are close to one another, and given the binomial error bars on a single run, they may not be statistically distinguishable.
Second, the result is a reminder for the wider industry that agentic coding benchmarks are vulnerable to contamination in ways static question-answer sets are not. When an agent has a shell and the open web, public fixes, commit history and discussion threads are all reachable. Leaderboards that do not control for this can reward retrieval over engineering skill. Vals.ai's policy of publishing only sandboxed, offline runs makes its numbers a more conservative measure of genuine issue-solving ability, although they are not directly comparable with leaderboard entries produced under different network conditions.
For teams comparing coding agents, the practical lesson is to check how a benchmark handles network access, image delivery and harness grading before trusting a headline score. Multimodal tasks add another variable, since models must handle images supplied in the prompt, and Vals.ai standardizes this by attaching the downloaded files to the first message.
The article does not name the company behind the Muse Spark models, nor does it provide a full cross-vendor leaderboard in the excerpt, so conclusions about rankings among major labs cannot be drawn from this text alone.