# Do More Sub-Agents Make Better Code?


![](https://leafw-blog-pic.oss-cn-hangzhou.aliyuncs.com/parallel-subagents-main-agent-cover.png)

Many agent products now let a main agent spin up a batch of sub-agents to work in parallel. But how well does multi-agent collaboration actually work, and is it always better than a single agent? Few people bother to compare.

I recently read a paper that does exactly that: *When Sub-Agents Work in Parallel: The Promises and Pitfalls of Dynamic Concurrency in Long-Horizon Coding Tasks*, https://arxiv.org/abs/2610.10263 . The work comes from Nanjing University, Central University of Finance and Economics, and the University of Illinois Urbana-Champaign, by Han Li, HanHaoNing Li, Ziqian Jiang, and Yiling Lou. They turned the built-in sub-agent concurrency of three coding agents, Codex, Claude Code, and Kimi Code, on and off, and ran each setting on 354 tasks: 2,124 runs in total, with 11.2 billion tokens logged, which comes to $20,780.02 at standard API prices.

The findings are mixed. The longer the task, the more likely concurrency helps; on short tasks like SWE-bench Verified, turning on concurrency dropped the pass rate of Claude Code and Kimi Code by 24 and 18 points. Token usage was 1.41 to 3.31 times that of sequential runs, and wall-clock time was usually longer too. The whole paper revolves around one question: once sub-agents are running in parallel, can the main agent actually make use of their results?

Below I'll walk through it roughly in this order: background, experimental design, results, failure taxonomy, and takeaways.

## Background and Question

Earlier coding agents, such as TRAE (the paper cites ByteDance's open-source software engineering agent Trae Agent) and OpenHands, mostly used fixed workflows for multi-agent collaboration: how many sub-agents, what roles, who does which part, and where they sync were all hard-coded in advance. That works well on clearly structured tasks, at the cost of the developer having to figure out the decomposition beforehand. In many tasks the dependencies only become clear halfway through, and a pre-planned workflow no longer fits.

The "dynamic concurrency" studied in the paper means the main agent decides during execution, based on how the task is going, whether to spin up sub-agents, when to do it, and what work to hand out, and is then responsible for collecting results, aligning shared interfaces and files, and verifying the integrated output.

The authors point out that most existing evaluations look at single agents or multi-agent systems with fixed protocols, and the tasks are mostly bounded ones like SWE-bench that need only a few dozen lines of patch. Dynamic concurrency, however, targets long tasks such as generating a repository from scratch or continuous multi-version development, which require tens of thousands of lines of code. The paper asks four questions: how outcomes and resource consumption change, how sub-agents are dispatched, what failures occur, and under what conditions concurrency helps.

## How the Experiments Work

### Three Concurrency Mechanisms

The figure below comes from Figure 1 of the paper and shows how each of the three agents dispatches work.

![Dynamic concurrency mechanisms of the three coding agents](https://leafw-blog-pic.oss-cn-hangzhou.aliyuncs.com/parallel-subagents-three-agent-concurrency-mechanisms.png)

At the top is Claude Code's Dynamic Workflow. The main agent first writes a JavaScript workflow script, using `parallel()`, `agent()`, and `pipeline()` to arrange several parallel stages and join points. The example in the figure dispatches 8 sub-agents in three stages: gathering evidence (reproducing, tracing calls, inspecting code), implementation (fixing core code, updating tests), and checking (unit tests, integration tests, review). Intermediate results are passed between stages by the script, and the main agent stays responsive in the meantime. In the paper, Claude Code enables this feature via `ultracode`.

In the middle is Kimi Code's AgentSwarm. The main agent submits a whole batch of tasks in one call; sub-agents are generated from the same template and each assigned a different item, which in the figure is inspecting 8 modules. The main agent waits for all of them to finish, then gets back a single ordered report at once. It is enabled with `swarm_mode=true`.

At the bottom is Codex's multiagent v2. Sub-agents can dispatch further sub-agents; the figure has 3 direct sub-agents (exploration, fix lead, test lead) and 5 nested sub-agents, and the nested ones exchange findings across branches. The main agent does its own work while dispatching, and exchanges messages with sub-agents. It is enabled with two options, `multi_agent` and `multi_agent_v2`.

Each agent is paired with its own vendor's model: Codex with GPT-5.4, Claude Code with Claude Opus 5, and Kimi Code with Kimi K3. The combinations were fixed in July 2026, with reasoning effort set to the maximum. The authors' rationale is that during RL post-training a model may have adapted to its own harness's tool interface, and swapping models could break that fit.

### Five Benchmarks

The four long-task benchmarks are:

- RepoZero, with two task types: Py2JS asks for a Python package to be rebuilt in JavaScript, and C2Rust asks for a C interface to be rebuilt in Rust;
- ProgramBench, which provides only a reference executable and documentation and asks for a complete command-line program to be rebuilt from scratch;
- NL2Repo, which starts from natural-language requirements and an empty directory and builds an entire repository;
- LoopsBench, which completes multiple interdependent development stages in succession within the same repository.

The control is SWE-bench Verified, fixing one specific issue in an existing repository, representing bounded short tasks. The authors did stratified random sampling within each benchmark's original difficulty groups, sampling 25% of the long-task benchmarks and 20% of SWE-bench Verified. Here is Table 1 from the paper:

| Benchmark | Full set | Sampled | Easy | Medium | Hard | Runs |
| --- | --- | --- | --- | --- | --- | --- |
| SWE-bench Verified | 500 | 100 | 39 | 52 | 9 | 600 |
| Py2JS | 400 | 100 | 33 | 32 | 35 | 600 |
| C2Rust | 200 | 50 | 15 | 18 | 17 | 300 |
| NL2Repo | 104 | 26 | 6 | 12 | 8 | 156 |
| ProgramBench | 200 | 50 | 7 | 36 | 7 | 300 |
| LoopsBench | 112 | 28 | 4 | 11 | 13 | 168 |
| Total | 1,516 | 354 | 104 | 161 | 89 | 2,124 |

Each task was run once per agent per mode, so 354 × 3 × 2 gives 2,124 runs. Figure 3 of the paper plots the lines of code in each benchmark's reference solution: SWE-bench Verified averages only 14 lines, RepoZero about 2.5k lines, NL2Repo about 3.8k lines, ProgramBench about 8.4k lines, and LoopsBench about 26.8k lines on average.

### How the Comparison Is Set Up

In both modes, the model, prompts, environment, and time budget are identical. Concurrent mode uses each vendor's default settings; in sequential mode, Codex has both multi-agent options turned off, Claude Code has the Workflow, Task, and Agent tools disabled, and Kimi Code is set to `swarm_mode=false`, guaranteeing that no sub-agents are created. There are two metrics: task pass rate, the share of tasks fully solved; and test pass rate, the share of tests passed per task averaged across tasks, which shows how far unfinished tasks got.

## Results

### Pass Rates: Long Tasks Go Both Ways

Task pass rates from Table 2 of the paper (%, concurrent first, sequential second):

| Benchmark | Codex | Claude Code | Kimi Code |
| --- | --- | --- | --- |
| SWE-bench Verified | 71.0 / 71.0 | 59.0 / 83.0 | 59.0 / 77.0 |
| RepoZero | 40.7 / 40.0 | 26.7 / 42.7 | 38.7 / 37.3 |
| NL2Repo | 57.7 / 34.6 | 65.4 / 57.7 | 46.2 / 57.7 |
| ProgramBench | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| LoopsBench | 25.0 / 21.4 | 21.4 / 7.1 | 25.0 / 25.0 |

LoopsBench has the longest tasks, and that's where the gains are most concentrated: Codex went from 21.4% to 25.0%, Claude Code from 7.1% to 21.4%, up 14.3 points, and Kimi Code didn't change. All three agents improved their test pass rate on LoopsBench, by up to 6.0 points (Codex from 52.1% to 58.1%).

On the other long-task benchmarks the results are mixed. On NL2Repo, Codex rose from 34.6% to 57.7%, while Kimi Code fell from 57.7% to 46.2%. On ProgramBench every task pass rate is 0; by test pass rate, Codex (53.5% to 58.3%) and Kimi Code (1.5% to 1.7%) got a bit further under concurrency. On RepoZero, Claude Code dropped the most, with task pass rate going from 42.7% to 26.7% and test pass rate from 64.9% to 41.5%.

Results can also cancel out within a single benchmark. Codex barely moved on RepoZero, but broken down, it solved 2 fewer Py2JS tasks and 3 more C2Rust tasks, a net gain of 1 out of 150 tasks, hence 40.0% to 40.7%.

On short tasks concurrency brought no improvement: Codex stayed flat, Claude Code fell from 83.0% to 59.0%, and Kimi Code from 77.0% to 59.0%.

The authors pooled all tasks per agent and ran an exact McNemar test, which compares changes in success and failure on the same set of tasks under two conditions, then applied the Holm correction for multiple comparisons. Claude Code's and Kimi Code's overall drops are significant (p ≈ 5.1×10⁻⁷ and 0.013 respectively), and these overall drops include the long tasks; Codex's small gain is not significant (p ≈ 0.23). By overall solve rate, the change across the three agents ranges from an 11.9-point drop to a 2.3-point gain.

Ordered by code size and averaged across the three agents, the concurrent-minus-sequential difference in test pass rate is −8.29 points on RepoZero, −0.04 on NL2Repo, +1.24 on ProgramBench, and +4.31 on LoopsBench, matching the order of average code size from small to large. On this basis the paper says the larger the expected implementation, the bigger concurrency's average test-pass-rate advantage. Note that this trend comes from the averages of only four benchmarks.

### Do the Two Modes Solve the Same Tasks?

Figure 4 of the paper splits each agent's results on each benchmark into four buckets, with ProgramBench success counted as "test pass rate greater than 0." Summarized below (solved only concurrently / solved only sequentially / solved by both / solved by neither):

| Benchmark | Codex | Claude Code | Kimi Code |
| --- | --- | --- | --- |
| SWE-bench Verified | 9 / 9 / 62 / 20 | 5 / 29 / 54 / 12 | 4 / 22 / 55 / 19 |
| NL2Repo | 7 / 1 / 8 / 10 | 3 / 1 / 14 / 8 | 1 / 4 / 11 / 10 |
| ProgramBench | 5 / 2 / 39 / 4 | 3 / 3 / 0 / 44 | 5 / 1 / 0 / 44 |
| RepoZero | 4 / 3 / 57 / 86 | 0 / 24 / 40 / 86 | 6 / 4 / 52 / 88 |
| LoopsBench | 1 / 0 / 6 / 21 | 4 / 0 / 2 / 22 | 2 / 2 / 5 / 19 |

Almost every agent on every benchmark has tasks solved only under concurrency. The clearest case is Claude Code on LoopsBench: 4 tasks were solved only concurrently, and not one only sequentially. On short tasks, even though Claude Code's total score is much lower, it still has 5 tasks solved only under concurrency; Codex has the same total score in both modes, but each mode has 9 tasks the other didn't solve.

### The Cost: Tokens, Calls, and Time

Resource data from the same Table 2 (concurrent first, sequential second):

| Benchmark | Metric | Codex | Claude Code | Kimi Code |
| --- | --- | --- | --- | --- |
| SWE-bench Verified | Tokens (M) | 3.37 / 1.34 | 2.14 / 1.02 | 1.01 / 0.98 |
| | Time (min) | 5.1 / 4.5 | 27.8 / 17.5 | 12.3 / 11.5 |
| RepoZero | Tokens (M) | 2.47 / 0.89 | 4.03 / 2.42 | 1.33 / 0.82 |
| | Time (min) | 7.2 / 6.3 | 39.3 / 29.0 | 19.3 / 17.1 |
| NL2Repo | Tokens (M) | 8.00 / 2.87 | 8.55 / 6.53 | 2.70 / 2.21 |
| | Time (min) | 9.9 / 9.4 | 39.0 / 34.3 | 24.4 / 18.4 |
| ProgramBench | Tokens (M) | 27.05 / 9.19 | 6.02 / 6.18 | 11.64 / 6.05 |
| | Time (min) | 24.9 / 23.4 | 38.9 / 38.7 | 39.9 / 40.1 |
| LoopsBench | Tokens (M) | 51.65 / 11.45 | 21.73 / 15.52 | 41.49 / 11.29 |
| | Time (min) | 18.8 / 17.7 | 91.5 / 74.8 | 55.8 / 52.7 |

The number of shell calls (main agent plus all sub-agents) went up in every combination, often several-fold; Claude Code on LoopsBench went from 107.4 to 791.6 calls.

Aggregated by agent, tokens were 1.41 to 3.31 times those of sequential runs. The only case in the table where concurrency used slightly fewer tokens is Claude Code on ProgramBench (6.02 vs. 6.18 million). The extra tokens didn't buy proportionally more successes: spreading all tokens over solved tasks, all three agents spent more tokens per solved task. Codex, which improved slightly overall, needed 24.12 million tokens per solved task under concurrency versus only 7.69 million sequentially.

By average time, parallelism didn't make tasks finish faster. In 14 of the 15 agent-benchmark combinations average time went up; the only exception is Kimi Code on ProgramBench (39.9 vs. 40.1 minutes). On SWE-bench Verified, Claude Code averaged 27.8 minutes per task concurrently versus 17.5 sequentially, while solving fewer. There were faster cases too: of the 366 tasks solved by both modes, concurrency was faster on 102 (27.9%), and on 46 of them (12.6%) it was at least 20% faster.

### How Many Sub-Agents, and When

Figure 5 of the paper counts the sub-agents created per run and the peak number running at once. Across all runs, easy, medium, and hard tasks created an average of 6.69, 7.53, and 15.84 sub-agents, with average peaks of 3.66, 4.61, and 6.70. LoopsBench varies the most: from easy to hard, the average number of sub-agents created per task rises from 16.25 to 68.18; it accounts for only 14.6% of hard tasks but contributes 62.9% of the sub-agents in hard tasks.

The three agents' habits differ a lot. Codex leaves part of the implementation to the main agent itself and maintains only a small set of sub-agents, with average peaks between 1.21 and 3.00 across task groups. On hard LoopsBench tasks, Claude Code creates an average of 133.69 sub-agents with a peak of 25.46, as it keeps opening new batches in later workflow stages; Kimi Code creates 67.00, yet its peak reaches 37.46, because it sends out a large batch all at once.

The authors also tested the correlation between task difficulty and the number and peak of sub-agents created, with multiple-comparison correction. Of the 30 tests, the significant results the paper calls out are two on the number created: Claude Code on ProgramBench (ρ=0.470, p=0.015) and Codex on SWE-bench Verified (ρ=0.316, p=0.042). The paper's take is that the correlation is significant only in some settings, and that task structure and coordination mechanisms also affect dispatching. The authors' explanation is that a harder task may simply mean one core component takes more effort, without more independently separable subtasks.

On timing, the paper normalizes each trajectory from its first to last record onto 0 to 100%, and Figure 6 plots the average number of sub-agents running at each point.

- When dispatching starts: the median start time of the first sub-agent is at 4.0% of the trajectory for Codex, 22.1% for Claude Code, and 26.0% for Kimi Code. Codex hands control back to the main agent after each dispatch, while the other two first write a workflow or split up a batch.
- Whether it's really parallel: among each agent's 354 runs, the share of runs that created sub-agents but never had two running at the same time is 24.3% for Codex, and only 1.7% and 2.8% for Claude Code and Kimi Code.
- How long parallelism lasts: the median share of time with at least two sub-agents running at once is 23.9% for Codex, 52.7% for Claude Code, and 15.6% for Kimi Code.
- When it ends: the median end time of the last sub-agent is at 83.3% for Codex, 100.0% for Claude Code, and 93.6% for Kimi Code. Codex's main agent leaves some time at the end for its own implementation and checks, while Claude Code still spins up sub-agents during the review stage.

## 28 Failure Modes

### How They Were Labeled

The authors manually annotated all 1,062 concurrent trajectories, comparing each against the same agent's sequential trajectory on the same task, covering the main agent, all sub-agents, and the artifacts. They first built a codebook from 50 trajectories, then three annotators did the formal labeling, with each trajectory labeled independently by two people. Agreement was Cohen's κ = 0.77; this coefficient measures how much two annotators agree after removing chance agreement, and the closer to 1, the better.

The result is 4 categories, 13 subcategories, and 28 failure modes. Of the 1,062 trajectories, 650 had at least one failure mode, for 804 instances in total, since one trajectory can have several. Each subcategory is also labeled with a type: functional failures mean work nobody owns, information that didn't get through, writes that weren't isolated, progress nobody could see, or no acceptance check; performance failures mean inefficient coordination under the given execution budget.

Table 3 of the paper is summarized below (instance count and share of the 804 instances in parentheses):

| Category | Subcategory | Failure modes | Type |
| --- | --- | --- | --- |
| Task orchestration 229 (28.5%) | Missing owner 16 (2.0%) | Implementation has no owner 14 (1.7%); assembly has no owner 2 (0.2%) | Functional |
| | Pseudo-concurrency 47 (5.8%) | Work waits for pre-checks to finish 7 (0.9%); investigation runs serially 40 (5.0%) | Performance |
| | Load imbalance 166 (20.6%) | Critical path gets no resources 3 (0.4%); too many sub-agents exhaust the budget 154 (19.2%); single sub-agent's task too large 9 (1.1%) | Performance |
| Execution management 233 (29.0%) | Wrong timing for start, join, or result retrieval 166 (20.6%) | Sub-agent terminated too early 128 (15.9%); wrap-up too late 5 (0.6%); changes merged only after verification 7 (0.9%); implementation results never joined 9 (1.1%); verifier results never returned 17 (2.1%) | Functional |
| | Failure propagation 42 (5.2%) | Failure nobody picks up 42 (5.2%) | Functional |
| | Main agent doesn't actively monitor 25 (3.1%) | Completed results not used 13 (1.6%); blind wait until timeout 12 (1.5%) | Performance |
| Global context 75 (9.3%) | Missing task requirements 23 (2.9%) | Sub-agent's task description incomplete 23 (2.9%) | Functional |
| | Missing interface contract 5 (0.6%) | No agreed interface between sub-agents 5 (0.6%) | Functional |
| | Handoff overload 4 (0.5%) | Information lost in handoff 1 (0.1%); too much content in one handoff 3 (0.4%) | Functional |
| | Unfounded completion claim 18 (2.2%) | Overall completion declared without verification 18 (2.2%) | Functional |
| | Verbatim retry 25 (3.1%) | Retry without carrying over previous progress 25 (3.1%) | Performance |
| Shared state and merging 267 (33.2%) | Concurrent writes 213 (26.5%) | Cross-file ownership conflict 5 (0.6%); same-file conflict 42 (5.2%); source code overwritten 14 (1.7%); deliverable overwritten 120 (14.9%); final file tree overwritten 6 (0.7%); writes to a non-isolated workspace 26 (3.2%) | Functional |
| | Shared environment pollution 54 (6.7%) | Artifacts leak into the shared environment 54 (6.7%) | Functional |

Looking at individual failure modes, the three most common are too many sub-agents exhausting the budget (154), sub-agents terminated too early (128), and deliverables overwritten (120).

### Task Orchestration

This category is when the main agent didn't divide the work well between itself and its sub-agents, accounting for 28.5%.

Missing owner: some work isn't assigned to anyone. When Kimi Code rebuilt the verifiers repository in NL2Repo, it dispatched 5 sub-agents to handle the parser and scoring rules, utility scripts, docs, example environments, and a mock client, but nobody owned the core `verifiers/envs` module, and the final submitted package didn't pass a single official test.

Pseudo-concurrency: sub-agents run in parallel, but the work that decides whether the task gets done is still serial. When Kimi Code rebuilt the log highlighter tailspin in ProgramBench, 17 sub-agents simultaneously investigated different behaviors of the program, and none was responsible for implementation. The main agent waited for all reports to come back before starting to write on its own, and the run ended during the implementation planning phase, with 0 of 785 tests passing.

Load imbalance: too much effort goes into some parallel activities, leaving no time for the critical work. It accounts for 20.6% of all instances and 39.8% of Claude Code's failure instances. On the same tailspin task, Claude Code's main agent was setting up the Rust skeleton while having the workflow dispatch sub-agents to investigate ten behavior areas, plus review sub-agents. The investigation and repeated dispatching went on until the deadline, leaving the main agent with only a crate that didn't compile.

### Execution Management

This category is when things weren't managed well after the work was handed out, accounting for 29.0%.

Wrong timing: a sub-agent's work starts, ends, or gets adopted at the wrong time, and results that should have made it in missed the final submission. The seqtk task falls into this category; the figure below is the paper's diagram of the dispatch process.

![Sub-agent dispatch diagram of Kimi Code AgentSwarm on the seqtk rebuild task](https://leafw-blog-pic.oss-cn-hangzhou.aliyuncs.com/parallel-subagents-kimi-seqtk-dispatch.png)

seqtk is a command-line tool for processing biological sequences. The task was to rewrite a behaviorally identical program from its executable and documentation, with a 40-minute budget. Kimi Code's main agent first wrote the shared code, including the sequence reader, command dispatcher, and build script, then at minute 17.5 assigned all 22 subcommands to 20 sub-agents at once, all of which started within 11 seconds. The main agent then waited for the whole batch to return, using a two-hour sub-agent timeout, longer than the entire task budget. The sub-agents responsible for `listhet` and `split` reported done at minutes 22.3 and 26.7 (the two boxes on the right of the figure).

Yet the main agent never called another tool, and the planned integration and final tests never happened. At the deadline the remaining 18 sub-agents were aborted. The sub-agent responsible for `seq` never produced an implementation, so the official test calling `seq -Q 64` just got "unrecognized command," and in the end 92 of 440 tests passed. The paper's verdict: the dispatching itself was reasonable, but the timing and budget didn't line up.

Failure propagation: a sub-agent doesn't finish, and the main agent neither retries nor takes over. In Claude Code's canonicaljson migration, a sub-agent searching for counterexamples in parameter parsing was interrupted by rate limiting, and the main agent never restarted it or took over that verification before the timeout. In the end 147 of 148 official tests passed.

Main agent doesn't actively monitor: the main agent didn't reassess progress. In Claude Code's hopscotch-map migration in RepoZero, the main agent had already built the Rust version, checked it, and placed it in the output directory, but close to the deadline it started blocking on an unfinished sub-agent, waiting ten minutes, and the run was terminated before it gave a final reply. That artifact passes all 39 official tests.

### Global Context

This category is when agents working in their own contexts lack the information needed to connect the work, accounting for 9.3%.

Missing task requirements: when Kimi Code rebuilt icecream, the task required exporting the package metadata via `from icecream import *`, but the sub-agent responsible for packaging wasn't told, and the `__init__.py` it returned was untouched. The artifact passed all tests, but there's no evidence this requirement was ever handled.

Missing interface contract: when Kimi Code rebuilt python-fsutil, the sub-agents writing the implementation and the tests were assigned their own files, without a complete shared API contract. The sub-agent handling permissions returned an integer, while the test-writing sub-agent expected a string like "755", and other tests assumed different function signatures. The first integration test run reported 9 failures, and the main agent had to go back and align them one by one.

Handoff overload: in Claude Code's base58 migration, the workflow consolidated up to 60 findings per probe into an 80,000-character implementation brief. The sub-agent that picked it up didn't implement anything; instead it reinvestigated the environment, source code, and errors from scratch, and was interrupted before producing any files.

Unfounded completion claim: in Kimi Code's bidict migration in RepoZero, 4 sub-agents implemented different modules. After integration, the main agent only checked standard output and the exit code, not standard error, and declared the task complete even though it knew the exception output had been simplified and argument parsing was off.

Verbatim retry: in Claude Code's Clipper migration, a stuck design task was redispatched repeatedly with the same brief, and the replacement sub-agents couldn't see earlier findings, so they had to reread the source and rerun probes. After four attempts the deadline hit, with only a `Cargo.toml` in hand and none of the required Rust entry files.

### Shared State and Merging

This is the largest category, at 33.2%. Multiple agents read and write the same files, artifacts, or environments without coordination, and the paper draws an analogy to data races in traditional concurrent programs.

Concurrent writes account for 26.5% of all instances and 62.9% of Codex's failure instances. In Codex's Clipper migration, the main agent had already written and built `test5.rs`; later a sub-agent switched to a different module structure in the shared output directory, replacing both `test5.rs` and `Cargo.toml`. The final artifact passed the official tests, but the main agent's earlier understanding of the artifact was invalidated, forcing an unplanned extra round of integration.

Shared environment pollution: when Claude Code rebuilt clog-cli, two sub-agents each deleted and recreated `/tmp/t1` to hold test data. The Markdown sub-agent had just set up its data when the JSON sub-agent replaced that path, and the Markdown sub-agent, none the wiser, kept working with the wrong data. In another case, a sub-agent replaced the system-wide Git executable, and every later probe in the same run that called Git used the modified version.

Claude Code and Codex fail in different places: Claude Code's most common failure is load imbalance, while Codex's is concurrent writes. My guess is that Claude Code spins up the most sub-agents and runs in parallel the longest, while Codex's main agent writes code itself and easily ends up writing to the same files as its sub-agents; the paper doesn't attribute it this way directly.

## When Concurrency Helps

The authors also selected 131 pairs of trajectories where, for the same agent on the same task, the concurrent version had a higher test pass rate, or fully solved a task the sequential version didn't, or was at least 60 seconds and 20% faster without a lower test pass rate. Comparing the pairs, they identified three recurring patterns.

The first is complementary division of labor. When Kimi Code rebuilt justext in NL2Repo, the main agent implemented the core body-text extraction algorithm itself, while 8 sub-agents handled deliverable files and checks, and the main agent integrated all 8 results before final verification. Both modes passed all tests, and concurrency cut the time from about 734 seconds to about 468 seconds.

The second is parallel attempts, then picking one. When Codex rebuilt Halite in ProgramBench, the main agent and a sub-agent each built a version of this black-box game environment, and the main agent combined the sub-agent's map behavior with its own more complete protocol handling, passing 292 of 391 tests; the sequential mode failed to compile, 0/391. In NL2Repo's unidecode task, sub-agents implemented versions 1.3.8 and 1.4.0 respectively, and after comparing them the main agent adopted the 1.4.0 mapping table, passing more tests than sequential.

The third is independent verification. In Codex's LoopsBench security experiment suite task, while the main agent was implementing the full set of experiments, a cryptography sub-agent found that the ECDSA nonce-reuse attack needs the order of the P-256 curve, and the main agent used this to make the recovery algorithm generic. The concurrent artifact passed all checks while the sequential one failed. In NL2Repo's trimming task, a nested sub-agent probed the upstream implementation and found the correct naming rule for CSV headers, something the sequential version couldn't get fully passing.

What the three patterns have in common is that the main agent made use of the parallel work: integrating components, picking a better candidate, or adopting corrective evidence. The paper's advice to practitioners maps onto these three patterns: consider turning on concurrency when the task splits cleanly into independent components, when the same problem lends itself to several parallel attempts, or when an independent sub-agent can verify the main implementation; and weigh the potential gain against the extra tokens, tool calls, and time.

## Takeaways

The paper recommends turning concurrency on as needed and lists three situations where it's worth it; the points below extend from there.

1. For short tasks like fixing bugs or small feature changes, hold off on concurrency. On SWE-bench Verified, Claude Code dropped 24 points, Kimi Code 18 points, and Codex stayed flat, while the user pays for the extra tokens and time.
2. Before turning it on, check which of the three situations above your task falls into; if none, let the main agent do it sequentially.
3. Define a write scope for each sub-agent. Deliverables alone were overwritten 120 times. You can require sub-agents to write only to their own directories, with the final deliverable directory written only by the main agent; put the sub-agent's name in temp file paths, and don't let sub-agents touch system-level tools. The cost is an extra round of merging for the main agent.
4. Tell the main agent the time budget, require it to dispatch in batches and check progress midway, and leave time for integration and final tests. The seqtk main agent waited on a timeout longer than the task budget, sending out one batch and then just waiting.
5. Define interfaces before splitting the work. Have the main agent write out shared interfaces or type definitions first; the 9 integration failures in python-fsutil came from having no shared API contract, so return types and function signatures didn't match.
6. State explicitly in the prompt: when a sub-agent fails, the main agent should take over or redispatch, attaching existing findings and a narrower next step when redispatching.
7. Before declaring completion, the main agent should check the task description item by item itself, including standard error; once the artifact has passed its checks, don't block waiting on unfinished sub-agents.

## Limitations

- Each configuration was run only once. Agent execution is stochastic; the authors' justification is that Codex's sequential results on LoopsBench are close to those reported in the original LoopsBench paper, and the trends across multiple attempts in the original paper are consistent.
- Some benchmark samples are small. LoopsBench has only 28 tasks, so Claude Code going from 7.1% to 21.4% means going from 2 tasks to 6; NL2Repo has only 26.
- The conclusions are limited to three agent-model combinations and five benchmarks, using default concurrency settings. Different models, or users writing their own division-of-labor instructions, could give different results.
- The failure taxonomy relies on manual annotation; κ = 0.77 indicates good agreement, but the judgments still have a subjective component. The three helpful patterns come from 131 selected trajectory pairs, and the paper doesn't report how often each pattern occurs.
- The paper doesn't compare dynamic concurrency against fixed-workflow multi-agent systems; it only compares on versus off.

## Final Thoughts

Among the research directions the paper lists at the end is runtime monitoring and intervention: detecting stalled or failed sub-agents, reclaiming unfinished work, and then deciding when to collect results and when to end the run. The seqtk and hopscotch-map examples above both fall into the execution-management category of failures.

## References

- https://arxiv.org/abs/2610.10263