I tested (basically) every Multi-Agent System metric: they're mostly flawed
I tried every multi-agent collaboration metric I could find on one benchmark to see which ones actually measure the team rather than the task. 293 went in; 9 came out.
Authors
Hassam Gani
Affiliations
Columbia University
Published
Reading time
13 min
Failed “Reliable” (31)
MultiAgentBench KPI— model ranking flips between halves of the queues (τ = 0.33)
Token cost— no model effect once the queue is held fixed (p = 0.46)
Rounds used— model ranking flips between halves of the queues (τ = 0.33)
Score per token— model ranking flips between halves of the queues (τ = 0.00)
COMMA efficiency— no model effect once the queue is held fixed (p = 0.58)
Tool error rate— no model effect once the queue is held fixed (p = 0.13)
Message count— no model effect once the queue is held fixed (p = 0.16)
Balance (VillagerAgent)— model ranking flips between halves of the queues (τ = 0.33)
Write conflicts— model ranking flips between halves of the queues (τ = 0.33)
Tool permission use (ClawArena)— no model effect once the queue is held fixed (p = 0.80)
KPI concentration— no model effect once the queue is held fixed (p = 0.69)
Removal attribution— no model effect once the queue is held fixed (p = 0.86)
Progress rate AUC (AgentBoard)— model ranking flips between halves of the queues (τ = 0.33)
LLM calls (TheAgentCompany)— no model effect once the queue is held fixed (p = 0.86)
Critical Steps (MS-Eval)— model ranking flips between halves of the queues (τ = 0.33)
Steps per checkpoint (WebCanvas)— model ranking flips between halves of the queues (τ = 0.33)
Sink-agent tokens (MacNet)— no model effect once the queue is held fixed (p = 0.31)
Repetition rate (AgentQuest)— model ranking flips between halves of the queues (τ = 0.33)
Optima reward— model ranking flips between halves of the queues (τ = 0.00)
Mistakes (COMMA)— no model effect once the queue is held fixed (p = 0.09)
Communication density (Silo-Bench)— no model effect once the queue is held fixed (p = 0.26)
Question rate (ALEM)— model ranking flips between halves of the queues (τ = 0.33)
Instruction rate (ALEM)— no model effect once the queue is held fixed (p = 0.13)
Speaker consistency (MI)— no model effect once the queue is held fixed (p = 0.11)
Instantaneous coordination (MI)— no model effect once the queue is held fixed (p = 0.33)
Stall ratio (Magentic-One)— model ranking flips between halves of the queues (τ = 0.33)
Handoff events (ALEM)— model ranking flips between halves of the queues (τ = -0.33)
Under-delegation (MS-Eval)— no model effect once the queue is held fixed (p = 0.06)
Parallelism rate (SwarmBench)— no model effect once the queue is held fixed (p = 0.06)
Bottleneck agent PAC (HiveMind)— model ranking flips between halves of the queues (τ = 0.55)
Grounding accuracy— no model effect once the queue is held fixed (p = 0.13)
Figure 1. Each row is a test, and a metric drops out at the first one it fails. Click a row to see which ones left there and why. Colors are metric families.
I’m writing a paper where we aim to propose a new metric to eval collaboration in multi-agent systems, and a key thing I noticed reading other metric papers was that they begin with an analysis of a previous metric & then show why it’s inadequate for the thing they’re measuring. They then show how their proposed metric correlates more closely with a stronger proxy for that measure, such as what humans think.
So I decided to do the “exact” same thing.
Specifically, however, I decided to evaluate every MAS metric I could find—with the twist of doing so on my own benchmark ReOrgBench, which would allow me to do the measuring in a long-horizon, orchestrator-centric setting where changes to the task can occur mid-run, mimicking how humans would use a MAS such as Claude Code or Codex.
A key feature of ReOrgBench is that it uses no LLM-as-a-judge: code compares the final state of each run with predefined checkpoints, including whether required steps were taken and whether prohibited ones weren't. I cataloged 391 metrics from 128 MAS papers from ~the last 3 years and then merged them into 293 distinct ones. I then kept filtering them out till there were ultimately only 9 left—of which I still wouldn't say are necessarily infallible. The values of these metrics were determined from 60 episodes: Fable 5.2, Opus, Sonnet, and Haiku 5.5 (all medium effort) on 15 of ReOrgBench's held-out tasks, averaging ~7.6M tokens per episode.
Here's what each task looks like: there's some ops queue at a made-up software company. A "human" prompts the orchestrator with whatever it needs done; it, in turn, recruits any number of agents with their own tools from any of 7 roles: generalist, engineer, analyst, support, finance, procurement, and legal. Each of these agents can also recruit another agent, and they can communicate with any other agent they're aware of. At some point, the human requests some modification (sorta how you or I will tell Codex or Claude, "yk waht acc do thsi"), and now the orchestrator must adapt, alongside directing the rest of the team on doing so.
2. The outcomes
Fable 5.2
0.969
Opus 5.5
0.924
Sonnet 5.5
0.671
Haiku 5.5
0.604
0.000.250.500.751.00
Figure 2. Checkpoint score with 95% intervals, and strict pass@1 (the share of episodes where every single checkpoint passed). The per-queue tab shows each episode.
Here's what I found via failure analysis: Haiku gets basic stuff wrong, Sonnet primarily misses the exceptions, and both Fable and Opus lose their points primarily due to things that change while they're busy.
Only Fable and Opus ever pass, and Fable seems to be pretty ahead, but the checkpoint score shows that Sonnet and Haiku don't do too badly, and Fable and Opus are very close. These checkpoint score observations will prove very useful for filtering and later conclusions.
3. Can I even compute it?
The first question I asked when making the first cut from the 293 metrics was as follows: can I even compute it? The answer is mostly no. Specifically, it was no for 212 of the metrics. They either needed some expensive ablation, an LLM judge, a very specific setting, human labels, gold trajectories, or repeated runs. If you're resource-constrained or your MAS doesn't fit a specific box, these needs make those metrics generalize poorly. This left me with 13 communication, 25 efficiency, 15 outcome, 11 coordination, 2 robustness, 4 credit, and 11 role/structure metrics: a total of 81.
Of these 81, a lot were actually the same thing under different formalizations—total tokens and dollar cost for example are literally the same number here—and about a dozen were effectively the same as checkpoint scores or pass@1, while 5 were more like curves or descriptive views rather than singular numbers per episode, so that brought me down to 59. I then added 8 faithful adaptations of metrics that didn't quite fit as-is (GovSim's equality, for one) and 7 plain counts like team size or message count to get to 74. ReOrgBench's own metrics I left out entirely, since that'd sorta defeat the point.
4. Is it even defined?
To further filter them down, I then asked myself if these 74 even take values that are useful: they had to be defined on at least 80% of the episodes, and no one value could be what over 80% of them took on. 6 didn't, so they went out. Some of these just never happen in these runs, like assignment conflicts, structural collapse, and ClawArena's read-only violations, which were zero in every single episode, some were almost always the same thing like ALEM's parse success or TravelPlanner's delivery rate, and AgentWorld's CCE is by definition 0 on a failed task, which here was 82% of the episodes; I did however keep its any-outcome variant.
5. Team or task?
The value a metric takes on should be much more influenced by the model being evaluated rather than the task when the tasks are similar in nature. To this end, I evaluated reliability via a Friedman test (p < 0.05) and checking that model ranking was consistent on both halves of the queues via Kendall τ ≥ 0.67.
Figure 3. Each dot is a metric. The higher it is, the more of its variance comes from the model; the further right, the more comes from the queue. The checkpoint score sits in the top left. Hover for details.
From here, we lose 31 metrics. A lot of these were cost metrics, which turned out to be mostly a function of the queue rather than the model: 63% of the variance in token use came from the queue and only 4% from the model. That's how you get score-per-token and COMMA's efficiency score both saying Haiku is the most efficient model, which isn't because it's actually collaborating efficiently but because it does less and failing doesn't cost much. Optima's reward was the funniest one though, ranking the models in a totally different order on each half of the queues.
6. Does it point the right way?
Besides the model having a stronger sway on the metric than the task, I needed to ensure that it actually had the correct sway; the metrics must point in the direction their authors intended, so I applied a validity check: that the within-queue Spearman ρ with the checkpoint score had the intended sign with |ρ| ≥ 0.25.
Figure 4. How each metric lines up with the outcome, flipped so that right means it agrees with its own paper. The tabs switch between the checkpoint score (Spearman ρ) and strict pass (AUC). Anything in the shaded band has |ρ| < 0.25, and starred metrics never said which direction is better.
19 metrics were eliminated: problems with action-outcome correlation, message cost, action-to-action mutual information, or plain being weak or directionless. Action count was a weird one: normally you'd think more actions means a worse, more wasteful team, but Fable took like 437 actions an episode and Haiku only 264, and that's honestly just because doing the job properly means you have to go through every single ticket.
7. Is it new information?
Here, I wanted to filter for something new—to leave us with a few metrics that say a lot instead of a few that don't say much—so I did a redundancy check: that there was a |ρ| < 0.9 with the score, alongside |ρ| ≤ 0.85 with any stronger metric.
Hover a cell. Blue is positive, red negative.
Figure 5. Spearman correlations between the 18 metrics left at this point, plus the checkpoint score. Blue is positive and red is negative; the solid blocks are metrics that are pretty much copies of each other.
Here, 5 metrics collapsed into others, leaving us with 13.
8. Beyond the score
Having 13 metrics that correlate with the checkpoint score is a start, however I still needed to make sure that the sway a metric has over the models isn't just the checkpoint score's sway in disguise; if it is, we could've just looked at the checkpoint score and stopped there. Specifically, for the final cut I took the residuals from regressing each of the 13 metrics on the checkpoint score within the queue, and then I checked whether the models still differed on those residuals via a Friedman test. From here, we lose 4 more metrics: parallelism, MS-Eval's staffing F1, GovSim's equality, and TeamCraft's planner cost.
CCE, any outcome
Message token share
Unnecessary path ratio (GEMMAS)
Redundancy rate (Scaling)
Equality (GovSim) · drops
Task offloading (PARTNR)
Idle rate
Parallelism · drops
Critical-path ratio
Staffing F1 (MS-Eval) · drops
Task throughput
Assignment diversity
Planner cost (TeamCraft) · drops
Fable 5.2Opus 5.5Sonnet 5.5Haiku 5.5right = better, per each metric’s paper
Figure 6. Model means on the 13 final candidates, both raw and with the checkpoint score held fixed. Right is better according to each metric's own paper, and the faded rows are the ones that dropped here.
That leaves us with 9:
CCE, any-outcome variant (AgentWorld)
unnecessary path ratio (GEMMAS)
redundancy rate (Towards a Science of Scaling Agent Systems)
share of output tokens spent messaging
task offloading (PARTNR)
idle rate (Coordination Matters)
critical-path ratio (MS-Eval's Critical Steps)
task throughput (checkpoints per round)
assignment diversity (Coordination Matters)
Note that given there are only 15 queues in the held-out split, getting lucky on 1 or 2 queues could conceivably let a bad metric slip through! To check that, I ran 1,000 bootstrap resamples of the 15 queues: all 9 of these still pass at least 97% of the time, while the ones that dropped out earlier basically never do, with token cost passing 0.1% of the time, COMMA's efficiency score 1%, and ClawArena's tool permissions 3%.
Notice how in Figure 6, once the checkpoint score is held fixed, Sonnet and Haiku go in opposite directions. Sonnet beats what its checkpoint score predicts on CCE and task offloading alongside having a lower critical-path ratio (so less serial work), while Haiku sits below what its checkpoint score predicts on all three. Why does Sonnet look so much better once you control for the score? Because as we saw in the failure analysis earlier, Sonnet's team actually coordinates fine and primarily loses points on exceptions like collateral edits and held customers, whereas Haiku gets basic stuff wrong like wrong notices and records it never moved.
There's something completely different going on with Fable and Opus in Figure 6 though. On the checkpoint score, Fable beats Opus by 0.045 and is never lower than Opus on any of the 15 queues, yet with the checkpoint score held fixed 7 of the 9 survivors can't tell Fable and Opus apart at all, and the other 2, the redundancy rate and assignment diversity, both say Opus is better than Fable. …what? So either Fable just isn't really worth the switch from Opus when it comes to MAS (lol), or none of these metrics can properly see the difference between the two strongest teams, which is the gap the metric in our paper needs to close.
9. What's still wrong with the nine
Now, the 9 metrics making it through the funnel doesn't necessarily mean that they measure collaboration. It just means they aren't obviously broken. When I looked at how each of the 9 metrics is computed, I found three problems with them. The first problem is that 6 of the 9 metrics are computed from which checkpoints passed. These are CCE, the unnecessary path ratio, offloading, the critical-path ratio, throughput, and assignment diversity. So part of what these metrics measure is the outcome again, which was already measured by the checkpoint score. If a team does all of the right work but gets one figure wrong, then that work gets counted as waste by these metrics.
I have a qualm with the remaining survivors too. The redundancy rate is calculated by taking the mean pairwise TF-IDF similarity of a team's messages, and while it has one of the biggest model effects in the study (W = 0.79), it still calls Fable more redundant than Opus! My suspicion is that TF-IDF similarity is mostly just picking up on how each model writes its messages rather than whether agents are repeating the same information. Task offloading, the critical-path ratio, and the share of output tokens spent messaging all move with how many agents got hired, so if an orchestrator just spawns more agents on a queue, it shifts those numbers without the team actually collaborating any better.
It seems to me that, for the two metrics that bugged me the most, we can fix the formula directly rather than throwing the metric out. For content redundancy, the fix is to look at what a message actually sends instead of its TF-IDF wording: a message only counts as redundant if every ID, amount, and reference in it was already sent to that recipient before. And for reference-seeded CCE, you seed the causal graph from the team's last writes to the objects that the reference solution touches, without knowing which checkpoints the team gets right or wrong:
Metric
Model effect (W)
Tied to outcome (ρ)
Left after outcome (W)
Fable vs Opus (p)
Split-half τ
Redundancy rate (Scaling)
0.79
−0.40
0.82
0.000
0.67
Content redundancy
0.27
−0.28
0.12
0.012
1.00
CCE, any outcome
0.80
+0.79
0.38
0.762
1.00
CCE, reference-seeded
0.64
+0.57
0.59
0.252
1.00
Table 1. The two flawed survivors and their repairs (shaded). W is the model effect with the queue held fixed, ρ is the within-queue correlation with the checkpoint score, and "left after outcome" is W on the residuals. The last two columns are the Fable vs Opus test and how well the ranking agrees across halves of the queues.
Both of the repaired metrics end up fixing the issue I had with the original metrics in Table 1. Specifically, the model effect on content redundancy drops to W = 0.27 from W = 0.79 on the original redundancy rate, which makes sense if the original redundancy rate had to do primarily with writing style, and content redundancy now gives the exact same ranking on both halves of the queues (τ = 1.00) while putting Fable ahead of Opus (p = 0.012). In Table 1, reference-seeded CCE provides a lower correlation with the checkpoint score than the original CCE (ρ = 0.57 vs 0.79) along with a stronger model effect on the residuals (W = 0.59 vs 0.38).
So, which metrics measure collaboration?
Overall, the 9 metrics that survived all measure where the team's effort went, rather than how much the run cost in tokens or what shape the team had. I think that makes sense, because collaboration is about whether actions were on the job or off it, whether agents were on the critical path or waiting around, and whether messages carried new information or repeats. Going through all 293 metrics also gave me a five-point checklist I didn't have going in: a collaboration metric needs to be defined on every episode, vary more with the team than with the task, be computed without knowing whether the task passed, normalize for team size, and still separate teams once the outcome is held fixed.
Hardly anything in the literature hits all five of those yet, and with the outcome held fixed, 7 of the 9 survivors can't tell Fable and Opus apart while the other 2 actually rank Opus above Fable. Heck, that gap is the whole reason we're writing our paper on this topic. The benchmark and all 60 episodes are up on Hugging Face, and if there's a collaboration metric I missed, send it my way and I'll run it through the same funnel!
Model % and Queue % are the share of each metric's variance that comes from the model and from the queue; W is Kendall's W for the model effect with the queue held fixed; ρ is the within-queue Spearman correlation with the checkpoint score; and the last column is how often the metric passed the reliability and validity tests across 1,000 resamples of the queues. Hover over a row to see why it dropped.