I tested (basically) every Multi-Agent System metric: they're mostly flawed

I tried every multi-agent collaboration metric I could find on one benchmark to see which ones actually measure the team rather than the task. 293 went in; 9 came out.

Failed “Reliable” (31)
  • MultiAgentBench KPI — model ranking flips between halves of the queues (τ = 0.33)
  • Token cost — no model effect once the queue is held fixed (p = 0.46)
  • Rounds used — model ranking flips between halves of the queues (τ = 0.33)
  • Score per token — model ranking flips between halves of the queues (τ = 0.00)
  • COMMA efficiency — no model effect once the queue is held fixed (p = 0.58)
  • Tool error rate — no model effect once the queue is held fixed (p = 0.13)
  • Message count — no model effect once the queue is held fixed (p = 0.16)
  • Balance (VillagerAgent) — model ranking flips between halves of the queues (τ = 0.33)
  • Write conflicts — model ranking flips between halves of the queues (τ = 0.33)
  • Tool permission use (ClawArena) — no model effect once the queue is held fixed (p = 0.80)
  • KPI concentration — no model effect once the queue is held fixed (p = 0.69)
  • Removal attribution — no model effect once the queue is held fixed (p = 0.86)
  • Progress rate AUC (AgentBoard) — model ranking flips between halves of the queues (τ = 0.33)
  • LLM calls (TheAgentCompany) — no model effect once the queue is held fixed (p = 0.86)
  • Critical Steps (MS-Eval) — model ranking flips between halves of the queues (τ = 0.33)
  • Steps per checkpoint (WebCanvas) — model ranking flips between halves of the queues (τ = 0.33)
  • Sink-agent tokens (MacNet) — no model effect once the queue is held fixed (p = 0.31)
  • Repetition rate (AgentQuest) — model ranking flips between halves of the queues (τ = 0.33)
  • Optima reward — model ranking flips between halves of the queues (τ = 0.00)
  • Mistakes (COMMA) — no model effect once the queue is held fixed (p = 0.09)
  • Communication density (Silo-Bench) — no model effect once the queue is held fixed (p = 0.26)
  • Question rate (ALEM) — model ranking flips between halves of the queues (τ = 0.33)
  • Instruction rate (ALEM) — no model effect once the queue is held fixed (p = 0.13)
  • Speaker consistency (MI) — no model effect once the queue is held fixed (p = 0.11)
  • Instantaneous coordination (MI) — no model effect once the queue is held fixed (p = 0.33)
  • Stall ratio (Magentic-One) — model ranking flips between halves of the queues (τ = 0.33)
  • Handoff events (ALEM) — model ranking flips between halves of the queues (τ = -0.33)
  • Under-delegation (MS-Eval) — no model effect once the queue is held fixed (p = 0.06)
  • Parallelism rate (SwarmBench) — no model effect once the queue is held fixed (p = 0.06)
  • Bottleneck agent PAC (HiveMind) — model ranking flips between halves of the queues (τ = 0.55)
  • Grounding accuracy — no model effect once the queue is held fixed (p = 0.13)
Still standing (9)
  • CCE, any outcome
  • Message token share
  • Unnecessary path ratio (GEMMAS)
  • Redundancy rate (Scaling)
  • Task offloading (PARTNR)
  • Idle rate
  • Critical-path ratio
  • Task throughput
  • Assignment diversity
outcomeefficiencycommunicationcoordinationstructurecreditadaptationrobustness
Figure 1. Each row is a test, and a metric drops out at the first one it fails. Click a row to see which ones left there and why. Colors are metric families.

I’m writing a paper where we aim to propose a new metric to eval collaboration in multi-agent systems, and a key thing I noticed reading other metric papers was that they begin with an analysis of a previous metric & then show why it’s inadequate for the thing they’re measuring. They then show how their proposed metric correlates more closely with a stronger proxy for that measure, such as what humans think.

So I decided to do the “exact” same thing.

Specifically, however, I decided to evaluate every MAS metric I could find—with the twist of doing so on my own benchmark ReOrgBench, which would allow me to do the measuring in a long-horizon, orchestrator-centric setting where changes to the task can occur mid-run, mimicking how humans would use a MAS such as Claude Code or Codex.

A key feature of ReOrgBench is that it uses no LLM-as-a-judge: code compares the final state of each run with predefined checkpoints, including whether required steps were taken and whether prohibited ones weren't. I cataloged 391 metrics from 128 MAS papers from ~the last 3 years and then merged them into 293 distinct ones. I then kept filtering them out till there were ultimately only 9 left—of which I still wouldn't say are necessarily infallible. The values of these metrics were determined from 60 episodes: Fable 5.2, Opus, Sonnet, and Haiku 5.5 (all medium effort) on 15 of ReOrgBench's held-out tasks, averaging ~7.6M tokens per episode.

1. The benchmark

Here's what each task looks like: there's some ops queue at a made-up software company. A "human" prompts the orchestrator with whatever it needs done; it, in turn, recruits any number of agents with their own tools from any of 7 roles: generalist, engineer, analyst, support, finance, procurement, and legal. Each of these agents can also recruit another agent, and they can communicate with any other agent they're aware of. At some point, the human requests some modification (sorta how you or I will tell Codex or Claude, "yk waht acc do thsi"), and now the orchestrator must adapt, alongside directing the rest of the team on doing so.

2. The outcomes

Fable 5.2
0.969
Opus 5.5
0.924
Sonnet 5.5
0.671
Haiku 5.5
0.604
0.000.250.500.751.00
Figure 2. Checkpoint score with 95% intervals, and strict pass@1 (the share of episodes where every single checkpoint passed). The per-queue tab shows each episode.

Here's what I found via failure analysis: Haiku gets basic stuff wrong, Sonnet primarily misses the exceptions, and both Fable and Opus lose their points primarily due to things that change while they're busy.

Only Fable and Opus ever pass, and Fable seems to be pretty ahead, but the checkpoint score shows that Sonnet and Haiku don't do too badly, and Fable and Opus are very close. These checkpoint score observations will prove very useful for filtering and later conclusions.

3. Can I even compute it?

The first question I asked when making the first cut from the 293 metrics was as follows: can I even compute it? The answer is mostly no. Specifically, it was no for 212 of the metrics. They either needed some expensive ablation, an LLM judge, a very specific setting, human labels, gold trajectories, or repeated runs. If you're resource-constrained or your MAS doesn't fit a specific box, these needs make those metrics generalize poorly. This left me with 13 communication, 25 efficiency, 15 outcome, 11 coordination, 2 robustness, 4 credit, and 11 role/structure metrics: a total of 81.

Of these 81, a lot were actually the same thing under different formalizations—total tokens and dollar cost for example are literally the same number here—and about a dozen were effectively the same as checkpoint scores or pass@1, while 5 were more like curves or descriptive views rather than singular numbers per episode, so that brought me down to 59. I then added 8 faithful adaptations of metrics that didn't quite fit as-is (GovSim's equality, for one) and 7 plain counts like team size or message count to get to 74. ReOrgBench's own metrics I left out entirely, since that'd sorta defeat the point.

4. Is it even defined?

To further filter them down, I then asked myself if these 74 even take values that are useful: they had to be defined on at least 80% of the episodes, and no one value could be what over 80% of them took on. 6 didn't, so they went out. Some of these just never happen in these runs, like assignment conflicts, structural collapse, and ClawArena's read-only violations, which were zero in every single episode, some were almost always the same thing like ALEM's parse success or TravelPlanner's delivery rate, and AgentWorld's CCE is by definition 0 on a failed task, which here was 82% of the episodes; I did however keep its any-outcome variant.

5. Team or task?

The value a metric takes on should be much more influenced by the model being evaluated rather than the task when the tasks are similar in nature. To this end, I evaluated reliability via a Friedman test (p < 0.05) and checking that model ranking was consistent on both halves of the queues via Kendall τ ≥ 0.67.

0%0%25%25%50%50%75%75%100%100%team = taskshare of variance from the queue (task)share of variance from the model (team)TheAgentCompany partialMultiAgentBench KPIToken costAction countRounds usedScore per tokenCOMMA efficiencyCCE (AgentWorld)CCE, any outcomeExtraneous effort (PARTNR)Tool error rateMessage countMessage densityComm. overhead (CooperBench)Message token shareBroadcast rateUnnecessary path ratio (GEMMAS)Redundancy rate (Scaling)Information diversity (GEMMAS)Graph density (Puppeteer)Hub centralizationBalance (VillagerAgent)Equality (GovSim)Task offloading (PARTNR)Idle rateParallelismCritical-path ratioWrite conflictsRedundancy (TeamCraft)Team sizeHierarchy depthStaffing F1 (MS-Eval)Tool permission use (ClawArena)PAC (AgentWorld)KPI concentrationRemoval attributionRetirementsDelivery rate (TravelPlanner)Progress rate AUC (AgentBoard)Orchestrator token share (ClawArena)LLM calls (TheAgentCompany)Planning cycles (PARTNR)Critical Steps (MS-Eval)Parallelism gain (SwarmBench)Task throughputCompletion efficiency (VillagerAgent)Steps per checkpoint (WebCanvas)Sink-agent tokens (MacNet)Repetition rate (AgentQuest)Optima rewardMessage tokens (BANDMAS)Mistakes (COMMA)Communication density (Silo-Bench)Network efficiency (NetSafe)Task distance (DecArena)Question rate (ALEM)Instruction rate (ALEM)Assistance seeking (SyncMind)Speaker consistency (MI)Instantaneous coordination (MI)Stall ratio (Magentic-One)Assignment conflictsAssignment diversityHandoff events (ALEM)Planner cost (TeamCraft)Action-action coordination (MI)Under-delegation (MS-Eval)Parallelism rate (SwarmBench)Structural collapseRead-only violations (ClawArena)Attribution entropyBottleneck agent PAC (HiveMind)Grounding accuracyParse success (ALEM)Checkpoint scoreCheckpoint scoreCCE, any outcomeToken costRounds used
Hover or tap a dot.
outcomeefficiencycommunicationcoordinationstructurecreditadaptationrobustness
Figure 3. Each dot is a metric. The higher it is, the more of its variance comes from the model; the further right, the more comes from the queue. The checkpoint score sits in the top left. Hover for details.

From here, we lose 31 metrics. A lot of these were cost metrics, which turned out to be mostly a function of the queue rather than the model: 63% of the variance in token use came from the queue and only 4% from the model. That's how you get score-per-token and COMMA's efficiency score both saying Haiku is the most efficient model, which isn't because it's actually collaborating efficiently but because it does less and failing doesn't cost much. Optima's reward was the funniest one though, ranking the models in a totally different order on each half of the queues.

6. Does it point the right way?

Besides the model having a stronger sway on the metric than the task, I needed to ensure that it actually had the correct sway; the metrics must point in the direction their authors intended, so I applied a validity check: that the within-queue Spearman ρ with the checkpoint score had the intended sign with |ρ| ≥ 0.25.

−1.0−0.5+0.0+0.5+1.0← worse outcome when the metric says betteragrees with its paper →TheAgentCompany partialCompletion efficiency (VillagerAgent)Task throughputCCE, any outcomeExtraneous effort (PARTNR)Staffing F1 (MS-Eval)PAC (AgentWorld)ParallelismParallelism gain (SwarmBench)Assignment diversityCritical-path ratioHierarchy depth *Team size *CCE (AgentWorld)MultiAgentBench KPIAttribution entropy *Steps per checkpoint (WebCanvas)Planner cost (TeamCraft)Optima rewardTask distance (DecArena) *Unnecessary path ratio (GEMMAS)Progress rate AUC (AgentBoard)Idle rateTask offloading (PARTNR)Equality (GovSim)Message token shareScore per tokenStall ratio (Magentic-One)Retirements *Redundancy rate (Scaling)Rounds usedBalance (VillagerAgent)Parallelism rate (SwarmBench)Message count *Bottleneck agent PAC (HiveMind)Comm. overhead (CooperBench)Handoff events (ALEM)Sink-agent tokens (MacNet)Information diversity (GEMMAS)Delivery rate (TravelPlanner)Under-delegation (MS-Eval)Instruction rate (ALEM) *Parse success (ALEM)Tool permission use (ClawArena)Orchestrator token share (ClawArena)Critical Steps (MS-Eval)Removal attribution *COMMA efficiencyInstantaneous coordination (MI)Redundancy (TeamCraft)Broadcast rate *Message density *Communication density (Silo-Bench) *LLM calls (TheAgentCompany)Token costTool error rateGrounding accuracyAssistance seeking (SyncMind) *KPI concentration *Mistakes (COMMA)Message tokens (BANDMAS)Speaker consistency (MI)Question rate (ALEM) *Write conflictsNetwork efficiency (NetSafe) *Graph density (Puppeteer) *Repetition rate (AgentQuest)Planning cycles (PARTNR)Action countAction-action coordination (MI)Hub centralization *
outcomeefficiencycommunicationcoordinationstructurecreditadaptationrobustness
Figure 4. How each metric lines up with the outcome, flipped so that right means it agrees with its own paper. The tabs switch between the checkpoint score (Spearman ρ) and strict pass (AUC). Anything in the shaded band has |ρ| < 0.25, and starred metrics never said which direction is better.

19 metrics were eliminated: problems with action-outcome correlation, message cost, action-to-action mutual information, or plain being weak or directionless. Action count was a weird one: normally you'd think more actions means a worse, more wasteful team, but Fable took like 437 actions an episode and Haiku only 264, and that's honestly just because doing the job properly means you have to go through every single ticket.

7. Is it new information?

Here, I wanted to filter for something new—to leave us with a few metrics that say a lot instead of a few that don't say much—so I did a redundancy check: that there was a |ρ| < 0.9 with the score, alongside |ρ| ≤ 0.85 with any stronger metric.

TheAgentCompany partialTheAgentCompany partialCCE, any outcomeCCE, any outcomeExtraneous effort (PARTNR)Extraneous effort (PARTNR)Message token shareMessage token shareUnnecessary path ratio (GEMMAS)Unnecessary path ratio (GEMMAS)Redundancy rate (Scaling)Redundancy rate (Scaling)Equality (GovSim)Equality (GovSim)Task offloading (PARTNR)Task offloading (PARTNR)Idle rateIdle rateParallelismParallelismCritical-path ratioCritical-path ratioStaffing F1 (MS-Eval)Staffing F1 (MS-Eval)PAC (AgentWorld)PAC (AgentWorld)Parallelism gain (SwarmBench)Parallelism gain (SwarmBench)Task throughputTask throughputCompletion efficiency (VillagerAgent)Completion efficiency (VillagerAgent)Assignment diversityAssignment diversityPlanner cost (TeamCraft)Planner cost (TeamCraft)Checkpoint scoreCheckpoint scoreTheAgentCompany partial × TheAgentCompany partial: +1.00TheAgentCompany partial × CCE, any outcome: +0.75TheAgentCompany partial × Extraneous effort (PARTNR): −0.75TheAgentCompany partial × Message token share: −0.44TheAgentCompany partial × Unnecessary path ratio (GEMMAS): −0.56TheAgentCompany partial × Redundancy rate (Scaling): −0.32TheAgentCompany partial × Equality (GovSim): +0.56TheAgentCompany partial × Task offloading (PARTNR): +0.51TheAgentCompany partial × Idle rate: −0.54TheAgentCompany partial × Parallelism: +0.69TheAgentCompany partial × Critical-path ratio: −0.74TheAgentCompany partial × Staffing F1 (MS-Eval): +0.63TheAgentCompany partial × PAC (AgentWorld): +0.75TheAgentCompany partial × Parallelism gain (SwarmBench): +0.62TheAgentCompany partial × Task throughput: +0.73TheAgentCompany partial × Completion efficiency (VillagerAgent): +0.79TheAgentCompany partial × Assignment diversity: +0.59TheAgentCompany partial × Planner cost (TeamCraft): −0.42TheAgentCompany partial × Checkpoint score: +1.00CCE, any outcome × TheAgentCompany partial: +0.75CCE, any outcome × CCE, any outcome: +1.00CCE, any outcome × Extraneous effort (PARTNR): −1.00CCE, any outcome × Message token share: −0.50CCE, any outcome × Unnecessary path ratio (GEMMAS): −0.80CCE, any outcome × Redundancy rate (Scaling): −0.47CCE, any outcome × Equality (GovSim): +0.43CCE, any outcome × Task offloading (PARTNR): +0.72CCE, any outcome × Idle rate: −0.54CCE, any outcome × Parallelism: +0.57CCE, any outcome × Critical-path ratio: −0.80CCE, any outcome × Staffing F1 (MS-Eval): +0.42CCE, any outcome × PAC (AgentWorld): +0.93CCE, any outcome × Parallelism gain (SwarmBench): +0.54CCE, any outcome × Task throughput: +0.73CCE, any outcome × Completion efficiency (VillagerAgent): +0.76CCE, any outcome × Assignment diversity: +0.85CCE, any outcome × Planner cost (TeamCraft): −0.19CCE, any outcome × Checkpoint score: +0.75Extraneous effort (PARTNR) × TheAgentCompany partial: −0.75Extraneous effort (PARTNR) × CCE, any outcome: −1.00Extraneous effort (PARTNR) × Extraneous effort (PARTNR): +1.00Extraneous effort (PARTNR) × Message token share: +0.50Extraneous effort (PARTNR) × Unnecessary path ratio (GEMMAS): +0.80Extraneous effort (PARTNR) × Redundancy rate (Scaling): +0.47Extraneous effort (PARTNR) × Equality (GovSim): −0.43Extraneous effort (PARTNR) × Task offloading (PARTNR): −0.72Extraneous effort (PARTNR) × Idle rate: +0.54Extraneous effort (PARTNR) × Parallelism: −0.57Extraneous effort (PARTNR) × Critical-path ratio: +0.80Extraneous effort (PARTNR) × Staffing F1 (MS-Eval): −0.42Extraneous effort (PARTNR) × PAC (AgentWorld): −0.93Extraneous effort (PARTNR) × Parallelism gain (SwarmBench): −0.54Extraneous effort (PARTNR) × Task throughput: −0.73Extraneous effort (PARTNR) × Completion efficiency (VillagerAgent): −0.76Extraneous effort (PARTNR) × Assignment diversity: −0.85Extraneous effort (PARTNR) × Planner cost (TeamCraft): +0.19Extraneous effort (PARTNR) × Checkpoint score: −0.75Message token share × TheAgentCompany partial: −0.44Message token share × CCE, any outcome: −0.50Message token share × Extraneous effort (PARTNR): +0.50Message token share × Message token share: +1.00Message token share × Unnecessary path ratio (GEMMAS): +0.42Message token share × Redundancy rate (Scaling): +0.51Message token share × Equality (GovSim): −0.17Message token share × Task offloading (PARTNR): −0.37Message token share × Idle rate: +0.46Message token share × Parallelism: −0.44Message token share × Critical-path ratio: +0.55Message token share × Staffing F1 (MS-Eval): −0.26Message token share × PAC (AgentWorld): −0.49Message token share × Parallelism gain (SwarmBench): −0.36Message token share × Task throughput: −0.48Message token share × Completion efficiency (VillagerAgent): −0.48Message token share × Assignment diversity: −0.54Message token share × Planner cost (TeamCraft): +0.13Message token share × Checkpoint score: −0.44Unnecessary path ratio (GEMMAS) × TheAgentCompany partial: −0.56Unnecessary path ratio (GEMMAS) × CCE, any outcome: −0.80Unnecessary path ratio (GEMMAS) × Extraneous effort (PARTNR): +0.80Unnecessary path ratio (GEMMAS) × Message token share: +0.42Unnecessary path ratio (GEMMAS) × Unnecessary path ratio (GEMMAS): +1.00Unnecessary path ratio (GEMMAS) × Redundancy rate (Scaling): +0.42Unnecessary path ratio (GEMMAS) × Equality (GovSim): −0.50Unnecessary path ratio (GEMMAS) × Task offloading (PARTNR): −0.54Unnecessary path ratio (GEMMAS) × Idle rate: +0.50Unnecessary path ratio (GEMMAS) × Parallelism: −0.48Unnecessary path ratio (GEMMAS) × Critical-path ratio: +0.55Unnecessary path ratio (GEMMAS) × Staffing F1 (MS-Eval): −0.29Unnecessary path ratio (GEMMAS) × PAC (AgentWorld): −0.77Unnecessary path ratio (GEMMAS) × Parallelism gain (SwarmBench): −0.45Unnecessary path ratio (GEMMAS) × Task throughput: −0.66Unnecessary path ratio (GEMMAS) × Completion efficiency (VillagerAgent): −0.66Unnecessary path ratio (GEMMAS) × Assignment diversity: −0.74Unnecessary path ratio (GEMMAS) × Planner cost (TeamCraft): +0.15Unnecessary path ratio (GEMMAS) × Checkpoint score: −0.56Redundancy rate (Scaling) × TheAgentCompany partial: −0.32Redundancy rate (Scaling) × CCE, any outcome: −0.47Redundancy rate (Scaling) × Extraneous effort (PARTNR): +0.47Redundancy rate (Scaling) × Message token share: +0.51Redundancy rate (Scaling) × Unnecessary path ratio (GEMMAS): +0.42Redundancy rate (Scaling) × Redundancy rate (Scaling): +1.00Redundancy rate (Scaling) × Equality (GovSim): +0.06Redundancy rate (Scaling) × Task offloading (PARTNR): −0.48Redundancy rate (Scaling) × Idle rate: +0.49Redundancy rate (Scaling) × Parallelism: −0.42Redundancy rate (Scaling) × Critical-path ratio: +0.43Redundancy rate (Scaling) × Staffing F1 (MS-Eval): −0.26Redundancy rate (Scaling) × PAC (AgentWorld): −0.42Redundancy rate (Scaling) × Parallelism gain (SwarmBench): −0.28Redundancy rate (Scaling) × Task throughput: −0.49Redundancy rate (Scaling) × Completion efficiency (VillagerAgent): −0.46Redundancy rate (Scaling) × Assignment diversity: −0.55Redundancy rate (Scaling) × Planner cost (TeamCraft): +0.10Redundancy rate (Scaling) × Checkpoint score: −0.32Equality (GovSim) × TheAgentCompany partial: +0.56Equality (GovSim) × CCE, any outcome: +0.43Equality (GovSim) × Extraneous effort (PARTNR): −0.43Equality (GovSim) × Message token share: −0.17Equality (GovSim) × Unnecessary path ratio (GEMMAS): −0.50Equality (GovSim) × Redundancy rate (Scaling): +0.06Equality (GovSim) × Equality (GovSim): +1.00Equality (GovSim) × Task offloading (PARTNR): +0.20Equality (GovSim) × Idle rate: −0.22Equality (GovSim) × Parallelism: +0.50Equality (GovSim) × Critical-path ratio: −0.39Equality (GovSim) × Staffing F1 (MS-Eval): +0.39Equality (GovSim) × PAC (AgentWorld): +0.63Equality (GovSim) × Parallelism gain (SwarmBench): +0.47Equality (GovSim) × Task throughput: +0.55Equality (GovSim) × Completion efficiency (VillagerAgent): +0.61Equality (GovSim) × Assignment diversity: +0.24Equality (GovSim) × Planner cost (TeamCraft): −0.45Equality (GovSim) × Checkpoint score: +0.56Task offloading (PARTNR) × TheAgentCompany partial: +0.51Task offloading (PARTNR) × CCE, any outcome: +0.72Task offloading (PARTNR) × Extraneous effort (PARTNR): −0.72Task offloading (PARTNR) × Message token share: −0.37Task offloading (PARTNR) × Unnecessary path ratio (GEMMAS): −0.54Task offloading (PARTNR) × Redundancy rate (Scaling): −0.48Task offloading (PARTNR) × Equality (GovSim): +0.20Task offloading (PARTNR) × Task offloading (PARTNR): +1.00Task offloading (PARTNR) × Idle rate: −0.40Task offloading (PARTNR) × Parallelism: +0.53Task offloading (PARTNR) × Critical-path ratio: −0.67Task offloading (PARTNR) × Staffing F1 (MS-Eval): +0.30Task offloading (PARTNR) × PAC (AgentWorld): +0.64Task offloading (PARTNR) × Parallelism gain (SwarmBench): +0.48Task offloading (PARTNR) × Task throughput: +0.62Task offloading (PARTNR) × Completion efficiency (VillagerAgent): +0.61Task offloading (PARTNR) × Assignment diversity: +0.75Task offloading (PARTNR) × Planner cost (TeamCraft): −0.06Task offloading (PARTNR) × Checkpoint score: +0.51Idle rate × TheAgentCompany partial: −0.54Idle rate × CCE, any outcome: −0.54Idle rate × Extraneous effort (PARTNR): +0.54Idle rate × Message token share: +0.46Idle rate × Unnecessary path ratio (GEMMAS): +0.50Idle rate × Redundancy rate (Scaling): +0.49Idle rate × Equality (GovSim): −0.22Idle rate × Task offloading (PARTNR): −0.40Idle rate × Idle rate: +1.00Idle rate × Parallelism: −0.59Idle rate × Critical-path ratio: +0.47Idle rate × Staffing F1 (MS-Eval): −0.28Idle rate × PAC (AgentWorld): −0.51Idle rate × Parallelism gain (SwarmBench): −0.57Idle rate × Task throughput: −0.61Idle rate × Completion efficiency (VillagerAgent): −0.59Idle rate × Assignment diversity: −0.66Idle rate × Planner cost (TeamCraft): +0.50Idle rate × Checkpoint score: −0.54Parallelism × TheAgentCompany partial: +0.69Parallelism × CCE, any outcome: +0.57Parallelism × Extraneous effort (PARTNR): −0.57Parallelism × Message token share: −0.44Parallelism × Unnecessary path ratio (GEMMAS): −0.48Parallelism × Redundancy rate (Scaling): −0.42Parallelism × Equality (GovSim): +0.50Parallelism × Task offloading (PARTNR): +0.53Parallelism × Idle rate: −0.59Parallelism × Parallelism: +1.00Parallelism × Critical-path ratio: −0.63Parallelism × Staffing F1 (MS-Eval): +0.72Parallelism × PAC (AgentWorld): +0.70Parallelism × Parallelism gain (SwarmBench): +0.91Parallelism × Task throughput: +0.84Parallelism × Completion efficiency (VillagerAgent): +0.85Parallelism × Assignment diversity: +0.45Parallelism × Planner cost (TeamCraft): −0.56Parallelism × Checkpoint score: +0.69Critical-path ratio × TheAgentCompany partial: −0.74Critical-path ratio × CCE, any outcome: −0.80Critical-path ratio × Extraneous effort (PARTNR): +0.80Critical-path ratio × Message token share: +0.55Critical-path ratio × Unnecessary path ratio (GEMMAS): +0.55Critical-path ratio × Redundancy rate (Scaling): +0.43Critical-path ratio × Equality (GovSim): −0.39Critical-path ratio × Task offloading (PARTNR): −0.67Critical-path ratio × Idle rate: +0.47Critical-path ratio × Parallelism: −0.63Critical-path ratio × Critical-path ratio: +1.00Critical-path ratio × Staffing F1 (MS-Eval): −0.52Critical-path ratio × PAC (AgentWorld): −0.78Critical-path ratio × Parallelism gain (SwarmBench): −0.60Critical-path ratio × Task throughput: −0.69Critical-path ratio × Completion efficiency (VillagerAgent): −0.73Critical-path ratio × Assignment diversity: −0.62Critical-path ratio × Planner cost (TeamCraft): +0.17Critical-path ratio × Checkpoint score: −0.74Staffing F1 (MS-Eval) × TheAgentCompany partial: +0.63Staffing F1 (MS-Eval) × CCE, any outcome: +0.42Staffing F1 (MS-Eval) × Extraneous effort (PARTNR): −0.42Staffing F1 (MS-Eval) × Message token share: −0.26Staffing F1 (MS-Eval) × Unnecessary path ratio (GEMMAS): −0.29Staffing F1 (MS-Eval) × Redundancy rate (Scaling): −0.26Staffing F1 (MS-Eval) × Equality (GovSim): +0.39Staffing F1 (MS-Eval) × Task offloading (PARTNR): +0.30Staffing F1 (MS-Eval) × Idle rate: −0.28Staffing F1 (MS-Eval) × Parallelism: +0.72Staffing F1 (MS-Eval) × Critical-path ratio: −0.52Staffing F1 (MS-Eval) × Staffing F1 (MS-Eval): +1.00Staffing F1 (MS-Eval) × PAC (AgentWorld): +0.53Staffing F1 (MS-Eval) × Parallelism gain (SwarmBench): +0.59Staffing F1 (MS-Eval) × Task throughput: +0.68Staffing F1 (MS-Eval) × Completion efficiency (VillagerAgent): +0.73Staffing F1 (MS-Eval) × Assignment diversity: +0.26Staffing F1 (MS-Eval) × Planner cost (TeamCraft): −0.48Staffing F1 (MS-Eval) × Checkpoint score: +0.63PAC (AgentWorld) × TheAgentCompany partial: +0.75PAC (AgentWorld) × CCE, any outcome: +0.93PAC (AgentWorld) × Extraneous effort (PARTNR): −0.93PAC (AgentWorld) × Message token share: −0.49PAC (AgentWorld) × Unnecessary path ratio (GEMMAS): −0.77PAC (AgentWorld) × Redundancy rate (Scaling): −0.42PAC (AgentWorld) × Equality (GovSim): +0.63PAC (AgentWorld) × Task offloading (PARTNR): +0.64PAC (AgentWorld) × Idle rate: −0.51PAC (AgentWorld) × Parallelism: +0.70PAC (AgentWorld) × Critical-path ratio: −0.78PAC (AgentWorld) × Staffing F1 (MS-Eval): +0.53PAC (AgentWorld) × PAC (AgentWorld): +1.00PAC (AgentWorld) × Parallelism gain (SwarmBench): +0.63PAC (AgentWorld) × Task throughput: +0.81PAC (AgentWorld) × Completion efficiency (VillagerAgent): +0.85PAC (AgentWorld) × Assignment diversity: +0.72PAC (AgentWorld) × Planner cost (TeamCraft): −0.36PAC (AgentWorld) × Checkpoint score: +0.75Parallelism gain (SwarmBench) × TheAgentCompany partial: +0.62Parallelism gain (SwarmBench) × CCE, any outcome: +0.54Parallelism gain (SwarmBench) × Extraneous effort (PARTNR): −0.54Parallelism gain (SwarmBench) × Message token share: −0.36Parallelism gain (SwarmBench) × Unnecessary path ratio (GEMMAS): −0.45Parallelism gain (SwarmBench) × Redundancy rate (Scaling): −0.28Parallelism gain (SwarmBench) × Equality (GovSim): +0.47Parallelism gain (SwarmBench) × Task offloading (PARTNR): +0.48Parallelism gain (SwarmBench) × Idle rate: −0.57Parallelism gain (SwarmBench) × Parallelism: +0.91Parallelism gain (SwarmBench) × Critical-path ratio: −0.60Parallelism gain (SwarmBench) × Staffing F1 (MS-Eval): +0.59Parallelism gain (SwarmBench) × PAC (AgentWorld): +0.63Parallelism gain (SwarmBench) × Parallelism gain (SwarmBench): +1.00Parallelism gain (SwarmBench) × Task throughput: +0.77Parallelism gain (SwarmBench) × Completion efficiency (VillagerAgent): +0.77Parallelism gain (SwarmBench) × Assignment diversity: +0.44Parallelism gain (SwarmBench) × Planner cost (TeamCraft): −0.48Parallelism gain (SwarmBench) × Checkpoint score: +0.62Task throughput × TheAgentCompany partial: +0.73Task throughput × CCE, any outcome: +0.73Task throughput × Extraneous effort (PARTNR): −0.73Task throughput × Message token share: −0.48Task throughput × Unnecessary path ratio (GEMMAS): −0.66Task throughput × Redundancy rate (Scaling): −0.49Task throughput × Equality (GovSim): +0.55Task throughput × Task offloading (PARTNR): +0.62Task throughput × Idle rate: −0.61Task throughput × Parallelism: +0.84Task throughput × Critical-path ratio: −0.69Task throughput × Staffing F1 (MS-Eval): +0.68Task throughput × PAC (AgentWorld): +0.81Task throughput × Parallelism gain (SwarmBench): +0.77Task throughput × Task throughput: +1.00Task throughput × Completion efficiency (VillagerAgent): +0.96Task throughput × Assignment diversity: +0.69Task throughput × Planner cost (TeamCraft): −0.59Task throughput × Checkpoint score: +0.73Completion efficiency (VillagerAgent) × TheAgentCompany partial: +0.79Completion efficiency (VillagerAgent) × CCE, any outcome: +0.76Completion efficiency (VillagerAgent) × Extraneous effort (PARTNR): −0.76Completion efficiency (VillagerAgent) × Message token share: −0.48Completion efficiency (VillagerAgent) × Unnecessary path ratio (GEMMAS): −0.66Completion efficiency (VillagerAgent) × Redundancy rate (Scaling): −0.46Completion efficiency (VillagerAgent) × Equality (GovSim): +0.61Completion efficiency (VillagerAgent) × Task offloading (PARTNR): +0.61Completion efficiency (VillagerAgent) × Idle rate: −0.59Completion efficiency (VillagerAgent) × Parallelism: +0.85Completion efficiency (VillagerAgent) × Critical-path ratio: −0.73Completion efficiency (VillagerAgent) × Staffing F1 (MS-Eval): +0.73Completion efficiency (VillagerAgent) × PAC (AgentWorld): +0.85Completion efficiency (VillagerAgent) × Parallelism gain (SwarmBench): +0.77Completion efficiency (VillagerAgent) × Task throughput: +0.96Completion efficiency (VillagerAgent) × Completion efficiency (VillagerAgent): +1.00Completion efficiency (VillagerAgent) × Assignment diversity: +0.67Completion efficiency (VillagerAgent) × Planner cost (TeamCraft): −0.59Completion efficiency (VillagerAgent) × Checkpoint score: +0.79Assignment diversity × TheAgentCompany partial: +0.59Assignment diversity × CCE, any outcome: +0.85Assignment diversity × Extraneous effort (PARTNR): −0.85Assignment diversity × Message token share: −0.54Assignment diversity × Unnecessary path ratio (GEMMAS): −0.74Assignment diversity × Redundancy rate (Scaling): −0.55Assignment diversity × Equality (GovSim): +0.24Assignment diversity × Task offloading (PARTNR): +0.75Assignment diversity × Idle rate: −0.66Assignment diversity × Parallelism: +0.45Assignment diversity × Critical-path ratio: −0.62Assignment diversity × Staffing F1 (MS-Eval): +0.26Assignment diversity × PAC (AgentWorld): +0.72Assignment diversity × Parallelism gain (SwarmBench): +0.44Assignment diversity × Task throughput: +0.69Assignment diversity × Completion efficiency (VillagerAgent): +0.67Assignment diversity × Assignment diversity: +1.00Assignment diversity × Planner cost (TeamCraft): −0.19Assignment diversity × Checkpoint score: +0.59Planner cost (TeamCraft) × TheAgentCompany partial: −0.42Planner cost (TeamCraft) × CCE, any outcome: −0.19Planner cost (TeamCraft) × Extraneous effort (PARTNR): +0.19Planner cost (TeamCraft) × Message token share: +0.13Planner cost (TeamCraft) × Unnecessary path ratio (GEMMAS): +0.15Planner cost (TeamCraft) × Redundancy rate (Scaling): +0.10Planner cost (TeamCraft) × Equality (GovSim): −0.45Planner cost (TeamCraft) × Task offloading (PARTNR): −0.06Planner cost (TeamCraft) × Idle rate: +0.50Planner cost (TeamCraft) × Parallelism: −0.56Planner cost (TeamCraft) × Critical-path ratio: +0.17Planner cost (TeamCraft) × Staffing F1 (MS-Eval): −0.48Planner cost (TeamCraft) × PAC (AgentWorld): −0.36Planner cost (TeamCraft) × Parallelism gain (SwarmBench): −0.48Planner cost (TeamCraft) × Task throughput: −0.59Planner cost (TeamCraft) × Completion efficiency (VillagerAgent): −0.59Planner cost (TeamCraft) × Assignment diversity: −0.19Planner cost (TeamCraft) × Planner cost (TeamCraft): +1.00Planner cost (TeamCraft) × Checkpoint score: −0.42Checkpoint score × TheAgentCompany partial: +1.00Checkpoint score × CCE, any outcome: +0.75Checkpoint score × Extraneous effort (PARTNR): −0.75Checkpoint score × Message token share: −0.44Checkpoint score × Unnecessary path ratio (GEMMAS): −0.56Checkpoint score × Redundancy rate (Scaling): −0.32Checkpoint score × Equality (GovSim): +0.56Checkpoint score × Task offloading (PARTNR): +0.51Checkpoint score × Idle rate: −0.54Checkpoint score × Parallelism: +0.69Checkpoint score × Critical-path ratio: −0.74Checkpoint score × Staffing F1 (MS-Eval): +0.63Checkpoint score × PAC (AgentWorld): +0.75Checkpoint score × Parallelism gain (SwarmBench): +0.62Checkpoint score × Task throughput: +0.73Checkpoint score × Completion efficiency (VillagerAgent): +0.79Checkpoint score × Assignment diversity: +0.59Checkpoint score × Planner cost (TeamCraft): −0.42Checkpoint score × Checkpoint score: +1.00
Hover a cell. Blue is positive, red negative.
Figure 5. Spearman correlations between the 18 metrics left at this point, plus the checkpoint score. Blue is positive and red is negative; the solid blocks are metrics that are pretty much copies of each other.

Here, 5 metrics collapsed into others, leaving us with 13.

8. Beyond the score

Having 13 metrics that correlate with the checkpoint score is a start, however I still needed to make sure that the sway a metric has over the models isn't just the checkpoint score's sway in disguise; if it is, we could've just looked at the checkpoint score and stopped there. Specifically, for the final cut I took the residuals from regressing each of the 13 metrics on the checkpoint score within the queue, and then I checked whether the models still differed on those residuals via a Friedman test. From here, we lose 4 more metrics: parallelism, MS-Eval's staffing F1, GovSim's equality, and TeamCraft's planner cost.

CCE, any outcome
Fable 5.2: −0.017Opus 5.5: +0.022Sonnet 5.5: +0.097Haiku 5.5: −0.101
Message token share
Fable 5.2: −0.002Opus 5.5: −0.022Sonnet 5.5: −0.039Haiku 5.5: +0.063
Unnecessary path ratio (GEMMAS)
Fable 5.2: +0.016Opus 5.5: −0.025Sonnet 5.5: −0.065Haiku 5.5: +0.075
Redundancy rate (Scaling)
Fable 5.2: +0.013Opus 5.5: −0.013Sonnet 5.5: −0.023Haiku 5.5: +0.023
Equality (GovSim) · drops
Fable 5.2: +0.037Opus 5.5: −0.031Sonnet 5.5: +0.004Haiku 5.5: −0.010
Task offloading (PARTNR)
Fable 5.2: −0.012Opus 5.5: +0.010Sonnet 5.5: +0.076Haiku 5.5: −0.074
Idle rate
Fable 5.2: −0.023Opus 5.5: +0.005Sonnet 5.5: −0.069Haiku 5.5: +0.087
Parallelism · drops
Fable 5.2: +0.218Opus 5.5: −0.007Sonnet 5.5: +0.057Haiku 5.5: −0.268
Critical-path ratio
Fable 5.2: +0.019Opus 5.5: −0.003Sonnet 5.5: −0.152Haiku 5.5: +0.136
Staffing F1 (MS-Eval) · drops
Fable 5.2: +0.024Opus 5.5: −0.013Sonnet 5.5: +0.007Haiku 5.5: −0.018
Task throughput
Fable 5.2: +0.002Opus 5.5: +0.000Sonnet 5.5: +0.011Haiku 5.5: −0.014
Assignment diversity
Fable 5.2: −0.008Opus 5.5: +0.011Sonnet 5.5: +0.032Haiku 5.5: −0.035
Planner cost (TeamCraft) · drops
Fable 5.2: −0.044Opus 5.5: +0.027Sonnet 5.5: +0.038Haiku 5.5: −0.022
Fable 5.2Opus 5.5Sonnet 5.5Haiku 5.5right = better, per each metric’s paper
Figure 6. Model means on the 13 final candidates, both raw and with the checkpoint score held fixed. Right is better according to each metric's own paper, and the faded rows are the ones that dropped here.

That leaves us with 9:

  • CCE, any-outcome variant (AgentWorld)
  • unnecessary path ratio (GEMMAS)
  • redundancy rate (Towards a Science of Scaling Agent Systems)
  • share of output tokens spent messaging
  • task offloading (PARTNR)
  • idle rate (Coordination Matters)
  • critical-path ratio (MS-Eval's Critical Steps)
  • task throughput (checkpoints per round)
  • assignment diversity (Coordination Matters)

Note that given there are only 15 queues in the held-out split, getting lucky on 1 or 2 queues could conceivably let a bad metric slip through! To check that, I ran 1,000 bootstrap resamples of the 15 queues: all 9 of these still pass at least 97% of the time, while the ones that dropped out earlier basically never do, with token cost passing 0.1% of the time, COMMA's efficiency score 1%, and ClawArena's tool permissions 3%.

Notice how in Figure 6, once the checkpoint score is held fixed, Sonnet and Haiku go in opposite directions. Sonnet beats what its checkpoint score predicts on CCE and task offloading alongside having a lower critical-path ratio (so less serial work), while Haiku sits below what its checkpoint score predicts on all three. Why does Sonnet look so much better once you control for the score? Because as we saw in the failure analysis earlier, Sonnet's team actually coordinates fine and primarily loses points on exceptions like collateral edits and held customers, whereas Haiku gets basic stuff wrong like wrong notices and records it never moved.

There's something completely different going on with Fable and Opus in Figure 6 though. On the checkpoint score, Fable beats Opus by 0.045 and is never lower than Opus on any of the 15 queues, yet with the checkpoint score held fixed 7 of the 9 survivors can't tell Fable and Opus apart at all, and the other 2, the redundancy rate and assignment diversity, both say Opus is better than Fable. …what? So either Fable just isn't really worth the switch from Opus when it comes to MAS (lol), or none of these metrics can properly see the difference between the two strongest teams, which is the gap the metric in our paper needs to close.

9. What's still wrong with the nine

Now, the 9 metrics making it through the funnel doesn't necessarily mean that they measure collaboration. It just means they aren't obviously broken. When I looked at how each of the 9 metrics is computed, I found three problems with them. The first problem is that 6 of the 9 metrics are computed from which checkpoints passed. These are CCE, the unnecessary path ratio, offloading, the critical-path ratio, throughput, and assignment diversity. So part of what these metrics measure is the outcome again, which was already measured by the checkpoint score. If a team does all of the right work but gets one figure wrong, then that work gets counted as waste by these metrics.

I have a qualm with the remaining survivors too. The redundancy rate is calculated by taking the mean pairwise TF-IDF similarity of a team's messages, and while it has one of the biggest model effects in the study (W = 0.79), it still calls Fable more redundant than Opus! My suspicion is that TF-IDF similarity is mostly just picking up on how each model writes its messages rather than whether agents are repeating the same information. Task offloading, the critical-path ratio, and the share of output tokens spent messaging all move with how many agents got hired, so if an orchestrator just spawns more agents on a queue, it shifts those numbers without the team actually collaborating any better.

It seems to me that, for the two metrics that bugged me the most, we can fix the formula directly rather than throwing the metric out. For content redundancy, the fix is to look at what a message actually sends instead of its TF-IDF wording: a message only counts as redundant if every ID, amount, and reference in it was already sent to that recipient before. And for reference-seeded CCE, you seed the causal graph from the team's last writes to the objects that the reference solution touches, without knowing which checkpoints the team gets right or wrong:

MetricModel effect (W)Tied to outcome (ρ)Left after outcome (W)Fable vs Opus (p)Split-half τ
Redundancy rate (Scaling)0.79−0.400.820.0000.67
Content redundancy0.27−0.280.120.0121.00
CCE, any outcome0.80+0.790.380.7621.00
CCE, reference-seeded0.64+0.570.590.2521.00
Table 1. The two flawed survivors and their repairs (shaded). W is the model effect with the queue held fixed, ρ is the within-queue correlation with the checkpoint score, and "left after outcome" is W on the residuals. The last two columns are the Fable vs Opus test and how well the ranking agrees across halves of the queues.

Both of the repaired metrics end up fixing the issue I had with the original metrics in Table 1. Specifically, the model effect on content redundancy drops to W = 0.27 from W = 0.79 on the original redundancy rate, which makes sense if the original redundancy rate had to do primarily with writing style, and content redundancy now gives the exact same ranking on both halves of the queues (τ = 1.00) while putting Fable ahead of Opus (p = 0.012). In Table 1, reference-seeded CCE provides a lower correlation with the checkpoint score than the original CCE (ρ = 0.57 vs 0.79) along with a stronger model effect on the residuals (W = 0.59 vs 0.38).

So, which metrics measure collaboration?

Overall, the 9 metrics that survived all measure where the team's effort went, rather than how much the run cost in tokens or what shape the team had. I think that makes sense, because collaboration is about whether actions were on the job or off it, whether agents were on the critical path or waiting around, and whether messages carried new information or repeats. Going through all 293 metrics also gave me a five-point checklist I didn't have going in: a collaboration metric needs to be defined on every episode, vary more with the team than with the task, be computed without knowing whether the task passed, normalize for team size, and still separate teams once the outcome is held fixed.

Hardly anything in the literature hits all five of those yet, and with the outcome held fixed, 7 of the 9 survivors can't tell Fable and Opus apart while the other 2 actually rank Opus above Fable. Heck, that gap is the whole reason we're writing our paper on this topic. The benchmark and all 60 episodes are up on Hugging Face, and if there's a collaboration metric I missed, send it my way and I'll run it through the same funnel!

Every metric, every test
MetricSourceFails atModel %Queue %WρKept in resamples
Assignment diversityCoordination Matters assignment diversitysurvives7490.62+0.64100%
CCE, any outcomeAgentWorld CCE, any outcomesurvives7670.80+0.79100%
Critical-path ratiocritical path / contributing actions (MS-Eval Critical Steps)survives43140.68−0.64100%
Idle rateforced-idle rate (Coordination Matters)survives51110.46−0.53100%
Message token shareshare of output tokens spent messagingsurvives5290.52−0.5097%
Redundancy rate (Scaling)Scaling Agent Systems: redundancy ratesurvives7190.79−0.4098%
Task offloading (PARTNR)PARTNR task offloadingsurvives49140.41+0.52100%
Task throughputtask throughput, checkpoints per roundsurvives73160.84+0.83100%
Unnecessary path ratio (GEMMAS)GEMMAS Unnecessary Path Ratiosurvives53100.58−0.5499%
Equality (GovSim)GovSim equality (1 - Gini)beyond the score31170.41+0.5296%
Parallelismmean agents acting per roundbeyond the score61140.69+0.75100%
Planner cost (TeamCraft)TeamCraft planner cost functionbeyond the score13620.43−0.58100%
Staffing F1 (MS-Eval)MS-Eval staffing F1 (static)beyond the score40280.57+0.76100%
Completion efficiency (VillagerAgent)VillagerAgent efficiency of completionnot redundant76120.84+0.85100%
Extraneous effort (PARTNR)PARTNR extraneous effortnot redundant7670.80−0.79100%
PAC (AgentWorld)AgentWorld PAC (mean)not redundant70110.73+0.76100%
Parallelism gain (SwarmBench)SwarmBench parallelism gain (tokens as latency)not redundant5890.56+0.69100%
TheAgentCompany partialTheAgentCompany S_partialnot redundant47170.78+0.90100%
Action countaction countvalid47200.49+0.610%
Action-action coordination (MI)action-action IC, MI (social-influence MARL)valid39190.65−0.650%
Assistance seeking (SyncMind)SyncMind assistance-seeking ratevalid42100.37−0.170%
Attribution entropynormalized attribution entropyvalid48140.53+0.600%
Broadcast ratebroadcast share of messagesvalid18180.20−0.020%
Comm. overhead (CooperBench)CooperBench communication overheadvalid26210.24−0.2347%
Graph density (Puppeteer)Puppeteer: communication graph densityvalid24330.35−0.410%
Hierarchy depthhierarchy depthvalid52180.64+0.640%
Hub centralizationorchestrator hub centralizationvalid52100.59−0.670%
Information diversity (GEMMAS)GEMMAS Information Diversity Scorevalid64120.85+0.1718%
Message densityScaling Agent Systems: message densityvalid24190.24−0.100%
Message tokens (BANDMAS)BANDMAS communication costvalid27150.39+0.280%
Network efficiency (NetSafe)NetSafe network efficiencyvalid38220.42−0.380%
Orchestrator token share (ClawArena)ClawArena-Team main-agent costvalid22260.36−0.079%
Planning cycles (PARTNR)PARTNR planning cyclesvalid31130.31+0.580%
Redundancy (TeamCraft)TeamCraft redundancy ratevalid18110.37+0.025%
Retirementsnumber of retirementsvalid43150.64+0.440%
Task distance (DecArena)DecArena task distancevalid48120.54+0.550%
Team sizeteam sizevalid46240.61+0.620%
Balance (VillagerAgent)VillagerAgent Balancereliable28230.34+0.3781%
Bottleneck agent PAC (HiveMind)HiveMind bottleneck agent (weakest PAC)reliable18220.27+0.2859%
COMMA efficiencyCOMMA efficiency (harmonic mean)reliable8480.04+0.021%
Communication density (Silo-Bench)Silo-Bench communication densityreliable3170.09−0.110%
Critical Steps (MS-Eval)MS-Eval Critical Stepsreliable10560.24−0.074%
Grounding accuracygrounding accuracy / executabilityreliable18220.12−0.150%
Handoff events (ALEM)ALEM cooperative event countsreliable12300.32+0.2346%
Instantaneous coordination (MI)instantaneous coordination, MI (emergent-communication pitfalls)reliable9170.08−0.011%
Instruction rate (ALEM)ALEM lexical style: instructionsreliable7370.12+0.100%
KPI concentrationMultiAgentBench KPI concentrationreliable2230.03−0.230%
LLM calls (TheAgentCompany)TheAgentCompany LLM calls per taskreliable4430.02+0.120%
Message countmessage countreliable12210.11+0.300%
Mistakes (COMMA)COMMA average mistakesreliable22170.14+0.240%
MultiAgentBench KPIMultiAgentBench milestone KPI (top-3 credit)reliable59160.61+0.62100%
Optima rewardOptima effectiveness-efficiency rewardreliable3690.39+0.5798%
Parallelism rate (SwarmBench)SwarmBench parallelism ratereliable13300.16+0.3454%
Progress rate AUC (AgentBoard)AgentBoard progress rate, area under the curvereliable63170.61+0.54100%
Question rate (ALEM)ALEM lexical style: questionsreliable5780.50−0.320%
Removal attributionremoval attribution on the causal graphreliable7180.02+0.030%
Repetition rate (AgentQuest)AgentQuest repetition ratereliable25130.25+0.480%
Rounds usedrounds usedreliable10680.42−0.3785%
Score per tokenScaling Agent Systems: success per tokenreliable5360.37+0.4690%
Sink-agent tokens (MacNet)MacNet sink-agent tokensreliable2680.08−0.187%
Speaker consistency (MI)speaker consistency, MI (emergent-communication pitfalls)reliable13250.14−0.290%
Stall ratio (Magentic-One)Magentic-One stall counterreliable52100.51−0.4599%
Steps per checkpoint (WebCanvas)WebCanvas efficiency scorereliable37190.37−0.5897%
Token costtoken costreliable4630.06+0.140%
Tool error ratetool-call error rate (TRAIL execution errors)reliable18220.12+0.150%
Tool permission use (ClawArena)ClawArena-Team tool permission proportionreliable2440.02−0.083%
Under-delegation (MS-Eval)MS-Eval under-delegation ratereliable15230.17−0.1421%
Write conflictsshared-object write conflicts (CooperBench merge conflicts)reliable12160.28+0.370%
Assignment conflictsCoordination Matters assignment conflictswell-defined00––0%
CCE (AgentWorld)AgentWorld CCE (0 on failure)well-defined25240.29+0.620%
Delivery rate (TravelPlanner)TravelPlanner delivery ratewell-defined3220.04+0.170%
Parse success (ALEM)ALEM action parse successwell-defined4350.06+0.090%
Read-only violations (ClawArena)ClawArena-Team read-only compliancewell-defined00––0%
Structural collapsestructural collapse flagwell-defined00––0%

Model % and Queue % are the share of each metric's variance that comes from the model and from the queue; W is Kendall's W for the model effect with the queue held fixed; ρ is the within-queue Spearman correlation with the checkpoint score; and the last column is how often the metric passed the reliability and validity tests across 1,000 resamples of the queues. Hover over a row to see why it dropped.

Cite this post
@misc{gani2026mas,
  author = {Gani, Hassam},
  title = {I tested (basically) every Multi-Agent System metric: they're mostly flawed},
  year = {2026},
  month = {nov},
  url = {https://hassamgani.com/blog/mas-metrics-funnel}
}

← All posts