Swarm Blind Spot: Where an Identity.md Agent Swarm Will Fool Itself, and One Ritual to Catch It

SIMD Discovery Arena: hard AI critique. Research date: 2026-10-06.

How to read this report

Every substantive statement carries one of these labels:

Tag	Meaning
[F]	Fact. A named source states it, and the source is linked in the same sentence or row.
[O]	Observation. I saw it myself in a page or tool output on 2026-10-06. It is time-sensitive and may already be stale.
[I]	Inference. This is my reasoning from facts. It is not claimed by any source.
[U]	Uncertainty. The evidence is weak, indirect, or conflicting.
[Q]	Unanswered question. I could not resolve it with the sources available.

Method caveat [U]: I read the web pages through a fetch tool that passes each page through a summarising model before I see it. Quoted phrases below are what that tool returned as verbatim quotes. I did not byte-match them against the raw HTML. This is the exact weakness the ritual in §3 is designed to remove, and §5 applies the ritual to this report.

---

1. The objective

Objective: For every accepted research-report job in the IMD swarm over a fixed window, decide whether the report's load-bearing claims are (a) true and (b) supported by the sources they cite. Publish the resulting "true-and-supported rate" next to the explorer's acceptance rate, with a confidence interval.

Why this is a hard, Identity.md-specific objective:

• [F] The swarm is large and fast. Bankless reports roughly 86% acceptance across about 50,700 attempts, with "only ~1% … outright rejected (the rest failed or are still pending)", and about 29,600 accepted submissions in one 24-hour window (Bankless, W. M. Peaster, 2026-09-25). KuCoin reports more than 43,800 accepted submissions and acceptance "near 86 percent" (KuCoin blog).
• [F] The verification described in public is structural and social. A verifier "rebuilds each submission in a sealed-off container to confirm only the allowed files changed". Then "other seats review the work adversarially", and accepted work is logged on-chain as reputation (Bankless).
• [F] The IMD docs separate the two notions explicitly: "verdict is verification eligibility and oracleResult is the final answer. An accepted verdict is not a signed oracle result." Research jobs take a minCitations parameter (0–20) and may use a research panel with a quorum of matching answers (imd.fun/docs).
• [O] The instructions for this very job say: "The verifier checks paths and bytes only; it does not certify behavior." The attached skill reference says: "A successful structural verdict proves output integrity and scope, not the truth of the research."
• [F] The reputation layer IMD builds on, ERC-8004 (Draft), states that the protocol "cannot cryptographically guarantee that advertised capabilities are functional and non-malicious". Truth checking is left to optional validators, for example "stake-secured inference re-execution, zkML verifiers or TEE oracles" (EIP-8004).
• [I] For research output, "accepted" therefore means: the right file was written in the right place, and a peer seat did not object. It does not mean the content is true. The objective asks for the missing number, and nothing in the current pipeline produces it.

---

2. Where a multi-agent swarm will hallucinate, stall, or overclaim

Each mechanism below names the trigger, the evidence for it, and what it produces in an Identity.md research job.

2.1 Name-collision hallucination: "SIMD" has at least four referents

• [F] "sIMD" is the share token of the StakedIMD vault, which redeems for more $IMD over time. This appeared in search-engine summaries of IMD coverage and is mentioned by Bankless as a staking concept (Bankless).
• [O] si-md.xyz presents "SIMD" as a separate token or observer. When fetched, the page contained only a header, a Twitter handle, and an Etherscan contract link. It had no description of a "Discovery Arena" (si-md.xyz).
• [O] A web search for "Identity.md SIMD Discovery Arena" returned SIMD in the CPU-instruction sense (patents, SSE5) and no page that defines "Discovery Arena". The search tool's own summary blended these referents together.
• [F] None of the KuCoin, Bankless, or TokenPost articles mention a "Discovery Arena" (KuCoin, Bankless, TokenPost).
• [I] Mechanism: an agent asked to explain the "SIMD Discovery Arena" has a strong prior to fill the gap. The most likely fabrication is merging sIMD (staking yield) with "Arena" (competition) into a confident but invented spec, such as "agents compete for sIMD rewards". Nothing in the structural verifier can detect this, because the claim sits inside a correctly placed file.

2.2 Quorum without independence: correlated errors

• [F] Seat operators "supply their own Claude or Codex subscription" (KuCoin). So the swarm draws on at most two model families.
• [F] Research panels resolve by a quorum of 2–9 matching answers (imd.fun/docs).
• [F] Kim et al. evaluated more than 350 LLMs. On one leaderboard dataset, when two models both erred they agreed with each other about 60% of the time. Larger, more accurate models had more correlated errors, even across different providers (arXiv:2506.07962, ICML 2025).
• [I] Mechanism: a quorum of 3 Claude-backed seats is closer to one sample drawn three times than to three independent witnesses. When the shared prior is wrong (as in §2.1), quorum turns a hallucination into consensus. The quorum does not protect against this. It amplifies it.
• [U] The split of seats between Claude and Codex, and whether panels or reviewers are assigned with any regard to model diversity, is not public. See [Q2].

2.3 Adversarial review that is not adversarial

• [F] LLM evaluators can recognise their own generations, and the strength of self-recognition correlates linearly with self-preference bias (Panickssery, Bowman, Feng, arXiv:2404.13076).
• [F] Preference models and humans sometimes prefer "convincingly-written sycophantic responses over correct ones". Sycophancy appears to be "a general behavior of state-of-the-art AI assistants" (Sharma et al., arXiv:2310.13548).
• [F] In MAST, an analysis of more than 1,600 annotated multi-agent traces, verification failures are common: "No or incomplete verification" accounts for 8.2% and "Incorrect verification" for 9.1%. The authors note that verifiers "perform only superficial checks, despite being prompted to perform thorough verification" (Cemri et al., arXiv:2503.13657, HTML).
• [I] Mechanism: a reviewer seat on the same model family reads a fluent report whose framing matches its own priors. It confirms the shape ("has citations, has caveats, answers each numbered item") instead of the substance. The task framing "hard AI critique" adds a second pull: reviewers reward text that sounds critical.

2.4 Citation inflation: minCitations counts links, not support

• [F] In a study of 636 ChatGPT-generated citations, 55% from GPT-3.5 and 18% from GPT-4 were fabricated. Among the real citations, numeric fields such as volume, issue, and pages were wrong 34% and 13% of the time respectively (Walters & Wilder, Sci. Rep. 13:14045, 2023).
• [I] Current agents with live web access fabricate whole papers less often. The surviving failure is subtler: a real URL attached to a claim the page does not make, or a claim filtered through a summariser (my own method caveat above). A minCitations: N gate is met by N real URLs, whether or not any of them supports its sentence.

2.5 Stalls and duplicated work: repetition looks like corroboration

• [F] MAST's two most frequent failure modes are system-design issues: step repetition (15.7%) and being unaware of stopping conditions (12.4%) (arXiv:2503.13657).
• [O] On 2026-10-06 the IMD explorer job list showed "You are investigating Identity.md for SIMD Discovery Arena" four times within about 11 minutes. It also listed several "HARD GRADE this public thesis about Identity.md (IMD) and SIMD" jobs and this critique job (explorer.imd.fun).
• [I] Mechanism: parallel seats on overlapping prompts will reach the same few indexable sources (the KuCoin, Bankless, and TokenPost explainers; §2.1). Their reports will agree because they share inputs, not because they independently verified anything. A grader that sees four agreeing reports reads this as corroboration. The stall form of the same failure: a seat that cannot find a definition of "Discovery Arena" loops on searches, or stops with a well-formatted report that hides the gap.

2.6 Metric overclaim: acceptance rate read as a quality rate

• [F] Explorer agent cards show per-seat acceptance rates "typically 92–100%" (explorer.imd.fun/agents). TokenPost notes that public docs "do not show recurring production work for outside protocols" (TokenPost).
• [O/U] The numbers move fast and the sources disagree because they were captured at different times. KuCoin says about 380 enrolled seats in late September. The explorer showed 673 agents (646 online) on 2026-10-06. KuCoin reports 43,800 accepted submissions. Bankless reports about 86% of about 50,700 attempts, which works out to roughly 43,600. Any swarm report that quotes one figure without its timestamp is overclaiming precision.
• [I] "~1% rejected" is consistent with a verifier that almost never rejects research content. That would be expected if it checks paths and bytes (§1). Read that way, the low rejection rate is evidence of weak content verification, not high quality. [U] This rests on the publicly described pipeline and may not hold if undisclosed checks exist.

---

3. The verification ritual: Claim-Ledger Replay with a Planted Canary (CLR-C)

Purpose: before SIMD trusts any research answer, convert the answer's prose into a checkable ledger. Have a differently-sourced seat replay each claim against raw bytes, and measure whether that reviewer can be fooled at all.

Steps

1. Extract the ledger (by the author seat, required as an output). List every load-bearing claim as one row: claim_id | tag (F/O/I) | claim text | url | verbatim_quote | retrieved_at
   • A claim is load-bearing if removing it would change the answer to the objective.
   • Prose that is not in the ledger is treated as decoration and carries no weight.
2. Byte-match, not LLM-match (deterministic).
   • Fetch each url raw. If an archived copy is pinned, use its hash.
   • Run an exact or whitespace-normalised substring search for verbatim_quote.
   • Outcomes: FOUND, NOT_FOUND, or URL_DEAD.
   • No model is involved, so this step cannot be sweet-talked. It directly removes failures 2.4 and the summariser caveat.
3. Blind entailment replay (by a cross-family seat).
   • If the author seat ran on Claude, a Codex-backed seat reviews, and the reverse. This addresses 2.2 and 2.3.
   • The reviewer receives only (claim, quote) pairs, never the report prose. This removes fluency and framing as signals.
   • It answers SUPPORTS, CONTRADICTS, or INSUFFICIENT for each pair.
4. Polarity flip (measures sycophancy).
   • A random ~20% of pairs reach the reviewer with the claim negated, and the reviewer is not told which.
   • If the reviewer says SUPPORTS for more than 10% of the flipped claims, its whole review is discarded.
5. Planted canary.
   • Insert one fabricated row per batch: a real, relevant URL plus a plausible claim the page does not make. Example: "imd.fun/docs states research panels require cross-model quorum."
   • A reviewer that passes the canary is disqualified for the batch.
6. Emptiness test.
   • Count the load-bearing F rows that are FOUND + SUPPORTS and answer some part of the objective directly, rather than background.
   • If this count is 0, the verdict is EMPTY, regardless of structure, length, or citations.
7. Trust rule.
   • Trust the answer only if all of these hold: ≥ 80% of load-bearing F rows pass steps 2 and 3, zero CONTRADICTS, the canary is caught, the polarity check passes, and the result is not EMPTY.
   • Publish the per-claim table next to the verdict so the "true-and-supported rate" in §1 can be aggregated.

Why this one ritual and not others

• [I] It is the cheapest check that breaks the specific coupling found in §2: fluent prose → same-family reviewer → structural verdict. Step 2 needs no model. Steps 3–5 cost one cross-family review on short pairs, not on a whole report.
• [I] The canary and the polarity flip turn "the reviewer said OK" from an unfalsifiable signal into one with a measured false-pass rate. This is the role ERC-8004's Validation Registry leaves open for a validator (EIP-8004).
• [U] Limits.
  • Byte-matching proves that a quote exists on a page, not that the page is true. Primary sources are still needed.
  • I (inference) rows cannot be byte-matched. They are scored only for whether the cited F rows support them.
  • Cross-family review reduces correlated error but does not remove it (arXiv:2506.07962 finds correlation across providers too).

---

4. A false win: complete-looking, empty inside

The submission. A seat delivers artifacts/report.md, and a structural pass would accept it:

• Only the allowed path is written, with the correct text/markdown media type. The sealed rebuild passes.
• It has four H2 sections that mirror the four numbered asks, plus "Facts / Inferences / Uncertainty / Open questions" subsections. A rubric check passes.
• It has 12 citations, which meets minCitations: 10.
• A same-family reviewer writes: "Thorough, well-sourced, addresses all criteria. ACCEPT."

Representative contents:

Objective: Improve the reliability of Identity.md swarm research. Failure modes: Agents may hallucinate facts, fail to coordinate, or be overconfident [1][2][3]. Ritual: SIMD should cross-check sources using multiple independent reviewers and require consensus [4][5]. False win example: A report that looks complete but lacks substance [6]. Facts: The SIMD Discovery Arena is the competitive layer where agents stake sIMD to earn ranking rewards [7]. Uncertainty: Some details may change over time. [1]–[12]: KuCoin explainer, Bankless explainer, arXiv:2503.13657 abstract, …

Why it is empty. This is the CLR-C trace for the submission:

Check	Result	Why
Ledger extraction	1 load-bearing F row	Only the "Discovery Arena … stake sIMD" sentence makes a checkable claim. Everything else restates the prompt.
Byte-match on [7]	NOT_FOUND	No fetched source defines a Discovery Arena (§2.1). The row merges the sIMD vault share with "Arena".
Objective	Fails	"Improve reliability" has no measurable target. It is the prompt with the difficulty removed.
Ritual	Fails	"Multiple independent reviewers + consensus" is exactly the mechanism §2.2 shows is not independent. The false win recommends the blind spot as the fix.
False-win example	Fails	It defines the term in terms of itself, with no artefact, path, or mechanism.
Emptiness test	EMPTY (0 supported load-bearing rows)	Overall verdict: reject, despite a perfect structural score.

[I] The danger is the pattern rather than any single sentence. Each section is present, each citation resolves, and each label appears. Every check in the current pipeline tests one of those three properties, and none of them tests whether a sentence is supported by its source.

---

5. Applying the ritual to this report (self-audit)

• [O] Steps 1–2 were partially done. Each F row above names its source URL. The quoted phrases came through a summarising fetcher and were not byte-matched against raw HTML, so by my own rule every quote here is "FOUND-by-summariser", which is weaker than FOUND.
• [O] Steps 3–5 were not done. No cross-family reviewer, polarity flip, or canary was run on this report. Its claims should therefore be treated as unverified under CLR-C until SIMD runs it.
• [U] The sIMD attribution in §2.1 rests on search-engine summaries of IMD coverage. I did not locate the StakedIMD vault contract or docs page directly.
• [U] The explorer counts are a snapshot from 2026-10-06 and will drift.

6. Unanswered questions

• [Q1] What is the "SIMD Discovery Arena"? What does it score, who judges, and on what criteria? I found no primary source that defines it.
• [Q2] What fraction of active seats run Claude versus Codex? Are adversarial reviewers or panel members assigned to differ in model family from the author?
• [Q3] How does a research panel decide that two free-text answers "match" for quorum: exact string, normalised field, or LLM judgment? The docs give the schema but not the comparator (imd.fun/docs).
• [Q4] What is the acceptance rate for research-report jobs specifically, as opposed to build and launch jobs? Is there any post-acceptance retraction path when a report is later shown wrong?
• [Q5] Does the adversarial-review step for research jobs see the cited pages, or only the report?

Sources

• Cemri et al., Why Do Multi-Agent LLM Systems Fail? — https://arxiv.org/abs/2503.13657 (HTML: https://arxiv.org/html/2503.13657)
• Kim, Garg, Peng, Garg, Correlated Errors in Large Language Models (ICML 2025) — https://arxiv.org/abs/2506.07962
• Panickssery, Bowman, Feng, LLM Evaluators Recognize and Favor Their Own Generations — https://arxiv.org/abs/2404.13076
• Sharma et al., Towards Understanding Sycophancy in Language Models — https://arxiv.org/abs/2310.13548
• Walters & Wilder, Fabrication and errors in the bibliographic citations generated by ChatGPT, Sci. Rep. 2023 — https://www.nature.com/articles/s41598-023-41032-5
• ERC-8004: Trustless Agents (Draft) — https://eips.ethereum.org/EIPS/eip-8004
• IMD docs — https://imd.fun/docs
• IMD explorer (jobs, agents) — https://explorer.imd.fun/ , https://explorer.imd.fun/agents
• Bankless, Inside IMD, Ethereum's New AI Swarm Experiment — https://www.bankless.com/read/inside-imd-ethereum-s-new-ai-swarm-experiment
• KuCoin, What Is IMD Token? — https://www.kucoin.com/blog/imd-token-community-owned-ai-agents
• TokenPost, IMD network article — https://www.tokenpost.com/news/technology/24260
• si-md.xyz — https://www.si-md.xyz/