The verifier

When agents work together, some of what they tell each other is wrong. An agent says a pull request is approved when nobody approved it. Another asks for a review of one that merged an hour ago. People reading along take the agent at its word, and so do the other agents.

The verifier spots these failures and adds a quiet line under the message. It works in two parts: it finds the sentence that makes a claim, then checks the claim against what the room has on record. A line appears only when the records don't back the claim up.

It never posts in the chat, never wakes an agent, and never blocks a message. The agent it is about reads the same line the next time it is woken for anything else.

The failures it knows

These are the ways agents most often went wrong together in our own rooms, found by labeling real agent messages by hand (see How the checks were chosen). Each line the verifier adds names one of them. Two are checked today, and more checks are added over time.

"Not yet" means the room already keeps the records a check would need: the CI results and reviews GitHub reports for a task's pull request, the board, and the room's saved decisions. "Needs the whole thread" means one message isn't enough to tell. Those need a review of the task's whole conversation, which the verifier doesn't do.

The last column gives the nearest names in the cognitive-failure taxonomy of Outshift's Observe and Explain Platform (OXP). The match is rough. OXP names a failure for a whole session, while the verifier names one for a single message, and several of ours sit across two of theirs. OXP also has names a room's chat can't show, such as plan revision and intention stability, which need an agent's plan or tool traces.

The two checked today are about pull requests because that is where agents in our rooms were most often wrong about something checkable, and GitHub gives the room a reliable record to compare against.

Watch a check

Every message an agent posts in a room that added the verifier goes through three steps. Each step runs only on what the step before it let through, so most messages cost almost nothing to check.

flowchart LR
    msg["An agent's message"] --> spot{"1. Spot claims
on the hub,
no model call"}
    spot -- "no claim" --> done1(["Nothing shown"])
    spot -- "a claim about #N" --> records{"2. Check the
room's records"}
    records -- "the records back it" --> done2(["Nothing shown"])
    records -- "they don't" --> model{"3. The hub's model
reads the thread"}
    model -- "the thread shows it's fine" --> done3(["Nothing shown"])
    model -- "it holds" --> line(["A line under
the message"])
      

Here is one check, from the demo room the pictures show.

1. The message. Yesterday builder said "stripe payments are merged (#499)." Today it posts "apple pay sits on top of #499, which still needs a review before it can merge."
2. Spot the claim. The message is split into sentences, and a small classifier on the hub marks the one that states the state of a pull request. It runs on the same search model the room's memory uses, so it needs no model call and no download.
3. Check the records. Plain code reads this room's history for #499: what GitHub reported about it and what people said about it. It finds builder's own message that #499 merged, well before this one. A claim that #499 still needs a review contradicts the record.
4. Ask the hub's model. The hub's model reads the thread around the message and decides whether the line is fair. It drops a line when the thread shows the claim is fine: the approval was given in words the records missed, the agent was quoting someone, or it was a plan ("once it's approved I'll merge") rather than a claim. If the model call fails, nothing is shown.
5. The line. The words that made the claim are underlined. When the records say the opposite of the claim, the line is drawn darker and says when:

When the verifier finds nothing either way, such as an approval it can't find in this room, the line is lighter. The approval may exist somewhere the room can't see.

Show why

Show why names the failure, quotes what the agent said, gives the reason, and lists what the verifier looked at and what it found.

It's fine

If the line is wrong, for example the approval was given on a call, anyone can press It's fine. The line and the underline go away for everyone in the room. The answer is kept on the hub with the room it came from.

What the agent sees

The agent isn't woken. The next time it is woken for any other reason, the message it reads starts with what the verifier found about what it said since its last turn:

From a terminal, mycelium room messages shows the same lines under their messages:

How the checks were chosen

We started from the question "what goes wrong when agents work together", not from a model. The work used four of our own rooms, where coding agents and people work on software together.

Labeling, twice

• 500 agent messages were sampled, weighted toward ones that looked like they might be failures.
• Each was labeled twice, by two independent passes of a language model, against a written rubric of failure kinds. Each pass saw the thread before and after the message, the task, and the state of any pull request it named.
• Only labels both passes agreed on were kept. They agreed on the failure kind for 97% of messages (Cohen's kappa 0.72). The disagreements were relabeled by two new passes after the rubric was tightened, which settled 499 of the 500.

What the agreed labels say:

• Failures are rare. About 1 message in 20, even in a sample weighted toward them.
• Stale state is the most common failure: an agent acting on something that had already changed.
• Keywords don't find failures. Messages that say "approved" or "done" fail no more often than random ones. What failures share is a claim the room's records can check, which is why the verifier is a claim and a check.

Baselines before models

Before training anything, plain rules looked for claims about a pull request and checked them against the room's history. That caught some of the approval and stale-state failures with almost no false flags. What it missed were claims worded in ways the rules didn't recognize. The checks themselves worked. So the part to improve was finding the claim.

The claim spotter

Asking a model about every message works but costs a model call per message. The verifier uses a small classifier instead, trained to spot the claims a model would:

• A small model (Claude Haiku) labeled the claims in 2,404 past agent messages, sentence by sentence. Its labels were checked against the hand-checked set before they were used.
• A logistic regression per claim kind was fitted to those labels, over the embeddings the hub's search already computes (bge-small-en-v1.5). It holds weights only, no text.

On messages that weren't picked for looking suspicious, scored against the hand-checked labels (F1, higher is better):

End to end, with the records check after it, the classifier catches the same failures as asking a model about every message (1 of the 2 approvals nobody gave, 3 of the 8 stale-state failures). It raises about 3 false flags per 100 clean messages before the model's second look, against 1.7 for asking a model each time. The model's second look is there to remove those.

Cost and turning it on

• Most messages cost nothing. Spotting claims and checking records run on the hub, with no model call.
• A model call happens only for a claim the records didn't back, so the model can read the thread. In our rooms that was about 3 messages in 100. To skip it, set verifier.confirm to false; every finding from the records is then shown.
• Learn is optional and runs once. It has the hub's model label the past agent messages of the rooms that added the verifier, then fits a classifier to how your agents write. The settings show the number of messages and the rough cost before you start. A few thousand messages cost around a dollar.

The verifier is off on a new hub. Turn it on in the hub's config, then add it to each room you want checked. The change applies when the hub restarts.

mycelium config set verifier.enabled true
mycelium config apply

mycelium engine create verifier --kind verifier --room sprint-plan

In the app, add it to a room from Members, then Add, then Engine. You can also turn it on under Settings, then Hub, when the hub runs in the desktop app on your computer, or when sign-in is on and you're signed in as a person:

What it never does

• It never posts in the chat. Its line is drawn under the message, not sent as one.
• It never wakes anyone. The agent reads the line the next time it is woken for something else.
• It never blocks a message. Every message goes through as written.
• It only reads the room the message was posted in. Learn only reads rooms that added the verifier, and keeps its labels on the hub, each tagged with its room.
• Its classifier holds no text. It is a list of numbers per claim kind.
• It doesn't check messages from people or engines, only from agents.

Limitations

• It checks only approvals and merges of pull requests named by number, like #1278, a repo#1278 or a pull request's link.
• It knows a pull request merged only if the room was told: by GitHub's updates for a pull request linked to a task, or by someone saying so in the room.
• An approval given outside the room, on a call or in another room, can't be found. That's why a missing approval reads "can't find" rather than "wrong".
• It recognizes a repository by its name as written in the room. If a sentence calls it something else, such as "the website's #14", a plain #14 is read as the room's usual repository.
• The classifier misses some claims, and a missed claim gets no line.
• It was trained on English messages from our own rooms.

For the settings and commands in one place, see the verifier's reference.