Review Findings Answered

The check Automatic review answered is RED until the automatic review has landed on a pull request and every inline thread that review opened has a reply from a person. It never skips, it never passes on no evidence, and a pull request the reviewer could not review is released only by a maintainer's visible waiver — or, when the internal reviewer itself is down, by the reviewer-unavailable degradation its own App records on the head, which defers the review rather than skipping it. It is the build of option B on #4299, decided by the maintainer on 2026-09-17.

• Workflow: .github/workflows/review-answered.yml
• Predicate and self-test: .github/scripts/check-review-answered.py
• Status: 🚨 REQUIRED on main since 2026-09-17 — measured that evening on ruleset 2128472, which now lists Automatic review answered beside Consolidate test results. The Rollout section below is kept as the record of how it got there; it is no longer the current state.

🚨 What being required FEELS like, because it is not obvious from the outside. An unanswered thread leaves the pull request mergeable_state: blocked with every build green — and blocked is the same word REST returns for a pull request merely waiting its turn in the queue. Three pull requests sat green, armed and silently OUTSIDE the merge queue for ~45 minutes on the evening this landed, and nothing in the REST view distinguished that from progress. If a green, armed pull request is not merging, read this check before anything else, and read the merge queue ITSELF rather than mergeable_state — the queue is one of the two things REST cannot express:

gh api graphql -f query='{repository(owner:"Systemorph",name:"MeshWeaver"){
  mergeQueue(branch:"main"){entries(first:20){totalCount nodes{position state
  pullRequest{number}}}}}}'

A totalCount that does not contain your pull request, while other pull requests merge through it, is the reading that separates "held" from "waiting".

🚨 It is answered by replying ON the thread, and nothing else does it:

gh api "repos/Systemorph/MeshWeaver/pulls/<n>/comments/<comment-id>/replies" -f body='…'

🚨 Quote the path. Unquoted, the shell reads <n> as a redirection and the command fails before gh runs — which looks like a broken instruction rather than a quoting mistake.

🚨 Answering every thread used to be NECESSARY, NOT SUFFICIENT — the check now finishes the job itself. Branch protection reads the check's pull_request run, not the pull_request_review_comment run your reply starts, so the verdict it read stayed an older failure on the same head while the newest run said GREEN, until somebody re-ran the pull_request run by hand. The lane's refresh job now does that re-run on its own after every GREEN verdict taken on a different event. The measurement, the self-refresh, and the manual fallback for the cases it cannot cover (a fork, or a red refresh job) are below, under "The check's log says GREEN and the pull request is still BLOCKED".

Why: a review was advisory

The ruleset main pr protection (2128472) carries a copilot_code_review rule, so every pull request into main is reviewed. The control plane (PrArming) arms a pull request once its review lane is satisfied; auto-arm.yml only DISARMS on a push. Nothing joined the two, so a merge waited for the required checks and never for the review — and on the pull requests whose checks are fast, the checks usually won.

The instances on the filing (#4269, #4291, 2026-09-14) were admitted before the reviewer had finished; condition 1 below holds those. In every miss measured after it — #4310, #4343, #4347, #4366, #4371, #4383 — the review had landed; it had not been read. A check that waits only for the review to complete (option A) would have held none of those, which is why option B asks whether each finding was answered.

The rate at the time this page was written, replaying the predicate over 60 merged pull requests (2026-09-16T08:48Z → 09-17T07:41Z), each as of its own merged_at: 32 of 60 would have been red at merge, all of them for unanswered threads; 126 findings, 86 of them unanswered at merge. None was red for a missing review. That is the cost of making the check required, stated before anyone decides to.

The rule

All three hold, or the check is RED:

1. The automatic review has landed — a review by the reviewer account, at a non-PENDING state, whose body is not a refusal. What makes it a review is who posted it, not how it is worded — see "Provenance, not presentation" below for why, and what requiring a recognisable shape cost. Two things release this condition — and only this one — without a review: a maintainer's waiver, and the internal reviewer's own reviewer-unavailable degradation.
2. Every thread the reviewer started has a person's reply — for every comment by the reviewer with no in_reply_to_id, at least one comment in that thread by an account of type: User (following in_reply_to_id to the root, so a reply to a reply counts).
3. The inputs were read completely — every read succeeded, and the comment listing is not shorter than the review_comments count the pull request reported before the listing began. Measured on 60 of 60 recent pull requests: listing and count agree.

A reply that says nothing still counts as answered. That limitation is known and accepted on #4299; the predicate is still strictly better than none.

The reviewer, as measured

Measured 2026-09-17 over 50 merged pull requests (#4487–#4568) through the REST endpoints the check uses:

Historical — superseded by the next section. Under the retired policy internal-code-review two reviewers were accepted: Copilot, and the internal GLM-5.3 reviewer of MeshWeaver.Plugins' PR steward, which posts through the systemorph-com App (COMMENTED or CHANGES_REQUESTED, never APPROVED). That policy planned for Copilot to leave the set; the next section reverses it.

Copilot is the reviewer again

Policy copilot-code-review reverses internal-code-review. GitHub Copilot reviews every pull request again; the internal reviewer is no longer required and is being switched off.

• The request. Every repository's ruleset carries copilot_code_review. Under this policy alone it carried review_on_push: true (review_draft_pull_requests: false), so Copilot reviewed every pushed head; One review per pull request below turns that back to one review.
• What branch protection requires. internal-review is not a required context in any repository. Core still requires Automatic review answered.
• This merge gate. It takes a landed Copilot review, as it always did. Every thread Copilot or the internal reviewer opened still needs a person's reply.
• The stage gate and the arm gate. They accept a landed Copilot review of the pull request: not PENDING, and a body that reads as a review, never a refusal (copilot_review_of_pull_request in check-review-answered.py — the head's own review first, otherwise one of an earlier head; see below). They check this before looking for an internal-review run, so no head waits out the stage gate's fallback for a reviewer that has been switched off.
• What does not count. A refusal on any head. The internal bot's review alone, because its verdict on a head is its internal-review check run. The self-test pins every one of these cases, and replacing copilot_review_on with one that finds nothing turns the acceptance cases red.
• One timing difference. Copilot's own pull_request_review event starts no workflow run (see The reviewer's own event cannot start a run here below). So a Copilot review releases a held stage gate on the next stage-advance.yml sweep, at most about 15 minutes later, rather than on the event itself.

One review per pull request

Policy review-once-per-pull-request narrows copilot-code-review: a pull request is reviewed ONCE, not once per push. With review_on_push: true every push bought a full new review round, and every round's new threads held the merge again until a person answered them — so a pull request that needed three fix pushes paid for four reviews and four rounds of replies.

• The gates. All three — Automatic review answered (evaluate), the stage gate (stage_readiness) and the arm gate (arm_readiness) — count a landed Copilot review against ANY head of the pull request as "reviewed". copilot_review_of_pull_request returns the newest landed review against the current head when there is one, otherwise the newest landed review against an earlier head, and the verdict's note names which head it was. pulls/{n}/reviews lists only this pull request's reviews, so every review it returns reviewed this pull request. The merge gate's condition 1 never looked at the head; the stage and arm gates did, and now do not.
• What is unchanged. Every thread a reviewer opened still needs a person's reply, whichever head it was opened on. A refusal is still not a review on any head. No review at all is still red.
• The ruleset. The copilot_code_review rule runs with review_on_push: false; review_draft_pull_requests keeps its value. .github/scripts/set-copilot-review-once.py applies that to the nine repositories idempotently (dry-run by default; --apply PUTs the ruleset with every other field as read, then reads it back and fails unless only that one parameter changed).
• The order. The gate change merges FIRST, the ruleset flip comes second. Flipped first, every new push would carry no review of its head while the old stage and arm gates still demanded one.
• Drafts. With review_draft_pull_requests: false a draft is not reviewed while it is a draft; the one review must then come when it is marked ready. Whether review_on_push: false still requests that review on ready-for-review was not measured when the policy was set. Check it on the first pull request opened as a draft after the flip. If no review arrives, the stage gate's fallback and the review-waived label are the existing exits, and review_draft_pull_requests: true is the remedy — the one review then lands while the pull request is still a draft.
• The self-test. The ONCE cases pin it: an older-head review with every thread answered is green on all three gates, an older-head review with an unanswered thread is red, an older-head refusal is red, and no review at all is red. With copilot_review_of_pull_request cut back to the head alone, every stage- and arm-gate ONCE case that reaches the thread condition turns red.

One account, two logins. A predicate keyed on either login alone sees half of the reviewer. The check matches on the account id or either login, and only for type: Bot, so a person who names their account Copilot is not the reviewer.

Exactly one review per pull request on all 50, always COMMENTED, usually on the first commit rather than the head. The ruleset carries review_on_push: false and review_draft_pull_requests: true: the reviewer reviews once, on open, drafts included, and never again unless someone asks.

A refusal is posted as a review. On #645–#654 (2026-07-25/26) the reviewer answered every pull request with a review, from the same account, whose whole body was:

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

So "a review by the reviewer exists" reads a quota outage as "reviewed". The check therefore classifies the body:

A refusal wins. Everything else the reviewer says is the review.

Provenance, not presentation — and what the other way cost

The body is read for one purpose: to separate a review from a refusal to review. What makes a review a review is that the reviewer account posted it at a non-PENDING state. That is is_reviewer plus the state check, and the body has no part in it.

Until 2026-09-18 landed additionally required the literal string Pull request overview, and this page said so, adding: "If the reviewer changes its format, every pull request reads red naming the new first line, and the fix is one marker in the script." The failure was therefore foreseen and accepted — sound reasoning for an advisory check, where a false red costs a reader a glance.

Two things then happened within a day of each other. The check became a required context (ruleset 2128472, 2026-09-17T21:15Z), and the reviewer dropped its overview block, posting a short verdict body instead — ### 🟢 Approval recommended, ### 🟡 Changes recommended, ### 🔵 Needs a closer look, plus a sentence or two. The marker matched nothing. Six core pull requests were genuinely reviewed and every one read "the automatic review has not landed".

Promotion changed the cost of "cannot-tell is never a pass" from a glance to a hard stop on every open pull request, with no exit but a maintainer waiver — and answering the findings could not clear it, because the unanswered-threads check is a second reason and the review-landed reason stood regardless. A whole lane was blocked while the reviewer had in fact reviewed everything.

So the marker was removed rather than updated. Chasing the format would have re-armed the same trap on the reviewer's next revision. A decorative substring is the reviewer's choice and can change without notice; the account id cannot.

🚨 The self-test could not have caught it. All three review fixtures carried the marker, so the suite proved the rule only on the side of the change where it held — a control with no case on the other side. The fixtures now include the three bodies measured on 2026-09-18, and each is verified to go red if the marker rule is restored.

When it is evaluated

There is no job-level if:, no path or branch filter and no continue-on-error: a skipped required context counts as satisfied, so every event evaluates the predicate in full. The script's self-test runs before the verdict in the workflow, and again in dotnet-test.yml's CI-shell lane so a pull request that breaks the predicate goes red on itself.

Evaluations are serialised per pull request for pull-request events and per queue entry for merge_group — the queue head ref carries the entry's sha, so a re-queue of the same pull request may evaluate alongside an older entry. That is the right granularity: what must not race is two evaluations publishing the same check-run name on the same commit, and two queue entries of one pull request are two commits with two check-runs. Nothing is cancelled mid-read, which also matters because cancelling a merge_group run ejects the entry rather than retrying it. A review arrives as one submitted event and one created event per inline comment; GitHub keeps one running and one pending run per group, so a burst of fourteen collapses to two evaluations, and the one that runs last read last.

🚨 The reviewer's own event cannot start a run here — so the check waits, briefly

Measured on this check's own pull request, #4575 (2026-09-17T08:05Z): the reviewer's pull_request_review event does reach the repository and GitHub does create a workflow run for it — run 35197843933, conclusion action_required, zero jobs. The triggering actor is Copilot, and the repository requires approval for runs triggered by a first-time contributor (actions/permissions/fork-pr-contributor-approval → first_time_contributors). The two pull_request_review_comment runs the review's own comments raised were action_required too.

So the event that would say "the review has landed" cannot evaluate anything, and a pull request whose review raises no findings would keep the red from its opened evaluation until somebody pushed again — a permanent red over a review that did land.

The check closes that itself: --wait-for-review 15. While the only thing missing is the review, the step re-reads every 30 seconds for up to fifteen minutes and then answers RED (the job's cap is 20 minutes). Nothing else is ever waited for — an unanswered thread needs a person, an incomplete listing needs another read. On #4575 the review landed 4m38s after the pull request opened; the range on #4299's thread is 3–12 minutes.

A person's reply is not gated: it re-runs the check within seconds, which is what turns a red into a green once the findings are answered.

The cheaper mechanism is a repository setting, not a wait: if runs triggered by Copilot no longer need approval, the review's own event evaluates the check the moment it lands and the wait becomes dead weight. That is a maintainer decision about the Actions approval policy, and it is the first item in Rollout.

Every read is REST — pulls/{n}, pulls/{n}/reviews, pulls/{n}/comments, issues/{n}/events, collaborators/{login}/permission — with the job's read-only GITHUB_TOKEN. The collaborator read is taken on every run for the pull request's author (when that is a person), so a token that cannot verify a waiver is discovered on an ordinary pull request rather than when a maintainer needs the waiver.

🚨 Three ways to be red, and only one of them is something you can do anything about

The check has one red and three causes, and until 2026-09-18 it printed one sentence for all three — "the automatic review must land. It usually arrives minutes after the pull request opens…". That sentence is correct for exactly one of them.

The middle row is the one that cost something (#4730). On 2026-09-18 the reviewer refused for quota from 11:39Z, and six pull requests — every one green on Consolidate test results, every one with auto-merge armed — sat blocked for over four hours reading "it usually arrives minutes after the pull request opens". Nothing was arriving. There were no findings to answer, and no push, re-run or new commit could change the answer, because the refusal is about the reviewer and not about the pull request.

So a refusal now names itself in all three places a reader looks — the run's headline (RED — UNREVIEWABLE (the reviewer REFUSED to review this pull request)), the step summary's heading, and the To go green line, which for a refusal says explicitly that pushing, re-running and replying all leave it exactly where it is.

🚨 And the run no longer WAITS for it. waiting_would_help existed for a real case — the reviewer's own event cannot start an evaluation here, so a review that raises no findings needs a bounded wait to be seen at all — but it asked "does the reason contain has not landed?", and the refusal reason is spelled "the automatic review has not landed — the reviewer posted, but not a review: …". So --wait-for-review 15 slept a quarter of an hour printing "waiting for the automatic review" at a reviewer that had already answered: it said no. Nothing arrives in that window by construction, the run then contradicts its own summary, and it spends a runner doing it. That is the same defect as the guidance line, one layer down, and it is why the discriminator has to be the field rather than the prose.

🚨 The VERDICT did not change and is not meant to. A refusal was red before and is red now; the check still fails in the safe direction, and the remedy is still a maintainer's. refused is a field on the verdict rather than a substring of the reason text, for the reason this page's own header gives about presentation-keyed reading — and the self-test asserts what the reader is told (says / never_says, over the guidance, the summary and the run headline), not only what the verdict is, because that is precisely the half that was wrong while the verdict was right. Each half has a negative control: remove the wait's not verdict.refused and the refusal-only wait case goes red; delete the headline and the refusal case goes red.

Decided since for the INTERNAL reviewer, still open for Copilot: what a structurally unavailable reviewer does to the merge gate. For the internal reviewer the answer is the reviewer-unavailable degradation (MeshWeaver.Feedback#86): its App records that it could not review, the gate releases condition 1, and a post-merge review is owed. A Copilot quota refusal is still held as described above — Copilot has no App of ours to record anything with, and it is being retired.

The waiver

A pull request the reviewer cannot review — a quota refusal, an outage, a change with no reviewable files — is released by exactly one thing: the label review-waived, applied by a maintainer.

• The label's presence is read from the pull request; who applied it is read from the REST issue events (the latest labeled/unlabeled event for that label).
• It is honoured only when that account is a person and its role_name on the repository is admin or maintain. A writer's label, a bot's label, or a label with no attributable event is refused, and the refusal names the account.
• It releases condition 1 only. Every thread the reviewer did open still needs a reply, waiver or not.
• It is never automatic, and the log names who waived and when.

The first remedy for a refusal is not the waiver: a maintainer re-requests the review, and a real review replaces the refusal.

🚨 An agent never applies the waiver. Agent sessions here run under the maintainer's own GitHub account, and the check reads an account's role, not who was at the keyboard — it cannot tell a maintainer's waiver from an agent's. The label event in the pull request's timeline is the audit.

The reviewer-unavailable degradation — the exit that needs nobody

Why it exists (MeshWeaver.Feedback#86). With the waiver as the only exit, the gate was circular: when the internal reviewer itself stops completing rounds — on 2026-09-29 they aborted at the 30-minute cap (MeshWeaver.Plugins#2564, #2565, #2568) — every pull request is held, including the one that repairs the reviewer, until a person is available to waive. Whether a change can merge must not depend on a person's availability, and a defect in the review step must not be able to block its own fix.

The rule. Condition 1 is also released when the pull request's head commit carries a check run that satisfies ALL of:

The newest completed internal-review run from that App on the head decides, so a later real round supersedes an earlier degradation (and runs from other Apps are never looked at). --as-of ignores a run completed after the instant; the merge-queue path reads the pull request's head, not the queue's merge commit. The check-run listing is read on every evaluation, and a failed or incomplete read is RED, like every other input. The gate and the fleet lane grant checks: read for it explicitly: measured on #5920 the listing answers without it, but only because core is public — a private caller's token would refuse it and hold every pull request.

Provenance, not presentation — the same principle as for the review itself. A neutral internal-review from any other App, a run under any other name, or that title at any other conclusion is not a degradation. The self-test carries a negative control for each field, and mutating the predicate to drop any one of them turns its control red.

It is a DEFERRED review, not a skipped one. The Plugins steward (Governance/PullRequestSteward, MeshWeaver.Plugins) re-kicks a failed round once, and posts the neutral run only for an infrastructure cause — never because a change was hard to review — naming that cause in the run's summary. A pull request merged on a degradation owes a post-merge review, recorded on the item. The GREEN verdict is never silent about it: the run headline reads GREEN — REVIEWER UNAVAILABLE, review deferred (degradation, not a review), a ::warning:: annotation carries the check run's id, title and summary, and the step summary heads with reviewer unavailable: degraded, review deferred.

It releases condition 1 only, exactly as the waiver does: every thread the reviewer DID open before it went down still needs a person's reply. When a degradation and a waiver both stand, the log credits the degradation — the system released it, nobody had to.

How the degradation turns the context green without a push

A predicate is re-asked only when an event starts the gate, and none of review-answered.yml's triggers fires when a check run completes. Measured and read, in order:

• The internal reviewer's events DO start runs here. Over the last 100 runs of review-answered.yml (2026-09-30), systemorph-com[bot] triggered 11 pull_request_review and 30 pull_request_review_comment runs, none action_required — unlike Copilot's (#4575, above). So a review that does land re-evaluates the gate by itself.
• check_run cannot be added to review-answered.yml. A check_run-triggered run executes on the DEFAULT branch with GITHUB_SHA = main's last commit (GitHub's event reference), and the event fires only for a workflow file already on the default branch. Its Automatic review answered check-run would therefore land on main's commit, where the pull request's branch protection never reads it — the "green in the log, BLOCKED on the pull request" shape below — and a red one would sit on main for every unrelated check run. Check runs created by GitHub Actions do not raise the event at all, so a re-run cannot loop.
• So review-answered-on-degradation.yml listens for it and RE-RUNS the gate. On a completed check run matching the contract, it lets any gate run still in flight for that head finish (one that read before the degradation would otherwise publish a stale red), then re-runs the newest pull_request run of review-answered.yml unless it already reads success. A pull_request run is the one branch protection reads (#4649), and re-running it is the remedy this page already names as legitimate. The listener is single-shot — the degradation completes once — so each of its API calls gets three attempts before it goes RED, naming the head it could not re-evaluate. Since 2026-10-03 it re-evaluates on ANY completed internal-review check run of the internal reviewer, not only the degradation: core #6014, #6017 and #6018 were reviewed after the gate's 15-minute wait had already published red (the review queued behind the reviewer's capacity bound for over 75 minutes), and nothing re-ran the gate when the review arrived. The listener decides nothing: the re-run applies the whole predicate, provenance included, so a drift in its filter can cost or miss a re-run but never turn a gate green.
• 🚨 Not yet observed end to end. A check_run workflow cannot run before its file is on main, so the first real degradation after the merge is the first observation. Read it there: the listener's run on main's head names the gate run it re-ran, and that gate run's new attempt must read GREEN with the degradation line.

Why --wait-for-review stays 15 minutes although the internal reviewer takes 23–30: the wait exists only because Copilot's event cannot start a run. The internal reviewer's review event and its degradation check run each re-evaluate the gate, so a longer wait buys nothing they do not already deliver — and would hold two runners (answered and lane) for half an hour on every pull request opened.

The fleet lane (node-repo-review-answered.yml) fetches the same predicate, so a satellite caller that moves its pinned sha past this change judges the degradation too; the re-run listener is core's own file, and a satellite that needs the context to turn green without a push adds the same listener naming its own caller workflow.

What it does not see

• A reply that says nothing counts as an answer (accepted on #4299).
• Resolution. REST has no resolved state for a review thread, so a thread resolved without a reply stays red. Reply; resolving is optional.
• Suppressed comments. The reviewer lists some findings only inside its review body (Suppressed comments (N)). Those are not threads and are not held.
• A human reviewer's threads. Only the automatic reviewer's threads are held.
• A finding that lands after the queue entry was judged. The merge_group evaluation reads the review once per queue build. Because the reviewer posts once, on open, and the pull-request check cannot be green before that review has landed, a new thread during a queue build needs someone to re-request the review while the entry is building.
• Its own edits. Like every check in this repository it runs the pull request's copy of the script, so a pull request that changes check-review-answered.py is judged by its own change. Read that file's diff yourself.
• Pull requests into other branches. The ruleset reviews only the default branch, so a pull request into another branch reads red; the check is not required there.
• 🚨 Every other repository in the fleet. This check exists here and nowhere else. See below.

🚨 This is a CORE-ONLY gate, and the rest of the fleet shows what that costs

Measured 2026-09-19, from branches/main/protection and every active ruleset:

🚨 MeshWeaver.Plugins is NOT covered, against the common assumption that it is. Every satellite carries a Copilot review for default branch ruleset, so the review is requested everywhere — nothing outside core requires it to be answered.

What that produces, over the last 20 merged pull requests of each of six repositories:

In all 55 the ratio is N of N — not one finding answered on any of them, never a partial. So outside core, findings are not occasionally missed, they are structurally not read: about twice the rate this repository measured before the gate landed (32 of 60, above).

It is not cosmetic. On 2026-09-18, five satellite pull requests merged with 13 unanswered findings between them. Assessed on the code: 11 of the 13 were real, and only two could be declined — one whose premise git itself rules out, one whose failure branch is unreachable. Those 11 reduce to 6 distinct defects, because three were raised twice (independently, on two repositories' copies of one file) and three were three sites of one root. Three of the six are in the platform's own canonical gen-manifests.py, replicated byte-identically into four repositories — including a --resolve that reported ✓ … the merge can be committed whenever git could not answer. Answered and fixed in #4775 after the merges; the remaining vintages in #4777.

Porting it is a workflow_call lane, never six copies — the predicate is ~900 lines with its own self-test, the settle wait and the event-class concurrency split derived from #4649. Six hand-copies is how gen-manifests.py reached five vintages (#1426). 🚨 And the rollout order is not optional: five of the six use CLASSIC protection, where an absent required context blocks every pull request in the repository forever (measured on Plugins#1453), so the lane lands observe-only, is watched publishing its context on live pull requests in that repo, and only then is the context added. Proposed with the full measurement in #4776.

The fleet lane — step 1 of 3 is landed

.github/workflows/node-repo-review-answered.yml is that lane. It fetches check-review-answered.py from this repository at its scripts-ref input, asserts the file's identity, runs the self-test on every call and never falls back to a local copy; it takes no checkout of the caller at all. The caller contract — the lane pinned to a 40-char core sha and that same sha passed as scripts-ref, in a separate review-answered.yml with the three pull-request triggers, the event-class concurrency block, and no job-level if:, path filter or merge_group: trigger — is written in the lane's header, because every one of those is either a skip-trapdoor or the #4649 eviction.

It is not dead code waiting for an adopter: core's own review-answered.yml calls it as a second job, lane, on every event, with scripts-ref: ${{ github.sha }} so both jobs judge with the same copy of the predicate. That job publishes lane / Automatic review answered, which is not required — the ruleset requires the first job, Automatic review answered, by name. The two must agree on every pull request; a disagreement is a defect in the lane.

What is still owed, and is not an agent's to do:

MeshWeaver.Plugins is the first adopter (MeshWeaver.Plugins#2727, at the maintainer's direction). Its staged pipeline (node-repo-stage-gate.yml) holds the heavy legs until the review has landed and been answered, but only when the stage gate runs with review-before-suites. On Plugins it runs with that off: the suites start in parallel with the review, and the gate's own log says "merging and arming still require the review landed and every thread answered". Arming does: the steward's PrArming checks it. A merge does not, because nothing in Plugins' classic protection reads the answer.

Measured over the 25 most recent merges on 2026-10-08, two merged with Copilot findings unanswered:

• #3115: one thread on head 0c60ab5bbc, merged by hand 54 minutes after the review;
• #2962: four threads.

Step 1 (the lane) is already landed. For Plugins, steps 2 and 3 of the table above go like this:

• Step 2 has two parts, in this order:
  • its roster row lands here as pending:;
  • then its review-answered.yml caller lands observe-only. The caller floats @main with scripts-ref: main, because Plugins' check-shared-lanes.py refuses a pinned lane. That repo's own rule overrides the lane header's "pin a sha".
• Step 3: the context review-answered / Automatic review answered is added to Plugins' classic protection, and only after it has been seen published on live pull requests there.

The same gap, measured wider — 240 merged pull requests

The 120-PR sample above was extended on 2026-09-19 to the 30 most recently updated merged pull requests in each of eight repositories (adding Memex, and deepening the six): 240 merged, 325 findings, 182 never answered, across 93 pull requests.

Core is 0 of 59, which is the check working — and it is also the control that makes every other row readable, because the same instrument finds both states. A zero here is a measurement, not a broken query. Memex is the worst of the eight and was not in the earlier sample.

A further 45 unanswered findings sit on 24 open DRAFTS (Plugins 34, of which #1910 alone is 13 of 13; Memex 11). Those describe code that never shipped and are deliberately left; several drafts are abandoned.

Treating the backlog: bound it, and state the bound

There are ~5,700 merged pull requests fleet-wide, so every sweep is partial. Report the denominator you actually swept and what is left, or the next session cannot tell a treated repository from an untreated one. Note that sort=updated is not sort=created: an older pull request that received a comment recently enters the window, which is why the windows differ in span per repo.

gh api "repos/Systemorph/<repo>/pulls?state=closed&per_page=100&sort=updated&direction=desc" \
  --jq '.[]|select(.merged_at!=null)|.number'
gh api "repos/Systemorph/<repo>/pulls/<n>/comments?per_page=100" \
  --jq '{findings:[.[]|select(.user.login=="Copilot" and .in_reply_to_id==null)]|length,
         replies:[.[]|select(.in_reply_to_id!=null)]|length}'

A finding counts as answered only when some comment's in_reply_to_id is that root's id. A PR-level comment answers nothing, and neither does resolving the thread.

Reply on every thread whatever the verdict. Four verdicts, and the declines need their reason on the record more than the acceptances do: real (fix it), real but cosmetic, obsolete (verify against the current file and quote the evidence), wrong (say so, with the measurement). A merged finding nobody answered and nobody declined is indistinguishable from one nobody read.

🚨 -F, never -f — the reply that posts its own filename

gh api -X POST repos/Systemorph/<repo>/pulls/<n>/comments/<id>/replies -F body=@reply.md

-f body=@reply.md sends the literal seven-to-twenty-character string @reply.md as the comment body, and the POST returns a normal comment id with a 201. So it reads as a successful reply; the thread's root now has a comment whose in_reply_to_id points at it; and every detector built on in_reply_to_id — including the sweep query above, and check-review-answered.py's own predicate — counts the finding as answered. The finding is untreated and nothing says so.

Measured on 2026-09-19: 100 posts across four sessions went out this way, each a @-prefixed filename or absolute path — Memex 41, Crm 24, Reinsurance 20, Manufacturing 15. Three of the four sessions believed they had replied and reported thread counts to prove it; the proof was the very field that cannot distinguish the two. The fourth caught it only by reading one reply back.

🚨 And the first audit of it was itself understated, for a structural reason worth keeping. A thread-centric sweep — enumerate the review threads on the swept pull requests, check their replies — saw 74 of the 100. The flag is a property of the call site, not of the thread, so it also hits PR-level issue comments (issues/comments, a different endpoint) and replies on the sweep's own pull requests, neither of which a thread-centric audit reaches. Re-audit by call site: every comment authored today on both endpoints whose body matches ^@. That found the remaining 26.

So the verification is read the body back and check its length, never the reply's existence:

gh api "repos/Systemorph/<repo>/pulls/<n>/comments?per_page=100" \
  --jq '.[]|select(.in_reply_to_id!=null)|"\(.id) \(.body|length)"'

A length near 20 is the bug. Repair with PATCH /repos/{o}/{r}/pulls/comments/{reply_id}, not a second reply — re-posting leaves the stub standing beside the real answer, and the thread then reads as two answers, one of them noise.

🚨 TWO flag mistakes produce a stub, and they leave DIFFERENT first characters. A sweep keyed on one of them misses the other, and the second is the worse instance because it looks less obviously wrong:

-F reads a file only with the @ prefix, so dropping it silently turns the path into the value. Measured 2026-09-19: the first mechanism produced most of the fleet's stubs, and the second produced a further batch that an ^@ sweep did not see. So do not key the sweep on a flag or on @ — key it on the shape of the body: a reply that is a single token with no whitespace is a path, whatever put it there. Re-audited on that key, 0 of 533 comments authored during the sweep were stubs; the earlier ^@-keyed pass could not have said so.

🚨 A stub also inflates every "how much is left" count, so a denominator taken before the repair is understated by its own stub total. A stubbed thread has a reply, so the sweep query above calls the finding treated — which means any backlog figure published while stubs are outstanding is a floor, and not by a knowable margin. Re-measure after repairing, and when reporting a backlog say whether the count was taken before or after. Two of this sweep's per-repo denominators were re-taken for exactly this reason.

The general rule, of which this is one instance: any gh api write whose payload came from a file is verified by reading the field back. -f and -F differ silently in both directions — -f treats @path as a literal string, and -F type-coerces a value that merely looks numeric or boolean — so the flag is the wrong thing to reason about. The response is not evidence either: it carries an id and a 201 whatever went in. Read the stored field and compare it with the source.

And the comparison is not a length check. @r_115_4028258629.md is 20 characters, which is short, not obviously wrong, and a genuinely terse reply would fail the same test. Assert a content signature you know is in the file — the verdict string the reply opens with — or byte-compare against the draft, allowing for the trailing newline GitHub appends.

This is the sweep's own instance of the defect class it exists to find: an answer that reads like a pass. The question to ask of any reply mechanism is the one that applies to a gate — if this had failed, would the output differ? Here it would not have.

🚨 A control can be the thing that cannot fail

The cases above are all checks that could not fail. The worse shape is a control that cannot fail, because a control is what you appeal to when you have stopped trusting the checks.

Measured 2026-09-19 on MeshWeaver.Education#335. A finding said a liveness census could not parse the heartbeat it exists to read, so wedged was false by construction and the instrument reported a clean pool over a real wedge. It was written up, agreed with, and given a control: fed the tick lines the script's own header quotes as the cause of a past incident, the census went from 0 of 5 wedged to 3 of 5, naming the window — a swing so clean it read as proof.

The parser was correct. The producer — ProcessLiveness.Describe() — renders every delta parenthesised and gap always as actual/threshold; its unit test asserts the rendered line, and a platform doc shows the same. The forms the control fed it (poolCompleted=+11343, a bare gap=10.00s) appear in no real log. The control had been built from an abbreviated transcription in a comment, so it proved something about the transcription and nothing about the parser.

Three things generalise:

• An example that merely APPEARS in the code under test is not a fixture. The header in question was not prose describing a format — it was a quoted log excerpt, which is exactly why it persuaded. A fixture is something the producer wrote (a format string, a golden file) or something a test asserts. Nothing else earns the name, however much it looks like captured output.
• A control that swings dramatically is not thereby a good control. A fabricated input makes a correct parser look broken, and the bigger the swing the more it reads as confirmation. Size of effect is not evidence of validity.
• The falsifying artefact was in the same repository the whole time — a sentence in the harness's own README stating that the real captured log does produce the window — and was read afterwards, while looking for somewhere to file the finding. Reach for the artefact that would falsify you before the one that would confirm you.

The real defect was the misleading transcription, which had by then generated the same false finding twice: once by the reviewer, once by the reader who agreed with it.

The two-part key: a read-back needs a baseline

This page's own sweep reported "182 of 182 findings answered, proven by read-back". The key it used was a substantive reply exists on this thread — and that half alone is unfalsifiable, because it passes on a reply an earlier session left. What makes the number mean anything is the other half: the same 182 were measured unanswered at sweep start.

So the baseline measurement is not a caveat on the result, it is half the instrument. State both halves or neither. The same applies to any "N of N done" claim built by re-reading a store you also wrote to.

🚨 A watcher's polarity decides whether a failed read waits or FIRES

The twin of "a broken query reports not-yet forever", and worse, because this one triggers an action. Measured 2026-09-19 at 09:08:35Z on MeshWeaver.Plugins: a blob watcher armed as

until [ "$(gh api "$PATH" --jq '.sha' 2>/dev/null || true)" != "$BASELINE" ]; do sleep 60; done
echo "ACT NOW — the file moved"

fired a false ACT NOW. GitHub answered the secondary-limit 403, --jq '.sha' returned the error body, and != against the baseline was true. Nothing had moved. || true plus != on an unvalidated value compose into any failed read is a positive result — and the announcement named a remedy (merge and push), which had it been trusted would have merged a stale canonical and burned a CI run during the very limit that caused it.

The polarity is the whole difference, and it is easy to get right by accident and wrong by accident:

So: write the condition so a failed read is FALSE, never TRUE — wait for the value you expect, never for difference from a value you remember. Where difference is genuinely what you need, shape-guard first (a sha must be 40 hex characters; a size must be digits) and make "could not read" its own printed outcome that backs off, rather than a silent third state folded into one of the other two.

The question to ask before arming any watcher is one line: what does this print on a 403? Ask it of the whole expression, including the || true you added so a transient failure would not kill the loop — that fallback is usually where the defect enters.

The safe form for posting

Build the whole request body as JSON with jq, and hand that one file to --input:

printf '%s' "$(jq -Rs '{body:.}' < reply.md)" > payload.json
gh api --method POST "repos/{owner}/{repo}/pulls/{n}/comments/{id}/replies" --input payload.json

🚨 The safety is jq's, not --input's. --input takes one file that is the complete request body — it has no per-field behaviour to confer any protection (Copilot review on #4788, which caught this page attributing it to the wrong half of the pipeline). What makes the recipe safe is that jq does the quoting and escaping, so no value is ever interpreted as a flag argument.

The one session of four that posted this way was the only one with no stub. It sidesteps the -f/-F question entirely rather than requiring anyone to remember which flag reads a file — which matters more than it sounds, because there are two ways to get that wrong, not one (below).

🚨 One defect, five copies — fix the canonical, then RE-COPY IMMEDIATELY

Findings cluster hard on vendored files, because the reviewer reads each repository's copy independently. Alongside the gen-manifests.py collapse above, the 2026-09-19 sweep found 25 of the 182 against five satellites' copies of scripts/resolve-platform.py, collapsing to five distinct defects in core's canonical — one of them reported three times over (fixed in #4779; the one deferred as needing a design decision is #4780). Patching a vendored copy alone is how this fleet reached five vintages of one script (#1426).

🚨 A canonical fix REDS EVERY SATELLITE THE MOMENT IT MERGES, so the re-copy is part of the same piece of work — not a follow-up. node-repo-validate.yml declares platform-ref with default: main, and scripts-ref falls back to it. A satellite whose validate: job passes no inputs — which is every one of them — therefore has the canonical fetched at core main, live, and check-resolver-copy.py has been hard-red since RED_FROM 2026-09-15. Measured 2026-09-19: core #4773 merged at 08:02:23Z and by 08:16Z every satellite's validate / Validate node repos — a required context in all of them — was failing with

scripts/resolve-platform.py has DRIFTED from the platform's canonical: 32 code line(s) differ
(76 raw), RED since 2026-09-15T00:00:00Z

Do not reason about this from a platform-ref: literal in a satellite's ci.yml. Those literals pin other jobs (compile-check, tag-modules, the pack lanes); the validate: job passes nothing and takes the default. Reading the wrong job's input produces the confident and wrong conclusion that a canonical fix is invisible to the satellites — it is the opposite, and the guard's own docstring says so ("a canonical fetched at @main is live on merge for every caller", MeshWeaver#4027). Read the run: the job log prints SCRIPTS_REF: main.

So the shape of the work is: fix the canonical, merge it, and re-copy into every satellite in the same sitting —

gh api repos/Systemorph/MeshWeaver/contents/.github/scripts/resolve-platform.py --jq .content \
  | base64 -d > scripts/resolve-platform.py

— verifying the guard then reports CODE-IDENTICAL (exit 0). A re-run does not help an open pull request, because the guard reads the branch's copy: that branch needs git merge origin/main after the re-copy lands on the satellite's main.

🚨 And re-check the canonical immediately before the merge, not only before the push. Measured 2026-09-19: the canonical moved twice in one morning (201,067 → 228,485), so a re-copy that was byte-identical when it was pushed was stale before anyone merged it — three satellites re-copied the first canonical, landed it, and were behind again within the hour. Education's session checked before merging rather than after and found its own main still red on the guard at the intermediate size, which is the check that saved a pointless merge. The window between waves can be shorter than the time a pull request takes to go green.

🚨 And while the copy is behind, the rest of the lane is silently ungated

This is the more expensive half, and it is a skip-trapdoor made by step ordering rather than by an if:. The drift check sits mid-job, so its failure skipped 16 subsequent steps of the same job (measured on MeshWeaver.SocialMedia#210, job 105867277548) — among them:

• Every PR-reachable secret in this repo is asserted by a preflight
• Every manifest.lock is current (and carries a version)
• Every module's version matches its content
• No mapping in this repo's workflows writes a key twice
• No pin comment names a commit this repo no longer pins

Each reported skipped, which under both protection mechanisms counts as satisfied. So for as long as a satellite's vendored resolver is behind, every pull request in it is unchecked by all of those, and the only visible symptom is one red about an unrelated file. Filed as #4784; the durable fix is if: ${{ !cancelled() }} on each independent guard, or one job per guard family, so that a single red reports rather than masks.

Controls

The predicate, replayed through the real REST adapter with --as-of on pull requests whose outcome is known:

The self-test (--self-test) states, for each of its fixture cases, the exact reasons the case must be red for, so a fixture cannot pass by being red for a second, unintended reason. Its negative controls were watched failing: twelve mutations of the predicate — a bot's reply counted as an answer, a refusal counted as a review, an unrecognised body counted as a review, any bot taken for the reviewer, the completeness check removed, a waiver releasing threads, a writer or a bot allowed to waive, --as-of ignored for comments, only direct replies followed, a PENDING review counted, a short sha accepted as a queue ref — and each one turned the self-test red.

To replay a pull request yourself:

python3 .github/scripts/check-review-answered.py --repo Systemorph/MeshWeaver --pr 4310 --as-of 2026-09-14T14:04:21Z

Observed end to end on #4575

The check's own pull request, in order, with the run that did the work:

The review that counts was submitted on the previous head (c496a0bffb) and the check is green on the new one (bf37e92a09): condition 1 asks whether the review landed, never whether it landed on the head, because the reviewer reviews once. And the 15-minute wait was exercised against #645 — three polls, then RED naming the quota refusal.

Rollout — done; kept as the record

🚨 This section is history, not instructions. The context was added to ruleset 2128472 beside Consolidate test results on 2026-09-17, and the check is required today (see Status above). Read the steps below as what was confirmed before the switch was thrown — do not re-run them as a plan, and do not read "the check lands non-required" as the current state.

Automatic review answered

What was confirmed before adding it:

1. Decide the approval policy for Copilot-triggered runs. Today they are action_required with zero jobs (measured, above), so the bounded wait is what makes the check self-sufficient. If the policy is relaxed, drop --wait-for-review to 0 and the job's cap to 5 minutes.
2. Watch it on live pull requests — red on open, green once the review lands on a pull request with no findings (through the wait), and green after the replies on one with findings.
3. ✅ Teach the merge-queue steward this check — done 2026-09-17. It read only the failed merge_group run of MeshWeaver Build and Test under the pull request's queue prefix, so an ejection caused by this check reached it with both outcomes wrong: no such run → rejected as unclassifiable, naming the wrong workflow; or an older failed build of the same pull request → it classifies a failure that is not why the entry was removed, and if that stale failure is a catalogued flake it re-queues a pull request whose findings are unanswered. merge-queue-steward.py now reads the other merge_group workflows first and rejects with kind=gate, naming the workflow — keyed on "not the test workflow" rather than on this check by name, so a gate added later is covered the day it runs rather than the day somebody remembers that file.
4. Decide the cost. 32 of the last 60 merges would have waited for replies.

The timing, re-measured 2026-09-17 (39 merged pull requests)

The original framing — "checks outran the reviewer" — is no longer the live mechanism, and the numbers say so:

The required gate finished before the reviewer submitted on 0 of 37. The denominator is 37 of the 39 because two merged pull requests carry no automatic review at all — #4584 and #4582, both with zero reviews of any kind, checked rather than inferred from the gap in the counts. With no review there is no submission time to compare against, so they are excluded from the comparison rather than counted as a win for either side. They are also exactly the pull requests this check would hold: no review landed, condition 1 unmet — so the two numbers disagreeing is itself a measurement, not an inconsistency.

The reviewer is reliably first. So what merges past a finding today is not a race — it is that the review lands, and nothing requires it to be read. That is precisely what this check asserts, and it is why the remaining step is the ruleset edit rather than any further engineering.

What an author does

Reply to every thread the reviewer opened — "fixed in <sha>", or why not — through the pull request page or REST:

gh api "repos/Systemorph/MeshWeaver/pulls/<PR>/comments?per_page=100"
gh api -X POST "repos/Systemorph/MeshWeaver/pulls/<PR>/comments/<comment id>/replies" -f body='Fixed in <sha>: …'

Never hand-request the review to turn the check green, and never apply review-waived yourself.

🚨 Posting a reply from a file needs -F and @ — miss either and you post the PATH

Reading a body from a file takes both the typed flag and the @ prefix: -F body=@reply.md. Two independent near-misses each post the path itself as the whole reply, and they are distinguishable by the body's first character:

Each row posts its own value verbatim, which is what makes the near-miss reproducible as written. 🚨 In the wild it does not look that obvious, because the path arrives in a variable — the shape that hid the missing @ from its author:

reply() { gh api graphql -f query='…' -f id="$1" -F body="$2"; }   # ← "$2" needs @, and has none
reply "PRRT_kwDOTQ2qnM6XczkO" /Users/roland/.mwgate/reply1.md

-f sends its value literally and never interprets @; -F interprets @value as a file but passes a bare value through unchanged. Both apply to gh api graphql variables as well as REST fields. Either way the call SUCCEEDS, in_reply_to_id is set, the thread renders as answered, this check goes GREEN, and the finding is never read by anybody. It is strictly worse than not replying at all, because it consumes the one signal — "this thread is unanswered" — that would otherwise bring somebody back to it. Worse still when the same loop goes on to resolve the thread, as MeshWeaver.Plugins#370's did: the finding then looks deliberately closed rather than merely answered.

Measured 2026-09-19: 100 stub posts across four sessions. On MeshWeaver.Plugins#370 four Copilot findings sat that way for six weeks, one of them a privilege-escalation shape in an admin-invokable action. The answers had been written — the files were still on disk — and only the posting failed.

🚨 Never key detection on the stub's shape. The body is whatever string the caller passed, and three shapes have already been observed: a bare absolute path, @ followed by an absolute path, and a single x. Two predicates that do not depend on shape, used together over every reply:

# SUSPECTS, not proof — a body with no whitespace at all
jq -r '.[] | select(.in_reply_to_id != null and (.body | test("\\s") | not)) | "\(.id) \(.body)"' comments.json
# every short reply — the backstop for a stub that DOES contain whitespace
jq -r '.[] | select(.in_reply_to_id != null and (.body|length) < 200) | "\(.id) \(.body)"' comments.json

🚨 Neither predicate is a gate, and the first is a high-signal SUSPECT list rather than proof. It has false positives — a bare commit SHA, or a lone #1234, is a legitimate whitespace-free reply — and false negatives, because a placeholder containing a space passes it, which is why the second predicate is the backstop and not a refinement. Both are scoped to in_reply_to_id != null: a top-level review comment is not a reply, and counting one as a stub is how a census overstates. Read every hit. The discriminator is what the body NAMES: a real answer names a commit or a finding, a stub names a file. The only thing here safe to automate is the repair verification below, where a byte-compare against the posted file is exact.

Repair with PATCH, never a second POST — a new reply leaves the stub standing beside it:

gh api -X PATCH repos/<owner>/<repo>/pulls/comments/<stub id> -F body=@reply.md   # -F, never -f

🚨 Verify by BYTE-COMPARING the fetched body against the file, not by the exit status, not by the reply existing, and not by its length. "A comment exists whose in_reply_to_id is the thread root" passes on the very stub it is meant to catch, and length cannot separate two paths of similar length. Fetch pulls/comments/<id> afterwards and assert equality with the file's contents.

Two traps when sweeping a repo for stubs, both of which produce a confident undercount:

• Page to exhaustion, and get the denominator from the Link header (rel="last"), not from a fixed page budget. A 25-page sweep of pulls/comments?per_page=100 caps at 2500 comments; MeshWeaver.Plugins has 46 pages, so such a sweep silently misses the oldest ~45% — which is exactly why #370's August replies were invisible to it while a newer pair was not.
• A secondary-limit 403 lands INSIDE --slurp output as a JSON object among the arrays. Its final element is then an error, not a short last page, so "the last page held 3 rows, therefore pagination was exhausted" is a false pass. Type-check every page (select(type=="array")) and count what you actually read. And note that /rate_limit answering 5000/5000 remaining while calls are refused is the secondary limit — the primary quota is not the signal.

🚨 The check's log says GREEN and the pull request is still BLOCKED

The check now refreshes itself. After a GREEN verdict on any event other than pull_request (and never for a merge_group entry, which is its own commit), the refresh job of node-repo-review-answered.yml (lane / Refresh the verdict branch protection reads in core) re-runs the newest pull_request run of the same workflow for the current head — the WHOLE run (POST …/runs/<id>/rerun), never only its failed jobs (check-review-answered.py --refresh-read-run). The whole run is the right call because what the refresh must produce is a fresh pull_request evaluation of live state, and every job of the lane reads live state. It is also the only call GitHub accepts for the commonest candidate: a pull_request run that a newer arrival REPLACED in the concurrency group while it was pending, so it was cancelled with ZERO jobs and published no check-run at all. For that run rerun-failed-jobs answers 403 This workflow run cannot be retried and rerun is accepted (measured on #6315, run 37807892371); the refresh used rerun-failed-jobs until then and went red on exactly that. The stage advance below keeps rerun-failed-jobs, because its held run is a full test run whose green jobs must not be repeated, and it never revives a cancelled run. The decision is the pure function refresh_action, covered by the script's --self-test. It re-runs that run only if it has completed, is not success, and started before this GREEN verdict was taken. A run that is in flight, already green, or started after the verdict is left alone, and the job prints why. That makes it idempotent per (head sha, answered state), with no loop and no polling: the re-run is itself a pull_request run, which never refreshes, and a re-run made with the workflow token raises no new event. Covered the same way: a pull request that went RED because answers were missing goes green when a later reply's run is GREEN, because that run refreshes too. refresh is the only job in the lane that writes, and it holds actions: write alone. A caller adopting the lane grants it on its uses: job and records the grant in .github/lane-caller-grants.yml in the same change.

The staged pipeline's hold is re-evaluated on the same reply events by the existing listener, stage-advance.yml → node-repo-stage-advance.yml. Its on-answer job fires on pull_request_review_comment: created and pull_request_review: submitted and re-runs the failed jobs of the head's held dotnet-test.yml run once stage 1 is green.

Manual fallback. The self-refresh cannot help in two cases. A fork pull request's token cannot re-run a workflow, so the job warns and names this remedy. And if the refresh job itself is red, it names what it could not read. In both cases, re-run the check's pull_request run:

# the run whose verdict branch protection actually reads
gh api "repos/Systemorph/MeshWeaver/actions/workflows/review-answered.yml/runs?head_sha=<HEAD SHA>" \
  --jq '.workflow_runs[] | select(.event=="pull_request") | .id'
gh api -X POST "repos/Systemorph/MeshWeaver/actions/runs/<ID>/rerun"

An empty commit does the same thing by a longer road — it gives the check a head sha it has never judged — but 🚨 --allow-empty does not mean "empty": it means "allow a commit that would be empty", and anything already staged rides along. Under pressure, with a half-staged tree, that pushes an unreviewed change while you are trying to repair a check. Refuse a dirty index first:

git diff --cached --quiet || { echo "REFUSING: the index is not empty — this would commit staged work"; exit 1; }
git commit --allow-empty -m "chore: one clean head for the review gate"
git push

Prefer the re-run above: it changes no history and cannot carry anything with it.

Measured on #4652 (2026-09-17), the first time the remedy was used: BLOCKED / rollup FAILURE → CLEAN / rollup SUCCESS / isInMergeQueue: true, within a minute of the re-run. Re-arming auto-merge first did not clear it — the rollup was not stale in the usual sense, it was citing a real, still-current failure taken in a suite nobody reads. Measured again on #4662 (2026-09-18): two pull_request runs re-run, both green, blocked → clean and into the queue.

🚨 This re-run is legitimate, and it is not "re-run and see". The check reads LIVE state, and that state genuinely changed when the replies landed: the first evaluation was correct when it ran, and so is the second. This is the one case where re-running a red is the right response rather than a way of hiding one — and it is safe to write down because if a thread is genuinely unanswered the re-run is red too. (Every other red in this repository still means investigate, not re-run.)

What is actually happening (measured on #4649, head 94fc73b4fdaa, 2026-09-17). Answering a thread fires both a pull_request_review and a pull_request_review_comment event. Fifteen runs of this workflow resulted; eight published a check-run; and the statusCheckRollup that branch protection consumes carried exactly three:

So the mechanism is not a latched rollup and not "the newest run loses":

1. A pull_request_review_comment run's verdict is invisible to branch protection. Both successes were from that event, so they were never candidates — by construction, not by timing.
2. The read-eligible runs that arrived after the replies were EVICTED. pull_request_review runs at 19:48:19Z and 19:48:28Z were cancelled in the one pending slot they shared with the comment-driven runs, so they published nothing and the last read verdict on that sha stayed an early failure.
3. Hence the symptom: the newest check-run reads success, the check's own log says "GREEN — 5 of 5 answered by a person", and the rollup correctly reports a real failure in a suite nobody was looking at. Re-arming auto-merge does nothing, because the rollup is not stale.
4. It is invisible in REST, because mergeable_state: blocked is also what a PR waiting its turn reports. The rollup is the only place the difference shows.

What was done about it. Two changes, and one option that is not ours to take:

• The concurrency group is split by event class (gate for pull_request / pull_request_review / merge_group, feedback for pull_request_review_comment), so the survivor of the gate lane is always a run whose verdict is read. That removes cause 2 deterministically.
• --settle-replies 60 makes that surviving run judge a state that has stopped moving, so it publishes the settled verdict rather than a snapshot taken mid-answer. Every gap measured inside an answering burst was 1–11 s (#4649, #4656, #4646); 60 s is ~5×, and the one 954 s gap in that sample was a separate later round that rightly gets its own evaluation. It softens no verdict — an unanswered thread that stays unanswered is still red when the wait ends — and it waits only while unanswered threads are the whole complaint, never for a missing review or an incomplete listing.
• Not taken, because it needs the maintainer: reporting the verdict as a commit status against the head sha (POST /repos/{o}/{r}/statuses/{sha}) instead of as a job's check-run. That is the only shape correct by construction — one context, last write wins, whatever event produced it — but it is a required-context rename plus a ruleset edit on 2128472, i.e. a change to how everything merges. Until that is decided, the two changes above plus the re-run remedy are what stands.

🚨 And a reply must be ON THE THREAD. A pull-request-level comment does not answer anything: the check counts replies whose in_reply_to_id is the reviewer's root comment, and its log names every thread it still considers unanswered. If the log disagrees with what you think you answered, read that list before reaching for the re-run.

The arm gate — auto-merge is armed only on a reviewed, answered head

Measured 2026-10-03/04 on MeshWeaver.Plugins: auto-arm.yml armed auto-merge on every open, push and undraft, BEFORE the internal review had run on the new head. On Plugins the review is comment-only and nothing required waits for it, so #2549 (6 of 6 findings unanswered), #2643/#2647 (reviewer unavailable) and Memex#641 merged unreviewed or unanswered, and agents disarmed by hand after every push (#2791 twice in an hour).

Arming is a decision, so it moved to the control plane (policy review-then-suites). The control instance's PR steward (MeshWeaver.Plugins PrArming, App systemorph-com) already reads every fleet pull request's review state and answered-findings verdict; it is the one place that arms. No workflow arms auto-merge any more — ArmedMergeMustTriggerMainsPushLanesGuard.NoWorkflowArmsAutoMerge fails the build if one does.

What the control plane arms on

It arms a pull request only when all of these hold, read on its current head:

1. it is open, not a draft, and its head branch is in the same repository (a fork never arms itself);
2. its base is the repository's default branch, read from repos/{o}/{r} and never assumed. An unprotected base has an empty required set, so an arm there is an immediate merge (MeshWeaver.Plugins#1685 merged 61 seconds after it opened, before its own CI started);
3. the head has its own review: a Copilot review submitted against the current head (policy copilot-code-review; commit_id equal to the head, not PENDING, not a refusal — copilot_review_on, see Copilot is the reviewer again above), or, while one is still posted, a completed internal-review check run of the systemorph-com App (id 4918443) on the head whose newest run is not the neutral Reviewer unavailable … degradation. The degradation releases the merge gate's review condition (a down reviewer must not hold every pull request), but it is not a review — arming on it would land an unreviewed change nobody decided to merge, so a person merges it by hand;
4. every thread the automatic reviewer opened has a reply from a person, following in_reply_to_id to the root, read from a provably complete listing — the same reviewer_threads / listing_incomplete predicate this gate uses, so the two can never disagree;
5. every required status check of the base is success on the head (required_checks_green, policy review-then-suites — see Staged Pull Request Pipeline).

The reference implementation of 1 and 3–5 is check-review-answered.py --arm-gate (arm_readiness), and its --self-test cases are the test vectors the control plane's port must pass. The arm is the GraphQL enablePullRequestAutoMerge mutation (or enqueuePullRequest on a base with a merge queue) carrying expectedHeadOid = the head it judged, so an arm that races a push is refused by GitHub instead of landing on the new, unreviewed head. It is attempted once per head and read back (autoMergeRequest / mergeQueueEntry); a head a person disarmed is not re-armed.

🚨 The port has to keep up with the reference. PrArming is a one-for-one port of arm_readiness, so a condition widened here is not live until the port carries it. Copilot's acceptance in condition 3 is exactly such a widening (the register lists the port as owed under copilot-code-review): until PrArming accepts a Copilot review of the head, a head reviewed only by Copilot is never armed by the control plane and needs a person's arm.

What stays in the workflow: the disarm

A push invalidates the review, and GitHub keeps auto-merge armed across a push by anyone with write access — so a pull request armed on a reviewed head would merge the next, unreviewed head the moment the required contexts pass. auto-arm.yml (still the fleet's one lane, still called by every repository through workflow_call, its name kept so no caller changes) therefore now does exactly one thing: on synchronize, if the pull request is armed, it takes the arm off (gh pr merge --disable-auto) and reads the result back. It reacts to the event itself, so a fast CI cannot outrun it, and it tolerates no failed step: a disarm that silently did not happen is the hole reopened. It uses the meshweaver-cloud token it always used (contents: write, pull-requests: write) — no checks permission, and no change to any caller's grants.

Related

• The Merge Queue — the queue this check runs in, and the steward named in the rollout
• Reading CI Signals — why a skipped required context counts as satisfied