A one-node lookup reads the owner, never a query cache
MeshQueries.FindNode(hub, path) is the Store's single-node read: the cover reads PluginContent
through it, coupons read CouponContent, the installer reads its manifest, and every
PluginGate verify-before-write decision — "is this grant still missing?", "did my write become
durable?" — is one of its answers.
It reads the node from its OWNER. A listing answers EXISTENCE (empty on absent, so a point read
never lands on a path that is not there); the per-node stream answers CONTENT. That is the
composition Doc/Architecture/CqrsAndContentAccess prescribes for a node that may not exist yet,
and it is the only shape in which the answer cannot be older than the node.
What it used to be, and what that cost
It used to be one synced query: hub.GetQuery("Store.Node:{path}", "path:{path}") — one node's
content out of a live collection. Three properties of that transport compose into a defect:
- The chain is cached for the life of the PROCESS.
MeshNodeStreamCache.GetQueryRawbuilds it asReplay(1).AutoConnect(1)and keeps it in a registry that is never rebuilt. The FIRST answer is therefore the answer every later caller in that process replays. - A cold first answer is an ordinary outcome, not an error.
MeshQuery's merge counts a provider that completes without anInitialas an EMPTY Initial by contract — the alternative was a consumer that hangs — so a provider that gives up under load hands the chain "there is nothing there", permanently. (Routing has its own cold shape:GetDelegateForPathanswers null while a partition's schema adapter is unresolved, and the query takes the cross-schema route instead of its own. Which of these fired on the public instance is NOT established — what is measured is the RESULT, below, and that a restarted replica does all of this at once while every plugin hub activates.) - Nothing corrects it. A gating pass re-writes an unchanged node, the upsert is a NO-OP at the
store (
IsNoOpUpsert), and a no-op publishes no change — so the one event that would refresh the chain never happens. The wrong answer is now permanent for that replica.
Measured on the public instance on 2026-09-16, in the 24 minutes after the 21:24Z restart (the incident is
Admin/_LogIncident/d1cd36f53a5f3a6c, 11,927 occurrences, and it went quiet again afterwards):
| what the gate logged, at Error | what the store held |
|---|---|
Codex, Antigravity, Grok, OpenCode: write did NOT become durable for …/_Access/Public_Access, …/Anonymous_Access — a provisioned partition lost a write |
all eight grants present at version 1, written 2026-09-13, untouched since |
Feedback: same line for Feedback/_Submissions/_Access/* |
both denies present at version 1, written 2026-08-03, untouched since |
Governance, BuildServer, WhatsApp, iMessage: reconcile is NOT CONVERGING — rewrote …/_Policy |
each _Policy in the exact shape PolicyMatches wants, versions 2–3, no version written on 09-16 at all |
Nine of the ten retained samples name a node that was durably present, in the shape the pass wanted,
for days. The value was never stale at the owner; it was stale in the lens — and
SyncedQueryMeshNodes says so in its own summary: "for single-node content reads on a known path
use GetMeshNodeStream instead — the synced collection is for live collections, not for fetching one
node by path."
The shape now
- Existence — the cached listing first (
path:{path} select:path, metadata only, so nothing here can be mistaken for a content read). 🚨 A NEGATIVE is never final: it is re-asked once against a FRESH one-shot query, which re-reads the store rather than replaying a cold answer. A POSITIVE is never re-asked — the index trails the store, so "the index has seen it" implies the store has it. The hot path (a node that is there) therefore still costs the cached chain plus a cached stream read; only the answer that authorises a WRITE costs a real read. - Content —
hub.GetMeshNodeStream(path).Take(1): the owner's own state, typed by this hub, the same read the GUI databinds to. - The cost, deliberately taken. A present node costs what it did before (a cached chain plus a cached stream read). An ABSENT node now costs one indexed single-path query per call — the entitlement probe on a paywalled cover is the hot one — and that is the case where a cached "no" is most expensive to be wrong about: a stale "no" there shows a paying customer the paywall (2026-08-03), and a stale "no" in the gate authorises a write.
- A fault is forwarded, never folded into "absent". Absence authorises a write; a failed read
must leave the caller's state alone (
PluginGate's targeted read skips that candidate and self-heals on the next pass). Collapsing the two is how a reconcile writes blind.
What this does not fix
Feedback/_Policyon the public instance is still the legacy blanketpublicRead: trueat version 198,113 with no version written since 2026-08-12 — 31,701 retained versions, none after that date — while the entitlement-privacy migration has been shipping since 09-12 and the gate reported rewriting that path on 09-16. Every other sample is explained by the lens; this one is not: either the write never reached the store or the store dropped it. Separating those needs the replica's own log ([MonotonicWriteGuard] CONFLICT on Feedback/_Policy/[PostgreSqlStorageAdapter] write to Feedback/_Policy … was REFUSED by the version condition), which is an operator read (Hosting/InstanceAction→Logs), not something a pass can measure about itself.- An unprovisioned partition is still PARKED, and that is the design — a package the registry
lists but nobody installed here has no partition, and
Doc/Architecture/PartitionedPersistenceis explicit that a write to an unprovisioned partition is refused rather than conjuring a schema. Provisioning belongs to the install, which runsPluginGate.SeedGatingas its own phase (PluginGate.PhaseName), so the parked state converges through the install and is reported once per hub lifetime at Warning — seeGatingWatcher.md. It is terminal and quiet by construction; it is not something the gate should retry into existence.
The other half: a write's outcome is the owner's ANSWER
Reading the owner fixes the read that decides whether to write. It does not fix a read taken
immediately AFTER a write, and that one cannot be fixed by choosing a better seam: the per-node
stream handle and the synced query both replay a cached last-known value, so a read microseconds
after a write legitimately answers with the pre-write state. So the gate no longer re-reads at all.
CreateOrUpdateNode emits the node as its owner then holds it — created, merged, or the
untouched existing one when the upsert was a no-op — and PluginGate.VerifiedWrite,
EnsureEntitlementPrivacy and EnsureEntitlementRootPrivacy judge that. It is race-free, one round
trip cheaper, and the owner is the one component that cannot be looking at a stale copy of its own
node.
What that verdict deliberately does not claim is DURABILITY — the owner accepting a shape is not the
store keeping it. That statement belongs to the next pass: a path this hub wrote, whose fresh read
finds the shape missing again, is the store losing writes (or a partition that has none), and
GatingWatcher.md describes how the watcher parks and classifies it. The privacy prerequisites fail
CLOSED, so the same race there refused an acquisition on a partition that was already private.
Pinned by PluginGateLandedVerdictTest (MeshWeaver.PluginCatalog.Test): a cold lens that never saw
the node must not make the pass write, fail, or report a durable node as lost — the test poisons the
lens exactly the way a replica does, by letting a SECOND host over the same store write the shape,
which is the topology OrleansTestBackingStore models ("what is shared is the STORE, not the change
feed").
A SWEEP reads listings — the owner only for what they cannot settle
Reading the owner is right for ONE node. It is wrong as the inner loop of a sweep, because every
owner read is a routed SubscribeRequest that ACTIVATES the node's own hub, its answer is routed back
to the reader's cache/* hub, and the warm mirror it leaves heartbeats for ten minutes. The
standard-pack reconcile (StandardPacks.EnsureForAllUsers, run on every Store hub activation — i.e.
on every roll — and EnsureForViewer on a Store visit) used FindNode per (user, pack): the root,
the manifest {user}/_Install/{repo}, and for a free pack the marker {plugin}/_Entitlements/{user}
and the grant {plugin}/_Access/{user}_Access. On a mesh where every user already had every pack it
answered "nothing to do" by activating up to four record hubs per pair. Those were exactly the
per-user dispatch targets that led the post-roll [ROUTE] Routing back-pressure crossings
(MeshWeaver#5389, #5118): named as the latest dispatch target in 21 of the 51 crossings sampled
after the #5571 fix, and the oldest leg in 6.
Each user now costs ONE prefetch of LISTINGS — the _Install children, hydrated through the same
synced read the tile refresh keeps (AppTileRefresh.ManifestsQueryId), and one fresh metadata-only
query over every free pack's marker and grant path — and StandardPacks.PacksToVerify settles a pack
only on POSITIVE evidence: the manifest records it with nothing still owed, and (for a free pack) an
entitlement record is listed. A positive is safe to take from a listing, since the index trails the
store. Everything else — no manifest, a missing item, a grown declaration, no listed entitlement, or
a prefetch that failed or never answered — goes to the unchanged per-pack FindNode path, so a stale
negative costs a check and never decides a write. A converged user therefore costs zero owner reads.
Pinned by StandardPacksBatchTests (Store/Installer Tests area), which counts invocations of the
per-pack path, the one place those legs are born.
A provisioning read with no verdict stays unknown
During Quality Time PR #2428 release preflight on memex.meshweaver.cloud on 2026-09-27,
a metadata search (namespace:Admin/Provision nodeType:Store/Provision, limit 3) listed
Admin/Provision/LearningRoadmap as Store/Provision, version 32, last modified
2026-09-12T12:49:55.54352+00:00. Coverage was the admin partition; the listing was truncated,
so it establishes nothing about the other requests.
Two MCP get calls for that exact path, separated by 20 seconds, returned:
Unavailable: Admin/Provision/LearningRoadmap — this read reached no verdict, so it is UNKNOWN whether this node exists. This is NOT 'not found': do not create, delete or recreate anything on the strength of it. Retry shortly. Cause: read of 'Admin/Provision/LearningRoadmap' reached no verdict within 10s
The deployed NodeType metadata then reported:
| field | observed value |
|---|---|
| path / node version | Store/Provision / 2027 |
| compilationStatus | Ok |
| lastCompileStartedAt | 2026-09-27T15:24:42.7798213+00:00 |
| lastCompileSucceededAt | 2026-09-27T15:24:50.4410742+00:00 |
| latestReleasePath | Store/Provision/Release/20260927152450-hxUA22yJ |
| latestAssemblyPath | Store_Provision/v2025-c003e001-af6f51d859cf.dll |
| compiledFrameworkVersion / compiledPlatformVersion | c003e001 / 3.0.0-ci.9464 |
| compiledModulesHash | 6ac9d6ba6277a2c7f0df13f58faadc798ba678b33a6d757c9bb3463a4c54f760 |
These are NodeType metadata readings. The instance activation's bound assembly and request-fate trail were not captured. The cause, current request existence and content, and later recovery were not established.
Controls in the same session read successfully: Admin/Provision/Signature, version 14, typed
ProvisionContent, packageId: Signature, state: Done, message ✅ Step 6/6 — Compile types;
and the private QA host rbuergi/Agent/quality-time-preview-2428-en, version 9, typed
AgentConfiguration. This was an older deployment example, so it did not currently block Quality
Time; actual provisioning was still pending CI. No request was rearmed, recycled, deleted or
recreated, and no logging configuration changed.
The observation was submitted to the control instance's bug-triage pool and read back at version 1,
Active, FeedbackContent, submitted New (the omitted-status default):
provisioning read finding.
The recorded read gap is submitted for triage. Deployment continues through the normal Store front door.