The sync/ Hub Population

A 26 h production replica held 9 398 MessageHub instances, of which 6 925 were sync/ hubs at RunLevel=Started — about 2.7 GB of a 3.5 GB live heap (#3432). This page is what the code says about where they come from and what is supposed to take them away. It is a narrowing, not a diagnosis: the sections marked 🚨 NOT ESTABLISHED are still open.

🚨 The cause was established on 2026-09-10, by experiment on a live replica: the READ path minted one sync/ hub per GetDataRequest. See The Read Path Minted a Hub Per Read, which also carries the live census that decomposes the population into its holders — and re-scopes the evicted-stream mechanism to 1 parked stream in 475.

1. A sync/ hub is not a population — it is one field of a stream

SynchronizationStream<T>'s constructor mints exactly one, unconditionally, for every stream:

// src/MeshWeaver.Data/Serialization/SynchronizationStream.cs
var syncHub = Host.RunLevel > MessageHubRunLevel.Started
    ? null
    : Host.GetHostedHub(
        SynchronizationAddress.Create(ClientId), ConfigureSynchronizationHub, HostedHubCreation.Always);
…
Hub = syncHub;
syncHub.RegisterForDisposal(_ => ReleaseHub());

And one cross-hub subscription builds two streams, hence two hubs:

Both carry the same ClientId, so the two hubs share an address string while living under different parents.

This settles the first of #3432's open questions. The dump's own arithmetic agrees:

6 925 Started sync/  +  1 495 Dead sync/  =  8 420
SynchronizationStream total                =  8 461     (0.5 % apart)

So there is no separate "sync hub leak" to explain. The question is entirely "why are there 8 461 live SynchronizationStreams", and each one costs its own Autofac ILifetimeScope and TypeRegistry (~390 KB, of which ~173 KB is framework metadata duplicated per hub).

2. What retires one — three paths, and only one of them applies here

🚨 This table is about the STREAM's lifetime. The registry-level account — which collection roots a sync/ hub, what takes it out, and which (RunLevel, IsDisposing) states can persist there — is §8 below, written against the code after the retire-and-replace change, and it is the half that says which of these hubs cannot be removed versus is removed only when X happens.

The middle row is #3321 step 3, and it is why the Dead bucket exists at all. It does not touch a hub whose parent is alive. So for a Started sync/ hub under a live parent — which is this issue's entire 6 925 — the idle sweep is the only reaper in the process.

MeshNodeStreamHandle.DetachUpstreams() is reachable from nowhere else, and it is also the only thing that collects the change-feed-parked predecessors described below.

3. The sweep's predicate, and why traffic defeats it

// src/MeshWeaver.Hosting/MeshNodeStreamCache.cs — Entry
public bool TryMarkIdleEvicted(TimeSpan idleWindow)
{
    lock (gate)
    {
        if (evicted || subscribers > 0
            || Environment.TickCount64 - lastActiveAt < (long)idleWindow.TotalMilliseconds)
            return false;
        evicted = true;
        return true;
    }
}

Window 10 minutes, swept every minute (MeshNodeStreamCacheOptions). Two conjuncts, and the option's own summary states what refreshes the second: "Every GetStream subscription, unsubscription and Update on the path refreshes the window."

🚨 The subscribers count is NOT the Rx-subscriber trap Workspace warns about. That warning ("the reduce chain subscribes to the stream ITSELF, so an evicted stream measures 2–3 subscribers and never reaches zero — implemented, measured and reverted") is about a different, abandoned mechanism at the workspace layer. Entry.subscribers is a purpose-built refcount moved only by TryAddSubscriber/RemoveSubscriber from the cache's own SharedView, and the internal hydration subscription bypasses it. It genuinely can reach zero. Anyone reasoning about #3432 from the Workspace comment alone will mis-attribute this.

What actually holds the sweep off is the other conjunct.

4. The change-feed orphan — the write both creates it and postpones its collection

🚨 Since #1174 this runs only for change events no mirror can be held to (a delete, a recreate, a version-less recycle broadcast); a versioned Updated keeps the stream — see Live Mirrors and the Change Feed. The measurement below predates that.

Workspace.EvictForPath runs on every change-feed event for an owner path and removes every identity's cached stream for that owner:

_evictedRemoteStreams[removed.Value] = 0;
ReclaimIfUnheld(removed.Value);        // disposes ONLY if no declared lease remains

The mesh-node cache's hydration is a declared holder for the entry's whole life — that is deliberate, and MeshNodeStreamHandle.Subscribe says so: "The lease lives exactly as long as this subscription. The shared mesh-node cache's hydration comes through here, so its entry IS the declared holder of the path's upstream for the entry's whole life — which is what makes every OTHER (write-scoped) stream for the same path reclaimable on eviction."

So on the first write to a cached path:

1. the entry's stream S1 is removed from _remoteStreamCache and parked;
2. ReclaimIfUnheld(S1) finds the hydration's lease and does not dispose it;
3. nothing re-points the hydration — the sweep "only ever CLOSES idle entries; it never re-subscribes anything (the 2026-06-08 rule)" — so the entry keeps reading S1 forever;
4. the next reader or writer misses the cache and builds S2, which becomes the cached one.

From then on the path holds two live streams — four sync/ hubs, ~1.5 MB — where the design intends one. Write-scoped streams still churn correctly (an unleased evicted stream is reclaimed at once), so the accumulation is a floor of one orphaned pair per cached, written path, not growth per write.

🚨 And the same act does both halves: the Update that fires the change-feed event which orphans S1 is also an Update that Touch()es the entry, pushing the 10-minute window out. A path written at least once every ten minutes can never be released, so its orphan can never be collected. The only reaper is switched off by the traffic that feeds it.

That is the answer to "is the retirement path missing, or present-but-not-firing?" — present, and structurally unable to fire on exactly the paths that generate the garbage.

5. Bounding

Neither cache has memory-pressure eviction or any other TTL:

• Workspace._remoteStreamCache — a plain ConcurrentDictionary keyed by (Owner, Reference, Identity), so a path read under N identities holds N streams.
• MeshNodeStreamCache._streams — a plain ConcurrentDictionary keyed by path. (The MemoryCache in the same file is the write-queue's, not this one's.)

So the live-stream count is bounded only by (distinct paths × identities) touched since process start, with the per-path floor above added on top.

6. 🚨 What is NOT established

🚨 Partly superseded. The Evicted-Stream Retention closes the "what holds them" half for one named mechanism, with a controlled two-arm test rather than a dump: a change-feed eviction parks a remote stream, and ReclaimIfUnheld returns without disposing whenever the stream carries no lease entry at all — so every unleased call site (including LayoutExtensions.GetControlStream, i.e. every rendered layout area) retains one stream, and two sync/ hubs, per change event. It also rules out static state and timers as roots, and replaces the referrer walk below with two field reads on the singleton Workspace. What stays open is the magnitude, not the mechanism.

• Whether the 8 461 are garbage. A portal with thousands of open Blazor circuits, each holding live layout-area streams, is supposed to look like this. Sections 1–5 explain the shape of the population, not that it is waste.
• The split between the per-path floor of §4 and legitimately-live streams. That is the number that decides whether #3432 is a leak with a fix or the per-hub DI cost the issue's own third outcome row describes.
• Whether #3427 moved this number. Still not re-measured; no deployment carried f41f8bda at the time of the last attempt.

7. The discriminator — a ratio, from data the existing plan already collects

#3432's plan calls for a ClrMD referrer walk, which is the expensive half and needs a dump that (measured) restarts the replica. §1 makes a much cheaper reading decisive, and it comes off dumpheap -stat, which that plan already takes:

count MeshNodeStreamCache+Entry alongside SynchronizationStream.

No referrer walk, no field traversal, no second dump — one extra line of a histogram the plan already produces. 🚨 It does not replace the referrer walk for the third row; it tells you whether you need one.

8. The registry account: what roots a sync/ hub, and what takes it out

Sections 1–7 are about the STREAM. This section is about the HUB, read off HostedHubsCollection and MessageHub as they stand today. It answers the question the dump's histogram poses — what can a Started sync/ hub be? — by construction rather than by inference, and it distinguishes cannot be removed from is removed only when X happens.

8.1 One root, and it is the parent's registry

A sync/{clientId} hub is a HOSTED hub. There is exactly one creation site (§1), and every hosted hub enters its parent's registry through one method:

// src/MeshWeaver.Messaging.Hub/HostedHubsCollection.cs — Track
private void Track(IMessageHub hub)
{
    messageHubs[hub.Address] = hub;
    hub.RegisterForDisposal(h =>
        messageHubs.TryRemove(new KeyValuePair<Address, IMessageHub>(h.Address, h)));
}

messageHubs is an instance ConcurrentDictionary<Address, IMessageHub> on the collection, whose lifetime is the parent hub's. There is a second root beside it, retiring, a reference-keyed dictionary of hubs taken out from under their address but still tearing down; Hubs enumerates both, which is why a census walking Hubs sees a retired hub too.

8.2 Three removals, all of them self-driven

Nothing else removes a hosted hub from its parent's registry. There is no sweep, no idle eviction, no cap, and nothing anywhere asks "is this hub still needed". Removal 1 is the hub disposing itself; removals 2 and 3 only move a hub that is already disposing. So the registry is a pure consequence of who calls Dispose(), and for a sync/ hub that is: the stream's own Dispose(), the parent's teardown, or a routed DisposeRequest — §2's table.

🚨 Removal 2 is lookup-triggered, not self-triggered. Retire-and-replace made the hand-out safe; it did not make the registry self-cleaning. A hub wedged at ShutDown that no Always lookup ever asks for again stays in messageHubs for the parent's life. That is #4883's wedge showing through, deliberately left there.

8.3 The (RunLevel, IsDisposing) cross-tab — which cells can persist

IsDisposing is disposalStarted, set as the FIRST statement of Dispose(), before it posts ShutdownRequest(Quiescing). The RunLevel only moves when the action block dequeues that request.

8.4 A second retention class the histogram cannot see: the registrant composite

A hub's registered cleanups live in one CompositeDisposable:

// src/MeshWeaver.Messaging.Hub/MessageHub.cs
private readonly CompositeDisposable disposables = new();
public IMessageHub RegisterForDisposal(IDisposable disposable)
{
    disposables.Add(GuardRegistrant(disposable));
    return this;
}

A plain registration is append-only. CompositeDisposable.Add never prunes, and nothing removes an entry added through RegisterForDisposal, so such a registrant is held — with everything its closure captured — for that hub's whole life. The ONE way out before the hub dies is a DETACHABLE registration (RegisterForDisposalDetachable, or SubscribeHeldUntilTerminal built on it): its handle removes the entry without disposing it, and it is to be used only once the registrant's disposal can no longer do anything (see the table below). A PLAIN registrant whose own subject is SHORTER-lived than the hub is therefore a monotone root, and it retains an object graph rather than a hub, which is why no MessageHub histogram can see it.

MessageHub.DisposalRegistrantCount is the reading that can, and it is printed in both disposal diagnostics as Registrants=. A hub whose registrant count climbs with the traffic it has served is retaining one object graph per unit of that traffic.

Two classes, both measured in MeshWeaver.Data.Test:

🚨 Still held by design, and NOT a leak this fix touches: a GetDataRequest over a LIVE workspace reference keeps shipping every change to its requester, so its subscription — and its registrant — live until the owner goes down. Whether anything should end such a read earlier (the requester's own Observe completes on the first answer) is a separate question, not measured here. A read that ends EMPTY on a healthy hub also stays registered on purpose: its caller is still owed the teardown NACK.

Related

• Stream Liveness and the Hub Reference — #3321 step 3, the parent-teardown hook that reclaimed the Dead half
• MeshNode Stream Cache — the entry, the refcount and the idle sweep
• Portal Heap Is Hubs — the five dumps, the measurement hazard, and the ClrMD recipe
• The Evicted-Stream Retention — the named retainer, the controlled experiment that established it, and the two-field-read measurement that closes #3432