Incident identity — who computes it, and what happens to the corpus when it changes
Systemorph/MeshWeaver.Plugins#1796, and the half #1795 deliberately left out.
An incident's fingerprint is its node id (Admin/_LogIncident/{fingerprint}). So the process
that computes it decides, for the whole corpus, what counts as "the same error happening again" —
and until this change that process was mw-log-watcher, an image with its own delivery lane.
What was measured
On the control instance, 2026-09-16/17, read-only:
| incident | reproduced from | still folding |
|---|---|---|
c399f09e8765b7ed (MeshWeaver#2387) |
SHA256("MeshWeaver.PluginCatalog.InstanceAutoRegistrationService\n0\n")[..8] |
2026-09-16T22:54:35Z |
b03482717d5ba39a (MeshWeaver#3659) |
same formula, …InstalledPackageRepairService |
2026-09-13 |
05f47831e3b7d6cd (MeshWeaver#2157) |
same formula, ProvisionPlan |
2026-09-13 |
That formula — category + eventId + exceptionType, no message — is the 2026-08-09 identity.
Four revisions later (#1002, #1170/#1171, #1787, #1795) the corpus was still addressed by it, which
has two consequences and they pull in opposite directions:
- Nothing splits. One incident per (category, exception type) means two unrelated defects raised from one class share a ticket forever. MeshWeaver#2387 was closed and reopened four times on occurrences of a different call site in the same log category.
- Nothing lands. Every correction to
StructuralLogIncidentIdentitywas invisible where it mattered, and each correction that DID reach an image re-addressed the corpus: the same fault starts computing a new id, opens a new incident and a new GitHub issue, while the old node sits open on a key nothing will ever compute again.
🚨 And a legacy node's identity cannot simply be recomputed offline. Admin/_LogIncident/9b70b639c4e77af3
— still folding bursts on the morning of 2026-09-17 — is reproduced by none of the plausible
payload shapes over its own stored category / exception / message / frame: the mint-time masking and
the fields the node retains have drifted apart. A sweep that recomputes the corpus is therefore
guessing, which is why the migration here is event-driven instead.
What changed
1. The portal resolves the identity, on ingest
LogIncidentIdentityResolution.Resolve(report, identity) runs in LogIncidentIngestService.Report.
The node is named after the PORTAL's answer; the reported fingerprint stays on the wire as the
reporter's key. A corrected identity now ships with the portal, which rolls continuously, and a
Code node implementing ILogIncidentIdentity finally overrides something — the extension point was
declared in the contract but nothing on the portal side consulted it.
Two shapes are never recomputed:
- A well-known id (
log-burst-header-only-{ns},log-ingest-refused-{ns}, …) — the pipeline's own findings are hand-written constants that BOTH sides synthesize, so recomputing one would fork it per reporter vintage. They are recognised by shape: a computed identity is always 16 lowercase hex characters, so the rule needs no new field on the wire and works for a watcher of any age. - A site fold (
Variants > 1) — the aggregator already collapsed a site that fanned out past its budget; splitting it here would open the fifty tickets the fold exists to prevent.
LogIncidentReport.EventId was added so the identity's site branch can be reproduced portal-side. A
watcher that predates the field sends nothing and it defaults to 0, which is what all but a handful
of log sites emit.
2. The legacy incident is CARRIED, not orphaned
When the reported id differs from the resolved one, the reported id names the legacy incident —
exactly, with no recomputation. On the first burst that lands on a successor,
LogIncidentIngestService.MigrateLegacy looks the legacy node up (an existence LISTING, never a
point read of a node that may be absent) and LogIncidentCorpusMigration does two writes:
- Fold the legacy history into the successor: occurrences, window, pods, samples, the shape ledger — and, only into an empty seat, the ticket (issue number and URL, repository, comment budget, draft, triage thread). One fault keeps ONE issue across the re-addressing, which is what stops a burst of duplicate issues when an identity revision lands.
- Supersede the legacy node:
Status = Superseded,SupersededBy = <successor>, nothing requested. It is kept, not deleted — it is the audit trail its GitHub issue was opened from — and the control plane never triages, files or comments from it again.
Idempotent, and it never mints a third node. FoldedFrom on the successor is the idempotency
key: a replayed report folds nothing a second time, so an inherited occurrence count cannot be
double-counted. The only nodes a migration touches are the legacy one and its successor. A legacy
node already superseded by a different successor contributes its relation but not its numbers.
At most one listing per incident, ever. The gate is the successor's stored
ReporterFingerprint: a node that has one has already been through here.
3. A bucket can be answered — the per-shape ledger
Re-addressing fixes the future; it does not make the incidents that ALREADY cover several defects
answerable, and a site fold is a bucket by design. LogIncident.Shapes is a bounded ledger
(12 rows, ShapesEvicted counts what fell out) keyed by what the current identity function computes
for each burst ALONE — so each row is the id that shape would have if it were split — with its own
FirstSeen / LastSeen / Occurrences.
That is the property that lets an issue be closed with evidence: an incident's own LastSeen
belongs to whichever shape fired last, so on a bucket it cannot say whether the defect the issue is
about has stopped. The shape's row can. The filer prints the table on the issue and on every
recurrence comment, with the sentence that reads it correctly.
3a. The shape key sees a path SHAPE the identity deliberately masks (Plugins#2770)
Awaiting the Observability owner's choice. This is option 2-variant of the three in #2770 (a split action; a path-shape discriminator in the identity; document per-sample reading). It changes the SHAPE key only, so it re-keys no incident.
Keying rows on the identity alone was not fine enough. Admin/_LogIncident/3c9841dde50bfad4 held
ten samples of one masked message that were two defects — three of the every-path-missing install
(Plugins#2173) and seven of the rbuergi/Agent/voice ≠ Voice casing collision (MeshWeaver#4817,
fixed by Plugins#2188) — and its ONE shape row's lastSeen (2026-09-20) reported the fixed defect,
while the unfixed one last fired on 2026-09-19. The identity masks every path and quoted literal, so
both normalized to the same text.
LogIncidentIdentityResolution.SplitIdentity now reads one more thing from the raw line:
StructuralLogIncidentIdentity.PathShape. Its atoms are every quoted literal and every path segment;
when two are equal ignoring case and unequal ordinally ('Voice' vs …/voice) the line is labelled
case-only-path-collision. Prose words are not atoms, and id-shaped atoms are skipped by the
parser's own id thresholds (a guid, 16+ hex, or 8+ hex with a digit AND a letter) — so a word such as
Cafe against …/cafe is still a collision. The label is a constant, so no path text reaches a key.
One report can hold both defects. The watcher groups bursts by the path-blind fingerprint, so a
single window can put both shapes into one report, and its few samples cannot count them apart. The
watcher therefore sends LogIncidentReport.ShapeCounts: one entry per shape it saw, counted over
EVERY burst, with that shape's window and latest line. The portal re-derives each entry's key from
that line and RecordShapes books one row per shape with its own count. A report from a watcher
that predates the field falls back to one vote over the samples — the structural key first, then
the label among that key's samples — and books the whole report on the winner.
- No label (every line before this change, and every report with no parsable sample) → the key is exactly the identity function's answer, as before. Existing rows keep receiving their lines, and that key is still the successor's id the day the incident is split.
- A label →
DiscriminatedShape(key, label), a 16-hex token in its ownshape\n…namespace. It passesIsComputedIdentity, so a successor can be named after it, but it is NOT what the identity function answers for the burst: a split onto it must mint that id, not recompute one. The new row's detail is prefixed[case-only-path-collision]so the two rows are tellable apart in the issue table.
The incident id (Resolve) is untouched; AShapeKeySeesACaseOnlyPathCollisionTest pins both the
split and the unchanged key 3c9841dde50bfad4 for the undiscriminated lines.
What this does NOT do
- It does not migrate quiet legacy nodes. A legacy incident nothing reports any more is never
visited, because the move rides the burst that proves the fault is still live. That is deliberate:
a node nothing fires is, by definition, the closable case — its
LastSeenis the evidence — and minting an empty successor for it would add a node and a ticket for a fault that stopped. - It does not rename anything in place. Node ids are never rewritten;
issueNumber, the triage thread path and the bot's comment history stay attached to the node that earned them until they are deliberately carried. - It does not close the concurrent file-or-fold race (two issues for one fingerprint in the same second, MeshWeaver#4463/#4464). That is a third mode on the same thread and needs its own change.
- 🚨 It moves the HASH onto the portal's delivery, not the NORMALIZATION the hash is taken over — see the section below, which is the half of #1796's argument that is still open.
🚨 The masking is still watcher-gated, and that is the same defect one layer down
Resolve recomputes the hash. It does not — and cannot — recompute the text:
// LogIncidentIdentityResolution.ToBurst(report)
return new LogBurst(
report.Category, report.EventId, report.Severity,
report.NormalizedMessage, // ← the WATCHER's LogLineParser.Normalize output
report.NormalizedMessage,
report.ExceptionType, report.TopFrame, report.NormalizedDetail);
The report arrives carrying NormalizedMessage already masked, by the watcher's copy of
LogLineParser.Normalize. The portal has the raw sample lines on the report but never re-derives the
normalized form from them, so every masking rule is still delivered by the watcher image —
Path, MixedHexId, EpisodeStamp, SpacedSubject, SlotPlaceholder, all of them.
That matters because the masking rules are where the folding is decided, and they are the ones that
keep changing. Measured 2026-09-19: the four rules that collapsed the 52-issue flood
(Plugins#2165/#2167/#2168, and #2166's two residual gaps) all live in Normalize, so none of them
can fold a single production burst until mw-log-watcher is rolled — and Plugins#2156 measures that
the deployed watcher was still running pre-2026-08-25 code on 2026-09-19, three fingerprints
reproducing from the 2026-08-09 payload formula seven seconds after their own log lines. Nothing rolls
that image: its lane publishes latest and says so in its own summary ("the Deployment adopts it on
its next roll — latest moving is delivery, not a restart"), and the watcher has no
Hosting/Deployment record, so the sanctioned InstanceAction route does not reach it either.
So the last bullet under Reading a corpus that is mid-migration is true only of the hash. A
ReporterFingerprint != Fingerprint costs nothing for addressing; it still costs the whole of the
masking, and therefore the fold.
Two candidate repairs, neither taken here, and the choice is a scope call:
- Re-normalize portal-side from the raw lines the report already carries, which puts the masking
on the same continuous delivery the hash now rides. The risk is precisely the one this page warns
about above — the sample lines on the wire are a capped sample, so a portal-derived
NormalizedMessageneed not equal the watcher's, and any divergence re-addresses the corpus. The per-shape ledger andMigrateLegacyalready absorb a re-addressing, which is what makes this thinkable at all. - Give the watcher a delivery lane — a
Hosting/Deploymentrecord and a Roll, so a masking fix reaches production the way every other fix does. Smaller change, and it fixes the class (Plugins#2156 is the same gap wearing a different symptom) rather than one consequence of it.
Reading a corpus that is mid-migration
- A node with
SupersededByis history; read its successor. - A node with
FoldedFrominherited one or more legacy nodes; its counters include theirs. ReporterFingerprint != Fingerprintmeans the reporting watcher is running an older identity generation than the portal — which is normal and no longer costs anything for addressing. It still costs the masking: see "The masking is still watcher-gated" above.