SRE Infra Command Center Kusto

An active engineering workspace where operational intelligence continuously works in the background — an elite squad embedded next to your Kusto engineer.

Not a dashboard. Not a chat wrapper. An AI end-to-end engineering platform that navigates infrastructure across MCPs, agents, deployments, incidents, code, and live signals — correlating information, surfacing decisions, and coordinating action on complex interconnected systems at a high level.

Enables system-level reliability visibility across the entire operational ecosystem: incidents, deployments, mitigations, dependencies, capacity, and operational signals — without leaving the engineering workflow.

The Agent Team

The extension runs a two-phase agent pipeline on every refresh cycle:

Phase 1 — Deterministic (instant, always runs)

Results appear in the panel and sidebar immediately — no LLM needed.

Phase 2 — LLM Investigation (async, runs via GitHub Copilot)

┌──────────────────────────────────────────┐
│  Coordinator — two-stage routing         │
│  1. Free phrase-match narrows the skill  │
│     catalog to a shortlist (instant)     │
│  2. Only if genuinely ambiguous (>1      │
│     candidate): a small LLM call picks   │
│     the best fit from that shortlist     │
└──────┬───────────────────────────────────┘
       │ Drill In
       ↓
┌──────────────────────────────────────────┐
│  🧠 Investigator (big-prompt LLM,        │
│  e.g. @SRECopilot) — runs the full skill │
│  playbook + ICC operating doctrine       │
└──────┬───────────────────────────────────┘
       │ if evidence points to another domain
       ↓ (automatic — no operator click)
┌──────────────────────────────────────────┐
│  🧠 Specialist — a REAL second LLM       │
│  session, content-matched skill + agent  │
└──────┬───────────────────────────────────┘
       │ finding merges back into the report

The Coordinator stays cheap on purpose, but isn't hardcoded. A pure keyword shortlist doesn't scale as more specialists join the roster — overlapping trigger phrases start stealing each other's incidents (a DM alert once misrouted to an unrelated Eventhouse skill on a short substring hit). So the shortlist itself is still instant and free, but picking the winner from it is a real LLM decision whenever there's more than one plausible candidate — reading the actual skill descriptions instead of just scoring keyword overlap. The same two-stage matcher (matchInvestigationSkillAsync) is used for the panel's routing display AND the actual skill a Drill-In runs.

No GitHub Copilot? The LLM steps silently fall back to the phrase-match winner — Phase 1 always runs.

Sidebar — SQUAD

The SQUAD section in the Activity Bar sidebar shows every squad member and their live status:

• Coordinator → current focus area + recommended skill
• Members → all 6 deterministic agents with live capacity/deployment status
• @SRECopilot → suggested skill from the last coordinator decision

The ACTIVITY section shows real LLM findings as they arrive, followed by the decisions log.

How the Azure Kusto SRE Agent participates in the background

The existing Azure Kusto SRE Agent (the kusto-sre-agent instance — no agent is provisioned by this extension) is wired in as a live squad member. When opsCommandCenter.sreAgentBackgroundParticipation is enabled, it participates inside the squad's analysis on each refresh cycle — not just on an explicit drill-in.

It joins the squad's knowledge bus between Round 1 and Round 2 of the LLM agent team:

Refresh tick
   │
   ▼
1. Pull live-site incidents → correlate → prioritize → rank groups
   │
   ▼
2. Coordinator routes focus, selects specialists
   │
   ▼
3. ROUND 1 — inline specialists run in parallel (capacity, deployment,
   mitigation, networking, security) → each emits a finding
   │
   ▼
4. ┌─ BILLING GUARD (the SRE Agent is a billed Azure resource) ──┐
   │  • hash the incident context (severity queue + top group     │
   │    keys + deployment state)                                  │
   │  • same hash as last consult?  ─► SKIP (no call, no bill)    │
   │  • within cooldown (default 30 min)?  ─► SKIP (no call)      │
   │  • else ► consult the SRE Agent ONCE, record hash + time     │
   └──────────────────────────────────────────────────────────────┘
   │   (the agent receives the squad's Round-1 findings + context,
   │    and replies with its own independent assessment)
   ▼
5. SRE reply folded back into the bus as:
      • an AgentFinding  → a "sre-agent" card in the panel/activity feed
      • a broadcast AgentMessage (toAgent: 'all')
   │
   ▼
6. ROUND 2 — cross-pollination: the inline specialists re-run, now
   seeing the SRE Agent's message → they can revise/escalate findings
   │
   ▼
7. Synthesis → UI updates (panel, tree, status bar)

What this means in practice

• Quiet / steady cluster — the agent is not called on routine refreshes. The context hash is unchanged, so it costs nothing; it just sits in the squad silently.
• New incident or the top group changes — on the next refresh the hash changes → the agent is consulted once, immediately. Its take appears as a sre-agent finding card, and the other specialists react to it in Round 2.
• An incident keeps churning — it is re-consulted at most once per cooldown window (sreAgentParticipationCooldownMinutes, default 30), not on every 5-minute refresh.
• A call fails or returns empty — the guard state is left untouched, so it retries on the next refresh instead of going silent for the whole cooldown.
• No az login / no agent RBAC — the agent simply isn't available to you, so it auto-skips silently: one actionable hint in the SRE Agent output channel, then quiet for the rest of the session (it auto-recovers once you az login).
• Where to watch it — the SRE Agent output channel logs every consult (timestamp, @agentName, full reply); the squad/activity panel shows the sre-agent finding card alongside the inline specialists.

Enable & tune

The integration is read-only + chat only — a control-plane ARM GET to resolve the agent endpoint, then a chat POST. It never creates, updates, or deletes the agent, and requires az login.

Skill Routing

The extension routes to skills from .github/skills/ in the repo — nothing is copied or hardcoded.

When you click Drill In or the Coordinator auto-invokes @SRECopilot:

1. Coordinator picks the right skill name (e.g. sre-regression-analysis)
2. skillRouter.ts reads .github/skills/sre-regression-analysis/SKILL.md live from your workspace
3. @SRECopilot receives: coordinator reasoning + incident context + the full skill playbook

Adding a new skill? Just add .github/skills/<name>/SKILL.md — the extension discovers it automatically, no code changes needed.

Self-healing routing — mis-routes fix themselves (auto-fix + PR)

Drill-in replies are clean investigation reports: they lead with "<GroupKey> — combined investigation (N ICMs)" and never recite the ICC's operating rules or memory entries (those are applied silently).

When an investigation discovers the group was routed to the wrong skill, that finding is not report content either — ICC fixes itself in the background:

1. The agent reports the mis-route as a hidden machine line ([ICC-ROUTE-FIX: …], stripped from the display) or via the .icc/route-fixes.jsonl bridge file on harness drill-in paths.
2. The corrected route is applied locally immediately — the next Drill In on that group uses the right skill.
3. The fix is committed to /.icc/routing-overrides.json on an auto-created icc/auto-fix/… branch of this repo, and a pull request is opened for team review. Merged overrides are pulled by every ICC install on startup.
4. The only thing the operator sees is one closing note: 🔧 Auto-fixed a discovered skill mis-route: wrong → correct — PR #<id> with the PR link.

Escape hatch: opsCommandCenter.autoFixSkillRouting: false keeps the instant local fix but skips the commit/PR.

"Same issue as before" — recurring groups are remembered

The background investigation engine re-checks the top groups on every refresh, and a new alert landing on an existing group (e.g. EngineResponseFailureV3) is a legitimate reason to re-verify it. What used to get lost: the re-verification had no memory — every new ICM made the group look "changed", so a fresh, from-scratch investigation ran and nothing on the panel indicated it reached the same conclusion as last time.

src/investigation/investigationHistory.ts fixes that with a small durable digest (survives window reloads) per group: each fresh verdict is compared against the last confirmed root cause using the same token-overlap matcher the "being handled" ack gate already trusts (rcaMatches) — no per-monitor hardcoding. A match bumps a confirm counter and the card shows a recurring ×N badge plus "🔁 Same issue as before — confirmed N× since <date>"; a genuine change of root cause resets the anchor instead of falsely claiming "same issue" (a monitor group can legitimately cover more than one root cause across different clusters — the digest only ever claims recurrence when the conclusion actually held). A group with a confirmed recurrence also gets that history handed to the next drill-in, so re-investigating starts from "here's what was found before, verify it still fits" instead of zero.

"Same issue as another active group" — cross-group linking

Distinct from recurrence above (which compares a group to its own past): two different monitor-level groups can share one real root cause right now — e.g. MachinesAvailabilityV3 and CriticalAttentionCoverage both firing because of the same East US 2 EUAP regional drill. The panel is meant to cross-link these with a same issue ×N badge, but the match used to require the two independent investigations' rootCause text to be byte-identical — which two separately-run LLM verdicts of the same event essentially never are. It's fixed the same way as everywhere else: the same rcaMatches token-overlap matcher, computed server-side (withSameIssueLinks in extension.ts) and attached to each InvestigationResult as sameIssueGroupIds, so wording drift between investigations no longer hides a real shared cause.

Splitting groups by root cause — not by monitor name

A monitor-level group (EngineResponseFailureV3) is a correlation convenience, not proof its members share a cause. The investigation now judges that directly from evidence: it looks at what each cluster's own data actually shows and, when the group genuinely spans more than one causal chain, reports one bucket per distinct root cause instead of one flattened (and often wrong) verdict — each bucket carries its own label, confidence, [SRE]/[Customer]/Unknown owner tag, affected clusters, and its own "same issue as before" recurrence (keyed by cluster set, so a monitor with several real causes tracks each one independently instead of one anchor flip-flopping between them). The panel shows a ⑂ N causes badge and a mini sub-card per bucket. Discovery is entirely evidence-driven — nothing is hardcoded per monitor.

This is a first step: ranking, ack/"being handled", DRI-pointer eligibility, and sustained-post ICM watches still operate on the whole monitor-level group for now — only the investigation card and recurrence memory are bucket-aware so far. Making the rest of the pipeline bucket-aware (each root cause its own ranked, ack-able, ICM-postable entity) is a larger follow-up.

Mitigation vs. Root Fix — every drill-in ends with both

An investigation that stops at "here's what's wrong" leaves the SRE to figure out what to do. Every drill-in report now ends with two mandatory sections:

⏱ Mitigation (temporary) — the exact runnable command (KQL / admin command / control-plane action, real cluster and database names filled in from the evidence) SRE can execute right now, always led by the verbatim warning:

⚠️ TIMELY MITIGATION ONLY — this buys time but does NOT resolve the issue. The root cause below must still be fixed.

If no safe immediate command exists, the agent must say so and why — inventing one is forbidden.

🔧 Root fix (permanent) — where the durable fix lives, discovered, never guessed. The agent extracts the concrete artifacts from its own evidence (monitor id/name, workflow name, failing component, exception strings), searches the team's ADO scopes for them (ADO code search / az devops / code-search REST), and classifies the fix locus by what the found file is — never by repo name, since product code and Geneva monitor definitions share Azure-Kusto-Service:

Every cited repo + path must come from an actual search hit. On a surface with no ADO search tool the agent outputs the exact query and scope for a human to run instead — fabricating a path is a report-invalidating error.

Where it searches is declarative, per-team config (FixScope), merged from three sources: the built-in Kusto scope, the opsCommandCenter.fixRouting.scopes setting, and the team-shared /.icc/team-repos.json on the icc/team-memory branch (same zero-setup channel as team memory). Small ADO projects (≤60 repos) additionally get their full repo list auto-discovered daily and embedded in the prompt; huge ones (msazure/One) rely on code search. Escape hatch: opsCommandCenter.fixRouting.enabled: false.

Posting to ICM — human-approved, no agent-authored scripts

When you approve posting to ICM in a drill-in chat, the agent's only job is to report structured content — it never writes, generates, or runs a PowerShell script, and it never composes HTML. (The old contract told every drill-in to "go find Post-IccNotesHtml.ps1" — a script that wasn't even bundled into the installed extension — which is exactly why sessions used to go off script-hunting or improvising one-off posting scripts.)

The agent appends one JSON line (headline, what's happening, why it happened, evidence, owner-tagged next actions, a short 2–4 word cause) to .icc/icm-post-requests.jsonl. The ICC extension picks that up and:

1. Renders the card through one fixed template (src/agents/icmCardRenderer.ts) — the same ICC-blue #4d5ece "InfraCommandCenter · Combined Investigation" header every time, never agent-authored markup.
2. Opens a review webview showing, for every affected ICM, the exact rendered card, the exact title suffix it will apply ([ICC - <cause>] — e.g. [ICC - Product bug], [ICC - False positive]), and the ICC tag it will add.
3. Writes nothing until you click "Approve & Post." On approve, one generic script (scripts/Post-IccApprovedCards.ps1) tags the ICM, appends the title suffix, and posts the card — every step read-verified, and idempotent (a retry after a partial failure only completes what's missing).

Specialist hand-offs — a real second investigation, automatically

Not to be confused with the Phase 1/2 signal agents above (Capacity, Deployment, Mitigation, Networking, Security) — those are fast per-cycle scoring passes. This is about the deep drill-in: when its evidence points to another domain, the doctrine used to only let it narrate a hand-off ("Handing off to @capacity-copilot…") — the operator had to actually type that @-mention for anything to happen.

Now the drill-in also appends one JSON line (specialist focus, reason, what it found so far) to .icc/handoff-requests.jsonl, and the ICC runs that specialist as a genuine second LLM session — the same proven, read-only-tool-safe engine that powers the headless per-refresh investigations (src/runner/handoffRunner.ts) — automatically, no click needed. The specialist's skill and its display name are content-matched from the hand-off itself (same catalog/registry every other routing decision uses — never a hardcoded focus table). Its finding attaches to the same group's investigation card as a labeled "🤝 N hand-offs" section, and a real AgentBus pulse fires so the topology panel reflects an actual second agent working.

This first pass is sequential — the primary investigation finishes, then triggers the specialist. Fully parallel, simultaneous multi-agent dispatch is a larger, separate change.

The bridge-file hand-off runs out-of-band — it only ever shows up in the panel, never in the chat. An operator reading the drill-in reply has no way to see the "🤝 N hand-offs" card unless they go look at the panel, and there is no VS Code API to inject a delayed follow-up back into that specific chat session later (the same limitation already worked around differently for the drill-in recheck pass below — sending its follow-up immediately, back to back with the first message, rather than trying to retarget a session after a delay). So as of opsCommandCenter.handoffInlineContinuation (default on), the drill-in doctrine now has a third mandatory hand-off step: after naming the specialist and writing the bridge JSON line, it keeps investigating that domain itself, right there in the same reply, with its own tools, and folds a real finding into the combined report before finishing. Both mechanisms run side by side — the bridge/panel path is unchanged and still gives you the independently-skill-matched durable record; the inline continuation is what now actually appears in the response you're reading. Set opsCommandCenter.handoffInlineContinuation to false to go back to the old narrate-and-stop chat behavior (panel-only hand-off record).

Operator Memory — the squad learns from your corrections

Every correction an operator makes is captured automatically, remembered permanently, and shared with every other operator running the ICC. Correct the squad once, and no one on the team sees that mistake again.

How corrections are captured (no manual steps)

Drill In sessions run through the @icc chat participant. Clicking Drill In opens Copilot Chat addressed to @icc (the Infra Command Center participant). Unlike raw Copilot agents, every message in an @icc session flows through the extension, so it sees the whole conversation:

1. Before answering each turn, your message is scanned for corrections ("no, actually…", "the real fix is…", "don't suggest…") and preferences ("always include…", "next time…") — matches are saved instantly, together with what the agent had said that was wrong.
2. The model is instructed to emit [ICC-LEARN: …] / [ICC-PREF: …] tags when corrected — those are captured from its answers too.
3. Explicit commands work anywhere: !learn: <fact> · !prefer: <preference> · !skip: <never suggest this> · !lesson: <environment fact>.
4. The 📝 Log button (top bar) pops a one-line input to save a correction without leaving whatever session you're in.

The same capture pipeline runs on the panel's Ask box (Coordinator bar) conversations.

Escape hatch: opsCommandCenter.drillInViaIccParticipant: false reverts Drill In to raw Copilot agents (no auto-learning).

Where memory is injected

buildMemoryPromptBlock() renders every correction, preference, lesson, and skip-list entry into the system context of:

• every @icc Drill In session (so it never repeats a corrected mistake),
• every Ask box conversation,
• the external SRE Agent hand-off prompt,
• the headless mitigation verdicts shown in the panel.

Local store: sre-operator-memory.json (+ human-readable sre-operator-memory.md) in the extension's global storage.

Team sharing — zero setup (ADO repo backend, default)

Team memory lives in this repo itself: /.icc/team-memory.json on the dedicated icc/team-memory branch of msazure/One/Azure-Kusto-InfraCommandCenter — no PRs, no branch policies, main untouched. Auth reuses the extension's existing az token flow, so anyone who can run the ICC already has access.

• Push — ~8 s after any learning is captured, a debounced sync commits the merged file (ICC team memory: +N learning(s) [skip ci]). Entries carry the author's username.
• Pull — on startup and every agent cycle, teammates' learnings merge into the local store (deduped by type + normalized text). Toast: "Learned N team correction(s) from the ICC repo."
• Conflicts — concurrent pushes refetch-merge-retry once; the regular cycle catches the rest.
• The first operator to sync auto-creates the branch (from the default branch head).

Alternative/additional backends:

Prerequisites (each teammate)

1. PowerShell 7 — https://aka.ms/powershell-release?tag=stable
2. MSAL.PS module — Install-Module MSAL.PS -Scope CurrentUser
3. ICM access for OwningTeamId 23798.

The ICM scanner scripts (Query-ICM-Raw.ps1 + Query-ICM-Api.ps1) ship bundled with the extension under scripts/, so no repo clone is required.

Install from packaged .vsix

code --install-extension sre-infra-command-center-kusto-0.2.120.vsix

Then in VS Code settings, set:

• sreInfraCommandCenter.queryScriptPath (optional) — only needed to override the bundled Query-ICM-Raw.ps1 with a custom local copy. When empty, the bundled script is used (falling back to any open workspace folder).
• sreInfraCommandCenter.pwshPath (optional) if pwsh.exe is not on PATH.
• sreInfraCommandCenter.refreshIntervalSeconds (default 300).
• sreInfraCommandCenter.adoOrgUrl (default https://dev.azure.com/msazure).
• sreInfraCommandCenter.adoProject (default One).
• sreInfraCommandCenter.adoDefinitionIds (default 278508,390266).
• sreInfraCommandCenter.commsOwnershipHints (optional) — map Teams/Email escalation matches to active owners so Priority rows show "already working on it".
• sreInfraCommandCenter.workiqEnabled (default true) — query WorkIQ for Microsoft 365 mail/Teams evidence on top groups.
• sreInfraCommandCenter.workiqLookbackDays (default 20) — only search the last 20 days.
• sreInfraCommandCenter.workiqTopGroups (default 3) — enrich only the top few groups each refresh.
• sreInfraCommandCenter.workiqAccount (optional) — specific M365 account to use with WorkIQ.
• sreInfraCommandCenter.workiqTermMappings (optional) — add extra WorkIQ search terms for a monitor. Default includes TridentOperationStateSev2 -> PowerBIMetadataWorkspaceArtifactsQuotaExceededException.

Example:

"sreInfraCommandCenter.commsOwnershipHints": [
  {
    "pattern": "PowerBIMetadataWorkspaceArtifactsQuotaExceededException",
    "owners": ["alias1", "alias2"],
    "source": "teams",
    "note": "Escalated in Trident Sev2 bridge"
  }
]

WorkIQ example:

"sreInfraCommandCenter.workiqLookbackDays": 20,
"sreInfraCommandCenter.workiqTermMappings": [
  {
    "match": "TridentOperationStateSev2",
    "extraTerms": ["PowerBIMetadataWorkspaceArtifactsQuotaExceededException"]
  }
]

When WorkIQ finds evidence, Priority rows can show:

• owner already handling it
• latest evidence line
• latest email subject
• latest message/email link in drill-in context

The Comms Agent and the "being handled" lifecycle: owner verification feeds a new squad member — the Comms Agent (CM orb, after Priority Agent) — which hands per-group handling verdicts to the Priority Agent over the agent bus. When a verified human author stated a real root cause in their own message AND ICC's own drill-in agrees with that diagnosis, the group is auto-acked into a dedicated Being handled section: purple work-in-progress styling, "still open" chip, since-when timestamps, the owner's quote, and re-check/expiry countdowns. Severity and score are untouched — the card never leaves the list — but the pulsing DRI — take this first pointer, mitigation prompts, and escalations all skip it. A verified owner without a stated RCA (or someone who merely acked the ICM ticket) stays in the active queue. Handled is never forever: a fingerprint watch (blast radius: incident ids/clusters/regions/ severity/count) snaps the card back on drift with a changed badge, and after handledAckExpiryHours (default 24) of owner silence the ack expires — the re-check carries the last known root cause so WorkIQ answers "is this still the same issue?". Acks are team-shared via the icc/team-memory ADO branch (/.icc/team-acks.json), so every operator's ICC agrees who is handling what; Un-handle on the card revokes team-wide. Settings: sreInfraCommandCenter.handledRecheckHours (default 4), sreInfraCommandCenter.handledAckExpiryHours (default 24).

Owner verification (automatic + in-card): after each refresh the squad auto-verifies the owner of every Sev≤2 group plus the top workiqTopGroups ranked groups — the full owner-lookup doctrine runs through the WorkIQ CLI and the verdict is painted directly onto the card: Owner verified: <primary> is on it · also engaged: <names> plus per-person evidence lines (Teams channel / email subject + a human quote). The goal is deduplication: before jumping on an issue, see who is already actively working it (a name in automated IcM fields does not count — only real human-authored messages). Cards being checked pulse "Verifying owner…"; verified verdicts persist across refreshes (20-minute cache) and the Verify owner button (→ Re-verify once verified) forces a fresh lookup for any card at any time. Auto-verification signs in at most once per session; if WorkIQ isn't signed in it stands down and the buttons remain. If a manual verify fails (npx missing, unparseable answer), it falls back to the previous behavior — an owner-lookup prompt handed to Copilot Chat. Settings: sreInfraCommandCenter.workiqAutoVerifyOwners (default true) toggles the automatic pass.

Deployment sync uses non-PAT auth:

• az login
• az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 or set AZDO_ACCESS_TOKEN in your environment.

The panel auto-opens on startup; the status bar item $(pulse) SRE Infra Command Center reopens it.

Build from source

cd sre-infra-command-center-kusto
npm install
npm run compile        # tsc -> out/
npm run package        # vsce -> sre-infra-command-center-kusto-<ver>.vsix

Use F5 for the Extension Development Host while developing.

Pipeline

1. powershellRunner → invokes Query-ICM-Raw.ps1, parses JSON
2. adoRunner → polls ADO Build APIs for in-flight/recent deployment runs
3. correlationAgent → groups by monitor / region / cluster / service
4. priorityAgent → ranks groups by count × severityWeight
5. liveSiteAgent → builds top-line summary, flags emergency mode
6. mitigationAgent → suggests next steps for the #1 group (no action taken)

When ADO sync is enabled and at least one rollout is in flight, the decision card switches to "Approve next rollout" and opens the run page for operator approval.

All agents are pure TypeScript. Nothing is sent to any external API beyond ICM itself; drill-in / "ask the squad" hands off to Copilot Chat in the sre-infra-command-center-kusto chatmode.

Customizing for another team

Edit scripts/Query-ICM-Raw.ps1 — change OwningTeamId and the noise regex. Update build/release IDs referenced by the sre-infra-command-center-kusto chatmode (278508, 390266) for your service.

Link your team's repos (fix routing) — one declarative entry, no code:

// settings.json (or commit the same array to /.icc/team-repos.json
// on the icc/team-memory branch to onboard your whole team at once)
"opsCommandCenter.fixRouting.scopes": [
  {
    "team": "MyTeam",
    "orgUrl": "https://msazure.visualstudio.com",
    "project": "MyProject",
    "repoHints": ["My-Service", "My-Monitors"],          // searched first — optional
    "notes": "Monitors live in My-Monitors/geneva; automation in My-Service/ops"
  }
]

That's the entire onboarding: which repo holds the failing code, which file is the monitor definition, and whether a fix is SRE-ops or dev-backlog are all discovered by the agent at investigation time via ADO code search — there is no per-monitor or per-alert mapping to maintain, and projects with ≤60 repos are even auto-inventoried daily so the agent starts with the full repo map.

Commands

Settings

ICC Attribution — proving the value the command center delivers

Every piece of work the command center does is attributed so leadership can see — in hard numbers — how much engineer effort it replaced. Attribution is non-destructive by default: it is recorded as machine-readable telemetry regardless of whether ICC ever writes to a live ICM.

The seven contribution types

Each ICC interaction is stamped with one or more contribution types. Each maps 1:1 to an ICM keyword tag and to an estimate of the manual minutes / actions it replaced:

A single investigation accumulates types as it flows through the pipeline (correlate → prioritize → recommend → orchestrate → auto-route), and the same stable InvestigationId / CorrelationId / OrchestrationRunId is carried end-to-end — so the same incident is never double-counted.

ICM tagging — idempotent and additive

When opsCommandCenter.writeIcmTags is enabled, a user-initiated drill-in stamps the ICC tags onto the live ICM's Keywords via scripts/Tag-ICM.ps1:

• Idempotent — an ICM already carrying ICC:Routed is never tagged with it again; the PATCH is skipped entirely when nothing changed.
• Additive — every pre-existing keyword (ICC or not) is preserved; ICC only ever appends.
• Gated — off by default (it mutates production ICMs), and only the explicit drill-in writes; the background refresh loop never does.

When the flag is off, ICC still records the exact same tag set in telemetry, so the leadership reporting below works either way.

RoboKOP — Scenario 1 (Customer Inactivity)

RoboKOP owns the customer-reported ICM queue (the oldest 5 tickets, briefed each refresh — see src/agents/robokopAgent.ts). Beyond resolving owner/whyStuck for the queue card, it now performs a real ticket-resolution action itself, using the same ICM tool access it already uses to resolve owners by opening each ticket — ICC does not write to ICM for this feature, it only instructs RoboKOP in the brief and displays what RoboKOP reports back:

• RoboKOP validates the conditions — for each ticket it reads the ICM Discussion timeline and reports whether it is waiting on a customer reply and, if so, since when (customerWait in its reply). When idle ≥ opsCommandCenter.robokopInactivityThresholdDays (default 5) days, the brief instructs it to act.
• Tags the ticket ICC_ORC_Robokop (additive — keeps every existing keyword).
• Resolves the ICM.
• Posts a Discussion comment explaining the automatic closure and inviting the customer to open a new ticket if the issue recurs.
• Appends a title suffix — [ICC - Resolved waiting on customer more than N days] — keeping the existing title text before it, same [ICC - <cause>] convention the ICM-posting bridge (icmPostBridge.ts) and Register-IccWatch.ps1 already use elsewhere.

RoboKOP reports resolved: true per ticket only when it actually completed all four actions; ICC reads that back and shows a green "✓ auto-resolved by RoboKOP" line on the queue card. Gated by opsCommandCenter.robokopAutoResolveInactiveTickets (default on) — when off, the Scenario 1 instruction is simply not included in the brief and RoboKOP stays read-only triage.

Query the value over MCP

The bundled MCP server exposes the attribution roll-up so any agent (or a dashboard) can read it without the editor:

The same summary is also embedded in every persisted snapshot (agent-snapshot.json → iccTelemetrySummary), and the cumulative counters live in a truncation-proof sidecar (icc-telemetry.totals.json) so the all-time totals survive log rotation.

Three measurable outcomes you can show leadership

1. Engineer time saved (headline KPI). engineerTimeSavedMinutes — cumulative minutes ICC replaced, summed from the per-contribution estimates above. Report it as a daily/weekly total and break it down by contribution type to show where the savings come from. Durable across log rotation via the totals sidecar.

2. Manual actions eliminated. engineerActionsReduced — the count of discrete manual steps (clicks, lookups, correlations, hand-offs) ICC performed for the engineer, broken down by workflow type (pipeline, drill-in, auto-invoke). This is the "before vs after" action-count comparison.

3. Duplicate investigations prevented. duplicateInvestigationsPrevented — because every investigation keys on a stable InvestigationId, ICC detects when the same incident would have been triaged twice and suppresses the re-run. This counter is the direct evidence of wasted, redundant triage the command center removed.