Event-Driven AI Incident Triage

A large public-sector platform running hundreds of production services had a manual incident loop: an alarm fires, an engineer hunts through logs and runbooks, works out a probable cause, and raises a ticket — most of it retrieval, not judgment. I built an event-driven pipeline where a CloudWatch alarm triggers parallel log and knowledge retrieval, feeding an LLM-backed root-cause analyst that drafts a structured ticket with citations back to the specific log lines and runbook sections it reasoned from.

The interesting design decision was not using an autonomous agent to decide what to retrieve — the retrieval steps (logs, runbooks) are known in advance and independent, so a fixed parallel fan-out is faster, cheaper and simpler to test than a model reasoning its way to the same two calls. The model earns its place at the synthesis step, where it’s genuinely needed: turning heterogeneous evidence into one coherent diagnosis.

Time to a drafted ticket dropped from over 30 minutes to under 15 seconds. The system has no action authority — it writes a ticket, it can’t touch infrastructure — so the blast radius of a wrong diagnosis is a human reading a bad ticket and discarding it, not an automated action taken on bad information. Validated resolutions feed back into the knowledge base, so diagnostic quality compounds over time.