NOC incident correlation, 503 under load

Context

An NOC agent correlating a network incident hits a 503 on the topology service, overloaded by the very incident it's diagnosing. Retry with backoff, then fall back to the cached topology, flagging staleness.

The decision

Resilience here is a network SLA duty, not a nicety, the tools you depend on are least available exactly when the incident is worst, so the recovery policy is the design.

What most miss

Demos assume tools are up. In a real incident the dependency is overloaded precisely when you need it; the backoff and cache path is what keeps the agent useful under load.

Stakes

An agent that stalls on a 503 mid incident extends the outage it was meant to shorten.