incident-response-trainer
Incident response training · Rule-based scoring
DemoCatalogDiagnosticHistoryDashboardCoach Notes
Incident

Building 3 access switches at 99% CPU — intermittent connectivity loss

Network EngineeringDifficulty · Easy
← New scenario
Retry in progress
You have 1 previous attempt for this scenario. Submitting again will create a new attempt and show a comparison against your most recent response.
STP LoopDifficulty · easyHigh asset
[NOC ALERT] DIST-SW-03 CPU 99% — Bldg 3 users reporting drops
From
Network Monitoring <noc-alerts@acme-corp.local>
To
netops@acme-corp.com
Date
2026-04-19 11:08 UTC
Automated monitoring alert. DIST-SW-03 (Building 3 distribution) is running at sustained 99% CPU since 11:02 UTC. Users in Bldg 3 are reporting that network drops for ~5 seconds every 10–20 seconds, then comes back. Other buildings look fine. Latest config change on this switch was 9 days ago (port channel to ACCESS-SW-307 added). Please triage. — Network Monitoring
Evidence
DIST-SW-03 console + counters
# Last 20 lines of console
%STP-2-LOOPGUARD_BLOCK: Loop guard blocking port Gi1/0/24 on VLAN 10
%SPANTREE-7-RECV_TC: Received topology change BPDU on VLAN 10 (port Gi1/0/24)
%SPANTREE-7-RECV_TC: Received topology change BPDU on VLAN 10 (port Gi1/0/24)
%MAC-4-FLAP: Host 0011.2233.4455 flapped between Gi1/0/12 and Gi1/0/24 in VLAN 10
%MAC-4-FLAP: Host aabb.ccdd.eeff flapped between Gi1/0/9  and Gi1/0/24 in VLAN 10
... (TCN floods every ~3s) ...

# show processes cpu
CPU utilization for five seconds: 99%/45%; one minute: 96%; five minutes: 94%
PID  Runtime(ms)  Process            5Sec   1Min   5Min
116  3812440       Spanning Tree     58.4%  56.2%  55.8%
105   994320       HLFM-process      18.1%  17.4%  16.9%

# show interface Gi1/0/24
Gi1/0/24 is up, line protocol is up (connected)
Hardware is iGigabitEthernet, address is bcd1.7f00.1801
Last input never, output 00:00:00
Input queue: 982/2000/24/0 (size/max/drops/flushes); Total output drops: 0
... 60-second input rate 480000 bits/sec, 740 packets/sec ...
Affected asset
Name
DIST-SW-03 (Cisco Catalyst 9300)
Type
Distribution switch — Building 3 access layer
Owner
Network Operations · Building 3
Level
High
0 words

Grading is rule-based. Response is compared against a pre-written rubric.