Building 3 access switches at 99% CPU — intermittent connectivity loss

From: Incident Response Lead / Network Incident Lead
To: IT Leadership · Network Operations · Building 3 Support · On-call NOC
Subject: [SEV-2][INC-NET-STP03] Building 3 intermittent connectivity loss — STP loop suspected on DIST-SW-03 Gi1/0/24

Incident: INC-NET-STP03 · SEV-2 / P2
Status: Active network containment and triage in progress
Classification: Network Engineering · STP Loop · Intermittent Connectivity Loss
Affected asset: `DIST-SW-03` · Cisco Catalyst 9300 · Building 3 distribution switch
Detected: 2026-04-19 11:08 UTC

SITUATION & SUMMARY

Building 3 users are reporting intermittent network drops lasting approximately 5 seconds every 10–20 seconds. Monitoring shows `DIST-SW-03` has sustained CPU utilization near 99% since approximately 11:02 UTC.

Console and process evidence strongly indicates a Layer-2/STP instability centered around `Gi1/0/24` on VLAN 10. The switch is repeatedly logging Loop Guard blocks, received topology change BPDUs, and MAC flapping between `Gi1/0/24` and other ports. CPU output shows the `Spanning Tree` process consuming the majority of CPU, with additional load from the hardware learning/forwarding process.

This is not being treated as a generic CPU issue. The working diagnosis is an active Layer-2 loop or miswired downstream connection causing repeated STP topology changes and MAC table churn on Building 3.

SEVERITY & PRIORITY

This is rated SEV-2 / P2 because the affected device is a high-criticality Building 3 distribution switch and users are experiencing repeated connectivity drops.

The priority is to stop the loop condition while preserving enough evidence to confirm the root cause and prevent recurrence.

IMMEDIATE CONTAINMENT

* Preserve current evidence from `DIST-SW-03`, including console logs, CPU output, STP state, MAC flapping logs, and `Gi1/0/24` interface state.
* Administratively shut down or isolate `Gi1/0/24` if it is confirmed to be an access/downstream-facing port and not a critical uplink.
* If `Gi1/0/24` is connected to a downstream switch, unmanaged switch, conference-room jack, or recently changed port-channel path, isolate the downstream side and inspect the physical cabling.
* Monitor CPU utilization immediately after isolation to confirm the `Spanning Tree` process drops back to normal.
* Confirm that STP topology change notifications and MAC flapping stop after the port is isolated.
* Notify Building 3 support that a targeted port isolation may temporarily disconnect a local area, but should stabilize the broader building network.

NETWORK INVESTIGATION

* Run and preserve `show spanning-tree vlan 10 detail` to confirm the topology change source and last change interface.
* Review `show logging` around 11:02 UTC to identify when the STP instability began.
* Review `show mac address-table dynamic vlan 10` to identify all MAC addresses flapping through `Gi1/0/24`.
* Check `show interface Gi1/0/24 status`, interface description, cabling records, and patch-panel mapping.
* Check CDP/LLDP neighbor details on `Gi1/0/24` to identify the connected device.
* Verify whether `Gi1/0/24` connects to `ACCESS-SW-307`, an unmanaged switch, a wall jack, or a looped patch cable.
* Review the port-channel change from 9 days ago and confirm it did not create an unintended parallel Layer-2 path.
* Check EtherChannel/LACP state for mismatched bundle membership, suspended links, or one-sided channel configuration.
* Review whether BPDU Guard, Root Guard, Loop Guard, PortFast, and storm-control are correctly applied for this port type.

EVIDENCE PRESERVATION

* Preserve console logs showing `%STP-2-LOOPGUARD_BLOCK`, `%SPANTREE-7-RECV_TC`, and `%MAC-4-FLAP` messages.
* Preserve `show processes cpu` output showing `Spanning Tree` CPU load.
* Preserve `show interface Gi1/0/24` counters, queue drops, and packet rate.
* Preserve STP detail output for VLAN 10.
* Preserve MAC address-table output showing affected MACs flapping between ports.
* Preserve running configuration for `Gi1/0/24`, related uplinks, and any port-channel connected to Building 3 access switches.
* Preserve the change record for the port-channel added 9 days ago.

RECOVERY & NEXT STEPS

* Keep the suspected loop source isolated until cabling and downstream switch state are validated.
* Restore service only after confirming there is no physical loop, unmanaged switch loop, or EtherChannel/STP mismatch.
* After remediation, confirm `DIST-SW-03` CPU returns to normal and user connectivity stabilizes.
* Confirm no further MAC flapping or repeated STP topology changes on VLAN 10.
* Apply or validate BPDU Guard on user-facing access ports.
* Apply Root Guard or Loop Guard where appropriate on downstream switch links.
* Enable storm-control on access ports where supported.
* Confirm all port-channel links use consistent LACP configuration on both ends.
* Update monitoring to alert on repeated topology changes, MAC flaps, and STP CPU spikes.

STAKEHOLDER COMMUNICATION

* Notify IT Leadership that Building 3 impact is likely caused by an active Layer-2 loop or STP instability, not a WAN or core outage.
* Notify Network Operations to prioritize isolation and physical path validation for `Gi1/0/24`.
* Notify Building 3 support that one local port or downstream segment may be intentionally disabled while stabilizing the wider network.
* Provide an update after CPU, STP, and user connectivity are stable.

DO NOT

* Do not reboot `DIST-SW-03` as the first response; rebooting may temporarily hide the symptom without fixing the loop.
* Do not clear MAC tables, counters, or logs before preserving evidence.
* Do not globally disable STP, Loop Guard, or BPDU protections.
* Do not randomly shut uplinks or port-channels without confirming the role of the interface.
* Do not assume this is a hardware failure just because CPU is 99%.
* Do not close the incident after CPU drops; confirm the physical or configuration root cause.