incident-response-trainer
Incident response training · Rule-based scoring
DemoCatalogDiagnosticHistoryDashboardCoach Notes
Attempt report

Building 3 access switches at 99% CPU — intermittent connectivity loss

Network EngineeringSTP LoopDifficulty · Easy

Attempt 1 of 1 · cmq713g3d00020j3g59ijb0a0

← All attempts↻ Retry this scenarioNew scenario
Progress vs previous attempt

This is your first attempt for this scenario. Retry the scenario to generate a side-by-side comparison against your previous response.

Progression · Keep practicing

Stay on Easy · Network Engineering

4 signals are blocking advancement to Medium. Keep practicing at Easy until those areas stabilize. (Track: Network Engineering)

Track · Network Engineering
Easy
Sample · 5 recent attempts2 positive4 blocking

Signals helping

  • Dangerous action frequency. None in recent attempts
  • Recent retry improvement trend. Score is improving (+29.2 pts on later attempts)

Signals blocking advancement

  • Recent average score. 33 / 100 (need ≥ 75)
  • Recent pass rate. 0 of 5 passed (need ≥ 66%)
  • Rubric category coverage. 35% average (need ≥ 55%)
  • Consistently weak rubric areas. Recovery, Change record / log retention, Mitigation
Submission · what was sent and how you responded
STP LoopDifficulty · easyHigh asset
[NOC ALERT] DIST-SW-03 CPU 99% — Bldg 3 users reporting drops
From
Network Monitoring <noc-alerts@acme-corp.local>
To
netops@acme-corp.com
Date
2026-04-19 11:08 UTC
Automated monitoring alert. DIST-SW-03 (Building 3 distribution) is running at sustained 99% CPU since 11:02 UTC. Users in Bldg 3 are reporting that network drops for ~5 seconds every 10–20 seconds, then comes back. Other buildings look fine. Latest config change on this switch was 9 days ago (port channel to ACCESS-SW-307 added). Please triage. — Network Monitoring
Evidence
DIST-SW-03 console + counters
# Last 20 lines of console
%STP-2-LOOPGUARD_BLOCK: Loop guard blocking port Gi1/0/24 on VLAN 10
%SPANTREE-7-RECV_TC: Received topology change BPDU on VLAN 10 (port Gi1/0/24)
%SPANTREE-7-RECV_TC: Received topology change BPDU on VLAN 10 (port Gi1/0/24)
%MAC-4-FLAP: Host 0011.2233.4455 flapped between Gi1/0/12 and Gi1/0/24 in VLAN 10
%MAC-4-FLAP: Host aabb.ccdd.eeff flapped between Gi1/0/9  and Gi1/0/24 in VLAN 10
... (TCN floods every ~3s) ...

# show processes cpu
CPU utilization for five seconds: 99%/45%; one minute: 96%; five minutes: 94%
PID  Runtime(ms)  Process            5Sec   1Min   5Min
116  3812440       Spanning Tree     58.4%  56.2%  55.8%
105   994320       HLFM-process      18.1%  17.4%  16.9%

# show interface Gi1/0/24
Gi1/0/24 is up, line protocol is up (connected)
Hardware is iGigabitEthernet, address is bcd1.7f00.1801
Last input never, output 00:00:00
Input queue: 982/2000/24/0 (size/max/drops/flushes); Total output drops: 0
... 60-second input rate 480000 bits/sec, 740 packets/sec ...
Affected asset
Name
DIST-SW-03 (Cisco Catalyst 9300)
Type
Distribution switch — Building 3 access layer
Owner
Network Operations · Building 3
Level
High
Your submitted response
82 words
I would triage this incident as an emergency because one of the buildings is experiencing network issues.

The most likely reason the building is experiencing a network drop is the last change made to the switch (port channel to ACCESS-SW-307).

Therefore, I would remove the configuration, then test the network connection again and see what happens next.

If the network keeps dropping, I would use ping and traceroute to test the network connectivity where the ping is having issues and troubleshoot there.
Final score
9/ 100
82 words submitted
Verdict · Fail

The response is missing several critical incident response steps. Review the rubric and try again. Score: 9/100. Strongest area: Clarity & structure (61%). Weakest area: Fault understanding (0%) — expand this next time.

Category breakdown

Where points came from

coverage × weight = points
  • Fault understanding0/3 · 0.0 / 15
  • Topology / asset impact1/3 · 3.3 / 10
  • Prioritization0/2 · 0.0 / 10
  • Mitigation0/4 · 0.0 / 20
  • Diagnosis0/4 · 0.0 / 15
  • Recovery0/3 · 0.0 / 10
  • Change record / log retention0/3 · 0.0 / 10
  • Clarity & structure1/2 · 6.1 / 10

Strengths

No category reached 70% coverage.

Missing / weak

  • Fault understanding
  • Topology / asset impact
  • Prioritization
  • Mitigation
  • Diagnosis
  • Recovery
  • Change record / log retention

Dangerous actions detected

None detected in your response.

Learning · Coaching

Learn from this attempt

Post-submission coaching for this scenario. Score and verdict are unchanged — these notes are for your next attempt.

Why points were deducted

  • Mitigation0% coverage

    Identify Gi1/0/24 as the loop port and shut or err-disable it; never disable STP or no-shut all interfaces.

  • Fault understanding0% coverage

    Name the loop / TCN flood / MAC-flap pattern and explain why STP is doing the right thing here — CPU is the symptom, not the cause.

  • Diagnosis0% coverage

    Use `show spanning-tree`, `show mac address-table`, and `show logging` to pinpoint Gi1/0/24 — do not guess from CPU alone.

Model answer outline

From: Incident Response Lead (SOC on-call)To: IT Leadership · Affected Asset Owner · On-call SOCSubject: [SEV-3][INC-NET-EF5F1] Building 3 access switches at 99% CPU — intermittent connectivity loss — status update
Incident · INC-NET-EF5F1 · SEV-3 / P3Status · Investigating — containment in progressNetwork Engineering · STP Loop · EasyDIST-SW-03 (Cisco Catalyst 9300) · High criticalityDetected ~ 2026-04-19 11:08 UTC
Situation & summary

DIST-SW-03 (Bldg 3) is at 99% CPU because spanning-tree is hammering on a user-edge loop: loop guard is already blocking Gi1/0/24, BPDU TCN floods every ~3s, and the same MAC is flapping between Gi1/0/24 and Gi1/0/12 / Gi1/0/9. The trigger is the port-channel added to ACCESS-SW-307 nine days ago.

Severity & priority

Rated SEV-3 / P3. Treat as a P2 site-impacting incident: 5-second drops every 10–20s for every Bldg 3 user.

Prioritization & impact
  • Treat as a P2 site-impacting incident: 5-second drops every 10–20s for every Bldg 3 user.
  • Escalate to Network Operations and the on-site Bldg 3 team in parallel.
  • Be surgical — fixing the loop port is far cheaper than rebooting the stack.
Containment (actions taken / in progress)
  • Identify the loop port (Gi1/0/24) from STP and MAC-flap evidence, then shut or err-disable that single interface.
  • Keep BPDU guard / loop guard enabled; do not disable spanning-tree to silence the alert.
  • Avoid blanket no-shut or rebooting all switches as a guess fix.
Investigation (in progress)
  • Read `show spanning-tree` for VLAN 10 and confirm Gi1/0/24 is the topology-change source.
  • Use `show mac address-table` to show the same MAC flapping between Gi1/0/24 and Gi1/0/12 / Gi1/0/9.
  • Pull `show interfaces counters` and `show logging` to characterize the rate and the timeline.
  • Cross-check against the recent change (port-channel to ACCESS-SW-307 added 9 days ago) before blaming hardware.
Recovery & next steps
  • Enable BPDU guard on every PortFast user-edge port (`spanning-tree portfast bpduguard default`).
  • Add root guard / loop guard on uplinks toward the access layer.
  • Add UDLD on long-fiber uplinks so a unidirectional failure cannot quietly cause the next loop.
Evidence preservation
  • Snapshot pre/post `show running-config` and archive `show tech-support` if practical.
  • Export syslog around the cycle window and attach the BPDU / TCN samples.
  • Open a NOC ticket recording the offending port, the change ticket from nine days ago, and the operator who made it.
Stakeholder communication
  • Brief on-call NOC and the Bldg 3 user-rep once the loop is contained.
  • Document the root cause and follow-up so the next access-layer change does not repeat the pattern.
  • No customer-comm needed unless impact crossed an SLO threshold during the cycle.
Do NOT
  • Do not run `no spanning-tree` — the loop becomes permanent.
  • Do not reboot all switches as a guess fix.
  • Do not `no shutdown all` — you re-arm the loop.
  • Do not clear logs or the MAC address-table before the loop port is identified.

Dangerous actions to avoid

  • Do not run `no spanning-tree` — the loop becomes permanent.
  • Do not reboot all switches as a guess fix.
  • Do not `no shutdown all` — you re-arm the loop.
  • Do not clear logs or the MAC address-table before the loop port is identified.

How to improve next time

  • Read the STP / TCN evidence first — high CPU on a distribution switch is almost always a side effect, not the root cause.
  • Identify the offending port from MAC-flap entries before touching anything else.
  • Contain at the single port (shut Gi1/0/24 or rely on err-disable) instead of taking blanket actions.
  • Add BPDU guard / root guard so the next user-side mistake fails closed instead of looping.
  • Tie the fix back to the recent port-channel change so the post-mortem is honest about how the loop got introduced.
AI · Supplemental review

Request an AI review of this attempt

This AI review is supplemental coaching. It does not change your official score or verdict. The review is only kept for this page session and is not saved permanently.

Review language
AI Tutor · Explains your result

AI Tutor

This tutor explains your result. It does not change your score. Pick a question to see how the deterministic grading reached your verdict and where to focus next.

Generated deterministically from your graded result — no AI model was called.

Why did I get this score?

Your verdict was Fail at 9/100. That total is the sum of deterministic rubric points across 8 categories — each scores how much of its expected, ordered steps your answer covered, not an opinion about your writing. Your strongest coverage was Clarity & structure (61%). Points were held back mostly in Mitigation (0%), Fault understanding (0%), Diagnosis (0%).

Rubric focuscontainmentattackUnderstandinginvestigation
Next study step

Re-read the mitigation expectations for this scenario and list the concrete steps you missed.

This tutor explains your existing result. It does not change your score, verdict, or grade. Generated deterministically from your graded result — no AI model was called.

What should I improve first?

Focus on Mitigation first — it is your weakest rubric area at 0% coverage and carries weight 20. For this scenario: Identify Gi1/0/24 as the loop port and shut or err-disable it; never disable STP or no-shut all interfaces.

Rubric focuscontainment
Next study step

Rewrite your mitigation section as a short numbered checklist before your next attempt.

This tutor explains your existing result. It does not change your score, verdict, or grade. Generated deterministically from your graded result — no AI model was called.

How does my answer compare to the model answer outline?

Compared with the model answer outline, the most useful sections to study are the ones matching your weak areas. Re-read the outline's mitigation, fault understanding, diagnosis guidance and check which listed points you did not cover. The outline is a high-level checklist of expected points — use it to find gaps, not to copy a finished answer.

Rubric focuscontainmentattackUnderstandinginvestigation
Next study step

Pick one model-answer section you missed and add its key points to your next response in your own words.

This tutor explains your existing result. It does not change your score, verdict, or grade. Generated deterministically from your graded result — no AI model was called.

Which rubric area mattered most here?

Mitigation mattered most here: it carries the highest rubric weight (20), so coverage there moves your score the most. You covered 0% of it this time, worth 0 points.

Rubric focuscontainment
Next study step

Prioritise the highest-weight categories first; make sure mitigation is fully addressed before lower-weight ones.

This tutor explains your existing result. It does not change your score, verdict, or grade. Generated deterministically from your graded result — no AI model was called.

What should I study next?

Based on this attempt, study mitigation, fault understanding, diagnosis next. Coaching tip for this scenario: Read the STP / TCN evidence first — high CPU on a distribution switch is almost always a side effect, not the root cause.

Rubric focuscontainmentattackUnderstandinginvestigation
Next study step

Read the STP / TCN evidence first — high CPU on a distribution switch is almost always a side effect, not the root cause.

This tutor explains your existing result. It does not change your score, verdict, or grade. Generated deterministically from your graded result — no AI model was called.

Coach Notes

Open full notebook →

Save study notes for this attempt. They also collect in your mistake notebook.

Loading notes…