Incident Management Training for IT Operations

Level 2: Sprint 4

Simulated Technology Failure, Large Group

MISSION:  Each small group develops one IT incident response scenario to be exchanged with another group during breakout session. Unified Command will be simulated in the main session by Blackrock 3 staff.


Incident Commander: Drive the discussion and delegate tasks.

Scribe: Capture key events and decisions as the discussion unfolds. Their documentation should be compiled using the Scribe spreadsheet linked below so that it can be archived and easily shared.

LNO: Provide briefings as directed. May use the shared Scribe document to develop their briefing, or they may use their own notes. The briefing should be written electronically and saved on their local desktop so that they can access it when leaving to deliver a briefing to Unified Command or another group.


Part 1: Scenario Creation
The group must come up with a past, present or potential IT incident response scenario, including enough specific and accurate detail to convey the signs and symptoms of the current conditions. The group does not provide resolution actions or needs in this document.

An example of a scenario would be:

Customer Service Representatives (CSRs) are unable to log into the billing application to processor customer payments. This issue is only impacting new users trying to sign in; users already logged into the application are not impacted. The issue is impacting approximately 300 CSRs in the Northeast Division for our biggest customer. The incident was discovered at 09:00 CST. Initially, this was thought to be an issue on the client side as some IP changes occurred around the same time the issue started. However, after the change was reverted the issue persisted. All other desktop applications are functioning normally. Currently, the only impact is to the billing application.

Part 2: Briefing Delivery
Once the scenario is drafted, the LNO will return to the main session, where they will be assigned to an alternate breakout room to deliver a briefing.

Each group is then briefed, and any qualifying questions about the scenario are answered by the LNO that has joined their group. Once that task is complete, the LNO returns to the main room and is sent back to their original group by the instructor.

Part 3: Resolution Planning
The Incident Commander drives a discussion to resolve the scenario. The group must develop a primary resolution plan and a contingency resolution plan (Plan B). The Scribe should capture key events and decisions using the same methods as earlier in the exercise. The LNO should be preparing to deliver a briefing to UC or upon return to the main group.


● Identify SEV or P level for the incident.
● Create a list of SMEs, vendors, Executives, etc. that would be dispatched to the incident.
● Draw an org chart depicting all the incident responders identified above.
● Draft an overall Mission Objective for the response.
● Draft a CAN report for the primary resolution plan and Plan B.
● Identify any unique aspects or challenges of the response that may pose a challenge to the resolution effort.
● List the cadence and potential audiences for any communications briefings that need to occur outside the resolution effort.
● The LNO delivers the scenario briefing back to the entire group in the main session at the conclusion of the exercise. The briefing can represent the beginning, middle or end stage of the incident.

Click or drag a file to this area to upload.