Point in Time Exercise: Middle of Incident
02:46:23 There is a lot of background noise with one of the bridge participants. It sounds like someone is working from home and there are children’s voices in the background. One of the Network SMEs finally addresses the noisy SME on the phone. “Whoever is working from home, “he says, “put you phone on mute now please. We can hear your kids”. The discussion is disjointed.
02:48:22 One of the SMEs offer that there should be a script revision, and Pete says someone should go in and change host names, but it is not directed at anyone. The possibility that the host names may not have been correct in the release is brought up. There are a lot of frustrated comments made about another failed software release.
02:53:00 A number of people have dropped from the bridge as the software release team wanted an off-line discussion. No one announced this meeting, and the release team including Pete just disappeared from the bridge. Currently there are only two SMEs on the bridge, and they are feeling the bridge may have been lost, they are confused as to what is going on and what to do. They decide to continue their discussion on the host name issue and hope that the others will dial back in.
02:54:23 One SME comes back on the bridge and says that others will come back in a couple of minutes.
03:00:21 Customer service reports that one of their big customers, Acme Chemical Sales, is reporting that multiple customers are not able to log in and place orders. There is some discussion as to whether this is related to this incident, or another bridge should be opened. It is determined to open a separate bridge.
03:03:26 Customer service suggests to Pete that this a big problem and that he should make the appropriate notifications for a P1 (Priority 1) incident, which requires Executive and customer notifications at varied intervals. Pete ignores the discussion.
03:05:41 An unidentified SME asks, “have we let the customers know we are resolving the issue”? Pete says it is not resolved and there are still problems. Customer Service says we have an incident and we need to send the proper notifications.
03:12:54 The customer notifications have still not gone out. There is some discussion about whether this a performance degradation or a service disruption, but the customers are reporting they can’t log in. Pete determines that he will keep it a performance degradation.
03:14:54 An SME reports that the analytics do not look too bad. The only person that hears and comments on this is one of the database engineers.
03:19:02 An e-mail SME joins and there is a discussion about e-mail issues and one of the servers misbehaving. Pete is not sure what the problem is. Pete is looking for Bill, one of the e-mail SMEs.
03:29:11 Bill rejoins and says, “There are some problems, and we need to get the senior Executive on the call.”