Network failures due to signaling problem
Switzernet
2007-09-14
Due to SIP/SS7 signaling problem in Brussels the direct interconnection of Switzernet with Swiss PSTN was lost on 2007-09-13 starting from 18:00 until the mooring 2007-09-14.
|
On 2007-09-13 21:43, VZB wrote: Serverops have been triggered, we are awaiting updates soon. Quoting SWITZERNET: We have 600-Busy Everywhere for outgoing calls No incoming calls from PSTN Please fix ASAP *Please make sure that the script resetting the counter of calls is running.
On 2007-09-13 22:23, VZB wrote: please do force your traffic towards london. The brusselserver has a hard-disc malfunction :( . No ETR at the moment.
On 2007-09-13 22:31, VZB wrote: we created master ticket 2007091308433 with highest priority. I will brief you as soon as i know when the rebuild has finished. T.
On 2007-09-13 22:45, SWITZERNET wrote: We forced all our traffic to London 62.189.34.7
On 2007-09-13 22:48, SWITZERNET wrote: Please route the signaling of all our incoming traffic +41 21 550 0000 - +41 21 550 9999 +41 22 550 0000 - +41 22 550 9999 through the London server
On 2007-09-13 23:03, SWITZERNET wrote: Seems that the incoming calls are now working we moved away the outgoing traffic now we routed back to Verizon Swiss fix and Swiss mobiles. shall we keep? shall we route back the remaining traffic? Is it stable now? (do you force now our incoming calls through London only?)
On 2007-09-13 23:13, VZB wrote: yes, london takes traffic. Leave it this way, we will give advice as soon as we have an ETR for brussels. T.
On 2007-09-13 23:27, SWITZERNET wrote: It is not stable again ... the calls are going through occasionally ... I see that incoming calls are coming also from Brussels. Please make sure to force all incoming calls through the operating server - London. We can move away the outgoing traffic, but not the incoming:
On 2007-09-13 23:44, VZB wrote: in contact with serverops to shut down Brussels while fixing/rebuilding...
On 2007-09-13 23:52, VZB wrote: can you retest? Severops states diskissue has been fixed. Cheers, T.
On 2007-09-13 23:59, SWITZERNET wrote: Still very bad not able to establish any call a call attempt passes through after a delay of 2 or more minutes ...
On 2007-09-14 00:13, VZB wrote: Yes I can confirm, serverops is investigating, we do see also no processing of the call.
On 2007-09-14 00:42, VZB wrote: please do still route all calls towards london. Seems that brussels will not take calls as it should and we might take it out of service. I have escalated this and serverops are now checking with two people the machine. ...to be continued... t.
On 2007-09-14 01:52, VZB wrote: we have taken the brussels machine out of service. Engineering will investigate and rebuild starting business hrs.
On 2007-09-14 10:33, VZB-DE wrote: Server and services have been fully restored so it is working again on both Brussels and London. Please let us know if we can close the ticket. Regards
|
The load of calls through the VZB interconnection is shown below. The colored portion corresponds to the failure window.
|
Date |
Time |
Inbound+Outbound |
|
07-09-14 |
10:45 |
|
|
07-09-14 |
10:30 |
|
|
07-09-14 |
10:15 |
|
|
07-09-14 |
10:00 |
|
|
07-09-14 |
09:45 |
|
|
07-09-14 |
09:30 |
|
|
07-09-14 |
09:15 |
|
|
07-09-14 |
09:00 |
|
|
07-09-14 |
08:45 |
|
|
07-09-14 |
08:30 |
|
|
07-09-14 |
08:15 |
|
|
07-09-14 |
08:00 |
|
|
07-09-14 |
07:45 |
|
|
07-09-14 |
07:30 |
|
|
07-09-14 |
07:15 |
|
|
07-09-14 |
07:00 |
|
|
07-09-14 |
06:45 |
|
|
07-09-14 |
06:30 |
|
|
07-09-14 |
06:15 |
|
|
07-09-14 |
06:00 |
|
|
07-09-14 |
05:45 |
|
|
07-09-14 |
05:30 |
|
|
07-09-14 |
05:15 |
|
|
07-09-14 |
05:00 |
|
|
07-09-14 |
04:45 |
|
|
07-09-14 |
04:30 |
|
|
07-09-14 |
04:15 |
|
|
07-09-14 |
04:00 |
|
|
07-09-14 |
03:45 |
|
|
07-09-14 |
03:30 |
|
|
07-09-14 |
03:15 |
|
|
07-09-14 |
03:00 |
|
|
07-09-14 |
02:45 |
|
|
07-09-14 |
02:30 |
|
|
07-09-14 |
02:15 |
|
|
07-09-14 |
02:00 |
|
|
07-09-14 |
01:45 |
|
|
07-09-14 |
01:30 |
|
|
07-09-14 |
01:15 |
|
|
07-09-14 |
01:00 |
|
|
07-09-14 |
00:45 |
|
|
07-09-14 |
00:30 |
|
|
07-09-14 |
00:15 |
|
|
07-09-14 |
00:00 |
|
|
07-09-13 |
23:45 |
|
|
07-09-13 |
23:30 |
|
|
07-09-13 |
23:15 |
|
|
07-09-13 |
23:00 |
|
|
07-09-13 |
22:45 |
|
|
07-09-13 |
22:30 |
|
|
07-09-13 |
22:15 |
|
|
07-09-13 |
22:00 |
|
|
07-09-13 |
21:45 |
|
|
07-09-13 |
21:30 |
|
|
07-09-13 |
21:15 |
|
|
07-09-13 |
21:00 |
|
|
07-09-13 |
20:45 |
|
|
07-09-13 |
20:30 |
|
|
07-09-13 |
20:15 |
|
|
07-09-13 |
20:00 |
|
|
07-09-13 |
19:45 |
|
|
07-09-13 |
19:30 |
|
|
07-09-13 |
19:15 |
|
|
07-09-13 |
19:00 |
|
|
07-09-13 |
18:45 |
|
|
07-09-13 |
18:30 |
|
|
07-09-13 |
18:15 |
|
|
07-09-13 |
18:00 |
|
|
07-09-13 |
17:45 |
|
|
07-09-13 |
17:30 |
|
|
07-09-13 |
17:15 |
|
|
07-09-13 |
17:00 |
|
|
07-09-13 |
16:45 |
|
|
07-09-13 |
16:30 |
|
|
07-09-13 |
16:15 |
|
|
07-09-13 |
16:00 |
|
|
07-09-13 |
15:45 |
|
|
07-09-13 |
15:30 |
|
|
07-09-13 |
15:15 |
|
* * *
Previous failures:
2007-09-11 19:30 http://switzernet.com/public/070911-outage-outgoing-calls/
2007-09-11 11:50 http://switzernet.com/public/070911-outages-usa-bbcom/