Major problems in Geneva due to crashes of the Cisco IPSec tunnel gateway

Switzernet

2007-07-09

Site: Uunet-(gva2)/mci, CH-1228 Geneva

Today, on 2007-07-09 we lost the SIP connection of our Geneva servers due to problems with IPSec tunnel. The first lost of connectivity occurred within the time window between 11:30 and 13:00. The problem was solved by a hard reboot of the IPSec tunnel gateway. The second crash occurred at about 16:00. The hard reboot did not help. The network interface (NM-4E) was replaced. The connection was re-established at about 19:30. The crashes times can be identified in the histogram displaying the load of calls passing through the servers of Geneva.

Date	Time	Inbound+Outbound
07-07-09	20:00
07-07-09	19:45
07-07-09	19:30
07-07-09	19:15
07-07-09	19:00
07-07-09	18:45
07-07-09	18:30
07-07-09	18:15
07-07-09	18:00
07-07-09	17:45
07-07-09	17:30
07-07-09	17:15
07-07-09	17:00
07-07-09	16:45
07-07-09	16:30
07-07-09	16:15
07-07-09	16:00
07-07-09	15:45
07-07-09	15:30
07-07-09	15:15
07-07-09	15:00
07-07-09	14:45
07-07-09	14:30
07-07-09	14:15
07-07-09	14:00
07-07-09	13:45
07-07-09	13:30
07-07-09	13:15
07-07-09	13:00
07-07-09	12:45
07-07-09	12:30
07-07-09	12:15
07-07-09	12:00
07-07-09	11:45
07-07-09	11:30
07-07-09	11:15
07-07-09	11:00
07-07-09	10:45

The presence of a light load of calls during crash times is present since the RTP streams were not affected by the crash (crash only affected the SIP signalling between the two Geneva servers and the servers in Brussels and London, the SIP being further converted into SS7)

The crash affected the SIP signalling of both redundant servers in Geneva. The incoming calls, and the outgoing calls were blocked (outgoing calls were re-routed during the crash but CLI was not guaranteed).

Important note: IPSec tunnel gateway in Geneva is a vulnerable point and the redundancy on the level of SIP server is useless in case of problems with the tunnel end point.

Conclusion: redundant IPSec connection is required in Geneva.

* * *