Sat, 06 Sep 2008 17:07:18 +0200
About two hours ago, the University lost Internet access to all international sites. Users will likely experience this as an ERR_DNS_FAIL or ERR_CONNECT_FAIL error when trying to access internationally hosted web sites (including .za sites hosted outside the country).
The fault has just been reported to TENET's service desk and we're awaiting feedback from them.
Sat, 06 Sep 2008 17:34:21 +0200
It appears than one of the STM-1 circuits that provides TENET with international access has failed. What's confusing is that it is the Cape Town circuit and Rhodes normally routes via Johannesburg. I've asked TENET to investigate why this outage is affecting us. As yet there's no estimated time to repair.
Sat, 06 Sep 2008 21:35:35 +0200
The reason we're affected by the Cape Town failure is that, for some unexplained reason, Rhodes international Internet traffic was routed asymmetrically. Outgoing traffic routes via Cape Town and incoming traffic routes via Johannesburg. As a result we're affected by failures at both TENET's Breehead and Reefhead sites, instead of just the Reefhead site.
TENET has asked Neotel to fix the asymmetric routing as a matter of urgency. In the mean time, a workaround has restored our international Internet access.
Wed, 10 Sep 2008 07:25:40 +0200
There appear to be periodic, brief outages as they try to figure out why we're being routed incorrectly. These outages last between five and ten minutes, and there have been about three since my last post here.
Wed, 10 Sep 2008 18:44:48 +0200
QUOTE(TENET @ Sep 10 2008, 11:53AM)
This is a belated note on the failure of GEN3 international services on Saturday 6 September. The outage was caused by a fibre failure in London, but the diagnostics involved a lengthy detection process and co-ordination among several service providers. Once the cause was identified it took only minutes to fix, but it took almost six hours to get to the identification.
Since this outage takes GEN3 international services for August out of SLA, IS will be asked to identify possible service improvement actions that follow from the failure. TENET will undertake to implement better advisories for outages on weekends, probably by means of SMS notifications.