noticeboard.ru.ac.za

At about 6AM this morning, Rhodes lost all access to the Internet for reasons as yet unknown. The problem appears to be on the circuit between Grahamstown and Port Elizabeth, and is only affecting Rhodes.

You will likely experience this outage as an ERR_DNS_FAIL or ERR_CONNECT_FAIL error when trying to access websites hosted outside South Africa (even those ending in .za). When accessing secure (https://) web sites, your browser may report that the connection has timed out or the site is unreachable.

The problem was reported to TENET's service desk shortly after 7AM.

Apparently Neotel have a failure on their transmission equipment and there is an engineer on site attending to it. I've asked the service desk to confirm the location of the transmission failure, to ensure that it's actually plausible given our circuit (Neotel have been known to confuse faults in different parts of the country before).

Apparently the failure is on Infraco's network at Albany substation, which is just outside of Grahamstown (corner of Raglan Rd & R67). Contrary to what we've been told previously, nobody has attended to this yet -- a technician is en route from Port Elizabeth to Grahamstown to look at the problem. They're expected to arrive on-site at around 11AM. We'll have more information once they've actually arrived and found the problem.

I've just tried to get confirmation that a technician is actually on site now, and is actually trying to fix this problem. Neotel have promised to call back in a few minutes once they've confirmed this with Infraco.

Since we're now over the four hour MTTR mark, I've asked TENET's service desk to start escalating the matter (and particularly if they don't get a useful response from Neotel shortly).

We've had confirmation that a technician has arrived on site. Unfortunately it was the wrong technician -- it seems Infraco incorrectly assumed that the fault was power-related, and sent an Eskom technician to attend to the power problem; in reality the problem is the microwave link between Albany substation and Rhodes, and they need an Infraco technician.

I'm told that an Infraco technician is now on the way from Port Elizabeth, with an ETA of around 2PM. I've asked that they ensure that they have all spares that they might require, so that we're not further delayed by discovering that spares are needed and have to be brought from PE.

I'm also going to phone TENET's service manager in an attempt to reinforce this message.

TENET's service manager has unsuccessfully attempted to take this through several escalation steps within Neotel. In each instance, the designated person failed to answer their phone. I'm lead to believe this culminated with a message left with the CEO's secretary.

Nevertheless, we've determined that Infraco's technician is indeed on site at Albany, and is currently working on the issue.

This service has now been restored. The ultimate problem was a faulty patch cord in Albany substation. Given past history, I suspect rats...

I've just had confirmation from TENET's service desk that the cause of this failure was rodent damage -- a rat ate the fibre optic patch cable that connected Infraco's add-drop multiplexer to our microwave circuit. Given that this is the second time this has happened at Albany substation, I'm going to take this up at a service management level. Last time we were told that the problem was that a lack of routine maintenance was the cause -- nobody changed the rodent bait. Clearly that is still the case...