2008/08/11 - Unscheduled Internet Outage
At about 5:20AM this morning, we lost access to the Internet for reasons unknown. At the moment, you'll be unable to browse the web or access any resources off-site.

The problem appears to be Rhodes-specific, and has been reported to TENET's service desk. We're awaiting feedback from them
We've just been informed that the problem is with Neotel's microwave transmission equipment that links Rhodes, via Governer's Kop, to the point-of-presence at Albany substation just outside Grahamstown. A technician has been dispatched to investigate, and should be on-site within the next half an hour.
QUOTE(guy @ Aug 11 2008, 09:14 AM) <{POST_SNAPBACK}>
A technician has been dispatched to investigate, and should be on-site within the next half an hour.

The intermittent access we're seeing at the moment is apparently a result of Neotel's technicians attempting to isolate the fault. We've asked for an estimated time to completion, but as yet we haven't received one.
Neotel's just phoned me to let me know that the fault requires spares that have to be shipped from Johannesburg. The spares (and a technician to install them) have apparently boarded a plane and are en-route. There seems to be some confusion as to whether the fault is in Cape Town or Grahamstown, however. Assuming the earlier information is correct, and given the travel time from Port Elizabeth to Grahamstown, I suspect this means that the outage will persist for at least another two hours.
We've just been informed that the faulty part is a patch lead, and that it'll be arriving by overnight courier. This being the case, Internet access will only be restored during the course of tomorrow morning at the soonest.

This is clearly unacceptable, and so we've just escalated the matter to TENET's service manager. He's currently investigating, and has undertaken to get back to us with an update shortly.
Further to our last post, we've offered to make available to Neotel any of the spare fibre patch leads we have in stock. This covers pretty much any combination of fibre connectors, and we're hoping that it'll help resolve the problem sooner.
The root cause of this fault appears to be a faulty fibre optic patch lead at the Albany substation point of presence.

From our discussions with Neotel's sub contractors, it looks as if Rhodes might have an acceptable replacement for the faulty patch lead amongst the IT Division's spares. A technician is en-route to Grahamstown from Port Elizabeth and should be on campus in about an hour and a half. He'll evaluate what we've got available, and hopefully find something useful. If so, we'll travel to Albany substation and make an interim repair as soon as possible thereafter.
There is currently an interim workaround in place that's (temporarily) restored our access to the Internet. Neotel's subcontractors are making use of a fibre patch lead that they've borrowed from Rhodes. Sometime during the course of tomorrow, the patch lead will need to be replaced with one of their own. This will result in a second outage of a few minutes. We'll endeavour to post a scheduled time here if we know one.

The root cause of this problem turned out to be a somewhat large rat nesting in the add-drop multiplexer at Albany substation. It appears that at some stage during last night, said rat decided it'd be a fantastically good idea to gnaw through the fibre patch lead that connected Rhodes' microwave link to the ADM. The break in the fibre was almost unnoticeable to the naked eye, except under very careful examination.
We're informed that the current Internet outage is a result of Neotel's subcontractor replacing the temporary fibre patch we installed last night.

We expected this outage to last no more than a few minutes; we suspect that they've removed the operational patch before routing the new fibre lead, and that we're now waiting for them to thread fibre through conduit.

We've been in contact with TENET's service desk and asked that they escalate the matter as quickly as possible.