noticeboard.ru.ac.za

At about 6:25 this morning, Rhodes lost access to the Internet for reasons as yet unknown. Users will likely experience the outages as an ERR_DNS_FAIL or ERR_CONNECT_FAIL error when trying to access websites hosted off campus.

We have reported the problem to our service providers, and are awaiting feedback.

In light of Friday's prolonged outage and the fact that Neotel have already overshot their serviceability targets for the month, we've asked that this fault be escalated immediately to executive level within Neotel. This escalation is happening significantly earlier that it would normally.

We've just had feedback from Neotel indicating that this outage is a repeat of the latter part of Friday and is caused by an interface failure at Blanco substation (just outside George). A technician is en-route from Port Elizabeth to Blanco to attend to the problem and is expected to be on site by about 10AM.

Neotel apparently now have a technician on site in Blanco (George), and they're attempting to diagnose and repair the fault. They haven't given us an estimated time to restore, but have promised us feedback within half an hour.

It appears that their efforts in restoring our connectivity are somehow being hampered by a major fault on the SEACOM transit between Mtinzini & Rosebank, and by a fault between Rosebank and Port Elizabeth. We're unsure why either of these affect a circuit that runs from Grahamstown to Belleville in Cape Town?

It appears that the problem with restoring connectivity is that the equipment in Blanco is legacy, and spares are not readily available. When the fault occurred on Friday, other circuits were re-routed to free up an interface for us to use; it may be difficult to do that again. We're told, however, that technicians are currently on site and are working on the problem.

Based on the last information we heard, there may be another problem on the same link somewhere around the Tsitsikamma area. Details of what the problem is, or whether or not it is also affecting us, are unclear at the moment. Nevertheless, we're told that technicians are also en-route to attend this fault.

We've discovered that none of the Eastern Cape institutions (Rhodes, NMMU, etc) connect to Neotel's points of presence in either Port Elizabeth or East London; all are unprotected long distance point-to-point links to either Cape Town or to Durban. As a result of this, one possible resolution to our problem that's currently being investigated is moving the end of Rhodes' circuit from Neotel's Belleville PoP to their Port Elizabeth PoP. This solution is being investigated in parallel to resolving the interface problems at Blanco, and the first one ready will be implemented. Ultimately, however, this is a better long-term solution as it reduces the unprotected length of our circuit. Even if connectivity is restored at Blanco, we'll ask Neotel to schedule this change to happen out-of-hours at a later date.

We're now told that Neotel have located a fibre break 15.63 km from "Louter-water" (Tsitsikama) towards "Melkhout" (Humansdorp)

Apparently the fibre break is the result of one of the poles it was attached to being stolen. We're told that someone has been splicing the fibre as of about 12.30.

The fibre break around Tsitsikama has now been temporarily repaired, and service has been restored.

There will likely another (planned) outage during the course of tomorrow evening to effect a permanent solution to this problem. See http://noticeboard.ru.ac.za/post.5531717 for details.