noticeboard.ru.ac.za

Shortly before 9:20PM this evening the University lost access to the Internet as a result of a fault on Telkom's network. The fault has been reported to Telkom and their technicians should be investigating.

For some reason Telkom seems to have assumed that, as it was outside office hours, there'd be no-one available on the Rhodes side of their fibre optic to let them in (despite having been told to the contrary). As a result they've made no progress in rectifying this fault. This assumption having been corrected, they now inform us that they'll dispatch someone "as soon as possible".

I've just phoned Telkom CNC and I've been told that they've requested a technician do a site inspection. I'm not sure whether this means we're waiting for someone from Grahamstown or from Port Elizabeth but, as yet, they haven't arrived. We'll keep following this up during the course of the day and post updates here as we know things.

A Telkom technician is in our data centre working on this problem at the moment. Hopefully we're finally getting closer to resolving it.

At about 4.30PM Telkom's technician left our data centre saying that they thought the fault was somewhere on the ATM circuit between Grahamstown and Uitenhague. This effectively means they still don't know what's wrong, and they're still grasping at straws.

Our upstream service provider, TENET, is now aware of the fault and they've promised to take the matter up with Telkom from their end. On our side, we've indicated to Telkom that we're available during the course of the evening and that they should continue working on the matter. If I haven't heard back from them by 7PM, I'll follow up with their CNC once again. I'll continue to post updates as I learn things.

Still no real progress. At least part of the delay in resolving this problem seems related to the fact that Telkom can't remember what the enable secret on their router in our data centre is.

I'm told that the fault has now been escalated within Telkom's reporting structures and that they'll get a more senior technician to give me a call as soon as they can.

If I don't hear back from them soon I'm going to escalate the matter myself as Telkom are now very close to being in violation of their service level agreement.

Since Telkom are now (in our opinion at least) in violation of their service level agreement with us, I've now escalated this matter to our service manager. He was horrified to hear that we're still down and promises to make sure the matter is resolved ASAP. I've also escalated the matter within TENET and have their chief technical officer liaising with our service manager at Telkom to work out a solution.

With the intervention of our service manager, technicians were called out from Port Elizabeth to the exchange in Grahamstown at about 10.30PM last night. During the course of last night Telkom have tried replacing the ATM line cards in both the exchanges in Grahamstown and in Uitenhauge to no avail. So far as I can tell, however, nothing further has happened since 4.30AM this morning. Our service manager has promised to get back to us first thing this morning with an update. Depending on the results of that we may escalate the matter to the next level within Telkom.

As of about 9.30 this morning (almost exactly 36 hours later), Internet access has been restored. We're told the problem was with the configuration of an ATM router somewhere in Telkom's IPNet cloud.

At the moment our line will be a bit congested as things start to catch up. As a result you may experience slow and/or intermittent access for the next couple of hours.