noticeboard.ru.ac.za

Some time on or before Monday 31 March 2008, the University will be changing the way its Internet access is provisioned. This change is likely to disrupt Internet access for several hours when it happens. More details about how, why and when this will happen follow.

The University currently gets its Internet access from Telkom SA under a collective purchasing agreement known as the GEN2 agreement. Telkom has been providing us Internet access under this agreement since the begining of 2005. The GEN2 contract ended in December 2007, and Telkom will cease providing services under this contract on 31 March 2008.

In July 2007, South African universities, under the TENET umbrella, began the process of selecting suppliers for the next generation of higher education network, to be known as the GEN3 agreement. (Rhodes staff have been heavily involved in this process since the begining.) After a long selection process, a consortium consisting of Neotel and The Internet Solution was selected as the provider for this new network in October of last year. This consortium will replace Telkom as the provider of Internet access to higher education in South Africa. The intended commissioning date for the new network as 1 April 2008.

Unfortunately, because of its relative geographic remoteness, it doesn't currently look as if Neotel are going to reach Grahamstown in time to meet the 1 April deadline. Whilst they are optimistic, and have already installed some equipment on our campus, they currently predict that their connection to Rhodes will only be fully commissioned during the course of May. This means that, for a short period, we'll be in an interim phase where we're neither part of the GEN2 nor the GEN3 network.

Telkom has graciously offered to continue providing the University with Internet access during this transitional phase. They will be migrating Rhodes from their GEN2 network onto their commercial TI-DIS network, from which we'll get Internet access until such time as we're in a position to migrate onto the GEN3 network. (The GEN2 network will be disassembled from 1 April.)

Whilst we're using Telkom's TI-DIS service we will be provided with purely commodity Internet access, rather than the mix of commodity and research & education networking we've enjoyed for the last few years. This may have some (temporary) implications, particularly with respect to accessing information at other universities around the world. In particular, we may find that some of the electronic resources (journals, etc) provided by the library are not readily accessible during the few months we're on the TI-DIS network. It may also affect the SEALS Millennium system that's hosted at Rhodes. In addition, during the interim period, there may be some negative effects on the amount of Internet access (quota) we're able to make available to individuals at Rhodes. The IT Division will deal with these problems if and when they arise.

This first change, from the GEN2 network onto Telkom's TI-DIS network, will be happening some time during the course of next week. Unfortunately we're at the mercy of Telkom during this change -- we've been pressing Telkom for firm commitments about the timing of this transition and we've yet to get anything concrete. At the moment we have no idea when Telkom intend to make the change, nor how much notice we'll get. Unfortunately, we expect this change to disrupt Internet access for several hours.

There will be a subsequent change from TI-DIS to GEN3 some time within the next few months. Again no firm dates for this transition are known. This change, however, should be a lot less noticeable to end users. (The GEN3 migration strategy is a lot more flexible and inclusive that Telkom appear to be with their TI-DIS service.)

The lack of firm information about when these changes are going to happen as made the decision to send out notification somewhat complicated, and is why we've delayed this announcement for so long. We will continue to post status updates here until such time as the GEN2->TI-DIS transition is complete, and will send an announcement to the toplist@lists & studentnews@lists mailing lists once a firm date for this change is known. Please be assured that we wish to give all users at Rhodes as much notice as possible, so as soon as we know something definite, we'll pass that information on.

Despite repeated requests at all levels of Telkom we've been unable to get a firm commitment from them as to when they're planning to do the migration to their TI-DIS service. Their most recent correspondence states:



I suspect this means that the cut-over will happen any time within the next three and a half days.

We will have staff available over the weekend in case Telkom decides to do the cut-over then. However, as we don't know when they're planning on doing this, or what exactly their planned change involves, we expect that there'll be a prolonged outage and/or network instability when it happens. We'll deal with this as and when it becomes apparent that it is a problem.

Telkom have just indicated that the earliest we can expect to know the date & time that they're intending to do this work is during the course of tomorrow morning. We've still got no indication of what the work involves, or how long they expect it to take. From my earlier conversation with them, it seems unlikely that anything is going to happen tomorrow (Friday), but the change could be scheduled any time between Saturday and Monday (inclusive).

I've asked that they preferably make their changes over this weekend, so as to minimize the disruption to the first day of term. I've got no idea whether they'll honor this request.

We've just heard from Telkom again. Unfortunately, they have ruled out making the changeover on Saturday or Sunday. However, they would not commit to doing it on Monday, either.

The best-case scenario is that, sometime on Monday, Rhodes will be moved from the GEN2 network to the TI-DIS network and while that is happening, Rhodes will be without working Internet access for an unspecified length of time. The worst-case scenario is that Telkom will shut down the GEN2 network on midnight, Monday night, and Rhodes will have no access to the Internet until sometime after that when we're connected to the TI-DIS network.

Telkom have told us that they'll contact us again on Monday morning with more information. We'll pass on any more information that becomes available.

I've just been in communication with the Government Sales and ICT Business Solutions Executive within Telkom, the person who's ultimately responsible for the services that Telkom provide to Rhodes. He's given me a verbal undertaking that Telkom will not discontinue the GEN2 service until such time as all sites has properly migrated to their TI-DIS service. This rules out the worst-case scenario that was posted on Friday -- that we'll be without Internet from midnight tonight.

Telkom are still unable to give me a firm commitment as to when the cut over will actually happen. There have apparently been a number of technical problems relating to IP addressing that Telkom's staff need to resolve these before the change is authorized within Telkom. It is still possible that the migration will happen today, but it is probably more likely that it'll be delayed until tomorrow. Telkom has undertaken to give us a firm commitment in this regard by lunch time today. (This is the third such commitment.)

Telkom have recently confirmed that the change is not going to happen today. They're unable to confirm whether or not it'll happen tomorrow and have said they'll contact us "in due course".

We're now past the official end of the GEN2 agreement and Telkom seem to be honoring their undertaking to continue providing services using the GEN2 network until such time as they're in a position to migrate us to their TI-DIS network. The University is still obtaining its Internet access via the GEN2 network; the transition to TI-DIS has not yet happened. The minor disruptions to networking yesterday morning appear to be unrelated to this transition -- they were the result of some changes TENET coincidentally made at the same time.

There has been no further information about how and when Telkom intend to undertake this transition for the last day and a half. I've recently contacted them requesting an update, and they've promised to come back to us by lunch time. We'll continue to pass on any new information we receive.

Late this afternoon, various sections of Telkom informed us they anticipate that the change will only happen in "three to four weeks" or "four to six weeks". Whilst it isn't clear what this means in terms of a real deadline, what seems obvious is that the TI-DIS migration won't happen anytime in the next couple of weeks. The reason for the delay is apparently that their technical planning team have asked for more time to overcome the problems that've encountered. Telkom assure us that the GEN2 network will remain operational until they're ready to migrate us.

Given that Neotel currently appear to be well ahead of their anticipated installation schedule, it is possible that Telkom's delay might mean that we avoid the TI-DIS network entirely. We're currently hoping that Neotel are ready to migrate us to GEN3 before Telkom are ready to migrate us to TI-DIS. There are, however, no guarantees this will happen. Neotel's official schedule still remains in mid-May.

We've just received e-mail from TENET indicating that they've been told that the TI-DIS cutover will happen "within a few days". This contradicts yesterday's statement from Telkom. We're not quite sure which announcement to believe.

Despite the fact that we haven't yet transitioned off the GEN2 network, we're starting to see the impact of the other universities transition to GEN3 on our Internet bandwidth. Most things that used to be considered "higher educational" or "backbone" in the GEN2 era are currently been counted as "national". The reason for this is that a lot of the higher education sites people are trying to access are now on the GEN3 network, and traffic from the GEN2 network to the GEN3 goes via national peering.

This has drastically changed the University's traffic patterns. Whereas we've always had a fairly generous allotment of backbone bandwidth for inter-university traffic, this traffic is now competing with our normal national traffic. We suspect that this has resulted in extreme congestion on the national component of our GEN2 Internet bandwidth (we cannot confirm this, as GEN2 no longer has any traffic graphs in place to allow us to monitor this). This has increased latency at certain times of the day.

Users will notice this congestion as slow responses from national and South African higher educational sites, particularly at peak usage times. Note that national sites will include anything hosted in South Africa, such as the Akamai cluster which provides some of the content for facebook.com, microsoft.com, cnn.com, etc. Peak usage in the Rhodes context occurs between about 10AM and 11PM.

Unfortunately there's not a lot we can do about this, other than hope we migrate to either the GEN3 or TI-DIS network soon. The current situation wasn't one TENET, Neotel or Telkom planned for; we're not supposed to be in the situation we're currently in.

Late yesterday afternoon we received the following from Telkom after requesting an update on their progress:

During the course of today the University of the Free State went through a GEN2 -> TI-DIS migration similar to the one Rhodes is scheduled to undergo. We're informed that Telkom gave them absolutely no prior notice; they received a phone call informing them that the transition was underway. With a few minor exceptions, their access to the commodity Internet was disrupted for about half an hour. Unfortunately they still cannot access sites on the GEN2 network (and vice versa, which is why we cannot currently get to http://www.ufs.ac.za/ or send them e-mail). Telkom have promised this will be fixed by tomorrow. This will probably cause some e-mail, etc to be lost.

UFS' experiences with this transition are useful in informing us of what we can anticipate. In particular, the lack of prior notice is somewhat concerning and contradicts Telkom's earlier undertakings.

I've just had a phone call from Telkom asking to arrange a time for this cutover. They'd like to do the transition today, and we've proposed 3PM. They're now trying to confirm that they can have a technician available at that time, and will get back to me ASAP with confirmation. Failing that, the transition is likely to happen at 3PM tomorrow.

Telkom have indicated that they've resolved the GEN2<->TI-DIS problems, and that we're likely to be down for approximately two hours.

As soon as we've got firm confirmation that this is going ahead, we'll post that information here. Until then, all users should assume that we will be without Internet access between 3PM and 5PM today, and 3PM and 5PM tomorrow.

Telkom have just confirmed that the transition from GEN2->TI-DIS will happen at 3PM today. All users should expect to be without Internet access from 3PM to 5PM today (Tue 15 April)

Telkom currently have a local technician on site in our Struben data centre. Unfortunately, it appears that a previous transition (we suspect NMMU) has been delayed for some reason, and we're waiting for Telkom's project manager to confirm that they're ready to begin our transition. No changes have been made thus far. This probably means that the predicted two hour window will be extended into this evening.

The GEN2->TI-DIS transition has now happened. Everything went very smoothly. We've done fairly extensive testing of the new link, and we're reasonably happy that things are currently functioning as we expect them to.

If you notice unusual problems with access to specific sites at this stage, please report them to our Help Desk (support@ / 8288). Please ensure you give a full description of the problem, including the site you're trying to access, if you do so.