Thu, 06 Oct 2005 15:26:06 +0200
There will be a brief Internet access interruption starting shortly after 5:00 PM today. We expect that full service should be restored within 20 minutes.
Switching infrastructure between the TENET border router and firewalls will be replaced during the outage.
Thu, 06 Oct 2005 19:18:55 +0200
Unfortunately this took a lot longer than expected. Internet access, as well as access to some core services, was intermittent between 17:00 and 19:00.
The reason for this appears to be related to the way the University's core switch handles spanning tree. The point of the upgrade was to replace a faulty 100Mbps hub with a new 100Mbps switch. The hub originally connected our two Internet firewalls to the Telkom router in our machine room. At the same time, we intended to consolodate another, older, 100Mbps switch (serving the Rhodes side of our firewalls) in the same rack into the new switch.
The original hub and switch each had independent uplinks to the core switch on different VLANs. A single new switch was configured to segregate the two VLANs on different ports, with the idea that we could simply plug cabling across between the old and the new. The configuration of the new switch was tested before work began and everything appeared to be okay. Unfortunately, we didn't anticipate the effect of having two uplinks to the core switch plugged into the same switch (albeit into ports on two completely isolated VLANs).
Since spanning tree is enabled by default, but isn't used at all at Rhodes, the new switch was configured to do spanning tree. This has never been a problem in the past, but we've never had two uplinks to the same switch. It appears that one of the switches involved (most likely the core switch) saw the same MAC address coming in on different ports and, in spite of the fact that they were on different VLANs, interpreted this as a loop and disabled one of the ports. We're not sure why it did this and it took a while for us to work out what was going on.
The two uplink configuration has now been replaced with a single tagged-trunk containing both VLANs.
We appologise for any inconvenience this may have caused.