2011/07/05 - Power Outages on Campus
During the course of the day, there have been a number of power outages affecting various sectors of the campus.

Whilst there doesn't seem to be any official news from the Electrical section or the Municipality, we've managed to piece together a (perhaps inaccurate) picture of what's happened. Our understanding is recorded below, both because it has a direct impact on ICT services, and also as general information in lieu of any formal notification.

It'd seem that the root cause of the problem is that the University has been placing excessive load on the municipality's infrastructure This has caused one of the two main feeder cables to the University to fail. As of about 5PM, the municipal electricians were uncertain whether the cable itself was faulty, or whether the problem was with the switchgear at one or other end of the cable. They were investigating in an attempt to find the fault. The time-to-resolution will depend on the exact nature of the fault.

In the mean time, the municipality has restored power to campus using only one of the two cables. This presumably means that there's about half the normal amount of power available to the campus. As a result, the University has been asked to shed some load internally, and the Electrical section has done just this. It isn't clear what criteria has been used to do this, or how frequently they're switching loads. However, this is the likely cause of any outages that have occurred this afternoon. Our monitoring system shows that different areas of campus have been affected at different times, and that the areas currently without power include most offices and academic spaces on campus (conversely, most residences and spaces used by the National Arts Festival appear to have power.)

At the time of writing, the IT Division's Struben data centre has been without municipal power for the last seven hours (and for a total of 9.5 hours thus far today). This data centre houses most of the University's core ICT services, including components necessary for e-mail, Internet access, telephony, printing, administrative & library systems, and networking on campus in general. At present, the data centre is being powered by a standby generator. We estimate that we have at least fifteen hours of fuel remaining based on the current consumption levels, but we cannot accurately predict how long it will last. The fuel levels are automatically monitored, and a low fuel condition will signal an alarm at Campus Protection who will in turn contact IT staff. (We've asked CPU to do this at any stage during the night.)

Just before 8PM, the Electrical Manager indicated that power to our Struben data centre would be restored within an hour. Whilst this hasn't happened, we're hopeful that we'll get power back tonight. If not, we're likely to run short of diesel some time during the course of tomorrow. (If our predictions of usage are incorrect, it might happen sooner than this.) We hope to be able to refuel the generators before this happens, but refuelling a running generator at short notice is something we've never had to do before.

In addition, in the unlikely event of a fault or problem developing with the standby generator, we've no alternative source of power.

As such, all users of the University's ICT services (including services accessed from home or remotely) should consider these services to be at risk of complete failure until such time as normal power is restored. Should this occur, it'll likely take most of a working day for ICT services to be fully restored.
In addition to the risk described above, the load shedding is having a direct impact on some areas where electrical power is available and appears to be unaffected by the problems on campus. This is usually because networking in these areas is dependant on building on campus that doesn't currently have power. Areas that are known to be affected are:
  • Albany Museum
  • Celeste House
  • David Rabkin Project for Experimental Journalism (Grocotts)
  • Legal Aid Clinic
  • Somerset Corner
  • Summer Place
  • The Greens

which all depend on power being available in the Main Admin building and
  • Eleanor Brown / Canterbury Annexe

which depends on power being available at Beethoven house.
Shortly after posting this, power was restored to Struben. It is unclear whether the underlying cause of the outages has been resolved, or whether we'll experience further outages tomorrow.
I've had further confirmation of the problem. There are in fact four feeder cables coming onto campus, three from the West and one from the East. Two of the four cables are problematic, leading to the situation described yesterday. The fault still exists, and is being attended to by the Municipality.

To maintain power, the University needs to halve its electrical consumption. Load shedding is ongoing on campus. I've tried to obtain a schedule, but apparently the loads on campus are not consistent enough to do this. As a result, it would seem that loads are being shed and restored in response to peaks in demand. Once again, it would seem areas used by the National Arts Festival are getting priority. Buildings with standby generators are not.

We've been asked to conserve power as much as possible. The more power that can be conserved, the less severe the load shedding will be.

The IT Division is doing its bit by reducing the number of computer that are turned on in public labs. We've left a handful switched on in each open lab to allow people to continue to work.

Each of us can contribute, however. Please turn off any unnecessary electrical appliances. In particular, please turn off your heater or airconditioner.