Between 00:30 and 07:45 this morning, the University was without Internet access. The reason for the outage is that both of the University's redundant Internet firewalls crashed overnight, within six hours of each other. It is not entirely clear why this happened.

As a result of this outage, the University's DNS servers were unable to reach the global DNS root nameservers. Because our DNS servers run separate authoritave and recursive views (in line with best practices for security), an unfortunate side effect of such an outage is that over time we begin to lose the ability to resolve DNS internally. This will have resulted, for example, in erroneous reports of outages on http://bb.ru.ac.za/ as the monitoring machine would have been unable to resolve some DNS, even within the ru.ac.za zone. Steps have been taken to reduce the impact of an Internet outage on our internal DNS resolution by pre-seeding the recursive nameservers with information about the ru.ac.za zone.

Whilst Internet access was restored at around 07:45 this morning it will have taken some services, particularly those heavily dependent on DNS, a while to stabilise. Most notably, the IMAP server only became available at about 08:45.