noticeboard.ru.ac.za

2015/05/16 - reverse DNS problem (ERR_NOVALIDDNS)
guy
Sat, 16 May 2015 10:56:06 +0200
During the course of this morning, the global root name servers stopped publishing reverse DNS referrals for Rhodes' IP address space. This has a number of implications, most notably that the proxy servers are displaying an ERR_NOVALIDDNS error for most sites. Other services may be slow to access, as they need to wait for DNS timeouts.

This is not a problem with any system at Rhodes; it a problem with parts of the domain name system that are upstream of us. We're in contact with the , African Network Information Centre (AFRINIC, the regional Internet registry) to try and resolve this. We're also trying to find ways to work around the problem within those systems on campus that are under our control.
guy
Sat, 16 May 2015 11:28:33 +0200
The problem seems to affect all legacy IPv4 address space in AFRINIC's service region, not just Rhodes. We've confirmed that a number of other organisations with legacy space are also affected. Legacy space is IP address space that was assigned by the American Registry for Internet Numbers before the creation of an African regional Internet registry -- as an early adopter of Internet in Africa, Rhodes makes use of such space for assignments on campus.

AFRINIC's offices in Mauritius are currently closed for the weekend, and they do not publish emergency contact details. We're trying to find other ways to reach them to try and escalate this.
drs
Sat, 16 May 2015 11:50:41 +0200
A work-around has been implemented to fix the problems that this has been causing within the campus network.
guy
Sat, 16 May 2015 12:06:17 +0200
People off campus who're trying to access Rhodes' services may still be affected, since the underlying problem still has not been resolved.

We've managed to make contact with AFRINIC via a round-about means, and they're investigating.
guy
Sat, 16 May 2015 12:41:05 +0200
We've just heard back that the problem is bigger than just legacy assignments. It affects other assignments within AFRINIC's service region too. Due to the nature of DNS, once it fixed, it may take a day or so for all the issues that have resulted from this to be resolved.
guy
Sun, 17 May 2015 07:23:54 +0200
The problem was resolved during the course of yesterday afternoon. We have received the following explanation from AFRINIC:

QUOTE(AFRINIC @ May 16 2015, 20:44 PM)
During today we had an issue with the system that provisions reverse DNS delegation data for many of the *.in-addr.arpa zones associated with legacy IPv4 addresses.

This problem affected reverse DNS for a number of allocations to AFRINIC members from some of the legacy /8 blocks that are shared between multiple RIRs. For these shared /8 ranges, the majority RIR manages the relevant xxx.in-addr.arpa zone, using input from the other, minority, RIRs.

The legacy /8 blocks where AFRINIC is in the minority, have reverse DNS delegation data stored and managed in the AFRINIC WHOIS database like any other resources. This information is then extracted, reformatted and shared with the systems of the relevant majority RIR automatically.

Earlier today, we were alerted to a problem with this process whereby the DNS zone information provided to the other RIRs was, in some cases, missing records.

This was traced to a recently provisioned testing system which was erroneously synchronising test data into our DNS provisioning system (and FTP) in parallel with valid WHOIS data.

This also slipped past existing monitoring and built in error checking; the leaking test data was not valid in content and very minimal, but it was never malformed or corrupt.

The testing system in question has been fixed. None of the AFRINIC production systems had any technical issues apart from having incorect data inputs.

We will be reviewing our processes of systems provisioning and configuration management. And we sincerley apologise for any and all inconvieniences that may have resulted.
post.5532738