A number of core ICT services have recently failed or begun behaving erratically.
The underlying problem appears to be related to storage, and specifically the FibreChannel link between Struben and AMM. This is not directly related to the changes that occurred last night, but those changes may have exposed an existing, underlying problem with the fibre optic cable connecting our storage systems. It appears that this fibre pair is causing intermittent connectivity between various servers and our storage systems, which is resulting in servers repeated changing the storage path they're using to access storage. Services that have high disk I/O have been most affected byt this.
We're in the process of installing a new fibre cable to replace the one we suspect is faulty.
In the mean time we've disabled the faulty storage path in an attempt stablise the situation. We're in the process of determining, and restarting the affected services.