Unplanned network outage on Oct. 02, 2014

Last night, shortly before 6:00 pm CT one of the network routers in building 240 lost connectivity to another in 221. This caused a breakdown in large portions of the lab’s network fabric, both internally and externally. CIS tracked down and fixed the problem shortly after 8:00 pm CT. The result was that many, if not all, of our services lost connectivity during this time. Most services resumed normally after the issue was fixed, but a handful were left in unknown states until this morning. We believe we have now restored all services, but if you find something that isn’t responding or is responding erratically, please let us know and we’ll get it fixed.
CIS is investigating how such a disruption occurred since the network is designed to mitigate these types of disruptions.