The cause of our issues was an external Denial Of Service attack based around NTP (Network Time Protocol). It was targeted at a server here that used to provide that service to the outside world. We haven’t run an externally-facing NTP server in years, certainly not since we moved from 221. Blocking all access to …

First of all, the actual cause of the problem was a bad network link.  It was difficult to find, and we were just about to begin migrating the services out of the 221 data center into 240 when it was finally discovered. Here’s the link to Networking’s summary of what happened: https://slack-files.com/T025KBW9U-F025WFY2W-0c97c9 We discovered things were …

More details will come later today once we’ve hammered out what happened, but everything is back (or should be back shortly). Thanks for your patience, sorry for the troubles, and stand by for details. Love, Your humble team of tech monkeys

Still struggling to get things back up. We’re moving the database services over to a different datacenter in the hopes it will be able to start providing service there. There’s a lot of data to move. I’ll provide updates direct to twitter from this point on, and reserve e-mail/blog posts for more detailed info. If …

We’re still battling an issue we have yet to diagnose. At this point, we’re not expecting things to come back tonight. We’re working to move the services off the affected infrastructure to bring the critical services back. We’re getting outage pages into place for affected websites. Once we’re through this, I’ll provide a detailed explanation …

Our primary mysql database server is not allowing connections (or only sporadically allowing them), which is causing a number of services to fail including many websites. We’re working feverishly on getting it working again, but we have no ETA at this point since we’re still not able to get any real diagnostic info after multiple …

If you’re running MacOS 10.9 (Mavericks), please run software update to get the latest version (10.9.2). Aside from some bugfixes (including improved Exchange interopability), this contains a fix for a serious security issue with SSL certificate validation. This update was just released, and should be available via the App Store/Software Update. It can also be …

The instructions for reserving equipment at https://wiki.mcs.anl.gov/IT/index.php/Mail#Reserving_Equipment has been updated to reflect the new addresses and information. Thanks!

This work is happening at noon today. Thanks.