We were building the replacement disk array overnight (it was still building when I got in this morning), and I decided to see if I couldn’t get the original array into a useful state. Lo and behold, it had become responsive again overnight, and was back online. I did a few tests, and all looks …

The disk array housing the Y and Z drives appears to have failed in a catastrophic way. We’re reconfiguring an unused array tonight and will start to restore files from tape backup tomorrow. Once the restore starts, we should be able to provide an ETA.

All servers but one (BIOWINFS, the file server for Y and Z drives) are back. The storage array for BIOWINFS is in a bad state. We’re still trying to bring it back. No ETA yet. If we can’t get it back, we will need to restore from tape backups to new storage. I will post …

Power work in TCS has unexpectedly affected the servers hosted in TCS. We’ve restored power to the affected servers and disks and are in the process of recovering the machines. Will send an all-clear when this is complete. This affects the shared drives Y, Z, and linux fileservers. Any services not functioning as expected are …

A maintenance window of our GitLab services has been planned for 2017-06-28 (Wednesday) from 12:00 to 13:00 (CDT) in order to update the software. GitLab services provided by gitlab.cels.anl.gov and xgitlab.cels.anl.gov will be unavailable during maintenance. Service is expected to be restored by 13:00, and may be available sooner than that. If this window poses …

The scheduled reboot of the compute nodes has been completed. brad From: "Fritz, Brad" <[email protected]> Date: Thursday, June 22, 2017 at 12:14 PM To: "[email protected]" <[email protected]> Subject: MCS compute nodes We are planning to reboot the MCS compute nodes on June 23rd, starting at 9am CST, to apply updates. Servers will be rebooted in the …

The scheduled reboot of the login nodes is now complete. brad From: "Fritz, Brad" <[email protected]> Date: Thursday, June 22, 2017 at 9:36 AM To: "[email protected]" <[email protected]> Subject: MCS login nodes The MCS login nodes 1-4 will be rebooted this evening, June 22nd, at 5pm to apply updates. The logins will be rebooted one at a …

We are planning to reboot the MCS compute nodes on June 23rd, starting at 9am CST, to apply updates. Servers will be rebooted in the following order, one at a time. compute001.mcs.anl.gov thwomp.mcs.anl.gov stomp.mcs.anl.gov crush.mcs.anl.gov crank.mcs.anl.gov steamroller.mcs.anl.gov grind.mcs.anl.gov churn.mcs.anl.gov trounce.mcs.anl.gov thrash.mcs.anl.gov vanquish.mcs.anl.gov octagon.mcs.anl.gov octopus.mcs.anl.gov Only one compute node should be down at any given time. …

The MCS login nodes 1-4 will be rebooted this evening, June 22nd, at 5pm to apply updates. The logins will be rebooted one at a time, in order. Only one login node should be down at any given time. Brad

As a followup to the announcement I sent earlier, and after forwarding on the feedback I received (as did the others in the lab), we got a great response from BIS on this. I’ll just quote it verbatim, because it made me smile. This comes from Dave Salbego. You guys have forwarded some very informative …