Electrical issue in the machine room – partial downtime.

Early this afternoon, the fire department was called by the folks in the basement level who reported an electrical burning smell.  Around the same time, one of the power panels in the machine room (one that powers half of Jazz) went down.After some investigating, they discovered two issues.  The first was that water was leaking into the building.  The second was that is was leaking into a power distribution panel.  While both are serious issues, the latter is the one that’s really hosed us.I’m not going to explain the entire power layout, but here’s a summary so you can understand the scope of the outage.  Power Panel 1 (the panel that went down) is irrevocably tied to Power Panels 2 through 4 — this means that all four have to be down whenever there’s work being done on them.  As you may be able to surmise, they’re kind of working on the water-soaked panel.  On top of that, because it’s now a wet area, it will be some time before we can get any power back to them.  Specifically, at least 24 hours.So, Power Panels 1 through 4 will be down until *at least* tomorrow afternoon.  At or prior to that time, I’ll send another announcement on the status of things.  The total outage should not last beyond Wednesday afternoon, but could be shorter (or intermittent).This affects Jazz, the Breadboard cluster and full disclosure (SC5832), the NMPDR Mac cluster, two racks of TeraGrid, a blower fan for BG/L, the rack that houses cvs.globus.org and the DSL development machines, and the tape backup server.  We’re going to try to get what we can online through any unused power (of the right type for the equipment), but the machine room is pretty much at capacity in terms of power usage, so there’s no guarantee at this point.Updates will come as we have information for you.