Delayed email to mcs.anl.gov

This morning, we got reports of mail sent to addresses in “mcs.anl.gov” from outside Argonne were not making it to us.

A brief investigation showed that the Domain Name Service (DNS) records that identify where mail should be delivered (MX records) were missing for mcs.anl.gov.  We replaced the records, and now the DNS servers are answering with the correct information.  Mail is starting to come in again.

Here is the sequence of events. Yesterday afternoon, I was working in our DNS management tool cleaning up some records for domains we manage.  I was attempting to remove the MX records from one domain that didn’t need them, but evidently was working in the wrong window, and instead removed them from mcs.anl.gov.  As the old information slowly aged out of the various DNS servers across the internet, new queries to refresh the information were not returning anything.  So the mail servers would not know where to deliver the message.

Mail sent during this period should be retried automatically, however that timing is up to the external server.  You should not be surprised to see messages trickle in out of order.

On a personal note, I want to say I’m really sorry about this mixup — this one’s totally on me for not reading the “are you sure?” message closely enough. This will teach me to keep the number of browser tabs open in the low double-digit range.