[vox-tech] Email continuity in server OS transition (CentOS -> Ubuntu)

[bill broadley]

Hrm, well generally most (not all) email will retry for 24 hours or longer.  So 
the trick is to never return a failure, downtime isn't overly sensitive in most 
cases. To avoid failures use a test domain first that nobody else will notice if 
mail bounces.

There's one big decision to make:

 1.   Do you use migrate by changing DNS?  Use one IP address for the old
    machine and one for the new, and change DNS to a new IP when you are ready? 
    Changing DNS means that DNS caching at various layers might take days to
    fully settle.  You can set your TTL on the DNS records shorter, but some
    sites just (sadly) ignore the TTL.
 2. Have the new mail server take the old mail servers IP.  You can do with with
    an IP alias so the new mail server can accept connections from both IP
    addresses.

Because of the days and the complexity of receiving some email on a old server 
and some on the new I recommend changing the IP addresses and not DNS.

Once you are happy with the new mail server and successfully use a test domain 
to send and receive email without either side complaining.

 1. Email your old server from 2 domains and email from your old server to two
    domains.  On received email look closely at all headers for warnings.  On
    sent mail to gmail/yahoo look at the full headers on the receiving side. 
    Common complaints are wrong/broken/missing SSL certs, errors with SPF, DKIM,
    DMARC, or DNS lookups.  Fix problems on your old server *BEFORE* migration.
 2. on the old server turn off all IMAP (ssl 993 and non 143), smtp (ssl 25 and
    non 465), pop (ssl 995 and non 110), any web mail, mailing lists, or
    related.  At this point the maildir/spools should not have ANY changes.
 3. sync any hard coded email aliases
 4. sync (with scp or rsync) all mail spools (both inboxes and mail folders)
 5. Sync any mail archives from mailing lists
 6. Verify the new mail server has enough disk space before going further.
 7. double check everything
 8. change old server IP to something else
 9. change new server IP to mail server IP (or just add it with an IP alias)
10. bring up smtp, clamav, graylisting, anti-spam, but not IMAP, pop, or webmail
    (let the machine receive email, but not show it to users)
11. send a test email from at least 2 mail domains to your new mail server,
    watch the logs for warnings or errors.
12. send a test email from your new mail server to at least 2 outside mail
    domains and view the full headers on the destination, pay particular
    attention to any errors with DKIM, SPF, and dmarc.
13. if things look good enable imap, webmail, and mailing lists so your users
    can receive email
14. Get your backups working and test said backups.

I typically refresh the mail server with the best pieces I can find, my 
recommendations for any new mail server:

 1. Definitely use maildir over mbox, faster, safer, and more scalable.  Much
    easier to backup as well.
 2. Use sieve over procmail for mail filtering
 3. Enable graylisting, or at least consider it.  I think it's a win, but as
    with all things there's a compromise.
 4. Keep domains in a database, it's VERY nice to be able to add domains with a
    single command at any time
 5. Keep users in a database, it's very nice to be able to safely edit and not
    worry that you added the wrong character corrupts some config file
 6. Keep aliases and forwardings in a database.
 7. I prefer postfix, but use something current anyways, not sendmail
 8. I prefer dovecot, but use something current
 9. Use a smart delivery agent that knows about mail filtering, I like dovecot's
    deliver
10. Don't forget backups.