Copying mail to MS-Exchange server using IMAP

I’m not really proud of the fact, but my work place has decided to migrate the mail server (after several tests of open source platforms) to Microsoft Exchange. For that we need to copy the current mail boxes of users from the server being phased out to the new MS-Exchange mail server.

In previous migrations we’ve used the excellent imapsync program which is written in perl and connects using IMAP to two servers and copies messages and folders from one server to another. Nonetheless, trying to get imapsync to work with the beast that is MS-Exchange has proved to be daunting. This is not really a rant (although you can read it as thus – I won’t mind at all 😉 ), but trying to get the thing to work I searched the web high and low and found precious little information about migrating to MS-Exchange1, so this is basically a summary of my recollection of the process in hopes that others will find this information useful2.

To make a long story short, the main problem is IMAP flags – A standard feature in the IMAP spec, MS-Exchange doesn’t really support them, as discussed in the project page for something called imap2exchange (In this project the author appears to be using MS-Exchange web services, which is not what I had in mind, but he explains IMAP flags and the MS-Exchange problem with them quite clearly). Strictly speaking, any message that has more then just the completely obvious flags (specifically, Seen only) is rejected with an error message:

BAD Protocol Error: "Specified set of flags is not valid".

I’ve seen some people offer a solution to just use an older version of imapsync (possibly because it does not attempt to sync flags), but I have a better one –

imapsync offers an undocumented switch called --regexflag which allows you to run a regular expression on the message flags before storing the message to the target server. I use this feature to remove almost all the flags that MS-Exchange doesn’t like. So my command line for imapsync look like this:


imapsync --fast --subscribe --noauthmd5 --skipsize --syncinternaldates --host1 source.mail.server --host2 target.mail.server --password1 'source-secret' --password2 'target-secret' --skipheader '^Content-Type' --user1 source-user --user2 target-user --regexflag 's/receipt-handled//' --regexflag 's/$Forwarded Forwarded//' --regexflag 's/\Answered//' --exclude Junk

The first regexflag substitution gets rid of the Zimbra private “receipt has been handled for this message” flag (yes, the source server is Zimbra). The second one gets read of the commonly used “message has been forwarded” flag (and a private Zimbra copy of the flag), while the third gets rid of the very standard “message has been replied to” flag which MS-Exchange doesn’t understand either.

MS-Exchange additionally doesn’t understand any junk mail flags, to in my scenario I simply do not copy the “Junk” folder under the assumption that it contains only SPAM that users do not want anyway. Another problem is the standard Deleted flag which MS-Exchange doesn’t like either, and under the same logic as the junk problem I’m going to assume that users do not want to keep deleted messages and simply let imapsync fail on those emails (the author of the imap2exchange project thinks that we should move such messages to the MS-Exchange deleted items folder, and he of course is right, but I’m too lazy to do that).

The main problems with losing flags, is that some are very important – I’m not aware of any one that the Answered flag is very important to, but there are other useful flags such as “mark for followup” which are rather important.

And if this was not trouble enough, these are not the only problems. Another problem with the migration is that MS-Exchange fails to store messages which are too long, and imapsync reports:

NO This message is too big and can not be appended.

By default MS-Exchange rejects messages that are over 10MB in size. 10MB is a default size for other mail agents as well, for example postfix, and its a good idea to increase this immediately after you install your mail server. After you change the appropriate setting in the MS-Exchange server configuration, it should be alright.

We haven’t finished the migration yet, but considering the above caveats, it looking OK for now. Of course, living with an MS-Exchange mail server is a whole new ball game, and about that – at a different time 🙂

  1. Hopefully that means that not a lot of people are doing it ? []
  2. And in higher hopes that others will not need this information 😉 []

4 Responses to “Copying mail to MS-Exchange server using IMAP”

  1. Dan:

    Just a note: Zimbra doesn’t do read-receipts yet, but when it does it’ll expose it in IMAP using the standard “$MDNSent” flag from RFC 3503. If you’re seeing “receipt-handled”, it’s probably one of your clients. (We’d love to know which one!)

    And sorry for the forced migration to Exchange. We’ll be here for you if and when your workplace reconsiders their choice of mail server…

  2. Guss:

    Thanks for the note about the flags. As most e-mail accounts in the company were used only with Zimbra’s mail client and Zimbra’s MS-Outlook connector, I assumed all the non-IMAP RFC stuff is Zimbra generated. I don’t remember in which account I’ve seen the receipt-handled flag, but it could be either mine (also used with Evolution) or the CEOs (also used with Blackberry servers).

    About the migration – I’m not happy with it but there was so much resistance to a non-MS solution even after the Zimbra trial, that it was impossible to keep the Zimbra server. Most likely it had something to do with our first few attempts at email server which were all F/LOSS based and were all flops to some degree or another, so much that even though Zimbra had delivered, people still wanted their MS fix.

  3. Stuart:

    Thankyou!

    I’ve actually had to embark on a Zarafa -> Exchange migration (personal project gone arwy :-/) and thus far (with your original pointers) this seems to be working and has the plus of not duplicating messages for every run (useful for large mailboxes with 200K+ messages specially when you’re sharing Virtualbox hosts :)):

    imapsync –fast –subscribe –noauthmd5 –skipsize –syncinternaldates –host1 [sourcehost] –host2 [desthost] –password1 [sourcepassword] –password2 [destpassword] –skipheader ‘^Content-Type’ –user1 [sourceuser] –user2 [destuser] –regexflag ‘s/receipt-handled//’ –regexflag ‘s/$Forwarded Forwarded//’ –regexflag ‘s/\Answered//’ –sep1 “/” –prefix1 “” –useheader ‘Message-ID’

    Hopefully we’ll all be good now as I cowtow to the missus saying ‘to just give me my mail/calendars/contacts!’

    Stu

  4. Granite lazy susan:

    As typical this was a thoughtful post today. You make me want to preserve coming back again and forwarding it my followers…….

Leave a Reply