Migration of IMAP accounts of Support and Billing questions to dedicated distributed servers
Emin Gabrielyan
2009-10-04
Migration of IMAP accounts of Support and Billing questions to dedicated distributed servers
Steps remaining for completing the migration of the Billing account
Importing IMAP cache files as local mailboxes
IMAP servers and the folder hierarchy of Switzernet’s Billing and Support accounts are being migrated to dedicated servers. The load will be distributed across several machines. The previous hosted solution raised serious issues with simultaneous connections and will be soon abandoned.
Two copy/backup methods of source IMAP folders are compared. Though server based imap transfer techniques exist, both our examined methods are based on the end user Thunderbird application. In both methods all concerned IMAP folders on the source server are marked for offline use and we completely sync thunderbird with the server. Creation of local copies is finalized only after the folders are synced completely and without error (done as of 2009-10-02).
With the 1st method we copy the synced IMAP folders into local folders from within Thunderbird while staying in off-line mode (which is important). With the 2nd method, we shut down the Thunderbird application, and we copy the hierarchy of synced IMAP folders from the file-system into the space corresponding to the Thunderbird’s local folders (files must be copied only when you are sure that no Thunderbird process is running in background).
Several misplaced folders (not respecting the structure of the how-to doc) are corrected manually. All misused dashes (‘-‘) in folder names are replaced by underlines (‘_’) according to the convention (the html bookmarks to chapters describing particular IMAP folders in our how-to docs support only alphanumerical symbols and underlines, but not dashes).
The table below shows the message counts and mailbox sizes of IMAP folders (a) on the original IMAP server (see the first column), (b) in the local Thunderbird directory copied from within Thunderbird (see the second column), (c) in the local Thunderbird directory imported from IMAP cache files (see the third column), and (d) on the destination IMAP server after a successful upload (see the fourth column).
For the final transfer path (source IMAP server – local Thunderbird – destination IMAP server), we have chosen the 1st copy method based on the copy of folders from within Thunderbird (the application being kept in the off-line mode during the copy). The numbers of messages in mailboxes imported from the file-system (according to the 2nd method) are given for comparison only and they show frequent discrepancies. Interpretation of IMAP cache folders as local thunderbird folders works, but frequent mismatches of message numbers take place because certain labels (e.g. for deleted messages) in imported cache files are not interpreted properly. Moreover, the manual tags (colors) and flags associated with messages are also lost when importing the IMAP cache files as local mailboxes.
When copying from within Thunderbird the tags and flags associated with messages were preserved (though tags were lost when mailboxes, but not the individual messages, are further re-copied in the local space, but this shall be a bug of Thunderbird).
There are few counter discrepancies also with the copy carried out according to the 1st method (see the first and second columns). The discrepancies between the source and the local copy should be explained only by fresh messages (according to a few occasional checks) and hopefully do not indicate on any error. New messages arrived because, thunderbird went on-line in between when restarted for importing the IMAP cache based version.
Steps remaining for completing the migration of the Billing account
Establish
the automatic forwarding from the source server to the destination server.
Move
the team members to the new server but keep temporarily the access to the old
server.
When
the last or most team members are migrated to the new server, block the access
to the old server.
Transfer
all new messages (since 2009-10-02) from all IMAP folders of the source server
to the corresponding counterparts of the destination server. Transfer also the
unsorted emails of inbox.
Folder
names in the destination IMAP server are changed so as to solve the location-path
problem of answered messages in Thunderbird search tools. The convention is
easy to follow by looking at the names. Note that numbering order of all
folders must be refreshed (manually!) whenever a new IMAP folder is added. If
several IMAP folders are to be added in a short period of time, use temporarily
in new names ‘x’ for sequential numbers. Renumber all folders of the concerned
class at once, when no more types are considered for addition in a week, or so.
Comparison of mailbox counters on source and destination IMAP servers as well as on the local computer
|
On the source IMAP server |
Copy of source IMAP folders carried out from within Thunderbird in off-line mode after a complete successful sync (1st copy method) |
Rebuilding local Thunderbird folders from the files of IMAP sync/cache folders on the disk (2nd method) |
On the destination IMAP server (uploaded from local folders of the 1st copy method) |
|
Billing’s 01 Explain on IMAP |
Billing’s 01 Explain copied in off-line mode |
Billing’s 01 Explain rebuild from IMAP sync files |
Billing’s 1 Explain on the destination (zimbra) IMAP server |
|
|
‘blocked_free_calls’ is empty [solved]
Number mismatches (with 1st column): ‘account_balance’ (ok after refresh) ‘blocked_free_calls’ ‘english_invoice’ ‘offers’ ‘payment_methods’ ‘payment_note’ ‘pdf_format’ |
Not user for the transfer to the destination server |
No mismatches with the 2nd column |
|
Billing’s 02 Cash on IMAP |
Billing’s 02 Cash copied in off-line mode |
Billing’s 02 Cash rebuild from IMAP sync files |
Billing’s 2 Cash on the destination (zimbra) IMAP server |
|
|
Mismatches with the 1st column: ‘lost_payment’ |
Not used for the transfer to destination server |
No mismatches with the 2nd column |
|
Billing’s 03 Account on IMAP |
Billing’s 03 Account copied in off-line mode |
Billing’s 03 Account rebuild from IMAP sync files |
Billing’s 3 Account on the destination (zimbra) IMAP server |
|
|
Mismatches with the 1st column: ‘contract_cancel’ ‘email_bill_again’ ‘free_billed’ ‘refuse_to_pay’ |
Not user for the transfer to the destination server |
Mismatches with the 2nd column: 3,10,re (ok after refresh) 3,11,re (ok after refresh) |
|
Support’s 01 Reception on IMAP |
Support’s 01 Reception copied in off-line mode |
Support’s 01 Reception rebuild from IMAP sync files |
Support’s 4 Reception on Zimbra |
|
|
|
|
The hardware for the IMAP server is pending |
|
Support’s 02 Phone on IMAP |
Support’s 02 Phone copied in off-line mode |
Support’s 02 Phone rebuild from IMAP sync files |
Support’s 5 Phone on Zimbra |
|
|
|
|
The hardware for the IMAP server is pending |
|
Support’s 03 Service on IMAP |
Support’s 03 Service copied in off-line mode |
Support’s 03 Service rebuild from IMAP sync files |
Support’s 6 Service on Zimbra |
|
|
|
|
The hardware for the IMAP server is pending |
Importing IMAP cache files as local mailboxes
Before copying the file-system folders (according to the 2nd method), the empty zombie folders left in the file-system by deleted or renamed folders in the past, are removed manually as well.
The following log shows the details of procedures carried out before importing the IMAP cache files as local Thunderbird folders. The MSF index files are deleted (so the indexes can be rebuilt from the scratch) and empty mailbox files are created for empty folders (no such files exist for empty mailboxes in case of IMAP caching). Recall that these folders however are not used for uploading.
Emin.Gabrielyan@pc8 /cygdrive/f/people/emin.gabrielyan/thunderbird/Local Folders/archive.sbd/091002 billing@ support@ sync.sbd
$ find . -name "*.msf"
./billing.sbd/01 Explain.sbd/account_balance.msf
./billing.sbd/01 Explain.sbd/account_balance.sbd/answered.msf
./billing.sbd/01 Explain.sbd/blocked_free_calls.msf
./billing.sbd/01 Explain.sbd/blocked_free_calls.sbd/answered.msf
./billing.sbd/01 Explain.sbd/contact_me.msf
./billing.sbd/01 Explain.sbd/contact_me.sbd/answered.msf
./billing.sbd/01 Explain.sbd/credit_warning.msf
./billing.sbd/01 Explain.sbd/credit_warning.sbd/answered.msf
./billing.sbd/01 Explain.sbd/english_invoice.msf
./billing.sbd/01 Explain.sbd/english_invoice.sbd/answered.msf
./billing.sbd/01 Explain.sbd/explain_invoice.msf
./billing.sbd/01 Explain.sbd/explain_invoice.sbd/answered.msf
./billing.sbd/01 Explain.sbd/final.msf
./billing.sbd/01 Explain.sbd/final.sbd/answered.msf
./billing.sbd/01 Explain.sbd/howto_refer.msf
./billing.sbd/01 Explain.sbd/howto_refer.sbd/answered.msf
./billing.sbd/01 Explain.sbd/offers.msf
./billing.sbd/01 Explain.sbd/offers.sbd/answered.msf
./billing.sbd/01 Explain.sbd/online_access.msf
./billing.sbd/01 Explain.sbd/online_access.sbd/answered.msf
$ find . -name "*.msf" | perl -ne 's/(\.sbd\/answered)?\.msf$//; print' | uniq -c
2 ./billing.sbd/01 Explain.sbd/account_balance
2 ./billing.sbd/01 Explain.sbd/blocked_free_calls
2 ./billing.sbd/01 Explain.sbd/contact_me
2 ./billing.sbd/01 Explain.sbd/credit_warning
2 ./billing.sbd/01 Explain.sbd/english_invoice
2 ./billing.sbd/01 Explain.sbd/explain_invoice
2 ./billing.sbd/01 Explain.sbd/final
2 ./billing.sbd/01 Explain.sbd/howto_refer
2 ./billing.sbd/01 Explain.sbd/offers
2 ./billing.sbd/01 Explain.sbd/online_access
2 ./billing.sbd/01 Explain.sbd/payment_methods
2 ./billing.sbd/01 Explain.sbd/payment_note
2 ./billing.sbd/01 Explain.sbd/pay_yearly
2 ./billing.sbd/01 Explain.sbd/pdf_format
$ find . -name "*.msf" | while read s; do rm "$s"; done
$ find . -name "*.sbd" | while read s; do f=`echo $s | perl -ne 's/\.sbd$//; print'`; if [ ! -f "$f" ]; then echo $f; fi; done
./billing.sbd/02 Cash.sbd/canceled_payment
./billing.sbd/03 Account.sbd/contract_resend
./billing.sbd/03 Account.sbd/private_to_commercial
$ find . -name "*-*"
./billing.sbd/03 Account.sbd/address-change
./billing.sbd/03 Account.sbd/address-change.msf
./billing.sbd/03 Account.sbd/address-change.sbd
To do for the Support classes
The
same scenario shall be followed for the migration of Support accounts, but the
three class indexes “01”, “02”, and “03” shall be replaced by “4”, “5”, and “6”
in case we will be interested in a central backup account sharing Support and
Billing (maybe with imapsync tool).
Type
names in Support classes must be renamed according to same conventions (space
separated suffixes, such as “4,1” in type names and unique names, such as
“4,1,re” for formerly “answered” folders)
The
class folders (such as “4 Reception”) must contain control messages showing the
total number of type sub-folders, their names, and links to the bookmarks of
the how-to docs.
References:
A tool for facilitating incremental recursive IMAP transfers from one mailbox to another (bugs, misuse, or source server limitations caused us to abandon this solution)
http://freshmeat.net/projects/imapsync/
On-line statistics of answered emails of support@ and billing@ accounts of Switzernet:
http://switzernet.com/public/090309-support-answers/last/
http://www.unappel.ch/public/090309-support-answers/last/
The application generating the on-line statistics of answers:
http://switzernet.com/public/090309-support-answers/
http://www.unappel.ch/public/090309-support-answers/
Zimbra collaboration suite:
* * *
Copyright © 2009, Switzernet