Migrating from Mailman 2.1 to Mailman 3

This guide covers the migration steps required for porting from Mailman 2.1 to Mailman 3.

Why Upgrade?

Mailman 3 is the new version and actively developed, as compared to 2.1, which is now in maintenance mode and won’t receive any feature updates.

Some of the reasons that might convince you to migrate to Mailman 3 include

• a well tested and rewritten code base,

• support for REST API (if you need to integrate with other services)

• support for multiple domains in same installation (without listname collisions)

• A real database backend for settings and configuration

• Modern Web UI for list administration and subscription management

• Support for social logins (and possibly LDAP/openid integration)

• Much improved and interactive archiver/forum with ability to reply from within the Archiver

Other considerations

Before you upgrade, you should consider a few things like:

• URLs to archived messages will break, unless you take extra steps to keep them around. Upgrade mechanism makes sure to import all your archived messages in the new system, but, all the URLs to the new messages are going to be different.

  If you need your URLs for Mailman 2 archives to work, you can keep the HTML files generated for the archives around and your web server configuration for the archives intact (possibly with a notice to viewers that it is now a read-only archive, see this list for example).

• The above mechanism won’t work for private archives since the archives are gated with password and without a Mailman 2 list, there is no password. You can however import them to Mailman 3.

• Some configuration and settings aren’t available in Mailman 3’s UI yet, so even though those settings will be migrated to Mailman 3, you may not be able to change them from the Web UI today. All of those settings should be exposed in the UI very soon.

• Importing archives with hyperkitty_import will never send outgoing emails. The messages are only imported from the old archive to the new archive.

• hyperkitty_import can be run multiple times with an input mbox containing some or all of the same messages. Regardless of other settings, messages which exist in the archive will not be re-imported on subsequent runs.

• After running import21 to import a list, those members will immediately be able to receive and send mail, and generally participate in the list. They will be members at the level of the mailman3_core database. However if they choose to manage their subscriptions in the web UI, which offers many more options, it will require going to “Sign Up” in the upper right corner of the website, and creating a new Django account. See further discussion about this on the Architecture Page in the section “Understanding User and Member Accounts”.

Before you upgrade

• Make sure that you have no pending subscription requests as those will not be ported over to Mailman3.

• Make sure that you don’t have any pending emails in the digest mbox, if there are, you can force send the digests before moving to Mailman 3, as those won’t be upgraded to Mailman3.

Upgrade strategy

As of now, there isn’t a turn key solution to migrate all your lists from Mailman 2 to Mailman 3, although, the process is fairly automated. You need shell access to your installation in order to perform the upgrade.

Mailman 3 is split in two main parts, the Core engine which includes all the lists and their configuration and the Web UI, which includes the archiver and UI for information in Core.

Before you start with migration, you need a working Mailman 3 instance, you can see here for recommendations on installing Mailman.

Some key information that you should know about your Mailman 2 and Mailman 3 installations before you do the migration:

• Location of list configuration in Mailman 2: This is typically at $var_prefix/lists/LISTNAME/config.pck where $var_prefix is an installation dependent directory which is typically /usr/local/mailman or /var/lib/mailman.

• Location of list archives in Mailman 2: This is typically at Mailman 2: This is at $var_prefix/archives/private/LISTNAME.mbox/LISTNAME.mbox where $var_prefix is as above.

• Location of the bin/ commands in Mailman 2: This is at $prefix/bin where $prefix is an installation dependent directory which is typically /usr/local/mailman or /usr/lib/mailman

Steps for migration:

• Create the list you are trying to migrate in Mailman 3, for the purposes of this guide, we will call it foo-list@example.com

• Migrate the list configuration from Mailman 2 to Mailman 3 by running the following command [1]

  $ mailman import21 foo-list@example.com /path/to/mailman2/foo-list/config.pck
  

[1]

This, and in general all, mailman commands should be run as the Mailman user and not as root. Running some of these commands as root can create files owned by root that can’t be read by the Mailman user.

• Migrate the list archives from Mailman 2 to Mailman 3 by running the following command [2]:

  $ python manage.py hyperkitty_import -l foo-list@example.com $var_prefix/archives/private/foo-list.mbox/foo-list.mbox
  

[2]

If the Mailman 2 list does not predate Mailman 2.1, its LISTNAME.mbox file is probably in good shape, but all mailboxes should be checked for defects before importing. Certain defects such as missing Message-ID: headers or missing or unparseable Date: headers will be corrected or ignored by the import process. The one defect that will definitely cause problems is lines beginning with From in message bodies. These will be seen as the start of a new message. There is a cleanarch3 script in hyperkitty/contrib that can identify and fix most such lines, but it is not perfect. Cases have been observed where a post includes in its body a copy of some other message including the From separator. This will normally occur only on an old list which includes spam messages or other email problems in its subject matter, but is something to be aware of. Certain other message defects can cause the import to abort. There is a check_hk_import script in hyperkitty/contrib that can find and report messages with these defects.

After this, you will need to rebuild the index for this list:

$ python manage.py update_index_one_list foo-list@example.com

After this, you should be able to search your messages in Hyperkitty.

• Delete Mailman 2 list:

  $ $prefix/bin/rmlist foo-list
  

After this, you may need some additional steps based on if you want to keep your old archives around or not. To add a notice to your list archives, you can edit index.html available at the root of your mailing list archives.

You may also want to add a redirect at old list page to automatically redirect your users to Mailman3 list-info page. However, this process is fairly manual depending on type of webserver you are using.