Migrating from Mailman 2.1 to Mailman 3
=======================================

This guide covers the migration steps required for porting from Mailman 2.1 to
Mailman 3.


Why Upgrade?
------------

Mailman 3 is the new version and actively developed, as compared to 2.1, which
is now in maintenance mode and won't receive any feature updates.

Some of the reasons that might convince you to migrate to Mailman 3 include

- a well tested and rewritten code base,
- support for REST API (if you need to integrate with other services)
- support for multiple domains in same installation (without listname collisions)
- A real database backend for settings and configuration
- Modern Web UI for list administration and subscription management
- Support for social logins (and possibly LDAP/openid integration)
- Much improved and interactive archiver/forum with ability to reply from within
  the Archiver


Other considerations
--------------------

Before you upgrade, you should consider a few things like:

- URLs to archived messages will break, unless you take extra steps to keep them
  around. Upgrade mechanism makes sure to import all your archived messages in
  the new system, but, all the URLs to the new messages are going to be different.

  If you need your URLs for Mailman 2 archives to work, you can keep the HTML
  files generated for the archives around and your web server configuration for
  the archives intact (possibly with a notice to viewers that it is now a
  read-only archive, see `this list
  <https://mail.python.org/pipermail/security-sig/>`_ for example).

- The above mechanism won't work for private archives since the archives are
  gated with password and without a Mailman 2 list, there is no password. You
  can however import them to Mailman 3.

- Some configuration and settings aren't available in Mailman 3's UI yet, so
  even though those settings will be migrated to Mailman 3, you may not be able
  to change them from the Web UI today. All of those settings should be exposed
  in the UI very soon.

- Importing archives with hyperkitty_import will never send outgoing
  emails. The messages are only imported from the old archive to the new archive.

- hyperkitty_import can be run multiple times with an input mbox containing some or all
  of the same messages. Regardless of other settings, messages which exist in the archive
  will not be re-imported on subsequent runs.

- After running import21 to import a list, those members will immediately be
  able to receive and send mail, and generally participate in the list. They
  will be members at the level of the mailman3_core database.  However if they
  choose to manage their subscriptions in the web UI, which offers many more
  options, it will require going to "Sign Up" in the upper right corner of the
  website, and creating a new Django account. See further discussion about this
  on the Architecture Page in the section "Understanding User and Member
  Accounts".

Before you upgrade
------------------

- Make sure that you have no pending subscription requests as those will not be
  ported over to Mailman3.

- Make sure that you don't have any pending emails in the digest mbox, if there
  are, you can force send the digests before moving to Mailman 3, as those won't
  be upgraded to Mailman3.


Upgrade strategy
----------------

As of now, there isn't a turn key solution to migrate all your lists from Mailman
2 to Mailman 3, although, the process is fairly automated. You need shell access
to your installation in order to perform the upgrade.

Mailman 3 is split in two main parts, the Core engine which includes all the
lists and their configuration and the Web UI, which includes the archiver and UI
for information in Core.

Before you start with migration, you need a working Mailman 3 instance, you can
see `here <http://docs.mailman3.org/en/latest/>`_  for recommendations on
installing Mailman.

Some key information that you should know about your Mailman 2 and Mailman 3
installations before you do the migration:

- Location of list configuration in Mailman 2: This is typically at
  ``$var_prefix/lists/LISTNAME/config.pck`` where ``$var_prefix`` is an
  installation dependent directory which is typically ``/usr/local/mailman`` or ``/var/lib/mailman``.

- Location of list archives in Mailman 2: This is typically at Mailman 2: This is at
  ``$var_prefix/archives/private/LISTNAME.mbox/LISTNAME.mbox`` where ``$var_prefix``
  is as above.

- Location of the ``bin/`` commands in Mailman 2: This is at ``$prefix/bin`` where ``$prefix`` is an
  installation dependent directory which is typically ``/usr/local/mailman`` or ``/usr/lib/mailman``


Steps for migration:

- Create the list you are trying to migrate in Mailman 3, for the purposes of
  this guide, we will call it ``foo-list@example.com``

- Migrate the list configuration from Mailman 2 to Mailman 3 by running the
  following command [#fn1]_ ::

	$ mailman import21 foo-list@example.com /path/to/mailman2/foo-list/config.pck


.. [#fn1] This, and in general all, ``mailman`` commands should be run as the
  Mailman user and not as ``root``.  Running some of these commands as
  root can create files owned by root that can't be read by the
  Mailman user.


- Migrate the list archives from Mailman 2 to Mailman 3 by running the following
  command [#fn2]_::

	$ python manage.py hyperkitty_import -l foo-list@example.com $var_prefix/archives/private/foo-list.mbox/foo-list.mbox


.. [#fn2] If the Mailman 2 list does not predate Mailman 2.1, its LISTNAME.mbox
  file is probably in good shape, but all mailboxes should be checked
  for defects before importing.  Certain defects such as missing Message-ID: headers or missing
  or unparseable Date: headers will be corrected or ignored by the import process. The one defect
  that will definitely cause problems is lines beginning with From in message bodies.  These
  will be seen as the start of a new message.  There is a ``cleanarch3`` script in ``hyperkitty/contrib``
  that can identify and fix most such lines, but it is not perfect.  Cases have been observed
  where a post includes in its body a copy of some other message including the From separator.
  This will normally occur only on an old list which includes spam messages or other email problems
  in its subject matter, but is something to be aware of.  Certain other
  message defects can cause the import to abort.  There is a
  ``check_hk_import`` script in ``hyperkitty/contrib`` that can find and
  report messages with these defects.


After this, you will need to rebuild the index for this list::

	$ python manage.py update_index_one_list foo-list@example.com

After this, you should be able to search your messages in Hyperkitty.

- Delete Mailman 2 list::

	$ $prefix/bin/rmlist foo-list


After this, you may need some additional steps based on if you want to keep your
old archives around or not. To add a notice to your list archives, you can edit
``index.html`` available at the root of your mailing list archives.

You may also want to add a redirect at old list page to automatically redirect
your users to Mailman3 list-info page. However, this process is fairly manual
depending on type of webserver you are using.