Django for Mailman Admins

This is a brief primer on Django to help with administration of Mailman as it is used as a web framework within the web interface of Mailman 3.

Ideally, Django should just be a python dependency and the details should be wrapped behind configurations and settings in Mailman. However, due to the amount of functionality provided to us by Django, it would be a monumental effort to hide it completely. We often resort to referring Django documentation directly in our documentation and also rely on several deployment related features that Django provides.

There are three main topics that is important for system administrators that we are going to cover in this doc, configuration, management commands and running (deployment).

Configuration

Django expects its configuration to be provided via a settings module. The name of the module itself isn’t a constant and can be passed in as --settings command line parameter in Django commands and also be set using DJANGO_SETTINGS_MODULE environment variable.

The un-usual thing to note here is that it expects a module not a file path, so whatever the name of the module is, it should be importable by Django(in Python), which can be achieved by adding the directory the file settings.py exists in the PYTHONPATH environment variable. It works similar to the PATH environment varible that unix shell use.

Since it is using Python, you can use more Pythonisms in how the configuration is defined or loaded. For example, a common pattern is to keep settings.py as close to the one provided by the upstream project, but make the changes required in a settings_local.py and then import the configuration in settings.py.

# settings.py

<SNIP>

try:
    from settings_local import *
except ImportError
    # If the import fails, don't worry.
    pass

# settings_local.py

DATABASES = ...
SECRET_KEY = ...

This way, you can keep the “local” configuration overrides in a separate file. You can also take this a step further to put secrets into a secrets.py and import from there in settings.py if you want to separate secrets from other configuration.

Django upstream settings

Django’s upstream settings documentation is probably the most useful place to lookup the variables you might find in the Mailman provided settings.py but don’t fully understand how to configure.

All the settings are applicable directly, including things like LOGGING (to setup logging), DATABASES(to setup backend database configuration), USE_I18N(Internationalization of the interface), EMAIL_BACKEND (how to talk to local or remote MTA for sending emails).

Mailman-web configuration

On top of these base settings provided by Django, the configurations for Postorius, Hyperkitty and django-mailman3 all belong to the same settings.py (or settings_local.py, if you choose to use that).

There are other third party Django applications, which we reuse for various functionality in Mailman-web use similar configuration process via settings.py.

The most important packages for Mailman are:

  • django-allauth: Good place to find any configuration related to account management in the Web UI from email sending issues to contents of the email sent for account/email verification.

  • Django-haystack: This is used for search indexing of the emails in Hyperkitty. This is a good place to find various settings for setting up different search backends.

  • Django-qcluster: This will provide all the asynchronous jobs that are run via cron in Hyperkitty to update various internals in a routine fashion. Use this to see how different backends can be configured to store and run jobs and how to tweak the configuraiton of each backend.

Management Commands

This is probably the most useful (and confusing!) this about Django due to the sheer number of ways that this can be done. Django management commands provide a framework to write and run commands for Django web applications for various manual or other tasks.

Django’s documentation on management commands is a good place to start, but if it is too long for you, this is a short know how for Mailman admins.

There are three variants for running management commands, all of which can be use interchangeably, depending on how you installed. Package managers, Docker images and mailman-web all provide different ways to run the same commands.

  • python3 manage.py --settings settings --pythonpath <parent dir of settings.py> <commamd>: This command is probably the _most_ common one that you can see. You will notice that it accepts --settings and --pythonpath flags to provide the settings module of Django. You can also set the environment variables DJANGO_SETTINGS_MODULE and PYTHONPATH instead of passing the values to command.

  • django-admin --settings settings --pythonpath <parent dir of settings.py> <command>: This is exactly same as the above, replace the “manage.py” file with an installed command django-admin, obviating the need to find the manage.py file in your current installation. The same environment variables mentioned in the above marker works here as well.

  • mailman-web <command>: This is the simplified version of the above two commands, which comes with the mailman-web package. It defaults the settings.py to be present at /etc/mailman3/settings.py and automatically sets up the PYTHONPATH for Django to be able to import. You can also set MAILMAN_WEB_CONFIG environment variable to point to a different config file (not module!) and it will take care of adding it to the PYTHONPATH and setting DJANGO_SETTINGS_MODULE.

The documentation of Mailman is slowly going to lean towards using mailman-web command as it is simple, but third party packages and Django can emit erorrs that use any of the above three notations. Which is why it is good to know them. You can use mailman-web <command> for commands from Django (like, migrate or createsuperuser for example) or from third party packages (like qcluster or qinfo).

Running Django

Running Django based web applications follows a Python standard mechanism called WSGI, which describes how a web server communicates with Python applications. There is also ASGI for async based applicaitons, but Mailman currently doesn’t use it.

You can use any WSGI server to run mailman-web. Our current documentation recommends using uwsgi, but gunicorn is also a great option. Some web servers like Apache2 come with wsgi modules, which obviate the need for a separate WSGI server like uwsgi, gunicorn. So, if your current infrastructure is already using Apache2, this might be an option for you.

It is good to refer to uwsgi docs or gunicorn docs if you want to configure the aspects of running Django that aren’t already done in Django, which includes things like, access/error logging, number of threads and processes to run which can handle your expected load, SSL configuration (if you want SSL all the way through, instead of terminating at web server) etc.

Most (all?) WSGI servers will require only the path to the (1) Django’s settings and (2) WSGI “module”. This wsgi module, is often times just a wsgi.py file somewhere on PYTHONPATH. If you are using mailman-web then it comes pre-installed and you can refer to it as mailman_web.wsgi, without worrying about the file itself.

Since web servers are more hardened in terms of security and more capable of handling high load, _typically_, WSGI servers listen on a local addresses and Web server proxy requests from Internet to the Wsgi server.