Django for Mailman Admins
This is a brief primer on Django to help with administration of Mailman as it is used as a web framework within the web interface of Mailman 3.
Ideally, Django should just be a python dependency and the details should be wrapped behind configurations and settings in Mailman. However, due to the amount of functionality provided to us by Django, it would be a monumental effort to hide it completely. We often resort to referring Django documentation directly in our documentation and also rely on several deployment related features that Django provides.
There are three main topics that is important for system administrators that we are going to cover in this doc, configuration, management commands and running (deployment).
Configuration
Django expects its configuration to be provided via a settings
module. The
name of the module itself isn’t a constant and can be passed in as
--settings
command line parameter in Django commands and also be set using
DJANGO_SETTINGS_MODULE
environment variable.
The unusual thing to note here is that it expects a module not a file
path, so whatever the name of the module is, it should be importable by
Django(in Python), which can be achieved by adding the directory the file
settings.py
exists in the PYTHONPATH
environment variable. It works
similar to the PATH
environment varible that unix shell use.
Since it is using Python, you can use more Pythonisms in how the configuration
is defined or loaded. For example, a common pattern is to keep settings.py
as close to the one provided by the upstream project, but make the changes
required in a settings_local.py
and then import the configuration in
settings.py
.
# settings.py
<SNIP>
try:
from settings_local import *
except ImportError
# If the import fails, don't worry.
pass
# settings_local.py
DATABASES = ...
SECRET_KEY = ...
This way, you can keep the “local” configuration overrides in a separate
file. You can also take this a step further to put secrets into a
secrets.py
and import from there in settings.py if you want to separate
secrets from other configuration.
Django upstream settings
Django’s upstream settings documentation is probably the most useful place to lookup the variables you might find in the Mailman provided settings.py but don’t fully understand how to configure.
All the settings are applicable directly, including things like LOGGING (to setup logging), DATABASES(to setup backend database configuration), USE_I18N(Internationalization of the interface), EMAIL_BACKEND (how to talk to local or remote MTA for sending emails).
Mailman-web configuration
On top of these base settings provided by Django, the configurations for
Postorius, Hyperkitty and django-mailman3 all belong to the same
settings.py
(or settings_local.py
, if you choose to use that).
There are other third party Django applications, which we reuse for various
functionality in Mailman-web use similar configuration process via
settings.py
.
The most important packages for Mailman are:
django-allauth: Good place to find any configuration related to account management in the Web UI from email sending issues to contents of the email sent for account/email verification.
Django-haystack: This is used for search indexing of the emails in Hyperkitty. This is a good place to find various settings for setting up different search backends.
Django-qcluster: This will provide all the asynchronous jobs that are run via cron in Hyperkitty to update various internals in a routine fashion. Use this to see how different backends can be configured to store and run jobs and how to tweak the configuraiton of each backend.
Management Commands
This is probably the most useful (and confusing!) thing about Django due to the sheer number of ways that this can be done. Django management commands provide a framework to write and run commands for Django web applications for various manual or other tasks.
Django’s documentation on management commands is a good place to start, but if it is too long for you, this is a short know how for Mailman admins.
There are three variants for running management commands, all of which can be use interchangeably, depending on how you installed. Package managers, Docker images and mailman-web all provide different ways to run the same commands.
python3 manage.py --settings settings --pythonpath <parent dir of settings.py> <commamd>
: This command is probably the _most_ common one that you can see. You will notice that it accepts--settings
and--pythonpath
flags to provide the settings module of Django. You can also set the environment variablesDJANGO_SETTINGS_MODULE
andPYTHONPATH
instead of passing the values to command.django-admin --settings settings --pythonpath <parent dir of settings.py> <command>
: This is exactly same as the above, replace the “manage.py” file with an installed commanddjango-admin
, obviating the need to find themanage.py
file in your current installation. The same environment variables mentioned in the above marker works here as well.mailman-web <command>
: This is the simplified version of the above two commands, which comes with the mailman-web package. It defaults the settings.py to be present at/etc/mailman3/settings.py
and automatically sets up the PYTHONPATH for Django to be able to import. You can also setMAILMAN_WEB_CONFIG
environment variable to point to a different config file (not module!) and it will take care of adding it to the PYTHONPATH and setting DJANGO_SETTINGS_MODULE.
The documentation of Mailman is slowly going to lean towards using
mailman-web
command as it is simple, but third party packages and Django
can emit erorrs that use any of the above three notations. Which is why it is
good to know them. You can use mailman-web <command>
for commands from
Django (like, migrate
or createsuperuser
for example) or from third
party packages (like qcluster
or qinfo
).
Running Django
Running Django based web applications follows a Python standard mechanism called WSGI, which describes how a web server communicates with Python applications. There is also ASGI for async based applicaitons, but Mailman currently doesn’t use it.
You can use any WSGI server to run mailman-web. Our current documentation recommends using uwsgi, but gunicorn is also a great option. Some web servers like Apache2 come with wsgi modules, which obviate the need for a separate WSGI server like uwsgi, gunicorn. So, if your current infrastructure is already using Apache2, this might be an option for you.
It is good to refer to uwsgi docs or gunicorn docs if you want to configure the aspects of running Django that aren’t already done in Django, which includes things like, access/error logging, number of threads and processes to run which can handle your expected load, SSL configuration (if you want SSL all the way through, instead of terminating at web server) etc.
Most (all?) WSGI servers will require only the path to the (1) Django’s
settings and (2) WSGI “module”. This wsgi module, is often times just a
wsgi.py
file somewhere on PYTHONPATH. If you are using mailman-web
then
it comes pre-installed and you can refer to it as mailman_web.wsgi
, without
worrying about the file itself.
Since web servers are more hardened in terms of security and more capable of handling high load, _typically_, WSGI servers listen on a local addresses and Web server proxy requests from Internet to the Wsgi server.