========================= Django for Mailman Admins ========================= This is a brief primer on Django_ to help with administration of Mailman as it is used as a web framework within the web interface of Mailman 3. Ideally, Django *should* just be a python dependency and the details should be wrapped behind configurations and settings in Mailman. However, due to the amount of functionality provided to us by Django, it would be a monumental effort to hide it completely. We often resort to referring Django documentation directly in our documentation and also rely on several deployment related features that Django provides. There are three main topics that is important for system administrators that we are going to cover in this doc, configuration, management commands and running (deployment). Configuration ============= Django expects its configuration to be provided via a ``settings`` module. The name of the module itself isn't a constant and can be passed in as ``--settings`` command line parameter in Django commands and also be set using ``DJANGO_SETTINGS_MODULE`` environment variable. The unusual thing to note here is that it expects a **module** not a file path, so whatever the name of the module is, it should be **importable by Django(in Python)**, which can be achieved by adding the directory the file ``settings.py`` exists in the ``PYTHONPATH`` environment variable. It works similar to the ``PATH`` environment varible that unix shell use. Since it is using Python, you can use more Pythonisms in how the configuration is defined or loaded. For example, a common pattern is to keep ``settings.py`` as close to the one provided by the upstream project, but make the changes required in a ``settings_local.py`` and then import the configuration in ``settings.py``. :: # settings.py try: from settings_local import * except ImportError # If the import fails, don't worry. pass :: # settings_local.py DATABASES = ... SECRET_KEY = ... This way, you can keep the "local" configuration overrides in a separate file. You can also take this a step further to put secrets into a ``secrets.py`` and import from there in settings.py if you want to separate secrets from other configuration. Django upstream settings ------------------------ Django's upstream settings_ documentation is probably the most useful place to lookup the variables you might find in the Mailman provided settings.py but don't fully understand how to configure. All the settings are applicable directly, including things like LOGGING (to setup logging), DATABASES(to setup backend database configuration), USE_I18N(Internationalization of the interface), EMAIL_BACKEND (how to talk to local or remote MTA for sending emails). Mailman-web configuration ------------------------- On top of these base settings provided by Django, the configurations for Postorius, Hyperkitty and django-mailman3 all belong to the same ``settings.py`` (or ``settings_local.py``, if you choose to use that). There are other third party Django applications, which we reuse for various functionality in Mailman-web use similar configuration process via ``settings.py``. The most important packages for Mailman are: * `django-allauth`_: Good place to find any configuration related to **account management** in the Web UI from email sending issues to contents of the email sent for account/email verification. * `Django-haystack`_: This is used for **search indexing** of the emails in Hyperkitty. This is a good place to find various settings for setting up different search backends. * `Django-qcluster`_: This will provide all the **asynchronous jobs** that are run via cron in Hyperkitty to update various internals in a routine fashion. Use this to see how different backends can be configured to store and run jobs and how to tweak the configuraiton of each backend. Management Commands =================== This is probably the most useful (and confusing!) thing about Django due to the sheer number of ways that this can be done. Django management commands provide a framework to write and run commands for Django web applications for various manual or other tasks. Django's `documentation on management commands`_ is a good place to start, but if it is too long for you, this is a short know how for Mailman admins. There are three variants for running management commands, all of which can be use interchangeably, depending on how you installed. Package managers, Docker images and mailman-web all provide different ways to run the same commands. * ``python3 manage.py --settings settings --pythonpath ``: This command is probably the _most_ common one that you can see. You will notice that it accepts ``--settings`` and ``--pythonpath`` flags to provide the settings module of Django. You can also set the environment variables ``DJANGO_SETTINGS_MODULE`` and ``PYTHONPATH`` instead of passing the values to command. * ``django-admin --settings settings --pythonpath ``: This is exactly same as the above, replace the "manage.py" file with an installed command ``django-admin``, obviating the need to find the ``manage.py`` file in your current installation. The same environment variables mentioned in the above marker works here as well. * ``mailman-web ``: This is the simplified version of the above two commands, which comes with the `mailman-web package`_. It defaults the settings.py to be present at ``/etc/mailman3/settings.py`` and automatically sets up the PYTHONPATH for Django to be able to import. You can also set ``MAILMAN_WEB_CONFIG`` environment variable to point to a different config **file** (not module!) and it will take care of adding it to the PYTHONPATH and setting DJANGO_SETTINGS_MODULE. The documentation of Mailman is slowly going to lean towards using ``mailman-web`` command as it is simple, but third party packages and Django can emit erorrs that use any of the above three notations. Which is why it is good to know them. You can use ``mailman-web `` for commands from Django (like, ``migrate`` or ``createsuperuser`` for example) or from third party packages (like ``qcluster`` or ``qinfo``). Running Django ============== Running Django based web applications follows a Python standard mechanism called WSGI_, which describes how a web server communicates with Python applications. There is also ASGI_ for async based applicaitons, but Mailman currently doesn't use it. You can use any WSGI server to run mailman-web. Our current documentation recommends using `uwsgi`_, but `gunicorn`_ is also a great option. Some web servers like Apache2 come with `wsgi modules`_, which obviate the need for a separate WSGI server like uwsgi, gunicorn. So, if your current infrastructure is already using Apache2, this might be an option for you. It is good to refer to `uwsgi docs`_ or `gunicorn docs`_ if you want to configure the aspects of running Django that aren't already done in Django, which includes things like, access/error logging, number of threads and processes to run which can handle your expected load, SSL configuration (if you want SSL all the way through, instead of terminating at web server) etc. Most (all?) WSGI servers will require only the path to the (1) Django's settings and (2) WSGI "module". This wsgi module, is often times just a ``wsgi.py`` file somewhere on PYTHONPATH. If you are using ``mailman-web`` then it comes pre-installed and you can refer to it as ``mailman_web.wsgi``, without worrying about the file itself. Since web servers are more hardened in terms of security and more capable of handling high load, _typically_, WSGI servers listen on a local addresses and Web server proxy requests from Internet to the Wsgi server. .. _Django: https://www.djangoproject.com/ .. _settings: https://docs.djangoproject.com/en/5.1/ref/settings/ .. _settings topic: https://docs.djangoproject.com/en/5.1/topics/settings/ .. _django-allauth: https://docs.allauth.org/en/latest/ .. _Django-haystack: https://django-haystack.readthedocs.io/en/master/tutorial.html#configuration .. _Django-qcluster: https://django-q2.readthedocs.io/en/latest/configure.html .. _documentation on management commands: https://docs.djangoproject.com/en/5.1/ref/django-admin/ .. _mailman-web package: https://mailman-web.readthedocs.io/en/latest/index.html .. _WSGI: https://wsgi.readthedocs.io/en/latest/what.html .. _ASGI: https://asgi.readthedocs.io/en/latest/ .. _uwsgi: https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/uwsgi/ .. _gunicorn: https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/gunicorn/ .. _wsgi modules: https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/modwsgi/ .. _uwsgi docs: https://uwsgi-docs.readthedocs.io/en/latest/ .. _gunicorn docs: https://docs.gunicorn.org/en/stable/settings.html