minor #13597 Messenger process managers (tvlooy)

This PR was submitted for the 5.1 branch but it was merged into the 5.4 branch instead.

Discussion
----------

Messenger process managers

Add information about mysql timeouts, supervisor FATAL, systemd user services.

Or, how should you handle this MySQL timeout issue, supervisor restart (FATAL) issues? How do you people do this? What does the component author thinks about this?

I myself am using with systemd user services in production for a while and it's running okay.

Commits
-------

1560584 Add documentation of Supervisor BACKOFF strategy
09fd70c Messenger process managers

Author: wouterj

messenger.rst

@@ -486,23 +486,30 @@ Deploying to Production
 
 On production, there are a few important things to think about:
 
-**Use Supervisor to keep your worker(s) running**
+**Use a Process Manager like Supervisor or systemd to keep your worker(s) running**
     You'll want one or more "workers" running at all times. To do that, use a
-    process control system like :ref:`Supervisor <messenger-supervisor>`.
+    process control system like :ref:`Supervisor <messenger-supervisor>`
+    or :ref:`systemd <messenger-systemd>`.
 
 **Don't Let Workers Run Forever**
     Some services (like Doctrine's ``EntityManager``) will consume more memory
     over time. So, instead of allowing your worker to run forever, use a flag
     like ``messenger:consume --limit=10`` to tell your worker to only handle 10
-    messages before exiting (then Supervisor will create a new process). There
+    messages before exiting (then the process manager will create a new process). There
     are also other options like ``--memory-limit=128M`` and ``--time-limit=3600``.
 
+**Stopping Workers That Encounter Errors**
+    If a worker dependency like your database server is down, or timeout is reached,
+    you can try to add :ref:`reconnect logic <middleware-doctrine>`, or just quit
+    the worker if it receives too many errors with the ``--failure-limit`` option of
+    the ``messenger:consume`` command.
+
 **Restart Workers on Deploy**
     Each time you deploy, you'll need to restart all your worker processes so
     that they see the newly deployed code. To do this, run ``messenger:stop-workers``
     on deployment. This will signal to each worker that it should finish the message
-    it's currently handling and should shut down gracefully. Then, Supervisor will create
-    new worker processes. The command uses the :ref:`app <cache-configuration-with-frameworkbundle>`
+    it's currently handling and should shut down gracefully. Then, the process manager
+    will create new worker processes. The command uses the :ref:`app <cache-configuration-with-frameworkbundle>`
     cache internally - so make sure this is configured to use an adapter you like.
 
 **Use the Same Cache Between Deploys**
@@ -669,11 +676,25 @@ times:
     startsecs=0
     autostart=true
     autorestart=true
+    startretries=10
     process_name=%(program_name)s_%(process_num)02d
 
 Change the ``async`` argument to use the name of your transport (or transports)
 and ``user`` to the Unix user on your server.
 
+.. caution::
+
+    During a deployment, something might be unavailable (e.g. the
+    database) causing the consumer to fail to start. In this situation,
+    Supervisor will try ``startretries`` number of times to restart the
+    command. Make sure to change this setting to avoid getting the command
+    in a FATAL state, which will never restart again.
+
+    Each restart, Supervisor increases the delay by 1 second. For instance, if
+    the value is ``10``, it will wait 1 sec, 2 sec, 3 sec, etc. This gives the
+    service a total of 55 seconds to become available again. Increase the
+    ``startretries`` setting to cover the maximum expected downtime.
+
 If you use the Redis Transport, note that each worker needs a unique consumer
 name to avoid the same message being handled by multiple workers. One way to
 achieve this is to set an environment variable in the Supervisor configuration
@@ -696,7 +717,7 @@ Next, tell Supervisor to read your config and start your workers:
 See the `Supervisor docs`_ for more details.
 
 Graceful Shutdown
-~~~~~~~~~~~~~~~~~
+.................
 
 If you install the `PCNTL`_ PHP extension in your project, workers will handle
 the ``SIGTERM`` POSIX signal to finish processing their current message before
@@ -712,6 +733,88 @@ of the desired grace period in seconds) in order to perform a graceful shutdown:
     [program:x]
     stopwaitsecs=20
 
+.. _messenger-systemd:
+
+Systemd Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+While Supervisor is a great tool, it has the disadvantage that you need system
+access to run it. Systemd has become the standard on most Linux distributions,
+and has a good alternative called *user services*.
+
+Systemd user service configuration files typically live in a ``~/.config/systemd/user``
+directory. For example, you can create a new ``messenger-worker.service`` file. Or a
+``[email protected]`` file if you want more instances running at the same time:
+
+.. code-block:: ini
+
+    [Unit]
+    Description=Symfony messenger-consume %i
+
+    [Service]
+    ExecStart=php /path/to/your/app/bin/console messenger:consume async --time-limit=3600
+    Restart=always
+    RestartSec=30
+
+    [Install]
+    WantedBy=default.target
+
+Now, tell systemd to enable and start one worker:
+
+.. code-block:: terminal
+
+    $ systemctl --user enable [email protected]
+    $ systemctl --user start [email protected]
+
+    # to enable and start 20 workers
+    $ systemctl --user enable messenger-worker@{1..20}.service
+    $ systemctl --user start messenger-worker@{1..20}.service
+
+If you change your service config file, you need to reload the daemon:
+
+.. code-block:: terminal
+
+    $ systemctl --user daemon-reload
+
+To restart all your consumers:
+
+.. code-block:: terminal
+
+    $ systemctl --user restart messenger-consume@*.service
+
+The systemd user instance is only started after the first login of the
+particular user. Consumer often need to start on system boot instead.
+Enable lingering on the user to activate that behavior:
+
+.. code-block:: terminal
+
+    $ loginctl enable-linger <your-username>
+
+Logs are managed by journald and can be worked with using the journalctl
+command:
+
+.. code-block:: terminal
+
+    # follow logs of consumer nr 11
+    $ journalctl -f --user-unit [email protected]
+
+    # follow logs of all consumers
+    $ journalctl -f --user-unit messenger-consume@*
+
+    # follow all logs from your user services
+    $ journalctl -f _UID=$UID
+
+See the `systemd docs`_ for more details.
+
+.. note::
+
+    You either need elevated privileges for the ``journalctl`` command, or add
+    your user to the systemd-journal group:
+
+    .. code-block:: terminal
+
+        $ sudo usermod -a -G systemd-journal <your-username>
+
 Stateless Worker
 ~~~~~~~~~~~~~~~~
 
@@ -2190,6 +2293,8 @@ middleware and *only* include your own:
     If a middleware service is abstract, a different instance of the service will
     be created per bus.
 
+.. _middleware-doctrine:
+
 Middleware for Doctrine
 ~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -2377,6 +2482,7 @@ Learn more
 .. _`streams`: https://redis.io/topics/streams-intro
 .. _`Supervisor docs`: http://supervisord.org/
 .. _`PCNTL`: https://www.php.net/manual/book.pcntl.php
+.. _`systemd docs`: https://www.freedesktop.org/wiki/Software/systemd/
 .. _`SymfonyCasts' message serializer tutorial`: https://symfonycasts.com/screencast/messenger/transport-serializer
 .. _`Long polling`: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html
 .. _`Visibility Timeout`: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html