Messenger process managers

Add information about mysql timeouts, supervisor FATAL, systemd user services.

Author: tvlooy

messenger.rst

@@ -462,22 +462,23 @@ Deploying to Production
 
 On production, there are a few important things to think about:
 
-**Use Supervisor to keep your worker(s) running**
+**Use a process manager like Supervisor or systemd to keep your worker(s) running**
     You'll want one or more "workers" running at all times. To do that, use a
-    process control system like :ref:`Supervisor <messenger-supervisor>`.
+    process control system like :ref:`Supervisor <messenger-supervisor>`
+    or :ref:`systemd <messenger-systemd>`.
 
 **Don't Let Workers Run Forever**
     Some services (like Doctrine's EntityManager) will consume more memory
     over time. So, instead of allowing your worker to run forever, use a flag
     like ``messenger:consume --limit=10`` to tell your worker to only handle 10
-    messages before exiting (then Supervisor will create a new process). There
+    messages before exiting (then the process manager will create a new process). There
     are also other options like ``--memory-limit=128M`` and ``--time-limit=3600``.
 
 **Restart Workers on Deploy**
     Each time you deploy, you'll need to restart all your worker processes so
     that they see the newly deployed code. To do this, run ``messenger:stop-workers``
     on deploy. This will signal to each worker that it should finish the message
-    it's currently handling and shut down gracefully. Then, Supervisor will create
+    it's currently handling and shut down gracefully. Then, the process manager will create
     new worker processes. The command uses the :ref:`app <cache-configuration-with-frameworkbundle>`
     cache internally - so make sure this is configured to use an adapter you like.
 
@@ -633,6 +634,132 @@ config and start your workers:
 
 See the `Supervisor docs`_ for more details.
 
+If a worker dependency like your database server is down, or timeout is reached,
+you can try to add reconnect logic, or just quit the worker (catch Exception,
+PDOException or do something more fine grained):
+
+    // SomeMessageHandler.php
+    public function __invoke(SomeMessage $message): void
+    {
+        try {
+            $thing = $this->someRepository->find($message->getThingId());
+            if (null === $thing) {
+                return;
+            }
+            $this->thingService->doStuff($thing);
+        } catch (\Exception $e) {
+            echo 'Received exception "'.$e->getMessage().'" from "'.get_class($e).'". Quitting worker';
+            exit(1);
+        }
+    }
+
+But, then you can end up in a situation where the supervisor job gets into a
+FATAL (too many start retries) state. You can prevent this by wrapping the
+Symfony script with a shell script:
+
+.. code-block:: bash
+
+    #!/bin/bash
+
+    # Supervisor sends TERM to services when stopped.
+    # This wrapper has to pass the signal to it's child.
+    # Note that we send TERM (graceful) instead of KILL (immediate).
+    _term() {
+        echo "[GOT TERM, SIGNALING CHILD]"
+        kill -TERM $child 2>/dev/null
+        exit 1
+    }
+
+    trap _term SIGTERM
+
+    # Execute console.php with whatever arguments were specified to this script
+    "$@" &
+    child=$!
+    wait "$child"
+
+    # Delay to prevent supervisor from restarting too fast on failure
+    sleep 30
+
+    # Return 1 so supervisor will restart the script
+    exit 1
+
+The worker would then look like this:
+
+.. code-block:: ini
+
+    ;/etc/supervisor/conf.d/messenger-worker.conf
+    [program:messenger-consume]
+    command=/path/to/your/console_wrapper php /path/to/your/app/bin/console messenger:consume async --time-limit=3600
+    ...
+
+.. _messenger-systemd:
+
+Systemd Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+While Supervisor is a great tool, it has the disadvantage that you need system
+access to run it. Systemd has become the standard on most Linux distributions,
+and has a good alternative called user services.
+
+Systemd user service configuration files typically live in a ``~/.config/systemd/user``
+directory. For example, you can create a new ``messenger-worker.service`` file. Or a
+``[email protected]`` file if you want more instances running at the same time:
+
+.. code-block:: text
+
+    [Unit]
+    Description=Symfony messenger-consume %i
+
+    [Service]
+    ExecStart=php /path/to/your/app/bin/console messenger:consume async --time-limit=3600
+    Restart=always
+    RestartSec=30
+
+    [Journal]
+    Storage=persistent
+
+    [Install]
+    WantedBy=default.target
+
+Now, tell systemd to enable and start one worker:
+
+.. code-block:: terminal
+
+    $ systemctl --user enable [email protected]
+
+    $ systemctl --user start [email protected]
+
+Then enable and start another 19:
+
+.. code-block:: terminal
+
+    $ systemctl --user enable messenger-worker@{2..20}.service
+
+    $ systemctl --user start messenger-worker@{2..20}.service
+
+If you would change your service config file, reload the daemon:
+
+.. code-block:: terminal
+
+    $ systemctl --user daemon-reload
+
+Restart all your consumers:
+
+.. code-block:: terminal
+
+    $ systemctl --user restart messenger-consume@*.service
+
+Logs are manged by journald and can be worked with using the journalctl
+command, but you do need elevated privileges (sudo) for that:
+
+.. code-block:: terminal
+
+    $ sudo journalctl -f [email protected]
+
+    $ sudo journalctl -f _UID=$UID
+
+See the `systemd docs`_ for more details.
+
 .. _messenger-retries-failures:
 
 Retries & Failures
@@ -1663,4 +1790,5 @@ Learn more
 .. _`Enqueue's transport`: https://github.com/sroze/messenger-enqueue-transport
 .. _`streams`: https://redis.io/topics/streams-intro
 .. _`Supervisor docs`: http://supervisord.org/
+.. _`systemd docs`: https://www.freedesktop.org/wiki/Software/systemd/
 .. _`SymfonyCasts' message serializer tutorial`: https://symfonycasts.com/screencast/messenger/transport-serializer