Skip to content

Commit 1560584

Browse files
committed
Add documentation of Supervisor BACKOFF strategy
This feature seems to be the official way to handle delays in restart.
1 parent 09fd70c commit 1560584

File tree

1 file changed

+14
-41
lines changed

1 file changed

+14
-41
lines changed

messenger.rst

+14-41
Original file line numberDiff line numberDiff line change
@@ -676,11 +676,25 @@ times:
676676
startsecs=0
677677
autostart=true
678678
autorestart=true
679+
startretries=10
679680
process_name=%(program_name)s_%(process_num)02d
680681
681682
Change the ``async`` argument to use the name of your transport (or transports)
682683
and ``user`` to the Unix user on your server.
683684

685+
.. caution::
686+
687+
During a deployment, something might be unavailable (e.g. the
688+
database) causing the consumer to fail to start. In this situation,
689+
Supervisor will try ``startretries`` number of times to restart the
690+
command. Make sure to change this setting to avoid getting the command
691+
in a FATAL state, which will never restart again.
692+
693+
Each restart, Supervisor increases the delay by 1 second. For instance, if
694+
the value is ``10``, it will wait 1 sec, 2 sec, 3 sec, etc. This gives the
695+
service a total of 55 seconds to become available again. Increase the
696+
``startretries`` setting to cover the maximum expected downtime.
697+
684698
If you use the Redis Transport, note that each worker needs a unique consumer
685699
name to avoid the same message being handled by multiple workers. One way to
686700
achieve this is to set an environment variable in the Supervisor configuration
@@ -702,47 +716,6 @@ Next, tell Supervisor to read your config and start your workers:
702716
703717
See the `Supervisor docs`_ for more details.
704718

705-
It is possible to end up in a situation where the supervisor job gets into a
706-
FATAL (too many start retries) state when trying to restart when something is
707-
not yet available. You can prevent this by wrapping the Symfony script with a
708-
shell script and sleep when the script fails:
709-
710-
.. code-block:: bash
711-
712-
#!/usr/bin/env bash
713-
714-
# Supervisor sends TERM to services when stopped.
715-
# This wrapper has to pass the signal to it's child.
716-
# Note that we send TERM (graceful) instead of KILL (immediate).
717-
_term() {
718-
echo "[GOT TERM, SIGNALING CHILD]"
719-
kill -TERM "$child" 2>/dev/null
720-
exit 1
721-
}
722-
723-
trap _term SIGTERM
724-
725-
# Execute console.php with whatever arguments were specified to this script
726-
"$@" &
727-
child=$!
728-
wait "$child"
729-
rc=$?
730-
731-
# Delay to prevent supervisor from restarting too fast on failure
732-
sleep 30
733-
734-
# Return with the exit code of the wrapped process
735-
exit $rc
736-
737-
The supervisor job would then look like this:
738-
739-
.. code-block:: ini
740-
741-
;/etc/supervisor/conf.d/messenger-worker.conf
742-
[program:messenger-consume]
743-
command=/path/to/your/app/bin/console_wrapper php /path/to/your/app/bin/console messenger:consume async --time-limit=3600"
744-
...
745-
746719
Graceful Shutdown
747720
.................
748721

0 commit comments

Comments
 (0)