Need to retry jobs that died with ProcessPrunedError

[nhorton]

We have some situations where we get ProcessPrunedErrors for jobs that were on killed workers (in Kubernetes). We need a way to have those retried.

on_thread_error seems like it might be appropriate, but it is very unclear what the lifecycle would look like using it since the examples are all just for error captures.

[rosa]

Oh, yes... on_thread_error is intended for Solid Queue's internal errors and wouldn't fit for this.

In theory, you should only get process pruned errors when a process dies unceremoniously (killed, like you said). On some occasions, there's something going on in the job that causes the process to be killed (eg. memory leak in the job that causes the OS to kill the process), so you don't want to retry these generally.

If your workers are getting killed frequently, could you address this instead of automatically retrying these jobs? Is there a way to signal proper termination to the workers instead of relying on the pod going away or something similar?