After a while, the workers stop pulling jobs from the queues, and I need to restart them

[vitobotta]

Hey there! We've been using SolidQueue for a while now, and we're really happy with it. But there's one thing that's been bugging us a bit. Sometimes, the workers just stop pulling jobs from the Postgres queues, and there's no error message or anything that helps us figure out what's going on. The workers don't crash either.

The only thing I've found that seems to work is to restart the workers. After that, they start pulling jobs from the queues again and processing them just fine.

I've been looking through the existing issues on Github, and I found one thread that describes a similar problem. But that one was related to the dev mode and was closed a while back. I remember having that issue back then too.

Do you have any suggestions of things I could try? Or is it just a bug?

Thanks a bunch!

[vitobotta]

Perhaps I should add a bit more details in case they might help:

• The workers are deployed in Kubernetes
• SolidQueue is using the main Postgres database
• The workers are connected to the database via a connection pooler (pgbouncer) in transaction mode.

[rosa]

Hey @vitobotta, with that information, I'm not quite sure what could be going on. I haven't experienced this or heard about anyone experiencing this 😕 I think what I'd do is to try to figure out what the workers are doing via strace or similar, and checking SQL logs if you can, to know whether the workers just stopped polling completely (setting ActiveRecord::Base.logger to 0 for all Solid Queue processes should log all the queries I think).

[vitobotta]

Hi @rosa ! What should I be looking for with something like strace? For now, I was thinking to set config.solid_queue.silence_polling = false so to see the messages when the polling happens. Is this a good idea or would it cause too much logging?

[rosa]

What should I be looking for with something like strace?

I'm not sure! 😅 It'd be to get an idea of what the workers are doing when they're not polling.

I was thinking to set config.solid_queue.silence_polling = false

Ah, yes! I had forgotten about that setting. It'll cause a lot of logging while it's working fine depending on your configuration (number of workers, polling intervals) but it could give some clues about what's going on when they get stuck.

[vitobotta]

I will go ahead with that then and let you know how it goes. Thanks!