Socket Mode: Slow message consumption when listeners do not immediately return ack()

[Mugenor]

(Describe your issue and goal here)

Category (place an x in each of the [ ])

• bolt (Bolt for Java)
• bolt-jakarta-socket-mode (Bolt for Java - optional modules)
• slack-api-client (Slack API Clients)
• slack-api-model (Slack API Data Models)
• slack-api-*-kotlin-extension (Kotlin Extensions for Slack API Clients)
• slack-app-backend (The primitive layer of Bolt for Java)

Requirements

We are using java sdk in socket mode with tyrus client.
I just found out that it runs message processing single threaded and:

1. it looks like it's not really intended.
   com.slack.api.socket_mode.SocketModeClient#initializeMessageProcessorExecutor
   It creates 10 (by default) jobs for processing messages with executor service created by the com.slack.api.util.thread.DaemonThreadExecutorServiceProvider#createThreadScheduledExecutor
   which creates a single threaded executor
   I've tried to test it by putting Thread.sleep for a couple of seconds in event processor and spamming messages and the message processing was very slow which proves the point.
   It looks like a bug to me.
2. It's hard to configure concurrency – it's configured in the clients constructor, the client is created via JakartaSocketModeClientFactory#create(Slack, String) method which is called in private method SocketModeApp#buildSocketModeClientFactory that doesn't use any top-level properties that can overwrite the behaviour.
   It would be nice to have an easy option to change concurrency.
3. It uses scheduled executor which significantly reduces the number of ExecutorServices that is possible to use for message processing. Is there a reason for using scheduled instead of a regular one? From my perspective switching to a regular one should simplify things, because it already encapsulates the queueing logic. + it would allow to use virtual threads executor
   If the used executor type were changed and the ability to provide the concrete implementation were given, it would mitigate the second problem in this list from my perspective

Let me know if I can clarify anything else

[seratch]

Hi @Mugenor, thanks for sharing the detailed feedback. Indeed, the message processor part could be improved for better runtime performance when listeners have slow synchronous code execution.

I did some quick prototyping to explore ways to improve the framework's internals today but I still need more time. A quick recap is that both 1) just replacing the scheduled executor service and 2) increasing the concurrency do not bring significant improvements when sync code in listeners are slow. If you're interested, you can try the code out here.

Now I am thinking that enhancing bolt-jakarta-socket-mode/bolt-socket-mode may bring much more positive impact. I will check if this idea works tomorrow.

As a quick suggestion for you, please avoid having slow synchronous code in your event listeners. As long as your code returns ack() immediately, the message processor won't get stuck. Here is a quick code snippet demonstrating how to run time-consuming operations asynchronously:

app.event(MessageEvent.class, (req, ctx) -> {
  // You can use your own ExecutorService too
  app.executorService().submit(() -> {
    // Do anything asynchronously here
  });
  // This line will be synchronously executed
  return ctx.ack();
});

The document should have clearly mentioned this, but using ExecutorService within a listener is the recommended way to implement complex and time-consuming event listeners. Nonetheless, the message processor can indeed be improved, and I will continue working on it.

Thanks again for sharing this feedback.

[seratch]

Qiuck update: I've submitted pull request #1411 resolving this issue. If the changes are fine, we will include new "concurrency" option in the next release. If you're interested in trying it out earlier, you can copy and paste the code into your project for now.