Serious performance issue with redis connection pooling due to CaseInsensitiveDict

[aldem]

I was doing performance tests in my app, and noticed that if I serve a large number of requests simultaneously, doing something like this:

async with redis.asyncio.Redis(connection_pool=redis_blocking) as conn:
    await conn.set(...)

... the number of requests per second is significantly lower (around 45-50%) compared to the case where I take a connection only once.

Since every request (as the app provides an API) requires taking a connection from the pool, I cannot do it once and then use multiple times - it will be even slower due to serialization (and will actually defeat the purpose of the pool).

Profiling revealed that the slowdown is primarily caused by the use of CaseInsensitiveDict in the Redis.__init__() method:

redis-py/redis/client.py

Lines 382 to 387 in a4df6b2

	self.response_callbacks = CaseInsensitiveDict(_RedisCallbacks)
	if self.connection_pool.connection_kwargs.get("protocol") in ["3", 3]:
	self.response_callbacks.update(_RedisCallbacksRESP3)
	else:
	self.response_callbacks.update(_RedisCallbacksRESP2)

Especially, update() is expensive, since there are many keys to update; the profiler shows that __setitem__() alone consumes approximately 50% of the time spent, most of which is attributed to the str.upper() method.

Replacing the dictionary with a simple dict reduced the time needed to acquire the client (not a connection) by a factor of 12, resulting in a performance boost of approximately 45% in the aforementioned use case, and its performance became nearly comparable to that of acquiring a connection only once.

Now the question - is there a reason to use CaseInsensitiveDict? I could not spot anything obvious in the code that requires case-insensitive keys, while the performance gain is significant.

Additionally, moving the initialization of self.response_callbacks to the pool (as they are constants anyway) could potentially lead to further performance improvements.

Thank you!

[petyaslavova]

Hi @aldem , thank you for pointing this out! We'll have a look at this.

[matemax]

Hi. Is there any updates? We have a same performance degradation in our project after update from version 5.2.0 to 6.0.0.

[aldem]

@matemax If you really need maximum performance, take a look at GLIDE - it has different API but is fully compatible with Redis. It is literally several times faster, uses single connection to the server (multiplexing & pipelining) and properly & transparently handles re-connections if Redis is down for a while.

[matemax]

@aldem Thanks for the idea!

[iamdbychkov]

@aldem Thank you for mentioning this library, that what I've been looking for!
I would like to highlight that multiplexing is a crucial feature that, in my opinion, is currently missing in redis-py. This absence significantly impacts the usability of redis-py in high-load production environments, where efficient handling of multiple connections is essential for optimal performance.