Add conj-reaper management command

[nicwolff]

We're running cacheops in a Django Rest Framework application that serves about 20,000 requests per minute, and our biggest conj sets – for which all filters were excluded from the DNF, and the table is rarely updated – had been growing to contain over 20 million keys.

On invalidation, this blocked Redis (and our entire application!) for many seconds while the invalidate EVALSHA looped over the keys in the conj set to delete query keys. (Even using UNLINK for the deletion of the conj set itself only helped halfway.)

So we've added this Django management command, which loops continually to scan cacheops' conj keys for already-expired cache keys and removes them. It can be configured with command-line options or environment variables to run as fast as your CPUs can accommodate, and to ignore conj sets under some reasonable size.

Some other users may find it valuable, so I submit it for inclusion with your excellent library.

[Suor]

I suspect if you are using ENV here then that's what you are using in your app. There is, however, a major way in Django - use settings everywhere, if you need to set something from ENV then assign a setting from env var in settings.py:

CACHEOPS_REAPER_SLEEP = os.environ.get('CACHEOPS_REAPER_SLEEP', 0.01)

There are many libs to support this like django-environ or dj_database_url. This way every library doesn't need to deal with it. And you can use:

from cacheops.redis import redis_client

redis_client.sscan(...)

While using it with:

CACHEOPS_REDIS="redis://redis/7" CACHEOPS_REAPER_SLEEP=0.05 ./manage.py reap-conjs ...

[bharling]

I'd prefer to see this implemented as a command in the library as with the invalidate_foo commands already available? or even just have it handled automatically without the developer needing to do anything.

if it was a library-level command it could easily be dropped into a periodic celery task or something?

[codeasashu]

I have memory issues with this library and I badly need this merged. Do we have any timeline for this?

[Suor]

if it was a library-level command it could easily be dropped into a periodic celery task or something?

When it needs to be implemented as a celery task. Or maybe a function, which would be easy to wrap into celery or any other bg task. Anyway not a function.

I have memory issues with this library and I badly need this merged. Do we have any timeline for this?

For this to be merged it needs to be cleaned up. It might be used as is though.

[Suor]

Implemented in #434.