Analysis of ES 7.16 Memory Consumption: Multiple Factors Leading to Memory Surge and the Mystery of 9GB Memory Stuck in GC and Optimization Exploration

[zhang18668]

The recent memory consumption of JVM is as follows:

The green area has optimized space for resident memory
The heap memory of ES is basically divided into segment memory Filter Cache、Field Data Cache、Bulk Queue、Indexing Buffer、Cluster State Buffer、 The result set fetch consumed by various aggregation queries

一、Segment Memory
This memory is not used by GC and currently only occupies 500M

2、Filter Cache

This memory is also not used by GC, currently occupying less than 500M

3、Field Data Cache
This memory is also not used by GC and currently only occupies less than 700M. We manually cleaned it up

4、Bulk Queue Indexing Buffer
Both of these are system default values, 10%, and the usage will not be very large

Cluster State Buffer
My ES version is 7.16, and now it's this state buffer. I can't confirm how much I've used. Currently, the disk usage of my cold node is only 45%. When I increased the utilization of the cold disk, I found that the memory skyrocketed severely,

7、 Fetch of ultra large search aggregation result set
I use scroll queries for everything, and I have also tried manually deleting all scorels without seeing a decrease in permanent memory

After overall investigation, it is still due to the fact that the larger the amount of data, the higher the memory. Also, where did the 9 GB of memory that has been stuck in GC go? How to optimize the memory usage of this part?

[DaveCTurner]

Thank you very much for your interest in Elasticsearch. Unfortunately the issue you have reported relates to Elasticsearch version 7.16 which is very old and has passed end-of-life. We will not investigate issues related to unsupported versions here on Github, so I am closing this to indicate that no action is needed from the Elasticsearch development team. It's possible that you will find a volunteer to help you with this issue on the community forums, but our strong recommendation would be to upgrade to a supported version of Elasticsearch as a matter of some urgency. If you can reproduce your issue on a supported version then please open a fresh bug report.

Quoting the bug report form:

Please also check your OS is supported, and that the version of Elasticsearch has not passed end-of-life. If you are using an unsupported OS or an unsupported version then the issue is likely to be closed.