Add dense vector off-heap stats to Node stats and Index stats APIs

[ChrisHegarty]

This change enhances the dense_vector section of the Nodes stats and Index stats APIs so that they report the desired size of off-heap memory for all indexed vectors. The dense_vector section of the Custer stats API remains unchanged.

The retrieval mechanism and structure of the new stats is the same across the various three stats APIs, but more fine-grained information is disclosed as when moving from Cluster -> Node -> Index API.

For Node stats, we aggregate the total byte sizes for all vectors, categorised by the data type. For example:

"dense_vector" : {
  "value_count" : 5,
  "off_heap" : {
    "total_size_in_bytes" : 27,
    "total_veb_size_in_bytes" : 3,
    "total_vec_size_in_bytes" : 23,
    "total_veq_size_in_bytes" : 0,
    "total_vex_size_in_bytes" : 1
  }
}

Index stats: same as Node stats with included field break down . For example:

"dense_vector" : {
  "value_count" : 5,
  "off_heap" : {
    "total_size_in_bytes" : 27,
    "total_veb_size_in_bytes" : 3,
    "total_vec_size_in_bytes" : 23,
    "total_veq_size_in_bytes" : 0,
    "total_vex_size_in_bytes" : 1,
    "fielddata" : {
      "bar" : {
        "veb_size_in_bytes" : 3,
        "vec_size_in_bytes" : 14,
        "vex_size_in_bytes" : 1
      },
      "foo" : {
        "vec_size_in_bytes" : 9
      }
    }
  }

The implementation accesses the actual statistics through reflection. This will be completely removed when Lucene exposes this, which is expected in Lucene 10.3, see apache/lucene#14426.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @ChrisHegarty, I've created a changelog YAML for you.

[rjernst]

I only looked at the reflection bits, and have a thought.

[benwtrent]

I realize it would be even more complicated, but there are all the bwc indices from Lucene (Lucene95, 92, etc.). Do we want to include those? Its probable that folks have data in those old indices.

Its likely fine to do a "good enough" job here and indicate that stats are only available on newer formats.

[ChrisHegarty]

let's add them, it's not difficult and will improve usability. When Lucene 10.3 arrives it will have support there too, and we can delete all this reflection code.

[ChrisHegarty]

@elasticmachine update branch

[ChrisHegarty]

@elasticmachine update branch

[ChrisHegarty]

@elasticmachine update branch

[benwtrent]

I am confused by delegate thing, maybe I am misunderstanding.

Do we want to add the other older readers later?

[ChrisHegarty]

Do we want to add the other older readers later?

The reason I've not added them yet is that they're hard to test. My latest thought is to generate an old index and stuff it into a unit test. Longer term we won't need this, since the Lucene readers will already be tested in Lucene itself, but for now we'd need some minimal sanity on the reflection code.

[ChrisHegarty]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[ChrisHegarty]

I added support for the old codecs, see 4638377. I tested these locally, since it's not straightforward to create these old indices.

[benwtrent]

I am confused by default behavior and if we always want to return these stats or if per-field stats are always returned.