Track vector disk usage by vectorReader.getOffHeapByteSize

[mayya-sharipova]

Currently IndexDiskUsageAnalyzer reports disk usage of vectors by

• Iterating through document values to access vector data
• Performing sample searches to force loading of the index structures
• using a sampling approach (only visiting a subset of documents based on log scale)
• tracking all bytes read during these operations

One problem of this approach is that it is very slow.
Another problem is that modifications to search algorithms and different encodings make it difficult to write definite test and assert expected results, hence a test failure such as #127689.

This modifies IndexDiskUsageAnalyzer for vectors to use a new introduced in Lucene 10.3 method vectorReader.getOffHeapByteSize. As all vector files are offHeap, we can rely on this method to report the precise disk
usage.

Closes #127689

[mayya-sharipova]

@ChrisHegarty This is a draft PR to test the correctness of this statement: "can we can rely on vectorReader.getOffHeapByteSize to report the correct disk usage"?

[ChrisHegarty]

@ChrisHegarty This is a draft PR to test the correctness of this statement: "can we can rely on vectorReader.getOffHeapByteSize to report the correct disk usage"?

Absolutely. The numbers reported by getOffHeapByteSize neglect the file format header/footers, etc, but they are minimal and not currently accounted for in the current implementation anyway. So I think you got it right here @mayya-sharipova. 👍

[ChrisHegarty]

LGTM

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[mayya-sharipova]

@elasticsearchmachine "Check labels"

[mayya-sharipova]

@ChrisHegarty I would like your input on this:

Now with DirectIO getOffHeapByteSize method will not report .vec anymore. Should we modify this pr:

• is a reader is direct IO reader (how easily to detect this?), use old method
• for all other readers use getOffHeapByteSize

[ChrisHegarty]

@ChrisHegarty I would like your input on this:

Now with DirectIO getOffHeapByteSize method will not report .vec anymore. Should we modify this pr:

• is a reader is direct IO reader (how easily to detect this?), use old method
• for all other readers use getOffHeapByteSize

I still need to port #128697 to the main branch. I've not done so yet, since we've had to disable direct IO temporarily.

Once #128697 is ported then we should be good to use getOffHeapByteSize always, even for the direct IO case - since the reader will know whether to report the raw vec usage or not. So no special handling required. Make sense?

[mayya-sharipova]

We had a discussion offline, and the approach we've decided to take is:

• get vectorReader.getOffHeapByteSize output and check it .vec size.
• if .vec size is 0, do manual calculation of this file based on element type

[mayya-sharipova]

@ChrisHegarty Can you please review the recent changes. Thanks a lot